When humans did penetration testing, "how far may we probe" was settled in a contract. Now that AI agents run the tests, that scoping decision has moved into a console wizard.
Amazon Web Services added a separate test scope for generative AI applications to Security Agent penetration testing. This is an extension of the existing service, not a new launch, and it is documented in the official penetration test guide. The point in one line: you can now test classic web vulnerabilities separately from LLM risks like prompt injection.
What was added: two test scopes
When you create a penetration test, the Test scope field now offers a real choice.
Web application (default):
full pentest across web and API risks
AI features are in scope but not the focus
Gen AI application:
focused on LLM risks in chatbots, agents, and other generative AI features
discovers the app's LLM-backed surfaces and tests them for prompt injection
Worth quoting the docs' own definition: prompt injection is crafted input that makes the model ignore its instructions and perform unintended actions. Where SQL injection targets database query syntax, prompt injection targets the model's instruction hierarchy. Different attack surface, different test harness — that is the logic behind the split.
Why split: web flaws and LLM risks are different exams
Under the shared name "penetration test," the two scopes ask different questions.
| Web application scope | Gen AI application scope | |
|---|---|---|
| Question asked | Are there holes in the app? | Does the model disobey instructions and do unintended things? |
| Typical risks | OWASP Top 10, SQLi/XSS, business logic flaws | Prompt injection, instruction override, out-of-intent behavior |
| Attack surface | URLs, APIs, auth paths | LLM-backed surfaces (chatbots, agents, generative features) |
| Payload style | Minimal-impact proof-of-vulnerability payloads | Crafted inputs that coax the model off its instructions |
| Risk exclusions | Categories excludable via Exclude risk types | Exclusion setting applies to the web scope only |
The last row matters in practice. Per the docs, excluding risk types applies only to the web application scope. The Gen AI scope is itself already a filter ("focus on LLM risks"), so there is no separate exclusion list. Picking a scope is picking a test policy.
The full picture: what scope does inside the five-step wizard
Scope selection sits at the very first step of the test creation wizard. The whole flow looks like this:
① Penetration test details: name, test scope, target URLs, service role
② VPC Resources (optional): network access for non-public targets
③ Credentials (optional): sign-in credentials, pre-verified by testing
④ Network configuration: classify every domain the test may touch
⑤ Additional configuration: app context, run options, risk exclusions
Step 4's domain classification deserves extra care for AI apps.
Target: URLs the test may attack (verified domains only)
Accessible: no attacking, transit allowed (login pages, CDNs, supporting services)
Out of scope: no access at all (destructive actions, sensitive admin functions)
If your chatbot sits behind login, provide credentials and mark the login page or CDN as Accessible. The docs' two preconditions carry over unchanged: test verified domains only, and run authenticated testing in non-production with narrowly scoped credentials. AI apps get no exemption.
Running it in production ops: budgets plus revalidation
The scope split shines brightest combined with the operating features added over the summer.
Cost control: Max task hours caps the work (minimum 20 hours, presets or custom)
→ on hitting the cap, findings found so far are kept and the run ends normally
Fix confirmation: per-finding revalidation (retest just those items, no full rerun)
→ Active (still exploitable) / Resolved (fix confirmed) verdicts, history linked
Find prompt injections with the Gen AI scope, fix them, then revalidate just those findings. Mixing web flaws and LLM risks in one run blurs "what was fixed and what was confirmed"; separate scopes keep the fix-revalidate cycle clean.
CodeBridge Mini Lab: verify with a regression set in an authorized environment
Turn the briefing's actions (an authorized test environment, regression tests for prompt injection and unauthorized tool execution) into an exercise.
① Set up the authorized environment:
[ ] Target domains verified (unverified domains cannot be attacked)
[ ] Non-production environment + ordinary-user credentials (no admin rights)
[ ] Out-of-scope paths marked (destructive actions, sensitive admin functions)
② Design the Gen AI scope test:
[ ] Direct prompt injection: inputs coaxing instruction override or role switching
[ ] Indirect prompt injection: hidden instructions in documents or web content (for RAG/search wiring)
[ ] Unauthorized tool execution: does the model call beyond its permitted scope?
③ Pin it as a regression set:
[ ] Save the cases and rerun on every model or prompt update
[ ] Confirm fixes via revalidation with Active/Resolved verdicts
[ ] Cap cost with Max task hours and confirm findings are preserved
Accumulate this sheet and you own a security regression set for your AI app. The real payoff of the scope split: the same exam, on the same scope, repeatable across model updates.
Conclusion: picking a test scope is writing a security policy
One line to close:
The exam that probes web holes and the exam that shakes the model's instructions are different exams. Don't blend them — run them separately.
This change may look like one added docs section, but it reads like a declaration: AI app security is now its own exam subject. Web flaw testing stays as is, while LLM risks get a dedicated Gen AI scope that confronts prompt injection head-on. The immediate to-do: stand up an authorized non-production environment and bundle injection cases plus out-of-scope tool execution cases into a regression set. That set is what protects you at the next model update.
Keep reading
- Why nothing should execute without permission in the computer-use era
- How the AI Cyber Index measures vulnerability agents
- A purpose-built model for finding leaked secrets
References
- AWS Docs: Create a penetration test (official scope and wizard guide)
- AWS Docs: Security Considerations for AWS Security Agent (minimal-impact payloads, scope control)
- AWS: AWS Security Agent Features (OWASP Top 10, business logic flaws)
- AWS What's New: budget controls and finding revalidation (Aug 19, 2026)
Want to learn this with guided, hands-on practice?
If you want practice controlling AI-generated output with verification criteria and constrained flows, organizing project rules and a verification environment with Claude Code connects directly to this article's regression-test design.