AWS published Orca Security's Discovery AI case on October 13: a natural-language agent that turns plain questions into Sonar queries. The numbers lead:

Input tokens: ~2M → ~140K per workload (93% down)
Cost: 90% down (vs the initial approach, Orca-reported)
Latency: ~60s → ~15s (75% down)
Live: thousands of queries a day since March 2026

Read it as a success story alone and you get half of it. The structure is the real content.

The first design: stuff all 900 in

Sonar is a domain query system with 900+ cloud data models (resource types like EC2 and S3) and 18 operators. Valid queries need exact model names, parent-child relations, and operators.

The first prototype took the simplest road:

All 900+ schema definitions
+ every relation type
+ many example queries
→ one prompt to the largest model

Quality was good and design partners liked it. Scale was not: ~2M input tokens and 60 seconds per workload, with complex multi-model queries needing repeated refinement.

A familiar trap:

"Put everything in, the model will pick"
→ ~890 of 900 schemas are junk tokens per request
→ biggest model × longest input × every time = bill shock

The replacement: 9 stages, right-sized models

Production is a 9-stage pipeline mixing RAG with schema-aware query construction. Simplified:

1. Context setup (question + org and cloud metadata)
2. Pattern match (IPs, CVEs, ARNs by regex, no LLM, instant return)
3. Guardrail check (Nova 2 Lite)
4. Embedding retrieval (Titan, 20 candidate examples)
5. Rerank (Cohere Rerank 3.5, down to 5)
6. Model scope selection (Haiku 4.5, 900 → the few needed)
7. Filter generation (Haiku 4.5, sees only narrowed schemas)
8. Validation + retries (static checks + Sonar API, Nova 2 Lite)
9. Response output

Two ideas carry it:

First, the generator sees only the selection, not everything.
  → input tokens 2M → 140K

Second, each stage gets the model that fits.
  → Haiku 4.5 reasons and generates, Nova 2 Lite guards and validates
  → never the largest model for everything

Cost fell 3 points less than tokens (90% vs 93%), a hint: more stages mean more calls and output tokens to offset. Keeping 90% after that is the point.

Humans stay in the loop: domain experts correct outputs into the example set through a labeling workflow, no redeploy needed.

What to discount before copying the numbers

- Orca-reported figures via an AWS customer post (vendor material)
- Baseline is the deliberately heavy full-schema prototype
- No dollar amounts, error rates, or latency distributions
- Teams already retrieving schema fragments will gain less

So 90% is not "you get 90% too" but "teams stuffing everything have this much to gain." The direction still holds: context engineering plus retrieval and validation structure bought the efficiency, no model swap required.

CodeBridge mini lab: a 30-minute prompt diet review

[ ] Any docs or schemas attached to every request? (audit all)
[ ] A per-request retrieval step for only what is needed?
[ ] Rules or regex fast-path before LLM calls?
[ ] Different models per stage (never max for everything)?
[ ] Output validation plus retry loop?
[ ] Per-task tokens, cost, and latency measured?

Start with the first item. A "just in case" schema in the prompt is where 2M tokens begin.

Conclusion: cut the context before upgrading the model

Before a bigger model, cut the tokens you send every time.

Select needed schemas, match models to stages, validate outputs. Those three made the 90%. Find your team's largest fixed prompt chunk this week and turn it into a retrieval step. That is the cheapest performance win available.

Further reading

References

Go deeper with a course

Designing retrieval quality, token cost, and latency together connects directly to this article's 9-stage story. Learn the structures linking search and generation.