Acme Support (demo)

Evidence

Model

claude-opus-5

Recommended model

claude-haiku-4-5

Calls in window

1,400

Measured window

30 days

Cost today

$128.99/mo

Cost after change

$25.80/mo

Replay samples attached

Yes

Replay verified

Yes

Mean judge score

95 / 100

Samples replayed

10

Samples attempted

10

Replay verification · 10 samples

Each of these is one of your own calls, re-run on the cheaper model and scored for equivalence by an independent judge.

• Sample 1 · replayed on claude-haiku-4-593/100

  “Same category, same priority, same owning rota. The candidate's summary is a few words shorter but names the same request.”

  Original output

  {"category":"feature-request","priority":"p4","team":"success","summary":"Customer wants business hours that can be set separately for each team."}

  Replay output

  {"category":"feature-request","priority":"p4","team":"success","summary":"Customer would like business hours that can be set separately for each team."}

• Sample 2 · replayed on claude-haiku-4-596/100

  “Both outputs route the ticket identically. Wording of the summary differs; the meaning a human would act on does not.”

  Original output

  {"category":"integrations","priority":"p4","team":"integrations","summary":"Customer wants webhook deliveries restored and the missed events sent."}

  Replay output

  {"category":"integrations","priority":"p4","team":"integrations","summary":"The customer wants webhook deliveries restored and the missed events sent."}

• Sample 3 · replayed on claude-haiku-4-592/100

  “Identical labels across all three enumerated fields. The summary paraphrases rather than copies, which the contract allows.”

  Original output

  {"category":"billing","priority":"p2","team":"billing-ops","summary":"Customer wants the duplicate seat charge credited and the invoice corrected."}

  Replay output

  {"category":"billing","priority":"p2","team":"billing-ops","summary":"Customer would like the duplicate seat charge credited and the invoice corrected."}

• Sample 4 · replayed on claude-haiku-4-595/100

  “The candidate matched the reference on category and team, and chose the same priority band. No material difference.”

  Original output

  {"category":"provisioning","priority":"p4","team":"platform","summary":"Customer wants their missing saved views back in the sidebar."}

  Replay output

  {"category":"provisioning","priority":"p4","team":"platform","summary":"The customer wants their missing saved views back in the sidebar."}

• Sample 5 · replayed on claude-haiku-4-598/100

  “Equivalent. The candidate dropped a redundant clause from the summary and was otherwise a character-for-character match on the labels.”

  Original output

  {"category":"integrations","priority":"p4","team":"integrations","summary":"Customer wants each Slack notification delivered once."}

  Replay output

  {"category":"integrations","priority":"p4","team":"integrations","summary":"Customer would like each Slack notification delivered once."}

• Sample 6 · replayed on claude-haiku-4-594/100

  “Same routing decision. The candidate's summary leads with the customer's request rather than the symptom, which reads slightly better.”

  Original output

  {"category":"billing","priority":"p4","team":"billing-ops","summary":"Customer wants the deleted workspace's seats taken off their billed seat total."}

  Replay output

  {"category":"billing","priority":"p4","team":"billing-ops","summary":"The customer wants the deleted workspace's seats taken off their billed seat total."}

• Sample 7 · replayed on claude-haiku-4-597/100

  “All enumerated fields agree. The summary is within the word limit in both, and neither leaks account identifiers.”

  Original output

  {"category":"feature-request","priority":"p4","team":"success","summary":"Customer wants business hours that can be set separately for each team."}

  Replay output

  {"category":"feature-request","priority":"p4","team":"success","summary":"Customer would like business hours that can be set separately for each team."}

• Sample 8 · replayed on claude-haiku-4-593/100

  “No difference that would change what the agent does next. Labels identical, summary reworded.”

  Original output

  {"category":"billing","priority":"p4","team":"billing-ops","summary":"Customer wants the refund for their cancelled annual plan paid out."}

  Replay output

  {"category":"billing","priority":"p4","team":"billing-ops","summary":"The customer wants the refund for their cancelled annual plan paid out."}

• Sample 9 · replayed on claude-haiku-4-596/100

  “Same category, same priority, same owning rota. The candidate's summary is a few words shorter but names the same request.”

  Original output

  {"category":"permissions","priority":"p2","team":"success","summary":"Customer wants to add a colleague to their shared inbox."}

  Replay output

  {"category":"permissions","priority":"p2","team":"success","summary":"Customer would like to add a colleague to their shared inbox."}

• Sample 10 · replayed on claude-haiku-4-592/100

  “Both outputs route the ticket identically. Wording of the summary differs; the meaning a human would act on does not.”

  Original output

  {"category":"billing","priority":"p4","team":"billing-ops","summary":"Customer wants the refund for their cancelled annual plan paid out."}

  Replay output

  {"category":"billing","priority":"p4","team":"billing-ops","summary":"The customer wants the refund for their cancelled annual plan paid out."}

How to fix it

1. 1.This call site averages 128 output tokens per call (spread 0.04) across 1400 calls.
2. 2.Change the model parameter at this call site from claude-opus-5 to claude-haiku-4-5 and re-run its evaluation set.
3. 3.Do not ship it yet: replay verification over the 50 attached sample calls will confirm output quality holds on claude-haiku-4-5 before you act on this.

Evidence

Model

gpt-5

Recommended model

gpt-5-mini

Calls in window

5,487

Measured window

29 days

Cost today

$54.84/mo

Cost after change

$10.97/mo

Replay samples attached

No

How to fix it

1. 1.This call site averages 357 output tokens per call (spread 0.10) across 5487 calls.
2. 2.Change the model parameter at this call site from gpt-5 to gpt-5-mini and re-run its evaluation set.
3. 3.This cluster has no prompt text (aggregate usage rows), so replay verification cannot run on it — validate the swap against your own evaluation set, or re-ingest this source with prompt capture enabled.

Evidence

Model

claude-opus-5

Recommended model

claude-sonnet-5

Calls in window

900

Measured window

30 days

Cost today

$57.16/mo

Cost after change

$34.29/mo

Replay samples attached

Yes

How to fix it

1. 1.This call site averages 701 output tokens per call (spread 0.04) across 900 calls.
2. 2.Change the model parameter at this call site from claude-opus-5 to claude-sonnet-5 and re-run its evaluation set.
3. 3.Do not ship it yet: replay verification over the 50 attached sample calls will confirm output quality holds on claude-sonnet-5 before you act on this.

Evidence

Model

claude-opus-5

Recommended model

claude-sonnet-5

Calls in window

1,300

Measured window

30 days

Cost today

$43.40/mo

Cost after change

$26.04/mo

Replay samples attached

Yes

How to fix it

1. 1.This call site averages 535 output tokens per call (spread 1.12) across 1300 calls.
2. 2.Change the model parameter at this call site from claude-opus-5 to claude-sonnet-5 and re-run its evaluation set.
3. 3.Do not ship it yet: replay verification over the 50 attached sample calls will confirm output quality holds on claude-sonnet-5 before you act on this.

Evidence

Signal

Prompt caching

Model

claude-opus-5

Calls in window

900

Measured window

30 days

Cacheable prefix

4,418 tokens

Already cached

0%

How to fix it

1. 1.Every one of this call site's 900 observed calls opens with the same ~4418 tokens of text, and pays the full input rate for them each time.
2. 2.Turn on provider prompt caching for this call site: mark the end of that stable block with a cache breakpoint (Anthropic `cache_control: {"type": "ephemeral"}` on the last stable content block, OpenAI and Google cache a matching prefix automatically once the request is shaped for it).
3. 3.For the breakpoint to hit, the prefix must be byte-identical on every call: order the request as system prompt, then tool definitions, then any static reference material, and keep everything volatile — the user's message, retrieved context, timestamps, request ids — strictly AFTER the breakpoint.
4. 4.The estimate above reprices the shared prefix at the cached-read rate and takes 10% off for cache writes and refills.
5. 5.If this call site also has a prompt_bloat finding, the two describe alternative remedies for the same tokens — trim the prefix or cache it — so their savings overlap rather than add.

Evidence

Signal

Prompt caching

Model

claude-opus-5

Calls in window

1,300

Measured window

30 days

Cacheable prefix

2,915 tokens

Already cached

0%

How to fix it

1. 1.Every one of this call site's 1300 observed calls opens with the same ~2915 tokens of text, and pays the full input rate for them each time.
2. 2.Turn on provider prompt caching for this call site: mark the end of that stable block with a cache breakpoint (Anthropic `cache_control: {"type": "ephemeral"}` on the last stable content block, OpenAI and Google cache a matching prefix automatically once the request is shaped for it).
3. 3.For the breakpoint to hit, the prefix must be byte-identical on every call: order the request as system prompt, then tool definitions, then any static reference material, and keep everything volatile — the user's message, retrieved context, timestamps, request ids — strictly AFTER the breakpoint.
4. 4.The estimate above reprices the shared prefix at the cached-read rate and takes 10% off for cache writes and refills.
5. 5.If this call site also has a prompt_bloat finding, the two describe alternative remedies for the same tokens — trim the prefix or cache it — so their savings overlap rather than add.

Evidence

Signal

Response caching

Model

claude-opus-5

Measured window

30 days

Repeated prompts

26

Duplicate calls

234

Sample prompt

Worked examples. Match the register, not the wording. EXAMP

How to fix it

1. 1.234 of this call site's calls repeated a request that had already been answered — 26 distinct prompt(s) sent verbatim five or more times each.
2. 2.Put a response cache in front of the provider, keyed on the exact tuple this detector groups on: model, system prompt and prompt text (hash them; do not key on the user or request id, or every lookup misses).
3. 3.Choose the TTL from how fast the underlying answer goes stale, not from the traffic: minutes for anything reading live state, hours to days for stable reference content, and invalidate on the write path when the source data changes.
4. 4.Do not cache prompts that are implicitly time-dependent ("as of today") or personalised without putting the varying part in the key.
5. 5.The estimate above is the observed cost of the repeats themselves, so it is what a 100%-hit-rate cache would have saved over this window; a shorter TTL saves proportionally less.

Evidence

Signal

System prompt bloat

Model

claude-opus-5

Calls in window

1,300

Measured window

30 days

Avg system prompt

2,913 tokens

Avg total input

3,901 tokens

System share of input

75%

Target share

30%

How to fix it

1. 1.This call site sends about 2913 tokens of system prompt on every call — 75% of its ~3901 input tokens.
2. 2.Read the system prompt end to end and cut the parts that are not needed on every call: stale policies, redundant "IMPORTANT" restatements, and few-shot examples beyond the two or three that actually move accuracy.
3. 3.Move large static reference material (tool catalogs, policy documents, schemas) out of the prompt and retrieve only the sections a given call needs.
4. 4.Whatever genuinely must stay on every call belongs in a cached prefix — see the caching_opportunity finding for this call site if one is present; those savings overlap with these rather than adding to them.
5. 5.The estimate above assumes trimming to 30% of the current input, not removing the system prompt.