<!-- PLATFORM: Hashnode | STATUS: draft-ready | DO NOT mass-repost identical copy -->

# Model Routing Without the Surprise Invoice: Cost Discipline for Agentic Engineering

Routing models across providers is no longer exotic. Unified gateways, OpenAI-compatible endpoints, and “pick a model per step” UX made it easy to stop hard-coding one API key. The hard part moved downstream: cost discipline—knowing which step burned money, stopping thrash before it becomes a bill, and encoding policy so juniors cannot accidentally upgrade every retry to the most expensive frontier model.

This Hashnode-style engineering note assumes you already understand why routing exists (task fit, failover, vendor diversity). We focus on making routing survive contact with agent loops and finance.

# Cost is a product of loop shape, not sticker price

Sticker price ($/M tokens) is a trap metric. Agentic sessions multiply cost through:

• Long context re-sent every tool turn
• Retries on flaky tool calls
• Silent “upgrade on failure” policies
• Speculative planning passes that never become commits
• Embedding + chat + vision stacked in one “feature”

A $2/M model that retries eight times with a fat prompt can beat a $15/M model that succeeds once with a tight brief. Measure cost per merged outcome (PR, preview URL, resolved ticket)—not cost per completion.

# A minimal cost model you can implement this week

Track four fields on every agent session (DB row, log line, or warehouse event):

1. task_type — plan | edit | review | extract | embed
2. model_id + provider_id
3. tokens_in / tokens_out / tool_calls
4. outcome — merged | abandoned | regenerated | failed_clearly

Join to your ticket id. After two weeks you will know which task types deserve frontier models and which should be scripts.

If you already run LiteLLM (or a similar proxy) in front of providers, persist the response metadata it already exposes—do not scrape invoices by hand.

# Routing policy that protects the budget

# Tier map (example—tune to your stack)

# Kill switches (non-negotiable)

• Max spend per session — hard stop with a clear error, not a quiet continue
• Max upgrades — at most one automatic bump (T2→T3), then fail clearly
• Retry budget — retries consume the same session budget; they are not free
• Context ceiling — refuse to reattach the whole monorepo; require path cites

“Fail clearly” is a feature. Silent spend is a bug.

# Alias discipline beats “latest-smart”

Dynamic aliases feel convenient and destroy reproducibility. For production agents:

• Pin model versions in config
• Diff your allowlist weekly against the catalog
• Treat alias changes like dependency bumps: changelog + smoke test on tool calling and JSON schema

Two endpoints advertising the “same” model can differ in quantization, rate limits, or tool support. Validate the agent path, not only chat playground quality.

# Observability: answer three questions in under a minute

1. Which provider served this completion?
2. Which task type is our top cost sink this week?
3. Did cost rise because volume rose—or because retries rose?

Persist provider id, model id, latency, token counts, and a hashed prompt fingerprint for sampling. Do not log secrets or full customer prompts in plaintext if your policy forbids it. Sampled fingerprints still let you cluster “expensive prompt shapes.”

OpenTelemetry-style spans around agent sessions make this boring in a good way: one dashboard, same muscle memory as services.

# Evaluation that finance will respect

Half-day protocol:

1. Pick five real tasks (not toy prompts).
2. Run each on T1, T2, T3 with identical briefs.
3. Score: correctness, time-to-green, cost, revert risk.
4. Freeze defaults. Re-run monthly—not daily—unless quality regresses.

Publish the table internally. Argument-from-anecdote dies when numbers exist.

For a structured buyer-oriented walkthrough of one popular unified router—and a checklist of what to verify before you commit spend—see Roseram’s OpenRouter review. Use it as a verification template even if your gateway is different.

# Agent-specific footguns

# Planner forever

Agents that replan every failure without shrinking the problem burn T3 casually. Force a “shrink or stop” rule after two failed plans.

# Tool chatter

Each tool call rehydrates context. Prefer batching file reads; forbid “list directory recursively” unless scoped.

# Review as rewrite

Using a frontier model to “improve” a finished PR often rewrites style and invalidates review. Constrain review prompts to smells, missing tests, and a11y—output a checklist, not a new codebase.

# BYOK fee math

Bring-your-own-key setups still have proxy fees, credit purchase friction, and unused reserved capacity. Model your traffic mix before standardizing.

# Team checklist (paste into the eng handbook)

• Session schema includes task_type, model, tokens, outcome, ticket id
• Default agent tier is T2 (or equivalent)—not max
• Session spend cap + upgrade cap enforced in code
• Pinned models in prod; weekly allowlist diff job
• Cost-per-merged-PR dashboard reviewed in standup weekly
• Tool-calling path smoke-tested on every allowed provider
• Security / PII policy encoded as allow/deny lists, not wiki hopes
• “Fail clearly” preferred over silent model upgrades

# What good looks like after 30 days

• Juniors know which tasks are allowed to touch T3
• Finance can forecast agent spend within a useful band
• Outages degrade to a fallback instead of stopping the team
• Retries are visible and declining
• Nobody is surprised by the invoice

# Closing

Model routing without cost discipline is just a more sophisticated way to set money on fire. Encode tiers, kill switches, and outcome metrics. Pin aliases. Measure cost per merged work, not cost per clever completion. Do that, and routing becomes infrastructure—quiet, boring, and budget-safe—while your agents keep moving.