Wattage puts AI-agent token costs behind a CI gate

Wattage is an open-source token-spend profiler and cost-regression gate for AI agents. Point it at an OpenTelemetry GenAI trace, and it prices each model call, identifies waste patterns, recommends fixes, and can fail CI when an agent becomes measurably more expensive.

×

The tool runs fully offline, requires no configuration file or API key, and accepts an OTLP JSON trace export. The basic command is:

''bash uvx wattage report trace.json ''

Users without existing traces can follow the project’s five-minute setup path, while a fixture is available for immediate testing:

''bash git clone https://github.com/faizannraza/wattage cd wattage && uv sync uv run wattage report examples/sample_trace.json ''

Recommended reading

World Model Optimizer targets frontier quality at lower cost

For the included sample, Wattage reports a Token Efficiency score of A (100) and a total cost of $0.0602. The trace contains 18,450 input tokens and 320 output tokens, with no cache or reasoning tokens, and produces no findings. Its vendored pricing snapshot is dated 2026-07-18-verified. Reports can also be exported as a self-contained HTML flame graph with --html report.html.

Detectors and convergence benchmarking

Wattage’s central feature is its convergence engine, which detects agents thrashing through loops without making meaningful progress. It can identify retries with changing timestamps, oscillations between two strategies, and stalls where every call is technically unique but nothing is learned—patterns that an exact-match duplicate detector cannot reliably catch.

The project compared its classifier with a real SHA-256 exact-match baseline across 10 hand-reviewed synthetic loops:

• Wattage: precision 1.00, recall 1.00, F1 1.00
• SHA-256 exact match: precision 1.00, recall 0.14, F1 0.25

The benchmark can be reproduced with uv run python -m benchmarks.harness. On a genuine captured three-turn agent trace, Wattage’s prefix_churn simulation reduced cost by 44.7%, from $0.000199 to $0.000110, by enabling prompt caching on a stable prefix. The project says the mechanism is the same at production scale, while the demonstration uses small dollar amounts.

Detectors, scoring, and CI gates

The tool normalizes traces into sessions → tasks → loops → iterations → calls and exposes three commands:

• wattage report prices calls and runs eight detectors: prefix_churn, cache_gap, verbosity, redundant_tool_calls, nonconvergence, retrieval_thrash, model_mismatch, and reasoning_overspend.
• wattage score and wattage badge generate a 0–100 Token Efficiency grade for dashboards, README files, or CI.
• wattage ci enforces cost and quality thresholds.

Each finding includes its cost in real dollars, a concrete fix, and a quality_risk tier: none, low, or review. Changes that could affect output quality, such as downgrading a model or reducing reasoning, count toward the score only when supported by a real --quality map.

Wattage does not guess. An unpriced model leaves its call cost at zero and causes wattage ci to fail with exit code 4. An unmeasured quality signal is reported as unmeasured rather than treated as safe.

A GitHub Actions integration can fail a build when the score falls below 80, cost rises by more than 5%, or any critical finding appears. It also posts a per-detector delta table to the pull request and emits SARIF and JUnit XML. The committed JSON baseline updates only after a run passes the gate; a second workflow, triggered by a push to the default branch, must commit the refreshed baseline and badge after merges. The project is licensed under Apache-2.0.