CostGuard
Self-hostable LLM reliability proxy — validity pre-filtering, fallbacks, circuit breakers, cost tracking, and trace replay
CostGuard is an open-source, self-hostable reliability layer for LLM-powered applications. It sits between your code and LLM providers as an OpenAI-compatible proxy, scoring every response, falling back across models, tracking exact cost, and benchmarking models on your own data with RealDataAgentBench — without requiring teams to build their own evaluation infrastructure.
Key Features
- Drop-in OpenAI-compatible API — Point the OpenAI SDK at CostGuard by changing only
base_url; streaming is supported, and validity and cost ride along in response headers. - Real-time Response Filtering — Every
/proxycall is scored by a fast lexical pre-filter (~1ms) inspired by RDAB findings. Scores are logged by default; setenforce: trueto reject responses below your threshold. - Automatic Fallback — On a provider error, timeout, or enforced rejection, CostGuard retries with the next model in your fallback chain.
- Per-Provider Circuit Breakers — CLOSED/OPEN/HALF_OPEN state management stops requests to a failing provider, and the state persists across restarts.
- Exact Cost Tracking — Per-call token accounting across 12 models and 5 providers, calculated to $0.000001 precision.
- Dataset Benchmarking — Upload CSV or Parquet files to get a cost-weighted model ranking grounded in 1,412+ RealDataAgentBench runs across 39 tasks.
- Trace Replay —
POST /replayreplays captured Tether traffic against an alternate model and returns the quality delta with a 95% bootstrap CI. - 6-Type Alerting Engine — Validity drops, cost spikes, failure rates, circuit breaker events, and consecutive rejections route to Slack or custom webhooks.
- Prometheus Metrics + Grafana — 13 metrics, an auto-provisioned Grafana dashboard, and opt-in OpenTelemetry traces.
Impact
Cuts LLM spend 10–20× vs GPT-4o defaults. Integrated 12 models across 5 providers with $0.000001 precision cost tracking.
Tech Stack
Python · FastAPI · Streamlit · Docker · Prometheus · Grafana · OpenTelemetry · SQLite · Pytest · Railway