CostGuard

Self-hostable LLM reliability proxy — validity pre-filtering, fallbacks, circuit breakers, cost tracking, and trace replay

CostGuard is an open-source, self-hostable reliability layer for LLM-powered applications. It sits between your code and LLM providers as an OpenAI-compatible proxy, scoring every response, falling back across models, tracking exact cost, and benchmarking models on your own data with RealDataAgentBench — without requiring teams to build their own evaluation infrastructure.

Key Features

  • Drop-in OpenAI-compatible API — Point the OpenAI SDK at CostGuard by changing only base_url; streaming is supported, and validity and cost ride along in response headers.
  • Real-time Response Filtering — Every /proxy call is scored by a fast lexical pre-filter (~1ms) inspired by RDAB findings. Scores are logged by default; set enforce: true to reject responses below your threshold.
  • Automatic Fallback — On a provider error, timeout, or enforced rejection, CostGuard retries with the next model in your fallback chain.
  • Per-Provider Circuit Breakers — CLOSED/OPEN/HALF_OPEN state management stops requests to a failing provider, and the state persists across restarts.
  • Exact Cost Tracking — Per-call token accounting across 12 models and 5 providers, calculated to $0.000001 precision.
  • Dataset Benchmarking — Upload CSV or Parquet files to get a cost-weighted model ranking grounded in 1,412+ RealDataAgentBench runs across 39 tasks.
  • Trace Replay — POST /replay replays captured Tether traffic against an alternate model and returns the quality delta with a 95% bootstrap CI.
  • 6-Type Alerting Engine — Validity drops, cost spikes, failure rates, circuit breaker events, and consecutive rejections route to Slack or custom webhooks.
  • Prometheus Metrics + Grafana — 13 metrics, an auto-provisioned Grafana dashboard, and opt-in OpenTelemetry traces.

Impact

Cuts LLM spend 10–20× vs GPT-4o defaults. Integrated 12 models across 5 providers with $0.000001 precision cost tracking.

Tech Stack

Python · FastAPI · Streamlit · Docker · Prometheus · Grafana · OpenTelemetry · SQLite · Pytest · Railway

Live Demo GitHub