RealDataAgentBench

Automated model-selection engine for production LLM deployments — 12 frontier models, 1,412+ runs, real-world tasks

RealDataAgentBench (RDAB) is an automated benchmarking system built to answer a production engineering question: which model should I deploy, and what will it cost?

It runs 1,412+ evaluations across 12 frontier LLMs on real data science tasks — EDA, feature engineering, statistical inference, ML modeling — and produces cost-accuracy trade-off data that directly powers CostGuard’s model-selection and recommendation engine.

What It Produces

Finding Production implication
GPT-4.1-mini (0.870) is statistically tied with top-ranked GPT-4.1 (0.875) at 65× lower cost per task Default to GPT-4.1-mini for most data-agent workloads
Free Llama 3.3-70B (0.798) scores above GPT-5 (0.780) — directional, as GPT-5 covered 23/39 tasks Benchmark open-source models before paying for frontier ones
Correctness and statistical validity correlate at only r = 0.43 Correctness alone is a misleading selection metric
Claude leads statistical validity; GPT-4.1-mini leads correctness Task-aware routing beats one-model-fits-all

Engineering Design

  • Multi-dimensional scoring — correctness, code quality, efficiency, and statistical validity — because a model that gets the right answer via the wrong method fails in production
  • Multi-run 95% confidence intervals for ranked models, with models below 80% task coverage flagged and excluded from the ranking
  • Published scorer calibration — the lexical statistical-validity scorer is measured against an LLM judge (Pearson r, Cohen’s κ), and every scorer’s known failure modes are documented
  • Category-level breakdowns across inference, feature engineering, EDA, modeling, and ML engineering — enabling task-aware model routing decisions

Benchmark Structure

39 scored tasks (43 in the suite) · 5 categories · 6 real-data tasks · 1,412+ runs · 12 models · multi-run 95% confidence intervals

Tech Stack

Python · LangChain · OpenAI API · Anthropic API · HuggingFace · scikit-learn · Pandas

Leaderboard GitHub