RelayOps
Production-shaped telecom support agent with scoped tools, guardrails, RAG citations, evals, live demo, and fine-tuned intent routing
RelayOps is a production-shaped AI customer-support agent for telecom and subscription support. It turns a chat request like “reset my device” into a scoped, auditable workflow: deterministic access gate, intent router, server-side tool enforcement, grounded RAG, independent guardrails, and per-turn traces.
What It Demonstrates
- Scoped tool use — Device reset and account lookup are enforced server-side against the authenticated customer’s scope, so prompt injection cannot widen access.
- Tiered routing — Keyword baseline, Complement NB, and Qwen2.5-1.5B LoRA classifiers sit behind the same
IntentClassifierinterface. - Hybrid RAG with citations — FAQ turns retrieve grounded knowledge-base snippets and cite sources; unverifiable turns escalate instead of fabricating.
- Independent guardrail layer — Blocks invented discounts/prices, PII leakage, and unsafe responses before they reach the customer.
- Agent evaluation — 7 adversarial end-to-end cases test scope refusal, billing escalation, guardrail blocking, unauthenticated actions, and cited FAQ answers.
Results
| Signal | Result |
|---|---|
| Intent classifier | keyword 0.49 -> Complement NB 0.93 -> Qwen LoRA 0.999 held-out accuracy |
| Adversarial intent set | Qwen LoRA 0.958 accuracy |
| Agent deterministic eval | 7/7 pass |
| LLM-as-judge | latest completed Gemini run: 6/7 pass, mean 4.6/5 |
| Deployment | Live Streamlit demo on Railway |
| Model artifact | LoRA adapter published on Hugging Face |
Tech Stack
Python · Streamlit · Docker · Railway · Qwen2.5 · LoRA · Hugging Face · RAG · MCP-shaped tools · guardrails · agent evaluation