Multi-Model QA Cascade — Claude-Orchestrated, 3-Provider, $0 ★ CENTERPIECE

Code: 02_ai-engineering-code/multimodel-qa-cascade/ (rewrite-qa.mjs, run.sh, test-providers.sh, README.md, output/sample-one-item.json)

Origin: tradetesttwo/manual/2026-03-03-qa-rewrite-pipeline/

Problem. 2,760+ flagged defects across exam questions, answer explanations, learning assets, and flashcards needed rewriting — fast, cheap, and without trusting any single model's word for "good."

What I built. A content-rewrite engine, run under Claude Code as orchestrator, that fans every item out to three providers in parallel:

Ollama (local, llama3.x via localhost:11434) — zero-cost local baseline.
Groq (llama-3.3-70b, free tier) — fast cloud pass.
OpenAI Codex (gpt-5.x via OAuth codex exec) — highest-quality pass.

Total spend: $0 (local + free tiers + login-based, no per-call billing).

The key idea — verify deterministically, decide with a human.

Every provider's output is scored by category-specific validators that are pure regex + structure checks, not an LLM (e.g. question options = 100 pts across structure/word-count/balance/no-meta-commentary/unique-options; pass ≥ 70). Same input → same score, always.
It's a comparison engine, not cascade-and-stop: the reviewer sees the original + all three rewrites side-by-side with per-check breakdowns, picks the best, and the tool emits SQL patches (apply-*.sql) using the highest-scoring version.
Production-grade ops: parallel fan-out, retry, --resume, --dry-run, --skip-ollama, rate-limit handling, preflight test-providers.sh.

Why it's the centerpiece for an AI-native role. It's the whole job in one artifact: orchestrating heterogeneous models by cost/speed/quality, refusing to let a model grade itself (deterministic validators), keeping a human in the decision loop, and shipping the result as executable patches.

Talk track (screen-share)

Here's the one I'd lead with. I had about twenty-eight hundred content defects to fix, and I didn't want to trust a single model. So under Claude Code as the orchestrator, every item fans out in parallel to three providers: a local Ollama model, Groq on the free tier, and OpenAI Codex through my login. Cost is literally zero. The trick is what happens next: each rewrite is scored by deterministic validators, pure regex and structure checks, no model grading itself, so the same input always gets the same score. It's not cascade-and-stop, it's a comparison engine, the reviewer sees all three versions with a per-check breakdown, picks the best, and the tool emits SQL patches. Multi-model routing, deterministic verification, human-in-the-loop, shipped as executable output. That's the job.

GG · Applied AI Engineering portfolio · synced to youdontneedmy.help design