MIS 752 · Lab 4 · richardyoung

GP versus the Specialist: a measured model-tier scorecard

A small free model against the largest model reachable for $0, on five clinical questions through one OpenRouter key — plus a 494M-parameter model run locally at three temperatures. No PHI: the questions are self-contained vignettes. Measured in Google Colab; numbers are this run's and will vary.

The 30-second read

PART A - one 494M-parameter model, one prompt, three temperatures local model : Qwen/Qwen2.5-0.5B-Instruct on cpu temperatures : 0.1, 0.5, 0.7, 1.0, 1.2 tokens/second : 2.2, 3.2, 3.2, 3.7, 3.2 PART B - the same 5 clinical questions, 3 model tiers questions asked: 5 (measured answers: 15) total API cost : $0.118310 liquid/lfm-2.5-2.6b:free full_credit 0.40 · coverage 0.57 · unsafe 0 4.3 s mean · 1066 out-tokens · $0.000000/question · grade 17.1 nvidia/nemotron-3-ultra-550b-a55b:free full_credit 0.80 · coverage 0.80 · unsafe 0 25.1 s mean · 512 out-tokens · $0.000000/question · grade 14.3 ~openai/gpt-astra-latest full_credit 1.00 · coverage 1.00 · unsafe 0 10.6 s mean · 450 out-tokens · $0.023662/question · grade 14.1 ROUTING - GP first, escalate what it flags escalated : 0/5 cascade accuracy: 0.40 self-doubt recall: 0.00 <- of the GP's wrong answers

Where the gap lives

One dot per model per question, joined by a connector, sorted so the biggest divergence is on top. The number at the right of each row is the specialist minus the GP; tie means they scored the same. Read the ties as the questions you should not pay for the big model on.

What accuracy can you buy?

The blue line is an oracle what-if: it escalates the questions the GP actually did worst on, so it is allowed to know the future. The red diamond is what really happened when the GP escalated only what it flagged as uncertain. The vertical distance between them is what the small model's missing self-awareness costs you. With five questions the curve is coarse by construction — add questions in the Explore step for a smoother one.

The scorecard

Latency, tokens, cost and reading grade live here rather than in a chart: with two or three models a table is easier to read than bars.

model coverage full_credit unsafe_claims refusal_rate hedge_rate reading_grade mean_seconds cost_per_q cost_per_correct liquid/lfm-2.5-2.6b:free 0.567 0.4 0 0.0 0.6 17.1 4.26 0.000000 0.000000 nvidia/nemotron-3-ultra-550b-a55b:free 0.800 0.8 0 0.0 0.4 14.3 25.13 0.000000 0.000000 ~openai/gpt-astra-latest 1.000 1.0 0 0.0 0.8 14.1 10.55 0.023662 0.023662

How this was graded

Every score comes from a rule you can read: `must_include` groups of synonyms (any word in a group counts) and `must_not_include` statements that would be flat wrong. coverage is partial credit across groups; full_credit requires every group and no banned statement. Cost is computed from prices fetched live from OpenRouter's /api/v1/models at run time, never hard-coded. Latency is wall-clock around each call. Reading grade is Flesch-Kincaid. This is a five-question pilot with a keyword rubric, not a clinical validation. Numbers are from one run and will vary; the pattern is the finding, not the digits.