Case Study

Physician-like retrieval: 6× better answers at 22× lower cost

A 35B model with ~3B active parameters got 6x as many answers right at 22x lower cost - not by compressing a patient's history to fit, but by searching it the way a physician does, one query at a time, against a live FHIR record. Trained on one customer's data, then tested on a Stanford benchmark it had never seen.

Model
Qwen3.6-35B-A3B, tuned
Active params
~3B
Serving
$0.54 / 1M flat
Domain
Longitudinal EHR
Key metrics & efficiencies
6x
More answers right
PhysicianBench Pass@1 climbs from 6.0 to 35.0 versus the incumbent frontier stack.
22x
Lower output cost
Flat $0.54 / 1M against $12 / 1M output on the prior frontier model.
35.0
PhysicianBench Pass@1
Ahead of Claude Opus 4.6 (31.7) and GPT-5.4 (27.7) on a held-out benchmark.
< $500
Total tuning cost
End-to-end post-training run across Scout, Compass, and Maestro.

Measured on PhysicianBench, Stanford's held-out benchmark for LLM agents working inside real EHR environments. The tuned model was never trained on it.

Assemblage Health Inc. is the company behind AVNii, an AI-powered clinical intelligence platform that helps clinicians understand the complete patient story while reducing the administrative burden around care. AVNii provides specialty-specific pre-charting across longitudinal records, concise discharge summaries, AI-assisted scribing, and clinical coding support including ICD-10 and SNOMED CT. Its premise is that important clinical signals live throughout the record - across encounters, notes, medications, labs, and imaging - not just in the structured fields.

Getting a model to actually read those records was the hard part.

The record is an archive, not a document

A longitudinal record for an older patient - or a younger one with a complicated history - runs to years of encounters, labs, imaging reports, medication changes, specialist notes, and discharge summaries. Drop one into an LLM and it will not fit in the context window, and not by a small margin. The frustrating part is that the standard workarounds fail quietly instead of throwing an error.

Truncation drops history
The 2019 note that explains why a patient presents the way they do in 2026 tends to be exactly the note that gets cut.
Chunk-and-embed RAG misses the thread
Ask about fatigue and you get the three notes that use the word - but not the medication change that caused it.
Summarization strips the signal
Dosages, dates, and sequence disappear - which is most of what clinical reasoning actually runs on.

What the customer needed was closer to how a physician handles an unfamiliar chart. You search, follow a thread, notice something missing, search again, and eventually you have enough to form a view.

Retrieval as a loop, not a single pass

PerceptEye's Compass agent runs a recursive, REPL-style search over a live FHIR EHR. There is no embedding index anywhere in it. The model writes short Python programs into a persistent Jupyter sandbox and calls FHIR search and read APIs directly, roughly the way a clinician pages through a chart.

01
Query a clinically motivated slice
A condition, a medication class, a date range, an encounter type. FHIR tools are exposed as functions inside the sandbox.
02
Read and check against the question
The model inspects what comes back and evaluates it against the question at hand.
03
Reformulate
A lab value out of range generates a follow-up query. A medication with no matching indication generates another. A referenced-but-unretrieved encounter generates another.
04
Repeat, then write the deliverable
Continue until there is enough context, then write a SOAP-formatted assessment with candidate remediations back into the workspace.

What comes out of the loop is not a chat response. It is a set of record fragments that bear on the question, in order, with their FHIR sources attached - assembled into a SOAP-formatted assessment with candidate remediations for a clinician to review. An agent that reports the boundaries of what it retrieved can be audited. One that quietly returns whatever happened to fit cannot.

The clinician stays in the loop

Nothing the system produces reaches a patient without review. The SOAP note and proposed remediations arrive as a draft; the physician edits, corrects, rejects, or approves. What gets compressed is the gap between being handed a 400-page chart you have never seen and having a defensible working assessment with the evidence attached. The judgment call stays with the physician.

Provenance tracking is what makes that review workable. Every claim in the assembled context points at a specific fragment of the record, so a reviewer can trace the reasoning back to the source rather than take it on faith. Fine-tuning used the customer's own de-identified FHIR records held in AWS HealthLake, plus synthetic data they provided. No PHI or PII was involved at any stage - not in retrieval development, not in fine-tuning, not in evaluation.

Three agents, one training loop

Post-training ran end to end through PerceptEye's three agents. The whole tuning run cost under $500 - and that number is close to the point. Capital is no longer the constraint on building a domain-specific model. The constraints are knowing what to train on, how to score it, and how to keep the loop from going sideways.

Scout
Reads FHIR tool specs and customer data to enumerate the tasks a physician would want to accomplish - the workflow surface training targets.
Compass
Writes the reward functions and runs the sandbox and REPL loop across multi-turn trajectories, producing advantages and rewards for each rollout.
Maestro
Monitors progress, tracks training, and provisions infrastructure - on Tinker, Fireworks, Baseten, or local compute. This run used Tinker.

The results

The system runs on a fine-tuned Qwen3.6-35B-A3B. It is a Mixture-of-Experts model, so while it holds 35 billion parameters, only about 3 billion are active on any forward pass - which is where the cost story comes from. We serve it at a flat $0.54 per million tokens. To find out whether the specialization transferred, we ran it against PhysicianBench: long-horizon tasks adapted from real primary-care-to-subspecialty consultations, spanning 21 specialties, graded against structured checkpoints, averaging 27 tool calls apiece.

PhysicianBench
Pass@1 — share of tasks answered correctly
Held-out benchmark of 21 specialties, ~27 tool calls per task. Higher is better.
Serving price
Output cost — $ per 1M tokens
List prices at standard tiers, no cache or batch discounts. Lower is better.
The sweet spot
Accuracy against output cost
Up and to the left is better: more answers right, fewer dollars per million tokens. The tuned specialist sits alone in that corner.
PhysicianBench Pass@1 percentage against inference cost - selected comparison
ModelPass@1In $/1MOut $/1M
PerceptEye-tuned 35B (3B active)35.00.540.54
Claude Opus 4.631.75.0025.00
GPT-5.427.72.5015.00
Prior frontier stack (incumbent)6.02.0012.00

List prices at standard tiers, with no cache or batch discounts applied. The tuned model bills at a single flat rate for input and output.

At 35.0 the tuned model sits ahead of every model in the comparison, taking six of the eight specialty groups outright. Against the incumbent frontier stack the customer had been running, overall Pass@1 moves from 6.0 to 35.0 - roughly 6x - while list pricing drops from $2 and $12 per million input and output tokens to a flat $0.54: about 4x cheaper on input and 22x cheaper on output. That 6.0 reads worse than it should: across 27 tool calls, general-purpose models tend to fail hard rather than gracefully, and one malformed call ends the trajectory. Closing that long-horizon gap is most of what tuning buys you.

The honest caveat

The models this beats are general-purpose - none were tuned for the customer's workflows or for PhysicianBench, and this is a selected comparison rather than the full leaderboard. It is not like-for-like. What it does establish is the commercial point: in a narrow, well-defined domain, a much smaller model trained on the right data with the right reward functions produces denser, more dependable behavior than a large model reasoning from general priors - and still holds up on held-out clinical agent tasks. Buying scale as a substitute for specificity works. It is just an expensive way to get there.

What this generalizes to

None of the architecture here is specific to healthcare. The pattern is a corpus too large and interconnected for single-pass retrieval, a domain where general models are competent without being dependable, a reviewer who needs to audit the reasoning rather than just receive an answer, and volume economics that make frontier API calls impractical. Legal discovery fits that description. So do claims adjudication, security investigation, and industrial diagnostics.

In the customer's words
“Our customers need to see the whole patient - not a compressed version of one. We had been working with a frontier model from a major cloud provider, and while it was impressive in general-purpose use, it struggled to reason reliably across the full breadth of a longitudinal history. PerceptEye improved the model's ability to reason across complex clinical records while delivering a far more sustainable cost structure. They move quickly without compromising rigor, and understand that AI in healthcare must support - not replace - clinical judgment.”
AH
CEO
Assemblage Health Inc.
About the partners
PerceptEye, Inc.

Builds agentic pipelines for autonomous post-training. Compass defines how the model learns by writing the rewards and running the multi-turn loop, and Maestro handles fine-tuning and training infrastructure.

Assemblage Health Inc.

The company behind AVNii, a clinical intelligence platform that brings together pre-charting, scribing, discharge summaries, and ICD-10 / SNOMED CT coding to give clinicians a complete view of the patient.

If you have a domain where a tailored model would beat a general one - and where reliability and token cost matter as much as they did here - reach out at hello@percepteye.ai.

A tailored model that beats a general one

PerceptEye builds agentic pipelines for autonomous post-training. Book a discovery call to see what a specialist could do for your domain.