Field-Level Ensemble OCR: Getting 74.8% Accuracy from Two Mediocre Vision Models

A registration clerk at one of our clinics squinted at the screen, deleted the garbled member ID the system had pulled from an insurance card photo, and typed it in by hand. Fourth card that morning. She’d been doing this for months — the OCR (optical character recognition, software that reads text from images) was “mostly” right, which meant it was also partly wrong often enough that staff had memorized which fields to double-check first.

I had shipped that OCR system.

Our single-model approach — minicpm-v:8b, a compact AI model that can look at images and read text from them, running through a self-hosted CardOCR API — was hitting about 56% exact match accuracy across 10 key fields on a 27-card test set. Nearly half the fields needed manual correction. Staff were spending almost as much time fixing OCR output as they would have spent typing it manually.

The fix was not a better model. It was routing each field to whichever model was already better at it.

The models failed in different places

I ran a systematic comparison of two open-source vision models — AI systems that can read text from images — against our ground truth dataset of manually verified card data. The two contenders:

• minicpm-v:8b — A compact multimodal model (processes both images and text), great at short, structured fields like member IDs and group numbers.
• llama3.2-vision:11b — Meta’s larger vision model, with a bigger language head (the part that turns what it sees into words), better at reading variable-length descriptive text.

I initially picked minicpm-v because it was faster and its overall accuracy looked better. That was the wrong comparison. Aggregate scores hide everything that matters; this one hid the fields staff actually had to fix.

The field-level breakdown was more useful.

FIELD_ROUTER = {
    "member_id":       "cardocr",
    "group_number":    "cardocr",
    "subscriber_name": "cardocr",
    "payer_id":        "cardocr",
    "plan_name":       "cardocr",
    "copay":           "cardocr",
    "payer_name":      "ollama",
    "rx_bin":          "ollama",
    "rx_pcn":          "ollama",
    "rx_grp":          "ollama",
}

def merge(cardocr_result: dict, ollama_result: dict) -> dict:
    return {
        field: (cardocr_result if src == "cardocr" else ollama_result).get(field)
        for field, src in FIELD_ROUTER.items()
    }

minicpm-v dominates structured ID fields — member IDs, group numbers, and subscriber names. llama3.2-vision wins on the pharmacy triplet (RxBIN, RxPCN, RxGrp — the codes that identify which pharmacy benefit plan covers the patient) and payer names. Neither model is good at everything. Together they cover each other’s blind spots.

If I had compared only overall accuracy, I would have picked minicpm-v (56% vs 53%) and missed that llama3.2-vision was significantly better on 4 out of 10 fields. I nearly made that choice. Field-level evaluation matters more than the aggregate score because the aggregate hid those four fields.

Route fields instead of picking a winner

The ensemble approach is dead simple: instead of choosing one model, send the card image to both, then route each field to whichever model is best at extracting it. Two 8–11B parameter models running on one GPU, each mediocre alone, outperform either one alone when the fields are routed intelligently. Before upgrading to a bigger model, I would check whether the models already have complementary strengths.

Insurance Card Image
        │
        ├──→ CardOCR (minicpm-v:8b)  ──→ member_id, group_number,
        │                                  subscriber_name, payer_id,
        │                                  plan_name, copay
        │
        └──→ Ollama (llama3.2-vision) ──→ payer_name, rx_bin,
                                           rx_pcn, rx_grp
        │
        └──→ Merge by field routing table

Both models run in parallel — a single model already takes 3-5 seconds, so running two concurrently costs almost nothing extra. Total latency is roughly 5-6 seconds: the slower of the two, not the sum of both.

This is the part of an ensemble that is easy to miss. Parallel execution makes ensembles cheap: two concurrent models cost max(model_a, model_b), not model_a + model_b. An insurance card upload already takes a few seconds, so the extra model is not noticeable here.

What changed

The ensemble moved the needle — not dramatically, but meaningfully:

“Exact match” means the extracted text matched the ground truth character-for-character. “Fuzzy match” gives partial credit for close-but-not-perfect extractions — a member ID like “WXD-1234” versus “WXD1234” counts as fuzzy-correct. A 3.4-point exact-match improvement means fewer fields for staff to correct per card. On a busy day with 50+ patient check-ins, that adds up.

Is 59.3% good enough? No. It is better than 55.9%, with plenty of headroom left. The shape of the improvement matters more than the absolute number: it came from routing, not a bigger model.

How It Runs in Production

The ensemble stays entirely inside extract_insurance_card(), the same method called by all five integration points: patient registration, the admin EHR system, the re-run endpoint, and two others. Zero downstream code changed.

It sits behind OCR_ENSEMBLE_ENABLED=true, a feature flag that turns the ensemble on or off without a redeploy. With the flag set to false, behavior is identical to the original single-model path. We shipped the code dark, flipped the flag in one clinic first, and could roll back instantly if something broke.

The service dispatches both models with a Python thread pool (a mechanism that runs multiple tasks concurrently on the same machine), then merges the results through the routing table:

# Both models run concurrently
with ThreadPoolExecutor(max_workers=2) as executor:
    cardocr_future = executor.submit(call_cardocr, image)
    ollama_future = executor.submit(call_ollama, image)

If Ollama fails or times out, the system falls back to CardOCR-only results. If CardOCR fails but Ollama succeeds, we use what we have. The ensemble path never returns worse results than the original single-model path.

AI models do not always return clean JSON, the structured data format programs use to exchange information. Ollama occasionally wraps its output in markdown code blocks, truncates mid-response, or produces mismatched braces. The integration layer strips markdown formatting, repairs truncated JSON where possible, and falls back to regex extraction — pattern-matching the text directly — as a last resort.

The system is live now at our clinics. Next step: expanding the test set and checking whether a third model — or a fine-tuned one trained specifically on insurance cards — can push accuracy past 80%.