Local Invoice Intelligence

Challenge

Enterprise document extraction workflows have two unsolvable tensions when using frontier APIs: privacy (invoice data contains vendor names, addresses, and financial figures that cannot leave the corporate perimeter) and cost (GPT-5.5 at scale for batch invoice processing is economically unviable).

The engineering question was precise: how close can a fully local model get to frontier accuracy on a real benchmark, with no fine-tuning, using only prompt engineering and hybrid transcription?

Architecture

The extraction pipeline is a four-stage local flow:

1. Hybrid Transcript — pdfplumber extracts text with layout preservation for text-native PDFs; Tesseract OCR activates as fallback for scanned pages. Both paths produce a normalized text transcript.
2. Field Schema — user defines extraction targets in YAML with type hints (string, date, money, number). The schema is compiled into a Pydantic model at runtime.
3. Local LLM Structured Output — Ollama runs qwen3:14b in no-thinking mode with constrained JSON output matching the Pydantic schema. Single model call per document.
4. Validation + Export — per-field type validation with warning metadata. Output is JSON for single documents or Excel (one row per PDF, one column per field) for batch folders.

The benchmark path runs via a DocILE adapter: same pipeline over 297 validation invoices with semantic scoring tolerant of OCR formatting artifacts, currency symbols, and date format variants.

Friction and Deep Diagnostics

The most counterintuitive finding: the "baseline-plus rescue" architecture (run extraction, gate on confidence, re-run low-confidence fields with a second model call) did not improve accuracy meaningfully. After extensive testing with shadow rescue and safer gating thresholds, net lift was within noise. The experiment is preserved in experiments/ for reproducibility, but it does not drive the product architecture. Knowing when to abandon a hypothesis is as valuable as building the feature.

The second friction point was qwen3:14b thinking mode. Extended chain-of-thought reasoning activates by default and significantly increases latency (~18s → ~35s per document) with no measurable accuracy improvement on structured extraction tasks. Explicit no-thinking mode via Ollama parameter was the fix — a reminder that reasoning tokens have a cost-accuracy tradeoff that varies by task type.

OCR quality was the dominant variance factor across models. The semantic scorer had to tolerate: address line breaks mid-field, $ vs empty string for zero amounts, DD/MM/YYYY vs YYYY-MM-DD, partial company name matches from OCR character substitutions.

Metrics and Validation

Benchmark: 297 invoices from DocILE validation set. All inference local on Apple Silicon.

Key finding: qwen3:14b outperforms GPT-5.5 on the hardest field — Gross Total — by 5.72 percentage points (76.43% vs 70.71%), at zero API cost. The 1.28pp aggregate gap is within practical tolerance for most private document workflows.