Challenge
Current RAG implementations treat retrieval as a commodity: embed, retrieve, generate. The two gaps this ignores are attribution (whose intellectual work fed this answer?) and economics (how do you pay fractional cents to an author for a paragraph retrieved?). Generic LLMs hallucinate or give opaque answers; they have no source of truth and no accountability mechanism.
The engineering constraint was building a system that is simultaneously grounded, verifiable, and economically attributable — all while supporting a local-first mode with zero API cost.
Architecture
The workflow runs as a LangGraph state machine with five primary nodes:
- Guardrail — input safety check; rejects off-topic or harmful queries before any retrieval
- Router + Query Expansion — classifies intent and generates sub-queries to improve recall
- Hybrid Retriever — combines ChromaDB vector search, BM25 lexical search, and Reciprocal Rank Fusion; optional cross-encoder reranking
- Royalty Ledger — for each retrieved chunk, looks up
catalog.jsonfor book price and total chunk count; credits fractional cost to a per-query ledger - Synthesis — grounded generation with source cards; each output citation maps to a specific author, book, and paragraph
Two inference profiles: local (MLX LLM + nomic-embed + local cross-encoder, zero API calls) and cloud (GPT-5 family with gpt-5-nano as router, gpt-5-mini as synthesizer, optimized for demo latency).
Friction and Deep Diagnostics
The hardest engineering problem was the semantic cache invalidation strategy. A naive exact-match cache would miss semantically identical queries with different wording. A pure vector cache would over-match unrelated queries with similar embeddings. The solution: cosine similarity threshold at 0.92, with cache keys stored as compressed query embeddings. Queries below threshold miss the cache and trigger a full graph traversal.
The second non-trivial problem was BM25 index freshness. When new books are ingested, the BM25 index must be rebuilt in full — BM25 is not incrementally updatable without a full reindex. The ingestion pipeline handles this with a staged approach: vector DB updates first, BM25 rebuilt second, catalog.json chunk counts patched last.
The "library gap" logging was built as an observability signal, not just an error handler. When the router classifies a query as out-of-scope, it logs the query to a gap ledger visible in the dashboard. This turns user failures into a content acquisition signal.
Metrics and Validation
| Signal | Detail |
|---|---|
| DeepEval scores | Faithfulness, answer relevance, contextual precision — tracked live in the dashboard |
| Semantic cache | Hit rate visible in observability view; threshold tuned to 0.92 cosine similarity |
| Local mode latency | ~2–4s per query end-to-end on Apple Silicon M-series, zero API cost |
| Cloud demo latency | ~1.8s average with gpt-5-mini synthesis |
| Royalty ledger | Exact fractional attribution per chunk; total reconciles against per-book pricing in catalog.json |
The micro-royalty system is novel: at scale, a platform could automatically compensate authors proportional to how often their writing is cited in AI answers — a first-principles approach to AI attribution economics.