Back to work

Agentic AI · RAG · Production Engineering

DocuMindAI

Multi-Agent Document Intelligence Platform

A document AI system designed to remain grounded and observable.

LangGraphHybrid RAGDense + BM25RRFCross-EncoderFastAPI
View on GitHub
DocuMind multi-agent architecture: a Supervisor routes a user query to a Retriever Agent (hybrid dense and BM25 retrieval, RRF fusion, cross-encoder reranking), optionally a Summarizer Agent, and a Synthesizer Agent that performs LLM generation over the retrieved evidence. A Critique Agent checks whether the answer is grounded; if not, a bounded corrective retry reformulates the query and routes back through the Retriever Agent before generation is attempted again.bounded corrective retryUser QuerySupervisor/ RouterRetrieverAgentDense+BM25RRF→RerankSummarizerAgentSynthesizer AgentLLM GenerationCritique Agentgrounded?AnswergroundedReformulatecorrective

Problem

Typical RAG demos stop at retrieving chunks and asking an LLM. The harder problem is what happens when retrieval is wrong, evidence is insufficient, or an answer needs to be corrected without allowing an unbounded agent loop.

A production-oriented multi-agent RAG system that routes queries, retrieves and reranks evidence, generates cited answers, critiques faithfulness, and performs bounded corrective retries when answers are not sufficiently grounded.

Architecture

User query, input middleware and a basic injection check, Supervisor, routing to Retriever, Summarizer, or direct synthesis, hybrid retrieval (dense + BM25), Reciprocal Rank Fusion, cross-encoder rerank, Synthesizer, citation validation, Critique, and if not faithful, query reformulation with a bounded retry, ending in a final answer with sources, trace, and cost.

DocuMind multi-agent architecture: a Supervisor routes a user query to a Retriever Agent (hybrid dense and BM25 retrieval, RRF fusion, cross-encoder reranking), optionally a Summarizer Agent, and a Synthesizer Agent that performs LLM generation over the retrieved evidence. A Critique Agent checks whether the answer is grounded; if not, a bounded corrective retry reformulates the query and routes back through the Retriever Agent before generation is attempted again.bounded corrective retryUser QuerySupervisor/ RouterRetrieverAgentDense+BM25RRF→RerankSummarizerAgentSynthesizer AgentLLM GenerationCritique Agentgrounded?AnswergroundedReformulatecorrective+ citation validation, Qdrant, PostgreSQL, Redis, Prometheus

Evaluation

Retrieval and generation are evaluated independently. The project includes a 137-question retrieval benchmark, faithfulness/grounding evaluation, abstention testing, live-call reliability checks, adversarial testing, and CI regression gates.

Retrieval evaluation

Improvement vs dense-only baseline; bars visualize relative delta magnitude, not absolute retrieval scores.

MRR+6.2 points
nDCG@10+5.0 points

Grounding / abstention

Eval fact

0.96–0.99

Faithfulness

Eval fact

92.9%

Correct abstention

Eval fact

100%

Citation validity

Reliability

Every query records agent nodes, latency, sources, token usage, retry behavior, and cost. Bounded retries prevent unbounded agent loops. In live traffic testing, the system recorded zero failures across 60 live calls and zero successful attacks across 12 scoped adversarial attempts.

Guardrails

Eval fact

0/12

Successful scoped adversarial attacks

Corrective retry

Eval fact

31.6%

Corrective-retry trigger rate

Eval fact

$0.00207

Added cost / query from retries

Schematic trace

Schematic traceSchematic node flow, no per-node timing/cost values are invented here. Aggregate values shown alongside are the measured ones from the metrics above.
Supervisor→Retrieval→Rerank→Synthesis→Critique→Retry triggered

Eval fact

31.6%

Corrective-retry trigger rate

Eval fact

$0.00207

Added cost / query from retries

Production

The system uses hybrid dense/sparse retrieval, RRF fusion, cross-encoder reranking, supervisor-led routing, and a critique node that triggers bounded corrective retries.

Observability

Production features include async document ingestion, JWT auth, Redis-backed rate limiting, request-level traces, structured and validated LLM outputs, timeouts and retries, and a CI gate for retrieval/faithfulness regressions.

Stack & infrastructure

ClaudeQdrantPostgreSQLRedisDockerPrometheusGrafanaGitHub Actions

Limitations

  • · A retry-exhaustion edge case is explicitly documented rather than hidden: when corrective retries are exhausted, the system falls back to a clearly labeled best-effort answer instead of failing silently.
GitHubAll projects
Next: MailTune