Back to work

RAG · Document Analysis

RepeatRadar

Semantic Repetition Detection for Long Documents

Finding semantically repeated ideas across a 100+ page document, not just repeated text.

GeminiFAISSFastAPIStreamlitMCP
View on GitHub
Chapters represented as nodes in an embedding-similarity graph. Most chapter pairs are weakly connected; a pair above the similarity threshold is flagged for the author to review as a candidate repetition.flagged: similarity> thresholdCh.1Ch.2Ch.3Ch.4Ch.5

Problem

While writing a PhD thesis, the same idea, definition, or argument can end up restated in more than one chapter without the author noticing: re-reading the whole document to catch it does not scale. Exact-text search does not help either, since the repeated idea is usually reworded, not copy-pasted, so finding it requires comparing chapters on meaning rather than on matching strings.

Pipeline

Chapters are chunked and embedded with Gemini embeddings, then indexed in FAISS for nearest-neighbour search. Chunk pairs whose similarity exceeds a fixed threshold are flagged as candidate repetitions and passed on for human review.

Chunk, embed, search, review

Chapters represented as nodes in an embedding-similarity graph. Most chapter pairs are weakly connected; a pair above the similarity threshold is flagged for the author to review as a candidate repetition.flagged: similarity> thresholdCh.1Ch.2Ch.3Ch.4Ch.5

Example

Illustrative repetition example

Illustrative repetition example

Chapter 2

The evaluation protocol separates retrieval quality from final answer quality so that improvements can be attributed to the correct stage of the pipeline.

Chapter 4

Retrieval and generation are assessed independently, allowing changes in answer quality to be traced back to the stage that produced them.

Suggested fix

RewriteShortenCross-reference

Review decision

AcceptRejectEdit

Illustrative passages, not real thesis text. No similarity score is shown; RepeatRadar flags candidates qualitatively above a fixed threshold.

Review

Suggested action

  • · Rewrite
  • · Shorten
  • · Cross-reference

Review decision

  • · Accept
  • · Reject
  • · Edit

RepeatRadar does not silently rewrite the document. It proposes a candidate and leaves the final editorial decision with the author.

Integration

The pipeline is served through FastAPI, with a Streamlit interface for reviewing flagged passage pairs across a document. The same repetition-check is also exposed over MCP, so it can be invoked as a tool from an MCP-compatible client instead of only through the web UI.

Limitations

  • · Flags candidate repetitions for human review rather than auto-rewriting: automatically rewriting already-reviewed thesis text would risk silently changing its meaning.
  • · Similarity threshold is tuned manually per document rather than learned.
GitHubAll projects
Next: ApplyPilot