Duplicate Detection

When you save a new idea, NeuralRepo checks your existing ideas for potential duplicates using vector similarity. This keeps your repository clean without requiring you to remember every idea you have ever captured.

How It Works

Duplicate detection runs automatically as part of the idea creation pipeline:

A new idea is created and queued for processing.
The queue worker generates an embedding using the @cf/baai/bge-m3 model.
The embedding is compared against all existing idea embeddings in Cloudflare Vectorize.
If an existing idea scores above the dedup threshold (default 0.75), a duplicate_detections record is created with status pending.
If an existing idea scores above the related threshold (default 0.5) but below the dedup threshold, a related relation is created automatically instead.

Viewing Duplicates

The dashboard shows a Duplicates badge in the navigation when pending detections exist. Click it to open the duplicates review panel, which displays pairs of ideas side by side with their similarity score.

# List all pending duplicate detections
nrepo duplicate list

# Output:
# Detection #1  Score: 0.92
#   Idea 42: "Background job processing system"
#   Idea 58: "Async worker queue for tasks"
#
# Detection #2  Score: 0.88
#   Idea 11: "Mobile habit tracker"
#   Idea 37: "Daily routine tracking app"

GET /api/v1/ideas/duplicates

Response:

{
  "detections": [
    {
      "id": 1,
      "idea_id": 58,
      "duplicate_of_id": 42,
      "score": 0.92,
      "status": "pending",
      "created_at": "2026-03-20T14:30:00Z"
    }
  ]
}

Resolving Duplicates

Every pending detection must be resolved with one of two actions: dismiss or merge.

Dismiss

If the detection is a false positive — the ideas are similar but distinct — dismiss it. The detection is marked as dismissed and will not appear again.

POST /api/v1/ideas/duplicates/1/dismiss

Merge

If the ideas are truly duplicates, merge them. Merging combines both ideas into one and cleans up the duplicate.

The merge process:

Bodies are concatenated — the absorbed idea’s body is appended to the kept idea’s body, separated by a horizontal rule.
Tags are unioned — all tags from both ideas are combined (up to the 20-tag limit).
A supersedes relation is created — the kept idea supersedes the absorbed idea.
The absorbed idea is archived — it is removed from active views but retained in the database for history.

# Merge: keep idea 42, absorb idea 58
nrepo merge 42 58

POST /api/v1/ideas/duplicates/1/merge

The merge endpoint uses the detection record to determine which idea is kept (the older one) and which is absorbed (the newer one).

Configuring the Threshold

The dedup threshold controls how similar two ideas must be to trigger a duplicate detection. Adjust it in your user settings.

Setting	Range	Default	Effect
`dedup_threshold`	0.1 - 0.9	0.75	Higher = fewer detections, only near-exact matches
`related_threshold`	0.1 - 0.9	0.50	Ideas above this but below dedup get a `related` relation

curl -X PATCH https://neuralrepo.com/api/v1/user/me \
  -H "X-API-Key: nrp_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"settings_json": {"dedup_threshold": 0.80}}'