Conversation

• R RSS - Really Simple Syndication RSS - Really Simple Syndication remote Aug 12, 2026, 11:59 PM You are here

  Deduping 64 AI feeds so I could count how many outlets cover the same story — four layers, one needed a language model

  I run an AI news reader (https://helucino.com — mine, free, no account needed to read) that pulls 64 feeds every 15 minutes. Reading them was easy. The thing I wanted was harder: rank stories by how many separate outlets are covering them. Five news websites independently publishing the same story is definitely a signal that the news is important — and to get it you must first know that four articles are one story. So dedupe was the problem I picked.

  Four layers, cheapest first:

  1. Canonical URL hashing — kills syndication and tracking-param variants.
  2. Exact normalized-title match — free, catches straight reprints.
  3. Title-token similarity — Jaccard + containment, but gated on shared named entities. Without that gate "Meta open-sources X" merges with "OpenAI open-sources X", which is definitely a wrong answer.
  4. Embeddings to shortlist, then a language model to decide — asked in both directions and these must agree, because one-directional prompting says yes far too easily.

  The output is a +N badge per row, and a sort order. Ranked orders purely by that count; Top applies the same count with an 18-hour half-life so recency still matters. A false merge is much worse than a missed one here — merging two stories hides one of them and inflates the other's count — which is why layer 4 has to agree with itself before it will collapse anything, and why complete linkage is checked both ways before two groups join.

  Current reality check: 63 of 64 feeds green. The one that isn't is Hacker News, which is a bit on the nose. Genuinely curious how others handle publishers that block feed readers — mirror, back off, or drop them?

  submitted by /u/Ok-Difficulty-7607
  [link] [comments]
  source