Conversation

• R RSS - Really Simple Syndication RSS - Really Simple Syndication remote Sep 14, 2026, 8:53 AM You are here

  Clustering high-frequency RSS feeds without paid tools or vector bloat

  A friend of mine wanted to set up an automated news forwarder for his community, and looking at existing options made the whole thing frustrating. Most commercial feed aggregators charge recurring subscriptions just for basic deduplication and webhooks.

  Even when you pay, you get almost zero granular control over publisher weighting, alert cadence, or how the output is tailored with custom prompts and analytics. Meanwhile, subscribing to active industry feeds floods the channel with duplicate alerts whenever a breaking wire drops.

  Instead of paying for SaaS tiers or overcomplicating things with vector databases and heavy pipelines, I built Celerum to handle this deterministically on tiny hardware like a cheap VPS or a Raspberry Pi. It is completely open source and runs as a single Go binary with an embedded SQLite database.

  Here is the approach I took to keep resource usage low:

  • RFC 7232 HTTP conditional polling: Feeds cache ETag and Last-Modified timestamps in SQLite. Unchanged feeds return 304 Not Modified with zero response body, finishing in under 50ms and consuming almost no bandwidth.
  • Algorithmic clustering instead of embeddings: Raw text gets cleaned, stripped of stop-words, and hashed into sorted 64-bit FNV-1a shingles. Breaking news inherently shares named entities, figures, and tickers, making lexical matching faster and less prone to semantic false positives than vector embeddings.
  • Inverted index candidate pruning: Before running similarity checks, an in-memory inverted index skips document pairs with insufficient shingle overlap. This bypasses over 80 percent of pairwise comparisons.
  • Allocation-free Jaccard scanning: Candidate pairs get evaluated with a two-pointer linear scan across sorted []uint64 slices with zero heap allocations, then merged using Disjoint-Set Union (Union-Find). It clusters 100 articles in about 2.4ms.
  • Velocity scoring: Clusters gain weight based on multi-source coverage, publisher tiers, and keywords, balanced by a linear time decay penalty so old stories naturally fade out.
  • Hybrid alert cadence: If multiple tier-1 outlets confirm a story quickly, it triggers an immediate alert. Quieter stories stay in a sliding window, and an hourly heartbeat ticker flushes the top stories so you do not miss context.
  • Decoupled synthesis and fallbacks: When an event qualifies, you have full control over the system prompt sent to an OpenAI-compatible LLM endpoint for the final briefing. If the AI provider times out or fails, it falls back to raw source links so delivery is never delayed or dropped.

  Right now it dispatches to Telegram or generic HTTP webhooks. The entire engine sits under 15MB of RAM while monitoring active sliding windows.

  The code and architecture documentation are up on GitHub: https://github.com/rafifmsn/celerum

  It is far from perfect, just something I put together over a weekend to solve an immediate headache without spinning up paid infrastructure. Always open to feedback from anyone running similar self-hosted pipelines.

  submitted by /u/rafifmsn
  [link] [comments]
  source