What this is about
An RSS bot that sends every new entry to a language model is quickly built and quickly expensive. Most entries are repeats, obvious noise or simply irrelevant to you. Yet every call is paid for, in money with a cloud API and in compute time with a local model.
The solution is a filter chain: cheap, deterministic stages sort out first, and the model sees only the rest.
The filter chain at a glance
- Deduplication: entries that were already processed are dropped.
- Keyword filter: positive and negative lists remove the obvious.
- Cache: similar or identical texts are not classified again.
- Classification: only now is the model asked.
- Daily budget: an upper limit on calls protects against outliers.
Each stage is cheaper than the next. That is why the model comes last.
Stage 1: duplicates via SQLite
Every entry gets a stable key, usually the GUID of the feed or, as a fallback, a hash of link and title. A table with this key as primary key does the check.
import hashlib
import sqlite3
def key_for(link: str, title: str, guid: str | None) -> str:
raw = guid or f"{link}|{title}"
return hashlib.sha256(raw.encode("utf-8")).hexdigest()
def is_new(db: sqlite3.Connection, key: str) -> bool:
cur = db.execute(
"INSERT OR IGNORE INTO seen(key) VALUES (?)", (key,))
db.commit()
return cur.rowcount == 1
INSERT OR IGNORE yields rowcount == 1 only the first time. That way you check and remember in one step.
Stage 2: keywords
A small negative list (for example advertising or prize draws) and a positive list with your topics sort roughly in advance. This is imprecise but very cheap. Important: whatever the filter discards should be logged so that you can later check whether it is too strict.
Stage 3: cache for classifications
Store the result of every classification together with a hash of the input text. If the same text shows up again in another feed, you read the result from the database instead of asking again. For news that several sources pick up, this saves a noticeable number of calls.
Stage 4: classification with a clear output format
Keep the prompt short and ask for a fixed answer, such as one of a few labels. Short answers are cheaper and easier to evaluate.
LABELS = ("relevant", "routine", "ignorieren")
def classify(text: str, ask) -> str:
prompt = ("Ordne die Meldung genau einem Label zu: "
+ ", ".join(LABELS) + ".\n\n" + text[:1500])
answer = ask(prompt).strip().lower()
return answer if answer in LABELS else "routine"
The text is truncated because long inputs drive up costs. Unknown answers fall back to a safe default label so the pipeline does not break.
Stage 5: daily budget
Count the calls per day in the database. Once the budget is reached, further entries are deferred without classification and processed the next day. That way a sudden flood of entries, for example after adding a new feed, cannot cause unexpected costs.
A note in the daily status message about how many entries were deferred makes this behavior visible.
Common mistakes
- The model is asked before deduplication.
- The cache key contains timestamps and therefore never hits.
- The keyword filter discards silently, without a log.
- There is no budget, so a bug in a loop triggers many calls.
When it is worth it (and when not)
The filter chain is worth it as soon as you have several feeds or use a paid API. Even the first two stages can reduce the number of calls considerably, but you have to measure the exact effect on your own feeds.
With few feeds and a local model that runs for free anyway, deduplication is enough. Keyword filter and cache then mainly add complexity and little benefit.