Python RSS bot: Filter noise with SQLite & LLM (without blowing your budget)

The first 24 hours of building a personal news monitor, crypto ticker, or industry-alert bot are pure euphoria. By the second week, you are frustrated. For two reasons:

1. The duplicate problem: the same story comes in through several feeds or gets re-processed every cron tick. Your Telegram bot ends up spamming you with the same info.
2. The cost problem: naively forwarding every raw item to a Large Language Model like OpenAI will burn through your credit card.

In this article we walk through a smart filter pipeline built on SQLite and disciplined prompt design. It gives you a robust, production-ready alert system that only delivers relevant hits — and keeps your API spend minimal.

1Memory — blocking duplicates with SQLite

Before any LLM enters the picture, we need to discard articles we have already seen. A simple in-memory list will not work, because the script restarts on every cron tick or systemd timer run. SQLite is ideal here: lightweight, file-based, zero server setup.

We create a table that stores the unique hash or URL of every processed item:

import sqlite3

def init_db(db_path="alerts.db"):
    conn = sqlite3.connect(db_path)
    cursor = conn.cursor()
    cursor.execute("""
        CREATE TABLE IF NOT EXISTS processed_articles (
            id TEXT PRIMARY KEY,
            title TEXT,
            fetched_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        )
    """)
    conn.commit()
    conn.close()

On every run we check whether the article_id (e.g. the RSS entry's URL) already exists. If it does, we skip the item immediately. This first stage alone can eliminate up to 90 % of unnecessary downstream work.

2The pre-filter — keywords over expensive AI

An LLM should never be your first line of defence. If you care about IPO announcements, you do not want to forward every weather or sports article to the OpenAI API.

Use a small, context-aware keyword list as a pre-filter — either with regex or simple if keyword in text checks. Only items that pass this coarse filter qualify for the more expensive AI classification.

3The LLM pipeline with structured output

Now the heart of the system: the LLM decides whether the article really meets your criteria. For automation to work, the model must not respond with prose — it has to return a clear, machine-readable decision (typically JSON).

Here is a battle-tested prompt pattern for this use case:

import openai

def evaluate_article_with_llm(title, summary, target_topic):
    prompt = f"""
    Analyze the following article and decide whether it contains a relevant piece of news on the topic "{target_topic}".
    Respond EXCLUSIVELY in the following JSON format:
    {{
        "relevant": true or false,
        "reason": "Short justification in one sentence"
    }}

    Article title: {title}
    Summary: {summary}
    """

    # Example call (e.g. with OpenAI or a local Ollama model)
    response = openai.chat.completions.create(
        model="gpt-4o-mini", # Pick a cost-efficient model!
        messages=[{"role": "user", "content": prompt}],
        response_format={ "type": "json_object" }
    )

    return response.choices[0].message.content

Wrap-up & the road to a production bot

With this three-stage pipeline (SQLite pre-filter → keyword check → LLM classification) you get a system that filters with high precision while keeping your API costs low.

Want to see how to package these components into a clean, object-oriented Python architecture, add error back-offs for failing feeds, and run the whole thing as a Docker container on a server without surprises?

The full book “Watch & Alert – Automated news and market monitoring with Python” walks you step by step through a production-ready build — including the complete open-source code on GitHub.