Where we started
My Watch-&-Alert setup currently pulls in 87 RSS and Atom feeds, primarily from IT security, open-source releases, and a few industry publications. On top of that come four webhooks from GitHub, one from Sentry, and a custom heartbeat endpoint. On average, between 12 and 40 alerts flow through per day; on peak days around CVE advisories or major releases, easily 200 within a few minutes.
The send pipeline has been running smoothly for months. Classification runs locally via Ollama with a small qwen2.5:7b; only when the Ollama host is unreachable does the system fall back to a cloud model. SQLite handles persistence, a systemd timer fires the next run every two minutes. The full send to Telegram was running synchronously in the main process: alert comes in, LLM finishes, requests.post to the Telegram API, done.
Then came that Tuesday. A large tech company released a tool suite, a popular open-source project shipped a major release, and an HSE newsletter dropped a batch update in parallel. Within four minutes, 28 alerts came crashing down. Telegram returned HTTP 429 for some of them, the rest went through. In my channel, seven messages were missing at the end — not because the system hadn't detected them, but because the send path had crashed into the rate limits during the burst. The system itself wasn't broken. It was just badly designed for exactly that case.
Requirements
Before I write code, I write down what the solution needs to do. That saves rework and stops me from later building something by the book that doesn't fit my setup. In our case, the requirements on a sensible send path are concrete:
- No alert may get lost. When Telegram returns a 429, the message needs to wait and try again later, instead of being dropped.
- Ordering doesn't have to be strictly preserved. If an older alert has to wait briefly so a newer one can go out faster, that's fine in monitoring.
- Backoff with jitter. Telegram sends a
Retry-Afterheader, but not always reliably. An exponential backoff with a random component prevents all waiting messages from hammering at once. - Persistent queue. If the system crashes or restarts during a burst, alerts that are already queued must not be gone. SQLite is there anyway, so I'll use it.
- Heartbeat stays visible. The Watch-&-Alert heartbeat pattern should keep running, so the send path needs to be designed so a stuck worker doesn't slow the heartbeat down.
- No new framework. No Celery, no Redis, no RabbitMQ. We stay with
asyncioand SQLite.
That list is honest, not exhaustive. I deliberately left out things that would show up in enterprise setups — multi-region failover, SLOs for send latency. This is an indie project, not a payment provider.
Three options compared
Before I dig into the solution, it's worth looking at the alternatives I seriously considered. Three options ended up on the list; one stayed.
Option 1: Naive throttling. The simplest path would just be to sleep one second after every sendMessage. One second per message, done. The upside: no code, no queue, five minutes of work. The downside: at 28 alerts in four minutes, you're looking at 28 seconds of backlog, and the next burst extends that linearly. In my case the system is single-threaded — when the send blocks for 28 seconds, the next systemd run goes nowhere, the heartbeat gets delayed, and the LLM pipeline backs up. That's not a solution, that's a workaround of a workaround.
Option 2: Coalescing per channel. Another idea would be merging several alerts into a single digest message. Telegram has a 4096-character limit per message, which is plenty for ten short alerts. That cuts message volume drastically and, frankly, it's the most elegant solution if you have lots of small notifications. The downside: you lose granularity. If a single alert needs to be clickable, that property vanishes inside the digest post. Plus, coalescing works poorly for long Markdown alerts, the kind you get after LLM classification. I kept coalescing as a second line of defense, but not as the primary protection.
Option 3: Dedicated send queue with backoff. The solution I ended up building is a dedicated send worker that pulls alerts from a SQLite table and ships them to Telegram with backoff. One token-bucket limiter per chat that strictly holds the send rate. That's more code than option 1, but structurally cleaner. The main process only writes to the queue, doesn't care about HTTP status codes anymore, and can move on to the next feed right away. This is also the variant described in the book in Chapter 13 under "Decoupling the send path".
The solution
The refactor has three building blocks: a new SQLite table, a worker, and a thin wrapper around the Telegram API. I'll show you the parts that actually matter.
First the table. It's deliberately kept simple, because SQLite doesn't need a high-throughput queue.
import sqlite3
from pathlib import Path
DB_PATH = Path("/var/lib/watchalert/state.db")
SCHEMA = """
CREATE TABLE IF NOT EXISTS outbox (
id INTEGER PRIMARY KEY AUTOINCREMENT,
chat_id TEXT NOT NULL,
payload TEXT NOT NULL,
attempts INTEGER NOT NULL DEFAULT 0,
next_attempt REAL NOT NULL,
created_at REAL NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_outbox_ready
ON outbox(next_attempt);
"""
def init_outbox(db_path: Path = DB_PATH) -> None:
with sqlite3.connect(db_path) as conn:
conn.executescript(SCHEMA)
next_attempt stores a Unix timestamp. The index lets the worker pull all ready messages in one shot. attempts counts the tries for later diagnostics.
The enqueue step replaces the old direct requests.post. It runs where the synchronous send used to live.
import json
import time
import sqlite3
from typing import Any
def enqueue_alert(
conn: sqlite3.Connection,
chat_id: str,
payload: dict[str, Any],
) -> None:
conn.execute(
"INSERT INTO outbox (chat_id, payload, next_attempt, created_at) "
"VALUES (?, ?, ?, ?)",
(chat_id, json.dumps(payload), time.time(), time.time()),
)
The worker is the heart of the system. It runs as an asyncio task, polls the outbox, sends messages with rate limiting and backoff, and respects Retry-After. I'm using aiohttp instead of requests so that waiting doesn't block other tasks.
import asyncio
import json
import logging
import random
import sqlite3
import time
from dataclasses import dataclass
from pathlib import Path
import aiohttp
logger = logging.getLogger("watchalert.sender")
PER_CHAT_INTERVAL = 1.05 # Telegram: ~1 Nachricht/Sekunde/Chat
MAX_ATTEMPTS = 8
@dataclass
class Job:
id: int
chat_id: str
payload: dict
attempts: int
async def send_one(
session: aiohttp.ClientSession,
token: str,
job: Job,
) -> tuple[bool, float]:
"""Return (ok, retry_after_seconds)."""
url = f"https://api.telegram.org/bot{token}/sendMessage"
try:
async with session.post(url, json=job.payload, timeout=10) as resp:
if resp.status == 200:
return True, 0.0
if resp.status == 429:
data = await resp.json()
retry_after = float(data.get("parameters", {}).get(
"retry_after", 1.0
))
return False, retry_after
text = await resp.text()
logger.warning("telegram %s: %s", resp.status, text[:200])
return False, 5.0
except (aiohttp.ClientError, asyncio.TimeoutError) as exc:
logger.warning("telegram transport error: %s", exc)
return False, 5.0
def backoff_delay(attempts: int, base: float = 2.0) -> float:
"""Exponential backoff mit Jitter, gedeckelt auf 5 Minuten."""
raw = min(base ** attempts, 300.0)
return raw * (0.5 + random.random())
async def worker(
db_path: Path,
token: str,
last_seen: dict[str, float],
) -> None:
async with aiohttp.ClientSession() as session:
while True:
with sqlite3.connect(db_path) as conn:
rows = conn.execute(
"SELECT id, chat_id, payload, attempts "
"FROM outbox WHERE next_attempt <= ? "
"ORDER BY next_attempt LIMIT 20",
(time.time(),),
).fetchall()
for row in rows:
job = Job(
id=row[0],
chat_id=row[1],
payload=json.loads(row[2]),
attempts=row[3],
)
wait = last_seen.get(job.chat_id, 0.0)
delta = time.time() - wait
if delta < PER_CHAT_INTERVAL:
await asyncio.sleep(PER_CHAT_INTERVAL - delta)
if job.attempts >= MAX_ATTEMPTS:
logger.error(
"drop alert id=%s after %s attempts",
job.id, job.attempts,
)
with sqlite3.connect(db_path) as conn:
conn.execute(
"DELETE FROM outbox WHERE id = ?",
(job.id,),
)
continue
ok, retry_after = await send_one(session, token, job)
if ok:
with sqlite3.connect(db_path) as conn:
conn.execute(
"DELETE FROM outbox WHERE id = ?",
(job.id,),
)
last_seen[job.chat_id] = time.time()
continue
delay = max(retry_after, backoff_delay(job.attempts))
next_at = time.time() + delay
with sqlite3.connect(db_path) as conn:
conn.execute(
"UPDATE outbox "
"SET attempts = attempts + 1, next_attempt = ? "
"WHERE id = ?",
(next_at, job.id),
)
last_seen[job.chat_id] = time.time()
logger.info(
"retry alert id=%s in %.1fs (attempt %s)",
job.id, delay, job.attempts + 1,
)
await asyncio.sleep(1.0)
The last_seen map is the per-chat limiter. It ensures that even if 30 alerts are queued, only one per second per chat goes out. The jitter in the backoff prevents all messages that Telegram rejected with the same 429 from knocking again in sync a second later. That's a classic anti-thundering-herd pattern.
Wiring it into the Watch-&-Alert main system is a two-liner. Instead of requests.post(...), the dispatcher now calls enqueue_alert(...). In our code path it looks like this:
def dispatch(
conn: sqlite3.Connection,
chat_id: str,
text: str,
) -> None:
enqueue_alert(
conn,
chat_id,
{"chat_id": chat_id, "text": text, "parse_mode": "Markdown"},
)The worker runs as its own systemd service. That matters because it has a different lifetime than the main loop. The main loop fires every two minutes, the sender runs continuously. In practice I handle that with two timer units, but that's a story for another article.
What we learned
Insight 1: HTTP 429 is not optional. If you use a Telegram bot for anything beyond tinkering, you'll hit the limit. The question isn't if, it's when. Bursts are the norm, not the exception.
Insight 2: Per-chat limiters beat global limiters. My first attempt was a simple "one message per 1.2 seconds" without chat context. That worked until I added a second channel. Then it throttled globally to 0.6 messages per second, and the second channel got alerts with unwanted delay. Per-chat state isn't a nice-to-have, it's mandatory.
Insight 3: Coalescing helps, but isn't a replacement for a queue. I kept it as an extra layer. Per channel, I now merge outstanding alerts every 30 seconds into at most three messages. That cuts the volume without replacing the queue.
Insight 4: Persistence beats in-memory. I briefly flirted with an asyncio.Queue. That would've been faster, but a crash mid-burst would have cost every alert waiting. SQLite has the advantage that the data is already there.
Insight 5: Jitter isn't cosmetic. Without the random component in the backoff, I could reliably reproduce Telegram returning a brief wave of 429s after the first one, when all waiting messages retried inside the same 100-millisecond window. With jitter, that disappeared.
How this fits with the book
Chapter 13 of the book covers sending the classified data over the three output channels Telegram, ntfy.sh, and email. That's exactly where, in the second edition, the note moved that the send path has to run asynchronously, decoupled from the classification loop — because Telegram imposes limits depending on load. The outbox mechanism I'm showing here is a direct implementation of the book's recommendation, supplemented with the concrete numbers from my setup. Readers of the book get the idea; readers of this code get a template they can adapt.
When you should rebuild this — and when you shouldn't
The effort pays off as soon as you regularly get bursts. Specifically: if your setup sends more than three alerts per minute to a single chat, any given day. If you serve several channels in parallel, much earlier than that. If you hook into third-party webhooks whose volume you don't control — GitHub in my case — the question isn't even a question anymore.
When not? If you have fewer than ten alerts per day and a single channel does the job, then a time.sleep(1.1) between messages is plenty. If you're building a multi-tenant product with hundreds of customers, aiohttp plus SQLite is the wrong foundation, then we're talking real message brokers. For the typical indie stack of RSS, a handful of webhooks, and SQLite, though, the solution shown here is exactly the right size: honestly complex, not over-engineered, easy to test, and above all robust against the Tuesday when everyone suddenly yells at once.