Where we started
We have three classes of feeds in the system: RSS 2.0, Atom 1.0, and a few mixed forms where a blog claims to deliver RSS but in reality serves up an invalid XML sandwich. On top of that come the usual suspects: encoding problems, missing fields, relative dates, HTML in the description field instead of CDATA, and one author who reshuffles his feed every few weeks. Before the rebuild we had roughly one or two messages per week in the log that we had to look at individually. With 30 feeds that's manageable, but we wanted to get away from the "close your eyes and push through" mentality.
The original pipeline looked like this: feedparser as a swiss army knife, a wrapper that shoves the items into a SQLite table, done. That worked without issues for a year. feedparser is great, no question. But we had two problems with it: First, for our own small system we wanted to reduce the dependency. We'd had a library before that, after a major update, smash our item structure into pieces. Second: when feedparser silently skips a broken feed, we only notice it when the heartbeat message "Feed X silent for 18 hours" arrives. We wanted to know more precisely what's going wrong.
The concrete situation last week: a feed, let's call it "security-feed-X", suddenly started returning HTML instead of XML because the author had migrated his blog to a new platform and forgotten to adjust the content type. Another feed had been sending a single broken item for days that tore through the whole loop. We had no idea where the error was hiding because feedparser just wrote "could not parse" into the log. With 30 feeds, "something isn't working" is too little information. We wanted structured diagnostics: which feed, which item index, which XML line. With a custom build we get that for free, because we throw the exception exactly at the point where we notice the problem.
Requirements
Before we got started, we wrote down five concrete requirements for the new feed component.
First: parse RSS 2.0 and Atom 1.0 cleanly. Both specs are small enough that we can work through them over a weekend. Second: with broken items, don't tear down the entire loop. A broken entry may at most trigger a heartbeat spike, not kill the whole cron run. Third: handle encoding and HTML entities robustly without us maintaining special handling for every feed. Fourth: a diagnostic log that tells us what's actually broken on a feed — with item index and field name. Fifth: performance doesn't need to be rocket science. 30 feeds, a few hundred items per run, under ten seconds for the whole loop. We can pull that off without caching too.
What we explicitly don't need: full OPML support, auto-discovery mechanisms, Readability-style content extraction. This is RSS, not Readability. Anyone who needs that should keep using feedparser — it's a fantastic tool for that job. For our 30 hand-picked sources, building our own is the better choice.
We deliberately decided against an elaborate state machine. The thing should stay readable so we still understand what it does half a year from now. We accept that we won't cover every conceivable edge case — better a system that cleanly eats the usual 95 percent and shouts loudly about the remaining 5 percent than a system that can do everything but nobody maintains anymore.
Three options compared
Option A is the obvious one: stick with feedparser. We know the library, it works for 28 of 30 feeds, only two blow up. We could do pre-validation, wrap the raw XML string in a try/except and only pass the clean stuff on to feedparser. That would be the path of least resistance. The trade-off: we don't solve the actual problem. The next time another feed migrates, we'll be back in the same spot. Plus, as described above, we'd already decided against the dependency for other reasons.
Option B: take a specialized, smaller library like atoma or a thin wrapper. atoma is tuned for speed and Atom focus, but it's no longer very actively maintained — the last release was over two years ago. We looked at other micro-libraries, basically they're just wrappers around ElementTree too. So we'd have foreign logic in play again that we don't control. With a library, the maintainer decides how to handle broken items, not us.
Option C: xml.etree.ElementTree from the standard library plus our own wrapper that understands the format-specific quirks of RSS and Atom. That's more code, but we decide the robustness strategy ourselves. We can make granular decisions: which fields are mandatory? How do we handle missing pubDates? When do we give up, when do we try again? This is our preferred option because it fits our "we understand what we run" philosophy.
A comparison in numbers so this doesn't sound like gut feeling: feedparser brings roughly 400 KB of code plus dependencies, atoma about 150 KB. Our own solution is about 180 lines of Python in the first iteration, no external imports beyond xml.etree.ElementTree, html, re, datetime, pathlib. For a one-person project like ours, custom development pays off exactly when the maintenance effort is smaller than the library cost. We estimate the effort as follows: one weekend for the first robust version, after that maybe an hour per quarter for new edge cases. The library would have saved us a weekend of dependency management, but we'd have less understanding of the format and less control over error types.
The solution
The architecture of our solution looks like this: one module feed_reader.py with a FeedReader class that has its own parse method per feed type. Both methods return a list of normalized FeedItem dataclasses. Errors are raised as exceptions with context, and the caller decides whether to skip the whole feed or just drop the item. The important thing for us was a strict separation: the parser is dumb and throws, the caller is smart and reacts.
from __future__ import annotations
import logging
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from xml.etree import ElementTree as ET
logger = logging.getLogger(__name__)
ATOM_NS = "{http://www.w3.org/2005/Atom}"
CONTENT_NS = "{http://purl.org/rss/1.0/modules/content/}"
@dataclass(slots=True, frozen=True)
class FeedItem:
title: str
link: str
published: datetime
source_feed: str
guid: str | None = None
summary: str = ""
The FeedItem is deliberately minimal. In the first iteration we had more fields, but most of them weren't needed for classification anyway. What really counts is a stable identifier (GUID), a publication timestamp, and a link. Everything else is bonus. If you later need more fields, adding a new property on a frozen dataclass is a one-liner.
The actual parse logic is split in two. First the root element is inspected to determine the feed type. Then the code branches into the respective parse method. Both use ElementTree, both throw speaking exceptions.
def parse(self, raw_xml: str, source_feed: str) -> list[FeedItem]:
"""Parst RSS 2.0 oder Atom 1.0 und gibt normalisierte Items zurück."""
try:
root = ET.fromstring(raw_xml)
except ET.ParseError as exc:
raise FeedParseError(
f"Ungültiges XML in Feed {source_feed}: {exc}"
) from exc
tag = root.tag
if tag == "rss":
return self._parse_rss(root, source_feed)
if tag == f"{ATOM_NS}feed":
return self._parse_atom(root, source_feed)
raise FeedParseError(f"Unbekannter Feed-Typ: {tag} in {source_feed}")Both parse methods share a collection of small helper functions that encapsulate typical RSS problems: encoding correction, date parsing with fallback strategies, HTML cleanup. These helpers are the heart of robustness. Anyone who can't parse an item cleanly isn't silently discarded but generates a warning in the log.
def _parse_published(self, raw: str | None) -> datetime | None:
if not raw:
return None
raw = raw.strip()
# RFC 822 für RSS 2.0, ISO 8601 für Atom
for parser in (parse_rfc822, parse_iso8601):
try:
dt = parser(raw)
return dt.astimezone(timezone.utc)
except ValueError:
continue
logger.warning("Unbekanntes Datumsformat: %s", raw)
return None
The date parsers are the second sore point. RFC 822 has various flavors, ISO 8601 is more forgiving but still not perfect. We try both strategies, log unknown formats, and return None. The upper layer then decides whether items without a date are discarded or go into the pipeline with a default timestamp. For us the order is important enough that items without a valid date are actually discarded — otherwise we'd have classified old posts as "new" again.
The most important part of the whole architecture, though, isn't the code, it's the calling side. We split the cron run into two phases: fetch first, then classify. If a feed is completely broken, it's skipped, the heartbeat is still updated — with the status "0 new items, error: ...". If only individual items are broken, the feed still gets processed. That prevents a single migraine feed from paralyzing the whole pipeline.
def process_feed(self, url: str, source_feed: str) -> int:
"""Holt und parst einen Feed, gibt Anzahl neuer Items zurück."""
try:
raw = self._fetch(url)
except FetchError as exc:
logger.error("Fetch fehlgeschlagen für %s: %s", source_feed, exc)
self._record_heartbeat(source_feed, 0, status="fetch_error")
return 0
try:
items = self.reader.parse(raw, source_feed)
except FeedParseError as exc:
logger.error("Parse-Fehler in %s: %s", source_feed, exc)
self._record_heartbeat(source_feed, 0, status="parse_error")
return 0
new_count = self._store_items(items)
self._record_heartbeat(source_feed, new_count, status="ok")
return new_countWhat we deliberately left out: a retry mechanism at the feed level within a single run. If a feed is broken in one run, we usually try again on the next cron run after an hour. Retrying within a run doesn't help at our volume and makes the heartbeats confusing. With a setup of 500 feeds and webhook input it would be different; we'll talk about that again then.
The _fetch method isn't shown here because it's less interesting than the parsing. Short version: HTTP with timeout, Accept: application/rss+xml, application/atom+xml, application/xml;q=0.9, explicit encoding handling based on the HTTP header with a fallback to utf-8. For the few hundred requests per day we rely on urllib from the standard library and avoid requests as an extra dependency. That fits our dependency mindset: as little as possible, as much as necessary.
What we learned
Insight 1: Keep mandatory and optional fields cleanly separated. We initially tried to parse a feed completely and then decide whether the item is "good enough". That's nonsense. Define upfront which three fields really matter — for us those are title, link, and published — and which fields are just bonus. Items without the mandatory fields get discarded, the rest still gets processed. This strict separation has in practice meant that we don't fall into discussions about "well, maybe it still works" when outliers show up.
Insight 2: Encoding detection is a lottery, but utf-8 plus a latin-1 fallback covers 99 percent. We first try the encoding announced by the server, then utf-8, then latin-1. If a feed comes in with an obscure encoding, it gets marked as "encoding_error" in the heartbeat overview, but the cron doesn't stumble. Over the past three weeks this strategy failed to cover exactly one case — we fixed that manually because we cared about the feed.
Insight 3: Date parsing is the most common silent bug. Local timestamps without a timezone are a classic. An item published at 11pm local time lands in the wrong order. We normalize everything to UTC as soon as we have a valid date. For dates we absolutely can't parse, we log the format so we can patch it up when we get a chance. That once cost us a complete sorting error on a German-language feed that we'd overlooked for a week.
Insight 4: Heartbeat does not mean "all good". We originally had a heartbeat that was simply set as soon as the cron run finished. That's worthless. We now distinguish four heartbeat statuses: ok, fetch_error, parse_error, encoding_error. Only then did it become visible that a certain feed had been throwing an encoding_error every morning for three weeks while still showing a green light. Without this differentiation we wouldn't have seen the data.
Insight 5: Custom builds only pay off when you understand the format. We invested an hour beforehand reading through the RSS 2.0 and Atom 1.0 specs. Without that background we'd have had to fight through dozens of blog posts only to end up realizing that the specs are actually quite readable. Anyone who doesn't want to read the specs should please stick with feedparser. You save not only time but also a lot of frustration.
Connection to the book
Chapter 9 ("Fetching data") covers exactly the interface between the outside world and our system. More specifically, it's about the three major data paths: RSS/Atom, webhooks, and manual sources. In that chapter we walk step by step through how you connect a data source without letting it drag down your system if it fails. The idea of the two-stage pipeline (fetch, then process) that we use here comes directly from that chapter. The heartbeat pattern and the question of "when is a feed broken enough to ignore" are also covered there in more detail. Anyone who has read this and wants to dig deeper into the concepts should jump to chapter 9 — especially the section on backpressure and the silent-alert problem is directly relevant to our scenario. The chapter also provides the surrounding material: SQLite schema for items, cron and systemd timer configuration, and the interplay with the classification pipeline in chapter 11.
When you should rebuild this — and when you shouldn't
Rebuild it if you run your own small monitoring system that aggregates 20 to 100 sources and you want to keep control over error diagnostics. If you just need a script that reads a single feed and nothing more, it's overkill. Rebuild it also if you feel like really understanding RSS — the specs are short, and the format is surprisingly stable after 20 years. It helped us become more confident when reading other people's XML structures, which also pays off when debugging webhook payloads.
Don't rebuild it if you run a multi-tenant RSS service or aggregate 5000 feeds. In that case you want a maintained library with an active maintainer, built-in OPML import, and auto-discovery. Our solution is deliberately tailored to an indie setup. Anyone who depends on universal compatibility is better served by feedparser, atoma, or commercial feed platforms. Also, if you plan to hand off your system to a team where not everyone knows Python, a well-documented library makes more sense in the long run than a 200-line custom build that only the original author understands.
In short: it fits our watch-&-alert project perfectly. For anything with professional demands on feed parsing or more than 100 sources, it's not intended for that.