Parsing RSS feeds robustly without a library

What this is about

Feeds are everyday business in every monitoring pipeline, and they are less reliable than the specification suggests. There is RSS 2.0, Atom 1.0 and hybrid forms. On top come wrong content types, missing fields, different date formats and encoding problems.

For many cases feedparser is the right choice. If you want to avoid the dependency or control exactly what happens on an error, xml.etree.ElementTree and urllib from the standard library take you surprisingly far.

Fetching and checking the response

Before parsing you check whether a feed has arrived at all. Some servers return an HTML page with status 200 on errors.

import logging
import urllib.request

log = logging.getLogger("feeds")

def fetch(url: str, timeout: int = 15) -> bytes | None:
    req = urllib.request.Request(url, headers={"User-Agent": "pipeline/1.0"})
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            ctype = resp.headers.get("Content-Type", "")
            body = resp.read()
    except OSError as exc:
        log.warning("Abruf fehlgeschlagen für %s: %s", url, exc)
        return None
    if "html" in ctype.lower() and not body.lstrip().startswith(b"<?xml"):
        log.warning("Kein Feed, sondern HTML: %s", url)
        return None
    return body

If you pass the bytes directly to the XML parser, it determines the character encoding from the XML declaration. That is more reliable than decoding yourself.

Telling RSS and Atom apart

RSS 2.0 has a root element rss with channel and item. Atom uses the namespace http://www.w3.org/2005/Atom with feed and entry. The easiest approach is to read the root element and branch.

import xml.etree.ElementTree as ET

ATOM = "{http://www.w3.org/2005/Atom}"

def parse_feed(data: bytes, feed_name: str) -> list[dict]:
    try:
        root = ET.fromstring(data)
    except ET.ParseError as exc:
        log.warning("%s: XML ungültig: %s", feed_name, exc)
        return []
    if root.tag == "rss":
        nodes = root.findall("./channel/item")
        get = _rss_item
    elif root.tag == ATOM + "feed":
        nodes = root.findall(ATOM + "entry")
        get = _atom_item
    else:
        log.warning("%s: unbekanntes Format %s", feed_name, root.tag)
        return []
    items = []
    for index, node in enumerate(nodes):
        try:
            items.append(get(node))
        except Exception as exc:  # ein kaputtes Item darf nichts abbrechen
            log.warning("%s: Item %d übersprungen: %s", feed_name, index, exc)
    return items

The decisive point is the loop with error handling per item: a single broken element is logged and skipped, and the rest of the feed stays usable.

Catching missing fields

Title, link and date can be missing. Set sensible defaults instead of failing, and require only what makes an entry worthless if absent, usually the link.

def _text(node, path: str, default: str = "") -> str:
    found = node.find(path)
    return (found.text or "").strip() if found is not None else default

def _rss_item(node) -> dict:
    link = _text(node, "link")
    if not link:
        raise ValueError("Item ohne Link")
    return {"title": _text(node, "title", "(ohne Titel)"), "link": link,
            "date": _text(node, "pubDate")}

For Atom the link is an attribute href on a link element, a peculiarity that is easy to overlook the first time.

Date formats

RSS mostly uses RFC 822, Atom uses ISO 8601. The standard library covers both: email.utils.parsedate_to_datetime for RSS and datetime.fromisoformat for Atom. Unreadable dates should not cause an abort but fall back to the time of retrieval.

Structured error messages

A message such as "feed broken" does not help. Log the feed name, the item index and the type of error, as in the example above. Then you see in the journal immediately which feed causes trouble, and a heartbeat can report how many feeds or items were skipped.

When it is worth it (and when not)

Rolling your own is worth it when you have few feeds, want to keep dependencies small or need detailed error diagnosis. The code is manageable and does exactly what you expect.

feedparser is the better choice when you process many different, partly badly formatted feeds, because the library handles many special cases that you would otherwise have to rebuild yourself. Extensions such as media attachments or namespaces of third-party services are covered there already.