Monitoring Container States and Reporting Failures

What it is about

A dashboard and an alerting system solve two different problems. A dashboard answers the question "What does it look like right now?" as soon as you check. An alerting system answers the question "Do I need to act now?" without you having to do anything. For small setups, a dashboard is often enough, provided you check it daily. Once services run unattended, you additionally need a pipeline that checks independently and speaks to you.

Such a pipeline consists of a few building blocks. A collector queries the state, a small SQLite database remembers the last known state, a classification decides whether a change is relevant, and a notification channel ensures you hear about it. A daily heartbeat confirms that the pipeline itself is still running. The whole thing is controlled via cron or a systemd timer.

Requirements

Before you start, clarify a few prerequisites. The script needs access to the Docker daemon, usually via the Unix socket. Read access is sufficient because you only want to query states and not control containers. A socket proxy with pure read permissions is the cleaner solution if the script runs in a container.

You also need a place for the last known state. SQLite is more than enough for this: one file, no extra services, easy to back up. For notifications, choose a channel you are constantly watching anyway, such as ntfy, Telegram, or email. Finally, set the interval. For stable services, one check per hour is sufficient; for sensitive services, five minutes makes sense. Shorter intervals rarely help because Docker often catches restarts on its own.

Options compared

You have several ways to monitor containers. The native Docker restart policy restarts crashed containers but does not tell you that something happened. A container that crashes and restarts every ten seconds often still looks "running" in the dashboard.

Health checks inside the container go one step further. They check whether the service actually responds and set the status to healthy or unhealthy. This is more precise information, but it alone does not generate a message to you.

A full monitoring stack with Prometheus and Alertmanager provides metrics and flexible rules. For small setups, it often involves more operations than the benefit justifies. Your own small pipeline lies in between: it is quick to build, checks exactly what you need, and remains comprehensible for you. Its disadvantage is that you have to maintain it yourself.

Implementation

The following script queries all containers, compares their state with the stored one, and reports only changes. It uses the Docker SDK for Python (pip install docker) and urllib for the ntfy message. The address of the ntfy topic is an example value that you replace with your own.

import logging
import sqlite3
import urllib.request
from pathlib import Path

import docker

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("container_watch")

DB_PATH = Path("data/containers.db")
NTFY_URL = "https://ntfy.example.org/container-alerts"  # Beispielwert


def init_db(conn: sqlite3.Connection) -> None:
    conn.execute(
        "CREATE TABLE IF NOT EXISTS state ("
        "name TEXT PRIMARY KEY, status TEXT NOT NULL, health TEXT)"
    )


def current_states(client: docker.DockerClient) -> dict[str, tuple[str, str]]:
    states = {}
    for c in client.containers.list(all=True):
        health = c.attrs.get("State", {}).get("Health", {}).get("Status", "none")
        states[c.name] = (c.status, health)
    return states


def notify(message: str) -> None:
    req = urllib.request.Request(NTFY_URL, data=message.encode("utf-8"), method="POST")
    with urllib.request.urlopen(req, timeout=10):
        pass


def run() -> None:
    DB_PATH.parent.mkdir(parents=True, exist_ok=True)
    conn = sqlite3.connect(DB_PATH)
    client = None
    try:
        init_db(conn)
        client = docker.from_env()
        known = {r[0]: (r[1], r[2]) for r in conn.execute("SELECT * FROM state")}
        for name, (status, health) in current_states(client).items():
            old = known.get(name)
            if old is not None and old != (status, health):
                log.warning("%s: %s/%s -> %s/%s", name, *old, status, health)
                notify(f"{name}: {old[0]}/{old[1]} -> {status}/{health}")
            conn.execute(
                "INSERT INTO state (name, status, health) VALUES (?, ?, ?) "
                "ON CONFLICT(name) DO UPDATE SET status=excluded.status, "
                "health=excluded.health",
                (name, status, health),
            )
        conn.commit()
    except docker.errors.DockerException as exc:
        log.error("Docker nicht erreichbar: %s", exc)
    finally:
        if client is not None:
            client.close()
        conn.close()


if __name__ == "__main__":
    run()

Start the script via cron or as a systemd timer at the chosen interval. An unreachable Docker interface should also trigger a message; otherwise, the pipeline goes silent exactly when something fundamental is broken. Add a notify call with its own text in the except branch for this purpose. A daily heartbeat that sends a "Pipeline running" confirmation covers the case where the script itself no longer starts.

For classification, you can later use a local model via Ollama or a cloud API, for example to summarize log lines of a crashed container. You do not need that for pure state monitoring. A fixed rule like "running to exited" is more reliable than a model and costs nothing.

Common pitfalls

The most common mistake is alerting on every check. If you report that a container is down on every run, you quickly get numb and ignore the messages eventually. Therefore, report only state changes and, if necessary, a reminder after a longer deadline.

The second stumbling block is comparing via the container ID. After an update with a new image, the container gets a new ID, but the name remains the same. If you use the ID as the key, you lose the history and report every updated container as new. The name or a stable label is the better key, as in the example.

Third, running alone is not a health statement. A process can be running and still not answer requests. Therefore, define health checks for important services and read out the health status, as the collector above does.

Fourth, the script itself needs to be monitored. If the cron job no longer runs, no message comes, and that looks exactly like "everything is fine." The daily heartbeat closes this gap. Also check that the system time is synchronized; otherwise, timestamps in logs and messages do not match other services.

When it is worth it (and when not)

Your own small pipeline is worth it if you operate one or a few Docker hosts and want to notice failures without maintaining an entire monitoring stack. The effort is manageable: one script, one file as a database, one notification channel. You decide yourself what is reported and can read every rule in the code.

It makes less sense if you already run monitoring with alerting. Then integrate the Docker metrics there instead of maintaining a second solution. Also, for a very large number of hosts, a script per machine does not scale well; centralized systems with agents are the better choice there.

A dashboard remains useful. It shows you the overview and facilitates manual interventions. The pipeline takes over the part that a dashboard cannot do: reaching you when you are not looking.