What this is about
Anyone who runs containers knows the problem: services come and go, but monitoring stays rigid. New Compose stacks rarely end up in the alert pipeline automatically. Tools that use Docker labels for service discovery have recently been discussed more often, even though little public detail exists about individual projects of this kind. This article therefore shows, independent of any specific product, how to build label-based discovery cleanly into an existing Watch and Alert pipeline.
Container environments grow quickly. A single host can run ten, twenty or more services at the same time, each with its own lifecycle. Classic monitoring based on static configuration files barely keeps up, because every change needs a manual entry.
Label-based discovery turns the principle around. Instead of maintaining a central list, you record the information directly on the container, for example with a Docker label such as monitor.enable=true. A collector reads these labels periodically and decides for itself what needs to be monitored. The concept is not new: Traefik and Caddy have used it for routing decisions for years. For monitoring it is still not established everywhere.
The advantage is that the configuration stays local. Whoever deploys a service decides in the same Compose file whether and how it is monitored. This shrinks the gap between deployment and observability, two areas that are otherwise maintained separately in many small setups.
Requirements
Before you integrate label discovery into your pipeline, settle three things. First: access to the Docker socket or the Docker API. Without it you cannot read labels, and the access itself is a security topic you should not underestimate.
Second: a data model for discovered services. The reference architecture of collector, SQLite, classification and notification assumes that discovered containers end up in a table that can be handled like any other feed entry. Without this normalization you have to build completely separate logic for container events, which raises the maintenance effort.
Third: a clear definition of "healthy". A label only says that a service should be monitored, not how. You also need a health check, a port check or an HTTP endpoint that the pipeline queries regularly. Without this second layer you only monitor the existence of the container, not its actual state.
If containers are additionally made reachable through an overlay network such as Tailscale, a fourth requirement appears: the pipeline has to know under which address a service is reachable inside the tailnet, not just through the internal Docker IP. This is where label-based service discovery and the network overlay meet.
Options compared
There are basically three ways to feed container services into an alert pipeline. The first is manually maintaining a configuration file in which every service to be monitored is listed explicitly. This is robust and easy to follow, but scales poorly once the number of services reaches double digits.
The second is label-based discovery, as described above. It scales well but requires discipline when assigning labels. If you forget a label, the service is missing from monitoring without any error message. This is a silent failure that only becomes noticeable when the service actually goes down.
The third is fully automatic discovery through the Docker events API, where every new container is captured regardless of labels. This covers everything, but it also creates noise from short-lived containers, for example from CI pipelines, which should never be monitored.
In practice a combination of label-based discovery and a health check layer is the most pragmatic middle way. You keep the explicit opt-in logic of the labels, but automate collecting and further processing.
Implementation
The following code shows how to read container labels, write them into a SQLite table and then run a simple health check. The collector ideally runs through a systemd timer, for example every two minutes.
from __future__ import annotations
import json
import logging
import sqlite3
import subprocess
from dataclasses import dataclass
from pathlib import Path
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("label_collector")
DB_PATH = Path("/var/lib/watchpipe/services.db")
LABEL_KEY = "monitor.enable"
PORT_LABEL = "monitor.port"
@dataclass
class DiscoveredService:
container_id: str
name: str
port: int | None
def init_db(db_path: Path) -> None:
db_path.parent.mkdir(parents=True, exist_ok=True)
with sqlite3.connect(db_path) as conn:
conn.execute(
"""
CREATE TABLE IF NOT EXISTS services (
container_id TEXT PRIMARY KEY,
name TEXT NOT NULL,
port INTEGER,
last_seen TEXT DEFAULT CURRENT_TIMESTAMP,
status TEXT DEFAULT 'unknown'
)
"""
)
def list_containers() -> list[str]:
result = subprocess.run(
["docker", "ps", "--format", "{{.ID}}"],
capture_output=True,
text=True,
check=True,
)
return [line.strip() for line in result.stdout.splitlines() if line.strip()]
def read_labels(container_id: str) -> dict[str, str]:
fmt = "{{json .Config.Labels}}"
result = subprocess.run(
["docker", "inspect", "--format", fmt, container_id],
capture_output=True,
text=True,
check=True,
)
raw = result.stdout.strip()
return json.loads(raw) if raw and raw != "null" else {}
def discover_services() -> list[DiscoveredService]:
discovered: list[DiscoveredService] = []
for container_id in list_containers():
labels = read_labels(container_id)
if labels.get(LABEL_KEY) != "true":
continue
name = labels.get("com.docker.compose.service", container_id[:12])
port_raw = labels.get(PORT_LABEL)
port = int(port_raw) if port_raw and port_raw.isdigit() else None
discovered.append(
DiscoveredService(container_id=container_id, name=name, port=port)
)
return discovered
def store_services(db_path: Path, services: list[DiscoveredService]) -> None:
conn = None
try:
conn = sqlite3.connect(db_path)
for service in services:
conn.execute(
"""
INSERT INTO services (container_id, name, port, last_seen)
VALUES (?, ?, ?, CURRENT_TIMESTAMP)
ON CONFLICT(container_id) DO UPDATE SET
name = excluded.name,
port = excluded.port,
last_seen = CURRENT_TIMESTAMP
""",
(service.container_id, service.name, service.port),
)
conn.commit()
except sqlite3.Error as exc:
logger.error("Database error while saving: %s", exc)
finally:
if conn is not None:
conn.close()
def main() -> None:
init_db(DB_PATH)
services = discover_services()
logger.info("Discovered services: %d", len(services))
store_services(DB_PATH, services)
if __name__ == "__main__":
main()
This collector does not replace health check logic. It only delivers the list of services to monitor. The next step in the pipeline reads this table, checks the port or HTTP endpoint, writes the status back and, if needed, triggers a notification through ntfy, Telegram or email. Deciding whether an outage is critical or can be ignored remains a separate step that builds on the same SQLite data.
Typical pitfalls
The most common mistake is mounting the Docker socket unfiltered into a collector container. That gives the process effectively root rights on the host. Use a proxy instead, such as a read-only socket reverse proxy, or restrict the collector to the host itself instead of a container.
Another pitfall is inconsistency between label and actual state. A container can be running while the service inside no longer responds. Pure label discovery tells you nothing about this state, so you need an active check in addition to passive discovery.
Name conflicts are a problem too. If two Compose projects use the same service name, both end up under the same name in the table unless you use the container ID as the unique key. The code above therefore uses the container ID as the primary key, not the name.
Finally, many people forget that stopped or removed containers do not disappear from the table automatically. Without cleanup logic that removes orphaned entries after a certain time, data junk accumulates over months and can cause false alerts if you later access an old entry by mistake.
When it pays off (and when it does not)
Label-based discovery pays off when you regularly start new container services and manually maintaining a configuration list becomes the bottleneck. Especially in environments with frequent Compose changes it noticeably reduces the coordination effort between deployment and monitoring.
It pays off less if you only run a handful of stable services that rarely change. In that case a static configuration is clearer and less error-prone, because missing labels cannot create silent gaps.
If an overlay network such as Tailscale is involved as well, the benefit grows further, because the reachability of a service depends on the network configuration anyway. In such cases the combination of label discovery and network overlay is more consistent than two separate, manually maintained lists.