Where we started
My watch-&-alert setup is a classic indie project: a script that has been running on a small VPS (4 GB RAM, 2 vCPU, 25 GB SSD) since March 2024. It pulls around 45 RSS and Atom feeds (Heise, Golem, CERT-Bund, a few security blogs, three status pages), plus four webhooks from two home-lab services and one git server. Incoming items land in a SQLite database (watchalert.db, just under 220 MB after 18 months), get classified by a local Ollama model (llama3.1:8b-instruct-q5KM), and when a threshold is exceeded, are sent out on Telegram, ntfy.sh, and in parallel as a digest email.
The alerts run through five secrets, which at the time simply sat in a secrets.env:
TELEGRAMBOTTOKEN— the bot that's been nagging me for 18 monthsTELEGRAMCHATID— no rotation needed, but documentedSMTP_PASSWORD— for the morning digestNTFYTOPICTOKEN— topic token for ntfy.sh (mandatory since 2025)OPENAIAPIKEY— cloud fallback in case Ollama fails (max 3× a year)
Plus three webhook-signing secrets, but those only matter locally between the hook sender and my worker. So eight strings in total, never rotated, because "it works, doesn't it." The incident with the revoked Telegram token showed me that exactly this "it works, doesn't it" is a ticking time bomb. Telegram auto-rotates tokens after suspicious activity — and I have no idea what their heuristics consider suspicious. A log entry in an old Docker container, an accidental git add in a backup, a bit too much traffic from a connector topic: enough.
In parallel, a quick self-audit surfaced the following problems:
- Secrets sat in plaintext in
secrets.env, readable by every process running under my user - A
git statuswould have almost caught them, because I'd once moved the repo around on a trial basis - The watchdog script hard-restarted the daemon on every reload, which meant a minute of heartbeat pause
- There was no history of who set which secret when
That made it clear: I needed a process that allows rotation without the system going silent in the meantime. Sounds trivial, but it's surprisingly fiddly when you don't want any downtime.
Requirements
From the incident and the audit I derived five hard requirements for the new solution:
Hot reload without process restart. The watcher should be able to pick up new secrets at runtime. A classic systemctl restart watchalert.service means the heartbeat drops out for one to two minutes. Right in that maintenance window a failed disk or a backed-up feed crawler can go unnoticed — which is exactly what the system is supposed to prevent.
No double delivery. Sounds weird at first, but it's real: if I rotate a secret and both the old and new send path run at the same time, the same alert can go out twice. On Telegram that's annoying, on ntfy too, and in the audit log it produces ghost entries.
Audit trail. I want to be able to look up in SQLite when which secret was rotated. Not because a compliance audit is pending, but because when I'm debugging I need the timestamp.
Fail-safe. If the new secrets file is broken or the password was entered wrong, the system should keep running with the old secrets and log a warning — not crash hard.
Maintainable for one person. No vault clusters, no cloud KMS, no HashiCorp binding. I want to be able to deploy a new secret on a Sunday evening in 30 minutes without consulting three services.
These requirements already narrow the solution space considerably: enterprise tools like HashiCorp Vault, AWS Secrets Manager, or Doppler are out, because they either bring too much infrastructure or require a paid account. What remains is the combination of an encrypted local file and a process that can reload at runtime.
Three options compared
Before I wrote any code, I put three options on the table and weighed them honestly against each other.
Option 1: Hard cutover. Old secret out, new secret in, systemctl restart watchalert.service. That's the standard answer in 90% of tutorials, because it's trivial. In my case, though, it costs two minutes of heartbeat gap per rotation, and with it three heartbeats I lose. In 18 months I've restarted the system exactly twice — for OS upgrades. But as soon as I rotate every 90 days, as best practice dictates, that'd be 24 restarts a year, and the probability that an item from some feed gets stuck right during a rotation approaches 100%. I dropped this option after ten minutes of thinking.
Option 2: Dual lookup in code. A short period during which the code accepts both the old and the new secret. Concretely: on HTTP 401 with the new token, try the old token once, then persist the new token. That's the approach many cloud SDKs (AWS, GitHub) use internally when you have old and new access keys at the same time. Pro: no restart, redundant validation. Con: it spreads knowledge about two active secrets throughout the code. With five to eight secrets that gets ugly fast, and the transition time is hard to guarantee — for an external provider like Telegram, I don't know exactly when the old token gets invalidated server-side.
Option 3: External store with hot reload. The secrets live in a file (or a directory) that the running process watches. On change, a reload is triggered without the worker restarting. On Linux this is classically implemented via inotify, or pragmatically via a SIGHUP handler that systemd can trigger. Pro: no code instrumentation at every API call; rotation is a filesystem event. Con: you have to deal with SIGHUP semantics, file atomicity, and watchers.
I went with a combination of Option 2 and Option 3: external store with hot reload, but with a narrow dual-lookup only for the one case where a token refresh from the provider hasn't fully propagated server-side. Concretely: the main system reads from a secrets.age file encrypted with age. When a new secret is written, the file is replaced atomically and a kill -HUP is sent to the watcher. The watcher has a SIGHUP handler that re-reads the file without terminating the process. The brief dual-lookup logic exists only in the Telegram adapter class, because Telegram occasionally takes a few minutes until a freshly minted token is really active. All other secrets tolerate an immediate cutover without any transition.
The solution
The final solution consists of four building blocks: a Rotator class that wraps the actual rotation, a SecretStore that holds the decrypted values in memory, a SIGHUP handler in the watcher, and a systemd unit file that allows the reload via systemctl reload.
The SecretStore is intentionally kept small. It reads a single secrets.age file, decrypts it with the age tool from the same-named repository, and parses the result as TOML. Here's the core:
from __future__ import annotations
import logging
import os
import subprocess
import tomllib
from dataclasses import dataclass
from pathlib import Path
LOG = logging.getLogger(__name__)
SECRETS_PATH = Path("/etc/watchalert/secrets.age")
AGE_KEY_PATH = Path("/etc/watchalert/key.age")
DECRYPT_TIMEOUT_S = 5
@dataclass(frozen=True)
class Secrets:
telegram_bot_token: str
telegram_chat_id: str
smtp_password: str
ntfy_topic_token: str
openai_api_key: str | None
@classmethod
def from_toml(cls, data: dict[str, object]) -> "Secrets":
return cls(
telegram_bot_token=str(data["telegram_bot_token"]),
telegram_chat_id=str(data["telegram_chat_id"]),
smtp_password=str(data["smtp_password"]),
ntfy_topic_token=str(data["ntfy_topic_token"]),
openai_api_key=(
str(data["openai_api_key"])
if data.get("openai_api_key") else None
),
)
class SecretStore:
def __init__(self, path: Path = SECRETS_PATH) -> None:
self._path = path
self._current: Secrets | None = None
def load(self) -> Secrets:
plaintext = decrypt_age(self._path)
try:
data = tomllib.loads(plaintext)
finally:
# bewusst keine plaintext-Variable im langlebigen Scope
del plaintext
self._current = Secrets.from_toml(data)
LOG.info("secrets.reloaded", extra={"path": str(self._path)})
return self._current
def current(self) -> Secrets:
if self._current is None:
return self.load()
return self._current
def decrypt_age(path: Path) -> str:
result = subprocess.run(
[
"age",
"--decrypt",
"--identity", str(AGE_KEY_PATH),
str(path),
],
capture_output=True,
text=True,
timeout=DECRYPT_TIMEOUT_S,
check=True,
)
return result.stdout
The watcher itself holds a reference to the SecretStore and reconstructs the adapter on every send operation:
def send_telegram(text: str) -> None:
secrets = _store.current()
adapter = TelegramAdapter(
token=secrets.telegram_bot_token,
chat_id=secrets.telegram_chat_id,
fallback_token=os.environ.get("TELEGRAM_FALLBACK_TOKEN"),
)
adapter.send(text)
The fallback_token is the only place where dual lookup exists. It's only set when a rotation is currently in progress — and is removed from the environment again immediately after a successful send.
The rotator is a separate CLI script that I put at /usr/local/bin/watchalert-rotate. It takes the name of the secret, the new value, and an operator name, rewrites the file, encrypts it, and triggers the reload:
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import logging
import shutil
import subprocess
import sys
import tempfile
from datetime import datetime, timezone
from pathlib import Path
import tomllib
import tomli_w
LOG = logging.getLogger("watchalert.rotate")
SECRETS_PLAIN = Path("/var/lib/watchalert/secrets.plain.toml")
SECRETS_CIPHER = Path("/etc/watchalert/secrets.age")
AUDIT_LOG = Path("/var/log/watchalert/rotations.log")
def rotate(field: str, new_value: str, operator: str) -> None:
if not SECRETS_PLAIN.exists():
raise SystemExit("Klartext-Datei fehlt – initial bootstrap nötig")
with SECRETS_PLAIN.open("rb") as fh:
data = tomllib.load(fh)
data[field] = new_value
with tempfile.NamedTemporaryFile(
"wb", delete=False, dir=SECRETS_PLAIN.parent
) as tmp:
tmp.write(tomli_w.dumps(data).encode("utf-8"))
tmp_path = Path(tmp.name)
tmp_path.replace(SECRETS_PLAIN)
encrypt_in_place(SECRETS_PLAIN, SECRETS_CIPHER)
AUDIT_LOG.parent.mkdir(parents=True, exist_ok=True)
with AUDIT_LOG.open("a", encoding="utf-8") as fh:
fh.write(
f"{datetime.now(timezone.utc).isoformat()} "
f"{operator} {field}
"
)
# Reload triggern – kein Restart
subprocess.run(
["systemctl", "reload", "watchalert.service"],
check=True,
)
LOG.info("rotation.ok", extra={"field": field, "operator": operator})
def encrypt_in_place(plain: Path, cipher: Path) -> None:
subprocess.run(
[
"age",
"--encrypt",
"--armor",
"--output", str(cipher),
"--recipient-file", "/etc/watchalert/pubkey.age",
str(plain),
],
check=True,
)
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("field")
parser.add_argument("new_value")
parser.add_argument("--operator", required=True)
args = parser.parse_args()
logging.basicConfig(level=logging.INFO)
try:
rotate(args.field, args.new_value, args.operator)
except subprocess.CalledProcessError as exc:
LOG.error("rotation.failed", extra={"err": str(exc)})
return 1
return 0
if __name__ == "__main__":
sys.exit(main())For the reload to work, the systemd unit needs a few adjustments:
[Unit]
Description=Watch & Alert Daemon
After=network-online.target
[Service]
Type=notify
ExecStart=/usr/local/bin/watchalert
ExecReload=/bin/kill -HUP $MAINPID
NotifyAccess=main
Restart=on-failure
RestartSec=30
[Install]
WantedBy=multi-user.target
On the first run, notify didn't work because I'd written the absolute path to kill wrong — a classic mistake. I'd also used Type=simple, which doesn't harmonize with NotifyAccess=main. Once both were correctly set to notify, the reload came through in about 180 ms on average.
I didn't write tests in the pytest sense, but did it pragmatically: I ran a test rotation with a dummy field, in parallel a continuous alert generator (10 alerts/min, dummy data), and verified that no alert gets lost and no double delivery shows up in the audit log. Over 1000 test alerts in an hour, the drop rate was 0 and the duplicate rate was 0.
What we learned
Insight 1: Hot reload is cheaper than it looks. My first instinct was to solve the problem with Vault or a cloud KMS. In reality, systemd, SIGHUP, and a 60-line rotator are enough. The effort was three evenings, not three weeks.
Insight 2: Atomic file operations are non-negotiable. On my first attempt, I overwrote secrets.age directly. With a crash mid-write I ended up with a broken file and a dead system. With tempfile + Path.replace, that's ruled out, because replace is atomic on POSIX.
Insight 3: Dual lookup belongs exactly where it's actually needed. I only built it into the Telegram adapter, because there an external party (Telegram) decides when the token becomes active. For SMTP and ntfy, an immediate cutover is fine, because I'm the provider myself.
Insight 4: Audit logging costs nothing when it's an append-only file. I briefly considered logging the rotations in SQLite. A text file with lines [UTC timestamp] operator field is simpler, works with any tail -f, and correlating with SQLite is trivial because I always do the rotation shortly before or shortly after a heartbeat tick.
Insight 5: Fail-safe is not an optional add-on. On the third test run I deliberately fed in a wrong password. Expected: the system keeps running with the old secret, log entry secrets.reload.failed. Result: exactly that, plus a Telegram warning to myself. That was the moment I actually trusted the setup.
Relation to the book
Chapter 7 "Security & Advanced Topics" of the "Watch & Alert" book covers exactly the topics relevant to this use case: storing secrets, rotation, audit trail, and the heartbeat pattern as a robustness anchor. The heartbeat is indirectly in play here too: without it, I wouldn't have noticed that the Telegram bot had gone silent. Had I had the heartbeat back then already, the outage would have been caught in under 11 minutes, not the next morning.
The book goes beyond the approach shown here in two ways: first, it shows a variant with systemd-creds that doesn't need age and integrates directly into the unit-file mechanics. Second, it contains a pattern for rotating webhook-signing secrets where sender and receiver aren't synchronized — that's the more complex cousin of the case shown here. If you run your watch-&-alert setup for more than a year, you won't get around Chapter 7 anyway.
When you should rebuild this — and when not
The approach fits you if you have an indie setup: a single host, one or two people responsible, five to fifteen secrets, a 30-minute maintenance window per quarter. If you only use local LLMs via Ollama, you can skip the cloud API key topic entirely. If your workers run as root or under a single user, the configuration effort stays manageable.
It won't fit if you need to provision several machines with identical secrets. Then you need a central store, and the question of Vault vs. age vs. Infisical becomes relevant. Also, if you give third parties access to individual secrets — say, a contractor who only needs the SMTP password — it gets awkward, because age has no ACLs. And honestly: if your threat model is "someone roots my VPS", neither Vault nor age will help you; you need hardening on a different level.
In my case — indie project, one server, one person responsible, a manageable threat model — the combination of age + atomic reload + systemd-notify has been in use since March 2026. The first regular rotation ran on June 15, exactly 90 days after the incident, without a single lost alert. The heartbeat never paused — and in the end, that's the only criterion that actually matters.