Where we started
My Watch & Alert system runs on a small VPS at Hetzner, Debian 12, 4 GB RAM, Ollama local for LLM classification. Around 30 RSS and Atom feeds are monitored from the IT security, Python ecosystem, and open-source tools spaces. On top of that, three webhook sources for GitHub releases and CVE feeds. On average, five to fifteen alerts go out per day, often more in the evening when mailing list feeds come in.
The tasks the system needs to run on a schedule are clearly defined: First, a feed poll every 15 minutes that fetches new entries from RSS/Atom/webhook, stores them in SQLite, classifies them via LLM, and fires off an alert via Telegram or ntfy.sh on a hit. Second, a heartbeat every 5 minutes — a silent "I'm still alive" message signaling that the service is still up; if it goes missing, ntfy.sh sends me a warning. Third, a nightly digest at 23:00 that sends the daily summary as an email so I can see first thing in the morning what happened.
For a year, the whole thing ran with a crontab that looked roughly like this:
*/15 * * * * /opt/watch-alert/.venv/bin/python /opt/watch-alert/poll.py >> /var/log/watch-alert/poll.log 2>&1
*/5 * * * * /opt/watch-alert/.venv/bin/python /opt/watch-alert/heartbeat.py >> /var/log/watch-alert/heartbeat.log 2>&1
0 23 * * * /opt/watch-alert/.venv/bin/python /opt/watch-alert/digest.py >> /var/log/watch-alert/digest.log 2>&1
Works. Looks tidy. But over the months it accumulated three problems that got on my nerves.
Problem 1: Silent failures. Cron doesn't send any notification on its own when a job fails to start or crashes. If /usr/sbin/cron is stuck, the disk is full, or the Python process segfaults, nobody notices. My 5-minute heartbeat was once down for three days without me noticing — because the heartbeat itself was the job that was supposed to raise the alarm. Classic single point of failure.
Problem 2: No dependencies. Watch & Alert needs both a network connection AND a running Ollama daemon for the poll. If the Ollama service hasn't come back up yet after an update and the cron job polls anyway, you get a barrage of 500 errors that you only discover in the log file. With cron you have to check that in the script itself — and then hope you got the order right.
Problem 3: Logging was a mess. >> /var/log/watch-alert/poll.log 2>&1 doesn't rotate on its own, grows forever, and isn't searchable. I ended up with three text files where I had to grep and tail for errors, with no idea when the last successful run was.
Requirements
Before migrating, I had to clarify what I expected from the new setup. Here's my list — low-bar enough for a one-person project, but specific enough that I can measure afterwards whether the migration was worth it.
- Observability: I want to see when a job ran, when it failed, and why. A single
journalctl -u watch-alert-pollshould be enough, without grepping through text files. - Dependencies: If Ollama isn't running, the poll job shouldn't start at all — or at least be clearly marked as waiting on a dependency.
- Catch silent failures: If the job fails repeatedly in a row, I want a notification via ntfy.sh. Not three days later when someone happens to notice the missing Telegram update.
- Backoff on failure: If the RSS feed server is briefly down, the next attempt shouldn't fire again immediately but wait a bit.
- Migration must not get more complex. If the setup requires more effort afterwards than before, the migration isn't worth it.
These requirements are deliberately not enterprise-grade. I don't want a central monitoring platform, Prometheus exporters, or six-figure damage figures that an outage supposedly caused. Just solid scheduling that doesn't quietly die on me in the background.
Three options compared
I seriously considered three approaches: keep Cron, switch to systemd timers, or a mix of both. Here's my honest comparison.
Option 1: Keep Cron, but harden it
You could armor up cron jobs with wrapper scripts: health check before the run, structured logs to /var/log/syslog, deadman switch via a second cron job monitoring the first, lock files against parallelism. It works technically, but it has the character of a duct-tape solution. You're building your own mini scheduling system that replicates the same properties as systemd timers — just worse, because it's not integrated into the init process and comes with no built-in tools for restart logic or dependencies. For an indie project with two or three jobs, it's simply too much effort.
Option 2: Fully switch to systemd timers
systemd timers replace cron jobs with .service and .timer units. The timer triggers the service, the service runs the job. Advantages: native integration into journald, dependencies via Requires=, After=, custom restart logic with Restart=on-failure, exponential backoff via RestartSec=, and the ability to manually test a job with systemctl start without manipulating the timer. Disadvantages: the unit file syntax takes some getting used to at first, and you need two files per job. For three jobs, that's six files, plus the directory layout under /etc/systemd/system/.
Option 3: Mixed operation
Classic, simple jobs — like a script that cleans up a directory once a day or clears a cache — stay in Cron. Complex, observation-worthy jobs move to systemd. That's pragmatic and reduces migration effort. That's exactly the path I took.
Side-by-side comparison
| Criterion | Cron | systemd timers | Mixed |
|---|---|---|---|
| Learning curve | minimal | medium | medium |
| Logging | manual | journald built-in | both |
| Dependencies | in the script | declarative | mixed |
| Failure notification | manual | via OnFailure= |
mixed |
| Effort for 3 jobs | low | medium | low to medium |
| Observability | poor | good | good for critical jobs |
| Backoff on failure | build it yourself | built-in | mixed |
| Persistent after reboot | no | yes (Persistent=true) |
mixed |
For my Watch & Alert setup, the trade-off between effort and benefit was ultimately the deciding factor. systemd timers aren't inherently better — they're better for jobs I actively want to monitor. For a pure cleanup script, that added value isn't there.
The solution: migration to systemd
The migration happened in three steps: heartbeat first, because it's the most critical. Then the poll job, because it has the most dependencies. I left the nightly digest in Cron for now, because it runs at low frequency and doesn't need real-time monitoring.
Step 1: Heartbeat as a systemd unit
First the service unit. It calls the Python script that writes a timestamp to SQLite and, on success, sends a silent ntfy.sh message to a separate topic. The latter isn't strictly necessary, but it helps with debugging.
# /opt/watch-alert/heartbeat.py
from __future__ import annotations
import logging
import sqlite3
from datetime import datetime, timezone
from pathlib import Path
import httpx
DB_PATH = Path("/opt/watch-alert/data/watch.db")
NTFY_TOPIC = "watch-alert-heartbeat"
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(message)s",
)
log = logging.getLogger(__name__)
def record_heartbeat(conn: sqlite3.Connection) -> None:
conn.execute(
"INSERT INTO heartbeats (ts) VALUES (?)",
(datetime.now(timezone.utc).isoformat(),),
)
conn.commit()
def notify_alive() -> None:
try:
httpx.post(
f"https://ntfy.sh/{NTFY_TOPIC}",
data="heartbeat ok",
timeout=5.0,
)
except httpx.HTTPError as exc:
log.warning("ntfy notify failed: %s", exc)
def main() -> int:
with sqlite3.connect(DB_PATH) as conn:
record_heartbeat(conn)
notify_alive()
log.info("heartbeat written")
return 0
if __name__ == "__main__":
raise SystemExit(main())
Plus the service unit at /etc/systemd/system/watch-alert-heartbeat.service:
[Unit]
Description=Watch & Alert Heartbeat
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
WorkingDirectory=/opt/watch-alert
ExecStart=/opt/watch-alert/.venv/bin/python /opt/watch-alert/heartbeat.py
User=watchalert
Restart=on-failure
RestartSec=30
[Install]
WantedBy=multi-user.target
And the timer at /etc/systemd/system/watch-alert-heartbeat.timer:
[Unit]
Description=Watch & Alert Heartbeat alle 5 Minuten
[Timer]
OnCalendar=*:0/5
Persistent=true
AccuracySec=10s
[Install]
WantedBy=timers.target
OnCalendar=*:0/5 means: every hour at minutes 0, 5, 10, … Persistent=true makes sure missed runs are caught up if the server was down at the trigger time. AccuracySec=10s defines the time window in which systemd is allowed to fire the timer — important because otherwise systemd tries to start the job with millisecond precision, which isn't needed for OnCalendar triggers and just wastes CPU.
Enable and start:
sudo systemctl daemon-reload
sudo systemctl enable --now watch-alert-heartbeat.timer
Checking status now takes two commands, and this is the point where systemd won me over:
systemctl list-timers --all | grep watch-alert
journalctl -u watch-alert-heartbeat.service -n 50 --no-pager
The first lists all timers with their next trigger time and last run. The second shows the logs of the specific service with timestamps, without me having to dig through text files. Those two commands cut my average debug time per incident in half.
Step 2: Poll job with dependency on Ollama
The poll job is more complex. It needs a network connection and a running Ollama daemon. In the service unit, this can be expressed declaratively:
[Unit]
Description=Watch & Alert Feed-Poll
After=network-online.target ollama.service
Requires=ollama.service
OnFailure=watch-alert-failure-notify.service
[Service]
Type=oneshot
WorkingDirectory=/opt/watch-alert
ExecStart=/opt/watch-alert/.venv/bin/python /opt/watch-alert/poll.py
User=watchalert
Restart=on-failure
RestartSec=60
StartLimitBurst=3
StartLimitIntervalSec=600
[Install]
WantedBy=multi-user.target
StartLimitBurst=3 combined with StartLimitIntervalSec=600 means: if the job crashes three times in a row within 10 minutes, it will no longer be automatically restarted. That prevents a broken Ollama container from flooding the system with failed attempts while simultaneously filling up the journal with identical stack traces.
OnFailure=watch-alert-failure-notify.service is the real game changer. This directive ensures that when the start limit is reached, a second service is triggered that sends a high-priority ntfy.sh message. That's the deadman switch I previously had to build by hand with Cron, and it works without any extra script.
The timer for this one is analogous to the heartbeat, just with a different interval:
[Unit]
Description=Watch & Alert Poll alle 15 Minuten
[Timer]
OnCalendar=*:0/15
Persistent=true
AccuracySec=30s
[Install]
WantedBy=timers.target
Step 3: Nightly digest stays in Cron
The digest job is low-frequency, only runs once a day, and a missed run at 23:00 can be manually triggered the next morning. Here the extra overhead of two unit files isn't justified. It stays in Cron; a wrapper script just checks that the SQLite DB is readable and writes the run to a small log file.
0 23 * * * /opt/watch-alert/.venv/bin/python /opt/watch-alert/digest.py >> /var/log/watch-alert/digest.log 2>&1
Step 4: Monitoring the timers themselves
Cron failures were my main problem. So now I monitor the timers themselves: a second systemd timer runs every 30 minutes and checks via systemctl is-active whether the critical units have run in the last 30 minutes. It sounds paranoid, but it's effective. That single timer has already saved me twice when an update broke the Ollama service and the heartbeat stopped coming through.
# /opt/watch-alert/timer_monitor.py
from __future__ import annotations
import logging
import subprocess
from datetime import datetime, timedelta, timezone
from pathlib import Path
import httpx
NTFY_TOPIC = "watch-alert-meta"
MAX_AGE_MINUTES = 30
log = logging.getLogger("watch-alert.meta")
def last_run_age(unit: str) -> timedelta | None:
result = subprocess.run(
[
"systemctl",
"show",
unit,
"--property=ActiveEnterTimestamp",
"--value",
],
check=True,
capture_output=True,
text=True,
)
raw = result.stdout.strip()
if not raw:
return None
last = datetime.fromisoformat(raw)
return datetime.now(timezone.utc) - last
def alert(text: str) -> None:
httpx.post(
f"https://ntfy.sh/{NTFY_TOPIC}",
data=text,
headers={"Priority": "high", "Tags": "warning"},
timeout=5.0,
)
def main() -> int:
critical = [
"watch-alert-heartbeat.service",
"watch-alert-poll.service",
]
for unit in critical:
age = last_run_age(unit)
if age is None or age > timedelta(minutes=MAX_AGE_MINUTES):
alert(f"{unit} last run {age} ago")
log.warning("%s stale: %s", unit, age)
return 0
if __name__ == "__main__":
raise SystemExit(main())
What we learned
Insight 1: systemd timers are no silver bullet. For trivial jobs with low frequency, crontab is still faster to set up and easier to read. Migration is only worth it where observability and dependencies matter. Moving your nightly digest to a systemd timer just because it sounds "more modern" is effort without benefit.
Insight 2: OnFailure= is the real killer feature. It turns a scheduled task into a self-monitoring service. In my Watch & Alert setup, it cut average response time from "next business day" to "a few minutes," because I now get a push notification as soon as the start limit is hit.
Insight 3: Persistent=true is gold on servers with irregular uptime. My small VPS gets shut down occasionally for maintenance. Without Persistent=true, the 15-minute poll slots during downtime would be lost. With the directive, they get caught up on the next start — though only a single run per timer, not a whole series.
Insight 4: journald logging isn't perfect, but good enough. The default view is capped at 1000 lines and you need --since to see older entries. For more convenience, I enabled persistent journald logs on my VPS (Storage=persistent in /etc/systemd/journald.conf). Costs a few hundred MB of disk, but worth it because I no longer have to guess whether a log entry is from yesterday or three weeks ago.
Insight 5: Mixing both worlds is legitimate, not a sign of laziness. If a setup does its job and doesn't constantly demand attention, it's well-set-up. I have two timers running on systemd, one job still in Cron, and all three have worked for six months without intervention. The temptation to squeeze everything into a single system is often bigger than the actual benefit.
Book reference
Chapter 14 ("Automatisieren") in the Watch & Alert book covers exactly this trade-off between Cron and systemd. It uses the book's concrete setup to show how the poll script, heartbeat, and digest work together as a system — and which design decisions matter when switching from Cron to systemd. If you want to look up the unit files from this article in the book, you'll find them in the repository under deploy/systemd/. The book also goes into detail on the heartbeat pattern that was only mentioned in passing here: a small script that monitors its own timer and triggers an escalation on failure — basically a meta-timer. I use the same pattern in step 4 of this article, just more compact. Anyone wanting to recreate the book project 1:1 should start with the heartbeat, because that's where the biggest learning effect lies.
When you should rebuild this — and when you shouldn't
You should switch to systemd timers when you have several critical jobs you actively want to monitor, and when you're already working with systemd — so Debian, Ubuntu, Fedora, Arch, basically any modern Linux server. If you notice you're slowly rebuilding a mini scheduling layer inside your cron job wrappers, that's the point where systemd becomes the better choice.
On the other hand, if you have a Raspberry Pi cluster running a single cron job and an outage doesn't interest you, stay with Cron. The effort of writing, enabling, and testing six unit files only pays off if you actually use the observability.
For my Watch & Alert project, the switch made sense because it addressed exactly the points that had worried me before: silent failures, missing dependencies, fragmented logging. Three jobs now run cleaner, one still runs in Cron, and I no longer lose sleep over a heartbeat that quietly vanished. Whether the same applies to you depends on how much effort you want to invest in observability — and how many hours of sleep you lose when a job fails at night.