Where we started
Our Watch & Alert system has been collecting feeds from four categories since the end of 2024: indie Python projects, DevOps tools, Linux kernel announcements, and security advisories. In total, that's 47 RSS and Atom sources, plus three webhooks from GitHub watch repos. Every incoming message first lands in an SQLite table incoming_items, is then scored by a classification LLM, and finally delivered via Telegram, ntfy.sh, or email. The whole setup is split across two machines: a Raspberry Pi 5 with 8 GB RAM as an always-on runner, and a Mac mini M2 as a development box.
The classification LLM was originally a cloud model. We kept it because the classification quality was good and the API was cleanly documented. But two things bothered us. First, the running costs. At around 120 articles per day and an average response length of 180 tokens, we land at roughly 6500 tokens per day. That's not a lot of money, but it scales. When we add more sources, the bill grows linearly, without the classification itself getting any better.
Second, data privacy. We process article content that we partly receive through public webhooks. Some of it contains internal notes or repository paths that aren't meant for external servers. Even though the cloud provider is reputable, we don't want to ship every snippet across the internet. For plain news items it doesn't matter, but for security advisories with specific CVE IDs, it's important to us.
The trigger for the switch was a Reddit post about a free, MIT-licensed AI agent that runs entirely locally (r/selfhosted). We looked at the architecture, read through the repository, and realized: this is exactly what we've been looking for in Watch & Alert for months. An agent that uses our existing Ollama installation, with no cloud dependency of its own, with a clean Python interface and tool calls.
Before the switch, we ran the cloud path in parallel for six weeks. Per article, we queried both the cloud model and the local Ollama model. Agreement on categorization was 89 percent, on urgency scoring 76 percent. That 11 percent deviation on category was important enough to dig into the topic properly instead of just flipping the switch.
Requirements
We set ourselves three hard requirements before we started.
The first requirement was local-first, no API keys. The agent must not need external services at runtime. When the internet connection goes down, classification must keep running. That rules out pure cloud agents and is also why we didn't want to just switch to another API-based provider. Even a local model with online license activation fails this criterion.
The second requirement was MIT license. We publish parts of Watch & Alert under a permissive license. An MIT dependency is fine to add, an AGPL or SSPL dependency is not. That narrows the choices, but it's necessary because we want to customize the agent. If you maintain your own project, pay close attention here — license questions can get expensive later.
The third requirement was heartbeat compatibility. Watch & Alert runs on a Raspberry Pi 5 with 8 GB RAM and a Mac mini M2 as a development machine. Both machines should be able to run the agent. The heartbeat needs to check the local process and activate the fallback on failure. The fallback logic was already in place; we just needed to wire up the new path.
On top of that came soft requirements: inference time per article should stay under 3 seconds, the model should run in under 8 GB RAM, and integration into our existing Python module classifier.py should take less than 200 lines of code. These numbers sound arbitrary, but they're experience values from 18 months of operation: more than 3 seconds per article gets uncomfortable on the Pi, more than 8 GB RAM kills other processes on the machine.
Three options side by side
We compared three realistic paths for replacing or augmenting our classification step.
Option 1: Cloud API with a better model. The easiest path would have been to keep the cloud model and instead switch to a more capable model. That would have slightly improved quality, but solved none of our underlying problems: still API keys, still costs, still data in transit. For an indie project with 120 articles per day, that's not scandalous, but it's also not satisfying. We dropped this option after one day, because it wouldn't have made any progress toward independence.
Option 2: Local Ollama with Llama 3.1 8B. We already had Ollama running on the Mac mini because we use it for other experiments. A quantized model with Q4_0 quantization uses about 4.7 GB RAM and delivers usable results for our classification tasks. Inference time sits at 1.8 to 2.4 seconds per article on M2 hardware. Advantage: no change to the model interface, we just swap the endpoint. Disadvantage: we had to rework the prompt engineering from cloud style to local models, because Llama 3.1 responds to system prompts differently than GPT models. That cost us about two afternoons.
Option 3: Self-hosted open-source agent with tooling. The agent presented in the Reddit post brings not just the model but a small framework for tool calls, memory, and structured outputs. It's written in Python, MIT-licensed, and runs headless. It expects an installed Ollama instance; the model is selected via tags. The appeal is that we can map not just classification but also the routing to ntfy.sh or Telegram through tool calls. Disadvantage: we're introducing a new codebase we have to maintain. For a one-person project, that's a deliberate decision.
After a week of testing, we went with option 3, because the structured output via JSON schema significantly simplifies the downstream processing in Watch & Alert. The agent hands us back a dict directly with category, priority, and summary, no string parsing required. In the actual code, that saved us about 60 lines of regex that were previously needed for edge cases with the cloud model.
The solution
The migration ran in four steps: install the agent, write the Watch & Alert integration, adapt the heartbeat, observe for two weeks.
For installation, we cloned the repository, installed the prerequisites in a virtual environment, and started the agent via a systemd user service. The service listens on a local port and accepts POST requests with JSON payloads. On the Mac mini it runs as a LaunchAgent, on the Raspberry Pi as a systemd user service with restart policy. Here's an excerpt from the systemd unit we use on the Pi:
[Unit]
Description=Watch & Alert Classifier Agent
After=network.target ollama.service
[Service]
Type=simple
ExecStart=/home/pi/.venvs/wa-agent/bin/python -m classifier_agent.server
Restart=on-failure
RestartSec=30
Environment=OLLAMA_HOST=http://127.0.0.1:11434
[Install]
WantedBy=default.targetThe Watch & Alert side no longer calls the agent directly, but goes through a small adapter class. That way, we can switch between the agent and a direct Ollama call at any time if the agent hangs:
from __future__ import annotations
import logging
from dataclasses import dataclass
from typing import Final
import httpx
logger = logging.getLogger(__name__)
AGENT_URL: Final[str] = "http://127.0.0.1:8765/classify"
TIMEOUT_S: Final[float] = 8.0
@dataclass(slots=True)
class Classification:
category: str
priority: int
summary: str
source: str # "agent" | "fallback"
class ClassifierClient:
def __init__(self, agent_url: str = AGENT_URL) -> None:
self._agent_url = agent_url
self._client = httpx.Client(timeout=TIMEOUT_S)
def classify(self, title: str, body: str) -> Classification:
payload = {"title": title, "body": body[:4000]}
try:
response = self._client.post(self._agent_url, json=payload)
response.raise_for_status()
except httpx.HTTPError as exc:
logger.warning("classifier agent unreachable: %s", exc)
return self._fallback(title, body)
data = response.json()
return Classification(
category=data["category"],
priority=int(data["priority"]),
summary=data["summary"],
source="agent",
)
@staticmethod
def _fallback(title: str, body: str) -> Classification:
# Trivial-Heuristik für den Notfall, dokumentiert im Buch.
lowered = (title + " " + body).lower()
priority = 3 if any(w in lowered for w in ("cve", "exploit")) else 5
return Classification(
category="uncategorized",
priority=priority,
summary=title[:200],
source="fallback",
)
The heartbeat checks the agent every five minutes. If it fails to respond three times in a row, the system sets a flag in the SQLite table runtime_state. On the next run, the orchestrator sees the flag and routes directly to Ollama instead of the agent. As soon as the agent responds again, the flag is cleared. This is exactly the pattern we've maintained since the start of Watch & Alert and which the book describes in detail.
After the migration, we ran both paths in parallel for two weeks and wrote the results to a log file. Per article, both the cloud model and the local agent were queried; both answers landed with timestamps in incoming_items.classification_log. The result was surprisingly clear: across 1340 articles in that span, agreement on categorization was 91 percent, the cloud path delivered a better urgency assessment on 4 percent of articles, the local agent on 3 percent. On the remaining 2 percent, it was a matter of taste. For our use case, the agent is more than sufficient.
What we learned
Insight 1: A local model is no silver bullet. At the start, we expected Llama 3.1 8B to be able to do everything the cloud model can. Reality is more nuanced. For simple categorization like "security" versus "DevOps", the local model is perfectly fine. With complex multi-category cases or articles with an ironic undertone, we saw outliers. We adjusted the prompt structure and now define at most one main category per article. Ambiguity is explicitly flagged as category=review.
Insight 2: Structured outputs save more code than they cost. We originally ran regex over the cloud model's free text, because the model didn't wrap its answers in stable JSON. With JSON schema and tool calls from the local agent, that whole parser goes away. That lowers the error rate for classification and makes the code more readable.
Insight 3: Heartbeat isn't a bonus, it's mandatory. During the two-week test run, the agent crashed three times. Once because of full RAM, once because of a stuck Ollama process, once because of a failed model update. Without the heartbeat, we'd have had silent failures we'd only have noticed on the next manual look at the logs. The heartbeat detected every failure within 10 minutes and activated the fallback.
Insight 4: MIT license is more than a formality. We customized the agent in two places: once to remove an extra tool we didn't need, and once to emit a model_version field. Both would have been problematic for our use case under a copyleft license. MIT lets us take the code, modify it, and keep maintaining it internally without endangering the overall project's license.
Insight 5: The way back must stay open. We deliberately built the adapter so that it activates the fallback not just on agent failure, but also via a configuration switch. That way, in an emergency, we can fall back to cloud without changing code. We haven't needed it yet, but it's reassuring. A system that only knows a single path is a system that goes down at every path problem.
Connection to the book
The book covers the heartbeat pattern in detail. The Watch & Alert setup with a local classification agent is exactly the concrete realization of the constellation the book presents in chapter 4 as a runnable example. The chapter on sources and classification walks through the prompt structure and JSON schema output we use here for the agent. The fallback chapter shows how to keep the cloud path as a safety net without blocking the local path. If you want to work through the setup from this article in detail, start reading at chapter 3. The systemd block above corresponds almost word-for-word to the listing in the appendix; the adapter code is an extended version of the classification skeleton from chapter 5.
When you should rebuild this — and when you shouldn't
The rebuild is worth it for you if you run your own small monitoring system, regularly process between 50 and 500 articles per day, and have a machine with at least 16 GB of RAM. If you develop on Apple Silicon, the switch is especially painless because the Neural Engine significantly speeds up inference. If you work with WSL2, it works too, but takes a bit more patience during the first model download and for path configuration between the Windows and Linux filesystems.
The rebuild isn't worth it if you process fewer than 20 articles per day, have no interest in model maintenance, or use a cloud provider you trust and that doesn't cost you anything. In that case, the existing cloud solution is pragmatic and the effort of rebuilding exceeds the benefit. Nor is the rebuild worth it with strongly fluctuating article volume, because you can't plan model load well in that case.
The rebuild is also not recommended if you don't have time for two weeks of parallel operation. The comparison mode isn't a luxury — it's the only way to realistically assess quality. Without that comparison, you risk blindly trusting a model that performs worse than the old one in your context. After three days, we'd almost decided to switch back because an outlier was overrepresented in the sample. It took the full two weeks to straighten out the picture.
If you bring the prerequisites, the step is smaller than it looks. We invested about 14 hours total in the rebuild — six for the code itself, four for tests, and four for documentation and heartbeat adjustments. In the book you'll find all the listings shown abbreviated here. Start with the heartbeat, not the agent. Anyone who secures operations cleanly can swap the model at any time.