What this is about
Classification is the step of a monitoring pipeline where content goes to a language model. With a cloud API, all texts leave your machine. That becomes a problem as soon as feeds contain confidential content or costs grow with volume. The alternative is a self-hosted model that runs on your own hardware.
Ollama is a tool that provides quantized models locally and offers a simple HTTP interface. There are also open agent projects that run models without third-party API keys. Which features a specific project has is described in its own documentation. This article is about the principle.
Why local?
- Privacy: texts stay on your machine. This is the most important reason when feeds contain internal or confidential content.
- Cost: there are no fees per call, only electricity and hardware. With large volumes this becomes predictable.
- Independence: no API key that can expire, no price change, no provider that shuts down a model.
There are drawbacks as well: smaller models are weaker than large cloud models on difficult tasks, response time depends on your hardware, and you take care of operation and updates yourself.
Model size and memory
Quantized models need much less memory than their originals. As a rough rule of thumb, a model with a few billion parameters in heavily quantized form fits into a few gigabytes of RAM, while larger models need correspondingly more and run noticeably slowly without a graphics card. Check the exact size in the model description and test with your real texts.
For classification with a few labels a small model is often enough. For summaries or hard distinctions a larger one is worth it.
The HTTP call
Ollama listens locally by default and accepts requests over HTTP. A call with a timeout and clear error handling looks like this:
import logging
import requests
log = logging.getLogger("llm")
OLLAMA_URL = "http://localhost:11434/api/generate"
def ask_local(prompt: str, model: str, timeout: int = 120) -> str | None:
try:
r = requests.post(OLLAMA_URL, timeout=timeout, json={
"model": model, "prompt": prompt, "stream": False})
r.raise_for_status()
return r.json().get("response", "").strip()
except requests.RequestException as exc:
log.warning("Lokales Modell nicht erreichbar: %s", exc)
return None
The return value None tells the caller that classification was not possible.
Fallback strategy
A local model can fail, for example when the machine restarts or the model is still loading. So plan deliberately what happens then:
- Try again after a short wait.
- Defer the entry and retry on the next run. For uncritical messages this is the simplest solution.
- For uncritical feeds, fall back to a cloud API, for sensitive ones never.
def classify(text: str, sensitive: bool) -> str | None:
answer = ask_local(text, model="klein")
if answer is None and not sensitive:
answer = ask_cloud(text)
return answer
The function ask_cloud stands for your own cloud call. What matters is the rule that sensitive feeds never leave the local path.
Operation and heartbeat
A local model is one more service that has to keep running. The daily heartbeat should therefore also report how many classifications succeeded locally, how many were deferred and whether the fallback was used. If the message fails to appear or the number of deferrals rises, the model service is the first suspect.
When it is worth it (and when not)
A local model is worth it when privacy matters, large volumes occur or you want to stay independent of a provider. It needs hardware with enough memory and the willingness to maintain one more service.
Stay with the cloud when feeds are public, volume is small and the quality of a large model tips the scale. A hybrid of both, as shown above, often covers both needs.