What this is about
API tokens, bot credentials and passwords should be renewed regularly, and immediately once misuse is suspected. In a pipeline with several components this is tricky: collector, classification and dispatcher may use the same key. If it is replaced while one service still uses the old one, that service's next call fails.
The goal is a rotation in which no service is ever left without a valid key.
The basic principle: two valid keys
The trick is a phase in which the old and the new key both work. The order is:
- Create the new key at the provider, do not touch the old one yet.
- Store the new key in the central place the services read it from.
- Switch the services over with a reload or restart and check that they work with the new key.
- Only then revoke the old key at the provider.
Not every provider allows two active keys. Where that is not possible, a short window is unavoidable. Place it deliberately in a quiet period and stop the services for those minutes.
Store secrets centrally and outside the code
So that one change is enough, secrets belong in an environment file or a secrets manager, not in the source code and not in several copies. The file should be readable only by the user that runs the service.
install -m 600 -o pipeline -g pipeline /dev/null /etc/pipeline/env
The services read the file through EnvironmentFile= in the unit or when the script starts. After a change a restart or reload is needed, because running processes keep their old environment variables.
Support both keys in the code
A robust variant is a client that knows a primary and a fallback key. If the primary key is rejected with an authentication error, it tries the second one.
import os
import logging
import requests
log = logging.getLogger("secrets")
def keys() -> list[str]:
found = [os.environ.get("API_KEY_NEW"), os.environ.get("API_KEY_OLD")]
return [k for k in found if k]
def call(url: str) -> requests.Response:
last = None
for key in keys():
r = requests.get(url, headers={"Authorization": f"Bearer {key}"},
timeout=10)
if r.status_code not in (401, 403):
return r
log.warning("Schlüssel abgelehnt, versuche den nächsten")
last = r
if last is None:
raise RuntimeError("Kein Schlüssel konfiguriert")
return last
During the rotation both variables are in the file. After the old key has been revoked you remove API_KEY_OLD.
Check before you revoke
Revoke the old key only when you are sure that no service still needs it. A simple test is one request per component with the new key and a look at the log for authentication errors. With several services a small check command is worthwhile that tests each credential individually and prints the result.
The heartbeat helps as well: if the daily status message fails to appear after the switch, some service did not pick up the new key.
Common mistakes
- Only one file was changed while a copy in another directory keeps running. Search for the old key value before you revoke it.
- The service was not restarted and is still running with the old value.
- The new key was committed to a Git repository. Keep secret files out of version control.
- The old key was never revoked. A key that is not revoked stays a risk.
When it is worth it (and when not)
The effort pays off as soon as more than one service uses the same credential or an outage would be noticeable. A fixed checklist and centrally stored secrets make every later rotation routine.
For a single script with one token, where a short outage bothers nobody, the simple procedure is enough: enter the new key, restart the service, revoke the old one. A fallback key in the code would be overkill there.