Zero-Downtime Secret Rotation in Our Watch & Alert Setup

Where we started

Our Watch & Alert setup has been running on a Raspberry Pi 4 for 18 months. We use RSS feeds for monitoring, store alerts in a SQLite database, and classify messages locally with Ollama. Secrets — Telegram tokens, third-party API keys — live in a config.yaml. Until now, rotation was done manually. A plain git pull wasn't enough because the keys were hardcoded.

During the last rotation, we had to take the system down to load the new config. The Telegram bot was offline for 45 seconds. For 50 users who rely on daily updates, that was unacceptable. The outage caused two alerts to never get sent. That was the last time we rotated manually.

Our setup has no Kubernetes clusters or microservices. It's a classic Python script on a single server, driven by cron. The secrets are critical for communication, but we don't have any compliance requirements to meet. The whole fix had to come in under 3 hours, including testing.

Requirements

The solution had to meet these criteria:

• Zero downtime during rotation
• No new dependencies (no Vault, no AWS Secrets Manager)
• Automation via cron (no manual steps)
• Integration with the existing SQLite workflow
• Solid logging support for troubleshooting

The implementation had to be 400 lines of Python or fewer. Rotation had to happen automatically within 90 days to handle expiry. We didn't want to introduce new tools — the solution had to live inside our existing project.

Three options compared

Option 1: Environment variables with service restart

Most DevOps workflows use environment variables for secrets. But we kept our config in a file. Switching to env vars would have required restarting the whole system. That meant 45 seconds of downtime — unacceptable for a system handling 50 alerts a day. The cost of switching was too high since we couldn't load the script via an environment variable.

Option 2: HashiCorp Vault

Vault is an established solution for secret rotation. We would have needed to stand up a Vault instance, which meant an extra VM and reconfiguring our API endpoints. Integration would have eaten 8 hours including testing. For a project with 10 feeds and 50 alerts a day, that was clearly overkill. The docs were complex, and we didn't have time to learn another tool.

Option 3: Dual-key system with SQLite updates

This approach used our existing SQLite setup. We stored the new key in a separate column and allowed a transition window during which both keys were accepted. The service checked both keys during rotation. The solution needed no new dependencies and slotted into our existing workflow. The implementation was more involved, but the benefits clearly won out.

The implementation

Step 1: Adjust the database schema

We added a new rotation_time column to the secrets table. The old key stays active until the next rotation, which lets us monitor the transition.

def add_rotation_column() -> None:
    """Add rotation_time column to secrets table."""
    with sqlite3.connect(DB_PATH) as conn:
        try:
            conn.execute(
                "ALTER TABLE secrets ADD COLUMN rotation_time TEXT"
            )
        except sqlite3.OperationalError:
            pass  # Column already exists

Step 2: Write the rotation script

The script generates a new key, updates the database, and sends a reload signal to the main process. It first checks whether the service is running.

def rotate_secret(new_secret: str) -> None:
    """Rotate secret and trigger reload."""
    with sqlite3.connect(DB_PATH) as conn:
        conn.execute(
            "UPDATE secrets SET value = ?, rotation_time = ? WHERE key = 'telegram_token'",
            (new_secret, datetime.now().isoformat())
        )
        conn.execute(
            "INSERT OR IGNORE INTO secrets (key, value, rotation_time) "
            "VALUES ('telegram_token_new', ?, ?)",
            (new_secret, datetime.now().isoformat())
        )
    if is_service_running():
        subprocess.run(["kill", "-HUP", str(os.getpid())])
    else:
        logger.error("Service not running during rotation")

Step 3: Transition window in the service

The service checks both keys before every Telegram call. After 30 days, the old key is deleted.

def get_telegram_token() -> str:
    """Get active token, prefer new one."""
    with sqlite3.connect(DB_PATH) as conn:
        cursor = conn.execute(
            "SELECT value FROM secrets WHERE key IN ('telegram_token', 'telegram_token_new') "
            "ORDER BY rotation_time DESC LIMIT 1"
        )
        return cursor.fetchone()[0]

Step 4: Automating via cron

The cron job runs the rotation every 90 days. There's a pre-rotation check to confirm the service is up.

# /etc/cron.d/secret-rotation
0 0 * * * /usr/bin/python3 /opt/watch-alert/rotate_secrets.py

Step 5: Error handling

If the new key doesn't work, the old one gets reactivated. We log every step to the SQLite database.

def test_new_token() -> bool:
    """Test new token before switching."""
    try:
        send_telegram_message("Test message", token=get_telegram_token())
        return True
    except Exception as e:
        logger.error(f"Token test failed: {e}")
        return False

What we learned

Lesson 1: A dual-key system needs a clear transition strategy

The transition window was critical. We set 30 days as the default. For critical keys like the Telegram bot, we shortened it to 15 days. A longer window would have increased the chance of things going wrong. 15 days was realistic for our small project.

Lesson 2: Logging is essential

Without logging, we wouldn't have been able to trace failures. We wrote every step to the SQLite database. One rotation bug led to a 30-second stall — the logs helped us find the cause. That 30-minute logging decision saved us 4 hours of debugging later.

Lesson 3: Testing in staging is non-negotiable

We tested rotation with 10 dummy alerts. That showed the service needed 0.2 seconds after a reload — not 45 seconds like a full restart. Staging tests were crucial for measuring that.

Lesson 4: Automation cuts down on human error

Manual rotation led to 3 cases where the key didn't get updated. Automation eliminated those mistakes. For 50 alerts a day, that's a big win.

Lesson 5: No new dependencies — that's the point

By sticking with SQLite and existing Python modules, we stayed inside our project. No new tools, no new services. That mattered for our small team.

How this connects to the book

Chapter 07 "Security — Deep Dive" in the book covers the importance of planned rotations. Our approach fits the book's principles perfectly. We didn't pull in external tools — we reshaped the system itself. The chapter emphasizes that "security starts in the code," and that's exactly what we did. The implementation shows how to get secure rotation with minimal moving parts.

When to rebuild this — and when not to

Rebuild this if:

• You run your own monitoring system on SQLite and Python.
• You monitor 10 feeds or fewer.
• You don't have compliance requirements.
• You want to automate rotation every 90 days.
• You don't want to bring in new tools.

Skip it if:

• You're running an enterprise system with more than 100 feeds.
• You already use Vault or AWS Secrets Manager.
• You have compliance requirements like PCI-DSS.
• You run a more complex infrastructure (Kubernetes, microservices).

The solution is built for indie developers and small teams. Enterprise setups have better tools available. But for us, this was the right call: simple, understandable, no overhead. If you run a similar setup, give it a try. It's worth it.