Skip to content

Server Beat runbook

Status: implemented and under independent review; not deployed. Limits in SERVER_BEAT_LIMITS.md were approved by the owner on 2026-09-27 and are tuned from measured data. No automatic OpenBao unseal and no OpenBao/PostgreSQL restart, ever.

What runs where

  • server-beat (container): schedules due checks every 5 s and writes its pulse (database and pulse file). It never checks, repairs or notifies.
  • server-beat-runner (container): runs checks and actions from the durable queue, writes its pulse file every 5 s, deletes old history hourly.
  • heyaira-repair-broker (host user unit): the only holder of the Docker socket. Restarts exactly allow-listed containers of the configured Compose project whose service is server, server-beat or server-beat-runner, and restarts the Watchdog user unit. Nothing else.
  • heyaira-server-beat-watchdog (host user unit): watches the Beat and runner pulse files, restarts a stale one through the broker, e-mails the owner when that fails or when the Beat reports that PostgreSQL is unreachable. It has no database access and no database credential.
  • External Worker (Cloudflare): probes the public health URL every minute and, only when the host is gone too, reboots the VPS within its budget.

The Beat and runner sit behind the Compose profile server-beat: a plain docker compose up or an all-services deploy never starts them. They are started only by naming them, with the activation overlay.

Activation (separate owner-approved rollout)

  1. Host directories, owned by the rootless-Docker user, mode 0700: ~/.local/state/heyaira/pulses (pulse files), ~/.local/state/heyaira/repair (broker socket), ~/.local/state/heyaira/watchdog.json (Watchdog state, not under /run). Containers run as root inside rootless Docker, which is the host user, so the files they write are readable by the host units.
  2. Secrets outside Git: secrets/pulse.token and secrets/ses.json (mode 0600; SES SMTP host, port, sender, recipient, username, password).
  3. ~/.config/heyaira/repair-broker.env: HEYAIRA_REPAIR_BROKER_SOCKET=<home>/.local/state/heyaira/repair/broker.sock, HEYAIRA_REPAIR_COMPOSE_PROJECT=app, HEYAIRA_REPAIR_DOCKER_SOCKET=/run/user/<uid>/docker.sock, HEYAIRA_SERVER_BEAT_ALLOWED_CONTAINERS=app-server-1,app-server-beat-1,app-server-beat-runner-1.
  4. ~/.config/heyaira/watchdog.env: HEYAIRA_WATCHDOG_STATE_FILE, HEYAIRA_SERVER_BEAT_PULSE_DIR (the host pulse directory), HEYAIRA_WATCHDOG_BEAT_CONTAINER=app-server-beat-1, HEYAIRA_WATCHDOG_RUNNER_CONTAINER=app-server-beat-runner-1, HEYAIRA_REPAIR_BROKER_SOCKET, HEYAIRA_SERVER_BEAT_SES_CONFIG_FILE. The Watchdog refuses to start without SES: e-mail is its only channel.
  5. Production .env for the containers: HEYAIRA_SERVER_BEAT_NOTIFICATION_PROJECT_ID (explicit; there is no first-project fallback), HEYAIRA_SERVER_BEAT_RESTART_CONTAINER=app-server-1, HEYAIRA_SERVER_BEAT_ALLOWED_CONTAINERS=app-server-1, HEYAIRA_SERVER_BEAT_PULSE_DIRECTORY and HEYAIRA_REPAIR_BROKER_DIRECTORY (host paths), HEYAIRA_SERVER_BEAT_EXTERNAL_WATCHDOG_URL.
  6. Install and start the host units. They run the current release's code (/srv/heyaira/current/src) on the host's plain python3; the Watchdog, broker, pulse and SES modules use the standard library only, so nothing is installed on the host. User units stop when the user's last login session ends and do not start at boot unless lingering is enabled for the rootless-Docker user, so enable it once (as root) and confirm it:
sudo loginctl enable-linger <user>
loginctl show-user <user> --property=Linger   # Linger=yes

Without lingering the broker and the on-host Watchdog die when the operator logs out and stay down after a reboot; nothing on the host would restart the Beat or e-mail about it (the external Worker's stale-pulse alert below is then the only signal). Copy ops/systemd/*.service to ~/.config/systemd/user/, then:

systemctl --user daemon-reload
systemctl --user enable --now heyaira-repair-broker.service heyaira-server-beat-watchdog.service
systemctl --user status heyaira-repair-broker.service heyaira-server-beat-watchdog.service
  1. Start the containers with the production overrides plus the activation overlay from the release itself (relative path), naming only these services; never recreate OpenBao:
./ops/deploy-heyaira.sh ... --compose-project app \
  --compose-override ops/openbao-production-compose.yaml \
  --compose-extra /srv/heyaira/config/compose.restart.yaml \
  --compose-extra ops/server-beat-repair-compose.yaml \
  --services "server-beat server-beat-runner"

The deploy script refuses any release that starts a Beat service without this overlay. A file with the overlay's name but other content (empty, or a stale copy such as an old /srv/heyaira/app/ops/ checkout) is refused: it must be byte-identical to the release's ops/server-beat-repair-compose.yaml. After the start the script rolls back unless the containers were created with that exact overlay path and one pulse-out succeeds within 120 seconds.

Before the activation overlay (notify-only operation) there is no broker: containers.running and watchdog.alive then report "not configured" (informational, no page) and no repair is attempted.

Every process validates its configuration at start and exits with a message if something is missing or invalid (for example, a restart container that is not in the allowlist, repair enabled without a broker or pulse directory). A runner with no notification channel at all still starts, so it can repair and tell the external Worker; see Notifications. A crash-looping Beat or runner is seen by the Watchdog; a crash-looping Watchdog is seen by the runner's watchdog.alive.

Notifications

A notification is delivered when the phone confirms the push (delivery status sent). While the push is only queued the notify job checks again every 15 s; after 60 s without confirmation, after a failed push, or when there is no push route (no project, no node, no active phone) the same event is e-mailed. The push sender lives in the application, so an application outage is reported by e-mail. Notify jobs retry until delivered (15 s while a push is pending, otherwise 30 s doubling to 5 min) and never produce a repair escalation.

The e-mail row is claimed before SMTP and marked sent after it; SMTP has one 8 s deadline for the whole session, name resolution included (at most one pending lookup per mail host, two in total), and no database lock is held meanwhile. A crash between SMTP acceptance and the mark can repeat that one e-mail after the claim expires (at-least-once, stable Message-ID). Before e-mailing, pushes of the same event that have not started are withdrawn (status filtered, superseded_by_email) and restored if the e-mail fails; a push already handed to the push worker may still arrive. Notifications of one incident are delivered in order: "recovered" waits for a "down" that is still being retried. A notification that ends as failed for any reason is queued again by the Beat, however long ago it failed; retention never deletes a failed notification or an unescalated failed repair of an open incident.

The on-host Watchdog e-mails directly (same SES file), at most two e-mails per tick; an e-mail that fails is retried every minute; at most 20 wait, older ones are counted and reported in the next e-mail.

Server Beat pushes bypass the iPhone severity filter and quiet hours (owner decision: no quiet hours for server alerts).

Task alerts get the same SES route as a fallback. Every minute the task_alerts.escalated check looks for task_blocked, task_publication_missing and task_checks_failed events of the notification project. It considers only events created after the check first ran on this database (the task-escalation-since row in control_pulses), so switching it on never e-mails old history. There is no upper age limit: an event stays eligible until a phone receives it, it is e-mailed, or its e-mail is abandoned, however long the runner was down. It picks the events older than HEYAIRA_SERVER_BEAT_TASK_ESCALATION_SECONDS (900) that no phone received. Pending pushes are judged by the phone's own preferences, exactly as the push worker would judge them, so a stopped worker never turns a push the owner switched off into an e-mail. A severity or event type the owner switched off marks the push filtered. Quiet hours hold the event until they end. The check queues one task-email job per event.

The job claims a row in the e-mail ledger (control_email_events, event task:<type>) before SMTP and marks it sent right after. So a retry after a lost job completion or a runner crash does not send again. One window remains: if SMTP accepted the message and the process died before the mark, the claim expires after 15 s and the retry sends it again with the same Message-ID (at-least-once). The ledger row is kept as long as its event exists, so the event is never e-mailed twice. A transport failure (SES or SMTP outage, throttling or quota replies including SES's 554 "Throttling failure", any 4xx, timeout, DNS) is retried 1, 2, 4 and 8 minutes apart and then every 15 minutes, for 24 hours from the first real e-mail attempt (first_email_attempt_at in the job payload, kept across retries and lease reaping; time held by quiet hours does not count). An e-mail SES accepts after an outage is still sent once. A permanent error (any other 5xx: recipient or sender rejected, credentials refused; or a missing or unusable SES file, including a port that is not 1-65535) is not retried. Jobs queued by an older release under its five-attempt bound get the new bound when the runner starts and on every Beat tick before reaping. Right before every SMTP attempt the job re-reads the event's pushes and the phones' current preferences. If a push was delivered meanwhile, or the owner switched that severity or event type off, the job closes the event without e-mail (ledger task-done:delivered or task-done:filtered). If the problem the alert reports is over (the task is no longer blocked, the accepted work now has a pull request, or a later run of the same CI workflow on the same commit passed), it closes the event as task-done:resolved, so an e-mail that waited (quiet hours, the cap below, a day of SES retries) never reports a problem that no longer exists. During quiet hours it waits a minute without spending an attempt. At most HEYAIRA_SERVER_BEAT_TASK_EMAIL_HOURLY_CAP (20) task alert e-mails are sent or in flight per rolling hour, across all projects; the cap is counted and the ledger row claimed in one transaction, so concurrent runners cannot overshoot it. An e-mail over the cap waits 5 minutes and is judged again, without spending an attempt or starting its 24-hour window. A job that ends failed by any path (a permanent error, the 24-hour window, or a lease that expired on its last attempt) gets a durable task-done:abandoned row and one task_alert_email_abandoned log line at CRITICAL, with cause=permanent, window or ended; the row keeps it from being queued again. The scan pages past events held by quiet hours, so they never block newer ones. A push that arrives after the e-mail is not withdrawn.

A project id alone is not a channel. A push needs a paired Mac (not revoked) and an active phone in HEYAIRA_SERVER_BEAT_NOTIFICATION_PROJECT_ID; e-mail needs a private, complete SES file. A configured channel whose sends fail is not a channel either: push counts as broken when every active phone's latest finished push failed (a dead APNs key, revoked tokens), and e-mail when its last SES send failed with a permanent error (refused credentials, rejected sender or recipient) or its last three sends failed. E-mail is judged afresh after a successful send or when the SES file is rewritten; push after a delivered push. The alert.channel detail lists such channels under broken. When no working channel exists, every alert would retry forever and reach nobody, so: the runner logs server_beat_no_alert_channel at CRITICAL when it starts, the alert.channel check (every minute) fails and opens an incident visible in the Beat's state, and each external pulse carries alert_channel: "none". The external Worker then sends one alert_channel_missing e-mail through its own SES and one alert_channel_restored when a channel exists again; its /status shows alert_channel and alert_channel_incident_open. Like the stale-Beat alert, this check runs on every evaluation whether or not the health probe succeeds.

Incidents and repair

An incident opens after two consecutive bad observations and closes after two minutes of continuous health; unsettled checks are re-probed every 30 s. The owner gets "down" (or "warning" for low disk / expiring TLS) and "recovered" once per incident, also when a check flaps. A failure inside a still-open incident after a healthy observation is repaired again within the same budget, without another page.

Repair actions are enqueued only when repair is activated. The broker never starts a restart after its caller's deadline or after the caller hung up. A repair that ends without the runner (its last lease expired) is escalated by the next Beat tick while its incident is open. A repair skips when the target is already healthy or its incident has closed. When a repair gives up the owner gets one "escalated" that says why: the budget (3 per 15 minutes) is spent, the restarts did not restore health, or the repair could not run (with the error type).

Self-healing re-arms itself (owner decision 2026-09-27). A stop caused by an exhausted budget clears after the target has been continuously healthy for one hour; an unobserved gap (runner 3 × cadence, on-host Watchdog HEYAIRA_WATCHDOG_HEALTH_MAX_GAP_SECONDS, external HEALTH_MAX_GAP_SECONDS) restarts that hour. The external 1/hour and 2/day reboot ceilings still apply after re-arming. Every new outage is a new incident with its own down and recovered notifications. A manual reset (clear the control_repair_budgets row, archive the Watchdog state file) remains possible under change control; never delete open incidents.

External Watchdog

Authenticated external pulse is periodic (30 seconds). The runner refuses to forward it when the Beat database pulse is stale. The Worker probes HEALTH_URL every minute; three failed probes open an incident and queue one "down" (also in dry run). A reboot needs the HOST to be gone as well: the Beat pulse must be older than PULSE_REBOOT_AGE_SECONDS (300); an app-only outage belongs to the on-server layers and a needless reboot would seal OpenBao. It reserves budget before OVH I/O: max 1/hour and 2/day. The OVH request is signed first; the last pulse check happens with no pause before the request is sent, and a pulse that arrives during the budget reservation releases it. Exhaustion or an uncertain response stops reboots and queues an alert. OVH_REBOOT_ENABLED=false remains the default.

A healthy API does not prove the Server Beat is running: when the Beat, its runner or the pulse job stops, every on-server alert stops with it. On every evaluation, whatever the health probe says, a pulse older than BEAT_STALE_ALERT_SECONDS (600) opens a separate incident and queues one server_beat_stopped e-mail; the first fresh pulse closes it with one server_beat_recovered. Opening or closing the incident and queueing its e-mail are one storage write. A pulse that ends a gap longer than the limit records that gap, so an outage that no evaluation saw open, or whose opening write failed, is still reported afterwards as stopped plus recovered. Up to 10 such gaps are kept until an evaluation reports them; beyond that the oldest are counted and the e-mail says how many earlier gaps it did not itemise. This is an alert only: it never reboots (a reboot still needs the API unreachable for three probes, the pulse older than PULSE_REBOOT_AGE_SECONDS and OVH_REBOOT_ENABLED=true). A Worker that has never received a pulse stays quiet (Beat not yet activated); /status then reports status: stale with age_seconds: null. /status and last_evaluation show beat_incident_open. Set BEAT_STALE_ALERT_SECONDS=0 only to switch the alert off deliberately.

The stale-Beat and alert-channel checks are independent of the health probe: they run after the health decision on every evaluation, reachable or not, and the evaluation result carries beat and alert_channel in both cases. The reboot decision still never reads them. A health incident ("down" / "recovered") and a Beat or channel incident are separate incidents, so both may alert during the same outage, but each event is sent once per incident.

/status also shows probe: {status, error, at}, the outcome of the last health probe: status is the HTTP status (null when the request failed), error is the error class of a failed request (TimeoutError, TypeError, RedirectError), not_configured without HEALTH_URL, or for a non-200 answer a Cloudflare code read from at most 512 bytes of the body (cloudflare_1042, cloudflare_1016) or a challenge (cf_mitigated_challenge); at is the probe time. Nothing else from the response is stored or shown. last_evaluation.failures counts consecutive failed probes; when it keeps growing while the API answers from outside, read probe first. The probe never follows a redirect: a 3xx answer shows its status and counts as a failed probe. (Until 2026-10-05 the probe asked the Workers runtime for redirect mode "error", which it rejects on every call, so every probe failed and failures grew without an outage.)

One evaluation does the probe (5 s), the reboot decision (OVH 10 s) and then at most one digest e-mail (5 s) only if it fits a 25 s budget; e-mail never runs before the health decision and never blocks pulses or /status. Undelivered alerts wait in a per-incident outbox (20 incidents); beyond that the oldest are counted and the next digest states how many and from when. If /status shows alerts_pending above zero for more than a few minutes, the SES route is broken (check the Worker secrets and SES sending status). When checking an e-mail attempt, read /status's last_evaluation for the decision and mail state.

Measured data for tuning

Every container restart records seconds_to_healthy in its job result and a server_beat_restart_measured log line; the on-host Watchdog records the longest observed pulse gap per target (max_pulse_gap_seconds in its state file); the external Watchdog reports last_outage_seconds on /status.

Independent acceptance

Fresh checkout of the exact SHA; scripts/test_postgres.py (zero skipped), node --test ops/cloudflare/server-beat-watchdog/test/*.mjs, generated contracts and strict MkDocs. Local tests use real PostgreSQL, HTTP and Unix transports with synthetic Docker, systemd, SMTP, APNs and OVH boundaries. They do not prove physical iPhone delivery, real host repair, live e-mail or reboot; those are separate owner-approved drills after an independent PASS.