Repair playbooks
Reflex can run a small set of autonomous repairs when the Brain confidence gate and your team policy allow it. Deploy safety comes first: health-gated rollbacks and audited actions, not a blanket “self-healing” claim.
Most catalogue playbooks are diagnostic suggestions (Brain fingerprint match + dry-run). Only the loops below execute without a human clicking Evaluate.
How autonomous repair works
- A watcher detects a known incident (uptime probe, host metrics, systemd unit status, cPanel health check, or a Pipeline health gate)
- Brain evaluates a fingerprint against your team’s minimum confidence
- If the gate passes and the playbook is wired to an executor, the agent runs the action
- A repair event records what ran
You can require human approval for destructive playbooks in team Brain settings. Mutating disk cleanup, process kill, and service restart are not dry-run-safe.
What is autonomous today
| Loop | Hosts | What it does |
|---|---|---|
| Pipeline deploy health gates / rollback | Any host using Reflex Pipeline | Failed post-deploy health checks roll the release back |
Node / Next.js keep-alive (nodejs_process_dead) | Sites with process manager + process name set | Restarts the PM2 app or systemd unit after consecutive HTTP failures, with a per-site cooldown |
| cPanel service restart | Root-mode WHM hosts with a linked agent | Restarts allowlisted panel services (httpd, MySQL, PHP-FPM, watchdog) when WHM reports them down |
| cPanel disk cleanup | Root-mode WHM hosts with a linked agent | Purges stale files from known-safe cPanel temp dirs and reports df |
Generic disk cleanup (disk_high) | Any linked agent | After disk.root.percent exceeds the server threshold, purges stale /tmp and /var/tmp files, rotated logs (*.gz, *.1), and old releases/ dirs (keeps the newest N) |
Generic systemd restart (systemd_unit_failed) | systemd hosts | Restarts allowlisted units (nginx, Debian php*-fpm, cPanel ea-php*-php-fpm, mysql/mariadb, redis, supervisor) when they are failed. Never sshd, systemd, or reflexd |
Generic OOM recovery (memory_exhaustion / php_fpm_oom) | Any linked agent | When memory.available_mb is below the floor (or memory.percent exceeds its ceiling), or kernel.oom_kill_total increases, restarts the PHP-FPM unit last seen active on the host (including cPanel ea-phpXX-php-fpm) |
PHP-FPM restart (php_fpm_restart) | Any linked agent | Restarts the resolved PHP-FPM unit after explicit reload/recycle fingerprints |
nginx reload (nginx_reload) | systemd hosts | Reloads nginx (systemctl reload) after matched reload/cert-deploy fingerprints |
nginx 502 (nginx_502) | systemd hosts | Restarts nginx on matched 502/upstream fingerprints or when http.server.upstream_errors_per_min exceeds the floor |
Queue worker restart (queue_worker_dead) | Hosts with a Reflex queue worker | Restarts the worker unit via worker.restart when worker.queue_metrics fails. On systemd production hosts the agent runs systemctl restart by default; set REFLEX_WORKER_SYSTEMCTL=0 on the agent to opt out. |
Compose container OOM (container_oom_kill) | Sites using Reflex Pipeline compose deploys | Restarts the compose project (docker compose restart) when kernel OOM kills increase on that host |
Runaway process (runaway_process) | Any linked agent | After two consecutive samples of an allowlisted process over CPU or RSS, sends SIGTERM. Site process_name values are included; same denylist as systemd restart |
For Next.js keep-alive: set process manager, process name, and optional upstream port / health check path on the site page. Auto-repair uses the team Brain confidence gate and a per-site cooldown.
What is not autonomous
Brain can still suggest other catalogue playbooks. Those matches evaluate and alert; they do not run as a generic loop. cPanel reseller playbooks (cpanel_reseller_*) are diagnostic-only — they fingerprint and guide, they do not write to WHM. Kubernetes pod recovery is not wired (no cluster credentials on the agent); compose hosts use docker compose restart instead.
The shipped YAML catalogue currently has 69 playbooks (counted from playbooks/**/*.yml id: fields). That number is asserted in CI so this page cannot silently drift.
Enabling playbooks
Built-in autonomous loops (table above) run when the team Brain confidence gate passes. Enable or disable them per server under Server → Playbooks. A disabled playbook still evaluates and alerts; it does not auto-execute.
Team-wide dry-run still applies: add the playbook under Settings → Repair playbooks with apply mode Dry-run only. A missing team row keeps the built-in auto-execute path. Destructive work still respects the team/server human-approval flag. Cooldowns are fixed in the dispatch services, not a dashboard field.
Notifications
Configure who gets notified when a playbook executes:
- Slack — Post to a specific channel
- PagerDuty — Escalate to on-call
- Email — Send to team members
- Webhook — Fire a custom webhook
Incident reports
Every playbook execution generates a detailed incident report in the dashboard, including trigger conditions, timing, diagnostic output, and resolution confirmation.