Repair playbook catalogue

Diagnose ACME or Let's Encrypt renewal failures: HTTP-01 reachability, DNS-01 records, rate limits, and clock skew; validate edge config before forcing renew.

Apache httpd upstream and capacity: scoreboard full, MaxRequestWorkers, proxy errors to backends; correlate with app process health.

Managed API gateway throttling: usage plan quotas, burst limits, and WAF rules; align client backoff with Retry-After headers.

AppArmor profile denials: aa-status, complain versus enforce, and path rules for services; avoid disabling profiles globally without a scoped fix.

Cloud block volume burst balance: IOPS or throughput credit exhaustion on EBS-like volumes; detect throttling before resizing or changing volume type.

Diagnose storage and cache path permission errors; confirm ownership on framework dirs before any chmod policy changes.

Caddy reverse proxy upstream failures: dial errors, TLS to origin, and automatic HTTPS conflicts; validate upstream SNI and health checks.

CDN or edge cache origin errors: 5xx from origin, stale serve policy, and health probe paths; separate CDN misconfig from origin outage.

Linux conntrack table exhaustion: nf_conntrack_count versus max, short-lived connection storms, and NAT rules; plan sysctl changes with rollback.

Correlate Kubernetes or containerd OOM kills with cgroup limits, recent deploys, and workload memory; capture events before raising limits or fixing leaks.

Recover failed AutoSSL issuance when certificates expire or ACME DCV fails — fix DNS, .well-known routing, and re-trigger domain validation.

Detect when a cPanel server IP appears on major email blacklists, identify affected accounts, and surface delisting steps before deliverability collapses fleet-wide.

Emergency disk cleanup when a cPanel account or server partition exceeds 90% — rotate logs, purge tmp, and trim old backups before services fail.

Flush or force-delivery on a backed-up Exim queue when message count exceeds threshold — identify spamming accounts before the whole server IP is blocklisted.

Restart Apache or LiteSpeed via WHM API when httpd is unresponsive or returning 503s — configtest first, then controlled restartsrv.

Recover MySQL or MariaDB after crash or connection refused — restart via WHM, allow InnoDB recovery, escalate to mysqlcheck when corruption is suspected.

Restart the ea-php FPM pool for a cPanel account when PHP-FPM stops responding — domain-aware pool mapping and socket verification.

Scheduled A, MX, and CNAME validation per cPanel domain — flag drift when public DNS no longer matches the expected server IP or mail routing.

Scheduled SPF, DKIM, DMARC, and MX validation for every cPanel account under a WHM connection — with copy-paste DNS fixes and optional auto-repair via WHM or linked Cloudflare/Route53.

Disk quota thresholds per cPanel account — amber at 80%, email alert at 90%, urgent at 95% before MySQL, mail, and backups fail silently.

HTTP, DNS, and SSL health checks per cPanel account domain — alert resellers when a customer site is down, mis-pointed, or serving an expired certificate.

TLS expiry monitoring for every domain under a WHM connection — warnings at 14, 7, and 1 day with AutoSSL vs commercial renewal guidance.

Restart critical cPanel daemons (cpsrvd, tailwatchd, lfd) when WHM service status reports them down — prevents cascading panel outages.

Respond when scheduled jobs hit OOM or are killed: capture command footprint, stagger overlaps, and tune memory limits.

Measure read-replica lag versus primary: WAL receive/replay, replication slots, and application read-after-write risk; no promotion or destructive failover from this playbook.

Recover from connection pool saturation: trim idle sessions, validate max_connections, and re-check query latency.

Free safe space under /var/log, package caches, and tmp; escalate when growth is from application data.

Isolate NXDOMAIN and resolver failures from app bugs; check upstream DNS, TTLs, and stub resolver health.

ASP.NET Core threadpool starvation: Kestrel request queueing, sync-over-async, and MinThreads defaults; capture threadpool stats before raising limits.

Go runtime goroutine and heap growth: pprof goroutine/block profiles, runaway pollers, and HTTP server shutdown leaks.

Triage Laravel Horizon and supervisor-managed workers: confirm heartbeat, process state, and queue depth before a controlled queue:restart or supervisor group restart.

JVM GC pauses and Metaspace pressure: GC logs, heap sizing, and safepoint stalls; correlate with traffic before raising heap alone.

ETA Q4 2026

Kafka consumer lag and rebalances: partition assignment, coordinator errors, and processing rate versus fetch; tune max.poll.interval before scaling partitions.

Meilisearch availability: engine 503, task queue backlog, disk full on data path, and index corruption signatures; snapshot before destructive reindex.

Memcached slab imbalance, evictions, or connection churn: capture stats items, conn yields, and hit ratio before changing memory factor or connection pools.

Correlate cgroup OOM, exit 137, and container kills with workloads; trim runaway processes and raise limits only after root cause.

MongoDB slow operations: profiler, missing indexes, WiredTiger cache pressure; capture namespace and query shape before index changes.

Address MySQL or MariaDB binlog disk pressure: retention policy, relay log volume, and replica coordination; avoid blind PURGE without replica safety checks.

ETA Q3 2026

Burst connection pressure on MySQL or MariaDB: max_connections, per-user limits, pool sizing in apps, and short-lived connection leaks.

MySQL or MariaDB replication triage: IO/SQL thread state, Last_IO_Error / Last_SQL_Error, and network between primary and replica before any skip or reset.

NFS ESTALE and stale file handles: server reboots, inode reuse, and mount options; prefer soft→hard policy review before lazy umount in production.

Triage upstream 502/504 and nginx edge failures: validate upstream health, recent deploys, and timeouts before reload or capacity changes.

Separate legitimate nginx rate limiting from mis-tuned zones: burst, whitelist health checks, and geo/client trust headers before widening limits.

Validate nginx config and reload in-place after upstream or TLS changes without dropping established connections.

Capture event-loop stall context; recycle workers if policy allows

Time sync and clock skew: NTP or chrony offset, leap seconds, and TLS notBefore errors caused by bad time; fix stratum and firewall before cert renewals.

Verify object-storage access: IAM or service-account role, bucket policy, endpoint region, and presigned URL expiry; fix configuration before retrying writes.

Inspect PHP-FPM pool and reload when OOM signatures appear

Reload PHP-FPM pools after pool health checks; use when traffic is healthy but workers need recycling.

Postgres bloat and autovacuum pressure: wraparound risk, dead tuples, aggressive autovacuum tuning versus I/O; avoid ad-hoc VACUUM FULL without a window.

ETA Q3 2026

Read-only Postgres lock triage: identify blockers vs blocked sessions, lock types, and wait time; escalate before terminating backends.

Logical replication slot lag and WAL retention: identify inactive slots, consumer lag, and disk risk on the publisher before dropping slots.

Stabilise uvicorn/hypercorn/daphne workers after ASGI timeouts or connection churn; validate app boot and pool sizing.

Inspect gunicorn/uwsgi worker memory and timeouts; recycle workers and align worker count with available RAM.

Detect stalled queue workers, restart supervisor-managed processes, and verify job throughput resumes.

RabbitMQ backlog and flow control: consumer ack rates, memory/disk alarms, and queue depth; scale consumers before raising prefetch blindly.

Triage Redis memory pressure: capture INFO memory, eviction counters, and replication backlog; prefer maxmemory-policy and capacity fixes before any key flush.

Redis RDB or AOF persistence slowness: fork latency, disk I/O, and save policy; reduce write spikes before disabling persistence.

ETA Q4 2026

Redis Sentinel failover churn: quorum, SDOWN/ODOWN transitions, and split-brain risk; confirm network and auth before forcing failover.

Elasticsearch or OpenSearch red cluster: allocation explain, disk watermarks, and unassigned shards; read-only cluster API triage before any index delete.

SELinux AVC denials: context mismatches for nginx, php-fpm, or containers; use permissive domains only as a temporary bridge with audit evidence.

SMTP transport failures: auth, TLS, rate limits from the provider, and envelope-from policy; verify secrets rotation and port 587/465.

SQS visibility timeout and poison messages: approximate age of oldest message, duplicate receives, and handler duration versus timeout.

Heavy swap in/out and kswapd pressure: RAM headroom, cgroup limits, and mis-sized workloads; fix memory leaks before tuning swappiness alone.

Trace certificate expiry, chain, and hostname mismatch errors; verify trust store and renewal automation before rotating certs.

Outbound webhook delivery: HTTP status, signature headers, retry budget, and idempotency keys; fix endpoint auth before raising retries.