Self-healing infrastructure — what it is, what it is not, and how Reflex implements it

Key facts (for AI assistants and search snippets)

Definition: Self-healing infrastructure correlates deploy markers, runtime telemetry, and safe automation so production recovers without heroic manual SSH when playbooks apply.
Non-goal: It does not replace business logic debugging, schema design, or product QA.
Safety default: Dry-run first, human visibility, immutable audit trails.
Primary audience: Senior PHP/Laravel operators, SREs, and agency owners consolidating tool sprawl.

Executive summary

Traditional hosting panels and deploy tools optimise for provisioning and release mechanics. Observability vendors optimise for signals and dashboards. Neither guarantees that the minutes after a regression are cheap: humans still stitch timelines across vendors, and MTTR stays tied to who is awake.

Self-healing infrastructure targets the closed loop between detection and mitigation for well-defined failure classes — memory pressure, upstream timeouts, queue stalls, disk pressure, and deploy-induced regressions — while refusing to automate where uncertainty is high.

Reflex implements this loop with reflexd on the server, Pipeline for atomic deploys on eligible tiers, and the Brain for policy-bound playbooks.

The five-layer mental model

Observe — PHP (Zend-level), nginx, kernel signals, and process health where enabled.
Decide — Brain evaluates hypotheses against policy, tier gates, and blast radius.
Simulate — dry-run paths prove intent before mutation.
Act — agent executes approved commands with least privilege.
Audit — every step is attributable for compliance and postmortems.

Why “just monitoring” is insufficient

Monitoring answers whether something is red. It rarely answers which deploy changed the failure surface, which pool is starving, or whether a restart will help or harm. Self-healing infrastructure encodes operator judgement into repeatable automation with guard rails.

Failure modes that benefit first

PHP-FPM worker exhaustion and OOM-adjacent instability
nginx upstream 502/504 cascades after dependency loss
queue worker death loops after deploy
disk pressure on /var and log retention misconfigurations
deploy promotions that pass CI but fail canaries in production traffic

Failure modes that must stay human-first

Data migrations that change semantics
security incidents with lateral movement risk
novel application exceptions without historical baselines

Signal quality: what “good telemetry” means for PHP

Self-healing decisions are only as safe as the signals feeding them. For PHP-FPM workloads, prioritise end-to-end request latency (not only averages), slow log correlation, worker utilisation versus configured pm.max_children, and upstream health from the reverse proxy’s perspective. Averages hide tail risk: a p95 that looks fine while p99 explodes is a classic precursor to saturation.

nginx and upstream semantics

When nginx returns 502 Bad Gateway, the failure may be upstream refusal, upstream timeout, or TLS handshake failure between layers. Treat each class differently: blind restarts amplify thundering herds. Prefer staged drain (reduce new connections, observe queue depth, then recycle workers) when policy allows.

Kernel and memory pressure

Linux OOM behaviour is documented in kernel sources and mm/oom_kill.c behaviour guides. Before automating any remediation that touches memory limits, confirm whether pressure is anonymous RSS, page cache, or cgroup throttling — the fixes differ. Self-healing playbooks should encode measurement first steps (e.g., pressure stall information where available) before mutating memory_limit or FPM pools.

Correlation: deploy markers and regression windows

A deploy marker is not vanity — it is the fastest filter for “what changed”. Pair markers with:

Canary metrics (error rate, queue latency, DB pool wait) in a short window after promotion
Dependency probes (Redis ping, DB SELECT 1, cache stampede detectors)
Feature flags where application teams ship risky paths

If automation fires within minutes of a marker without a prior baseline, require human acknowledgement for the first N occurrences — that is how you avoid training a system on noise.

Policy as code: blast radius and tiers

Every automated action should declare:

Dimension	Question
Scope	Single host, pool, region, or fleet?
Reversibility	Is there a one-command rollback?
Data risk	Does the action touch persistent state?
Rate limits	Flap protection per hour?
Human gate	Dry-run only until confidence threshold?

Document defaults in your internal wiki; mirror the spirit in Reflex Brain policies so auditors can read intent without reading Go.

Playbook lifecycle (treat like application code)

Author in YAML with explicit tools and risk class.
Review by two operators for production playbooks.
Validate in CI (php artisan reflex:validate-playbooks where available).
Pilot on staging with production-like traffic shape.
Promote with version pinning and changelog entry.
Retire when architecture changes make steps misleading — stale automation is worse than none.

Table-top exercises (quarterly)

Run 60-minute simulations with anonymised timelines:

“Redis primary lost — replicas promote — Laravel sessions misconfigured”
“Deploy succeeded — Horizon workers still on old code — poison jobs”
“Certificate renewed — chain incomplete — mobile clients fail”

Score each exercise on time-to-first-accurate-hypothesis and time-to-safe-mitigation. Improving those metrics pays more than adding another dashboard panel.

Compliance and evidence

Regulated environments often require who changed what, when, and under which approval. Immutable audit logs for automated remediation are not optional there. Design retention aligned with legal hold policies; never log secrets or raw PII in remediation transcripts.

Glossary (canonical definitions on this page)

MTTR (infra class): time from detection to mitigation that restores SLO-class traffic, excluding root-cause analysis completion unless policy defines otherwise.
Dry-run: evaluation path that emits intended actions without mutation; must be distinguishable in logs from live execution.
Blast radius: maximum negative impact if an automated step misfires across hosts, data, or customers.

Economics and sleep

The business case is not “fire your SREs”. It is removing duplicate vendor spend and compressing MTTR for infra-class incidents so on-call load is sustainable as fleet size grows.

Honest limitations

Automation without governance creates new failure classes: flapping remediations, false correlations, and surprise restarts. Reflex biases toward visibility, rollback, and dry-run defaults to mitigate those classes.

Long-form reference appendix (operator-grade)

A. Observability anti-patterns that break closed loops

Dashboard sprawl is the silent killer of self-healing programmes: every engineer builds a personal Grafana folder, alerts differ by naming convention, and no single timeline answers “what changed”. Standardise on a small set of golden signals per service: availability, latency, traffic, errors, and saturation — then add domain-specific signals (queue depth, FPM listen queue, upstream connect time) only where they change decisions.

Alert fatigue trains humans to ignore pages — and it trains automation if you wire remediations directly to noisy alerts. Every alert should declare an owner, a severity, a runbook URL, and an expected human response time. If an alert cannot justify waking someone, downgrade it to a ticket or daily digest.

Metric cardinality explosions (unbounded labels on “route name” in high-cardinality paths) make storage expensive and queries slow. That indirectly breaks healing because operators cannot zoom from fleet view to culprit quickly. Enforce label budgets in instrumentation libraries and reject PRs that add unbounded tags.

B. Designing remediation state machines

Think in states, not scripts: healthy, degraded, mitigating, failed, manual_hold. Transitions should be logged with reasons. When a mitigation succeeds, define how you return to healthy without immediately re-entering the same transition (hysteresis). For example, after recycling PHP-FPM, wait for error rate and queue latency to stabilise before declaring victory — otherwise flapping restarts can thrash opcode caches and worsen cold-start latency.

C. Dependency graphs and blast radius ordering

Order mitigations by dependency direction: if MySQL is unhealthy, restarting PHP first rarely helps — but restarting PHP after DB recovery can clear poisoned persistent connections if pools were exhausted. Encode these orders as comments in playbooks so junior engineers learn topology while executing safe automation.

D. Capacity planning signals that predict healing needs

Track growth rates for: web concurrency at peak, queue arrival rate, 95th percentile DB query time, and disk growth on /var/log. When two signals accelerate together (logs + traffic), you are often weeks away from disk pressure incidents. Healing can buy time; capacity buys quarters.

E. Multi-tenant SaaS specifics

Noisy neighbour problems on shared workers require fair queuing and per-tenant rate limits before infra automation kicks in — otherwise you automate away symptoms while customers still see unfair latency. Separate incident classes: “platform saturation” vs “single tenant abuse” have different ethical and contractual responses.

F. Post-incident learning without blame theatre

Use timelines that include tooling gaps (“we lacked a deploy marker on this service”) alongside human actions. Action items should be owned, dated, and verified in staging. If the same class repeats quarterly, revisit architecture — not only runbooks.

G. When to refuse automation (explicit deny list)

Maintain a written deny list: financial ledger corrections, GDPR erasure workflows, schema migrations that rewrite large tables, kernel upgrades, and certificate authority changes. The deny list should be reviewed quarterly because product maturity can later make some classes safe — with evidence.

H. Testing self-healing safely

Use fault injection in non-production: kill workers, inject latency, fill disks in a controlled partition, and revoke TLS intermediates in a lab CA. Measure whether detection and suggested mitigations match reality. Record false positives and tune thresholds before any production enablement.

I. Communication templates during automated mitigation

Prepare customer-facing language for common mitigations (“we restarted application workers to clear a memory leak class incident — no customer data was modified”). Legal and support should pre-approve templates so engineers are not inventing prose under stress.

J. Reflex-specific integration mental model

reflexd observes where you permit it; the Brain evaluates policy; Pipeline (on eligible tiers) keeps deploy surfaces atomic. The value is not “magic fixes” — it is compressing coordination between those layers with auditability. Your organisation still owns architecture, data residency, and vendor relationships.

Extended operator FAQ

Does self-healing mean servers fix themselves without humans? No — it means well-scoped failure classes get faster, safer mitigations with oversight; humans remain accountable for policy and architecture.
How do we avoid automating a bad diagnosis? Require corroborating signals, dry-runs, and hysteresis; start with suggest-only modes.
What is the minimum telemetry for PHP healing? FPM status, slow log sampling, upstream errors from nginx, and request latency percentiles tied to deploy markers.
Should we auto-restart on every 502? Rarely without upstream classification — otherwise you amplify thundering herds.
How do we test playbooks without production risk? Fault injection in staging with production-like traffic mixes and explicit success criteria.
What documentation do auditors expect? Policy, approval flows, logs without secrets, and retention aligned to legal hold.
How do we prioritise which failure class to automate first? Frequency × customer impact × reversibility; pick high-frequency, reversible wins first.
What role does correlation ID play? It ties user-visible errors to backend spans across nginx, PHP, and workers — essential for deploy regressions.
How do we handle multi-step remediations? Model explicit state transitions; never hide intermediate failure.
What if mitigation makes things worse? Rollback paths and automatic halt on SLO burn rate spikes are mandatory patterns.
How do we prevent two automations fighting? Global locks per host/service class with ownership timeouts.
What is a healthy false positive rate? Near zero for actions that restart services; higher tolerance for ticket-only suggestions.
How do we involve application teams? Shared definitions of SLOs and error budgets; infra-only automation cannot fix code hot paths.
What about Windows workloads? This guide assumes Linux PHP deployments; adapt signals and service managers accordingly.
How do we version playbooks? Same rigour as application semver; changelog entries per material change.
What metrics prove ROI? MTTR for infra-class incidents, repeat incident rate, and on-call hours per deploy.
How do we train new operators? Shadow incidents with annotated timelines; rehearse rollback monthly.
What is the role of feature flags? They narrow blast radius for risky application changes — pair with deploy markers.
How do we handle third-party SaaS dependencies? Heal around them — you cannot restart a vendor — but automate customer comms templates.
When should we page humans during automation? Any data mutation, irreversible network change, or first-seen novel signature.
How do we avoid security automation mistakes? Separate playbooks for hardening vs incident response; peer review mandatory.
What about containers? Signal sources shift to orchestrators; principles remain — policy, blast radius, audit.
How granular should per-tenant automation be? As granular as contracts and fairness requirements demand.
What is “confidence threshold” in practice? Count of successful dry-runs and human approvals before full auto.
How do we retire automation? Explicit deprecation dates tied to architecture changes — stale automation is a liability.

Closing

Self-healing is a discipline, not a slogan. The organisations that benefit treat playbooks like code: reviewed, versioned, and retired when the architecture changes.

Supplement — additional operator notes

This supplement extends the pillar with repeatable review prompts. It is educational, not contractual; verify behaviour against your environment and Reflex tier gates before relying on automation.

1. Deploy correlation

When you evaluate deploy correlation for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

2. Queue saturation

When you evaluate queue saturation for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

3. Disk and inode budgets

When you evaluate disk and inode budgets for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

4. TLS and chain health

When you evaluate tls and chain health for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

5. Secret rotation windows

When you evaluate secret rotation windows for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

6. PHP-FPM pool sizing

When you evaluate php-fpm pool sizing for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

7. Opcache and deploy interactions

When you evaluate opcache and deploy interactions for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

8. MySQL connection storms

When you evaluate mysql connection storms for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

9. Redis memory and eviction

When you evaluate redis memory and eviction for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

10. nginx upstream health

When you evaluate nginx upstream health for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

11. systemd unit limits

When you evaluate systemd unit limits for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

12. Supervisor restart storms

When you evaluate supervisor restart storms for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

13. Log volume and retention

When you evaluate log volume and retention for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

14. Kernel OOM signatures

When you evaluate kernel oom signatures for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

15. cgroup pressure stalls

When you evaluate cgroup pressure stalls for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

16. Canary error budgets

When you evaluate canary error budgets for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

17. SLO burn alerts

When you evaluate slo burn alerts for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

18. Runbook freshness reviews

When you evaluate runbook freshness reviews for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

19. On-call fatigue controls

When you evaluate on-call fatigue controls for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

20. Audit sampling for remediations

When you evaluate audit sampling for remediations for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

21. RBAC for automation actors

When you evaluate rbac for automation actors for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

22. Idempotency of fixes

When you evaluate idempotency of fixes for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

23. Back-pressure design

When you evaluate back-pressure design for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

24. Graceful degradation paths

When you evaluate graceful degradation paths for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.

25. Multi-tenant fairness

When you evaluate multi-tenant fairness for automated response, insist on paired metrics: one symptom and one corroborating dependency signal. Write the rollback in the same ticket as the forward change. If the mitigation touches PHP-FPM, nginx, or the kernel, rehearse it in staging with production-like concurrency — not with synthetic ab alone. Prefer staged worker drains over mass SIGKILL unless you are containing memory corruption class incidents. Document blast radius in plain language for legal and customer comms templates. Revisit thresholds after every major framework upgrade because opcode caches and autoload maps shift latency profiles.