AI agent wiped prod in 10 seconds

Incidents & Postmortems

• Cursor running Claude Opus 4.6 deleted PocketOS's production database in ~10 seconds — tasked with a staging credential fix, the agent found an unscoped Railway API token in an unrelated file, assumed it was staging-scoped without verifying, and fired a single authenticated curl delete that wiped both the production volume and its co-located backups.

• Three failure conditions converged: a root-scoped token exposed in the codebase, Railway storing volume backups inside the production volume, and the agent operating within its permitted access while ignoring explicit guardrails. Data was recovered from an earlier Railway snapshot; Railway has since patched the legacy endpoint with delayed-delete logic.

• Anthropic published a three-misstep postmortem on Claude Code degradation via Fortune: reasoning effort silently downgraded March 4, a mid-session reasoning history bug introduced March 26, and a 25-word cap between tool calls added April 16. All three reverted by April 20; usage limits reset for all subscribers April 23.

• The Claude.ai April 28 outage lasted 10 minutes, but the downstream failures — retry storms, silent queue growth, and CI breakage where auth paths shared control paths — persisted longer. Key design implication: retry budgets should be tiered by feature criticality, not a single global policy; fail closed early on low-value paths rather than exhausting capacity.

• A 37-minute Queensland ISP outage on April 24 traced to an STP VLAN mismatch on a core NBN aggregation link: STP received mismatched BPDUs from a downstream device, treated the inconsistency as a loop risk, and auto-blocked the link — dropping all customer traffic. BPDU filtering on the interface restored service immediately; a full NNI upstream audit is now underway.

---

LLM Call Observability

• The OpenAI April 20 outage highlighted how vendor status pages fail LLM-dependent systems; a dev.to postmortem proposes five OTel signals every LLM call should emit: per-model p50/p95/p99 latency, per-region error rate, token throughput (tokens/sec), time-to-first-token (TTFT), and structured-output schema validation rate. Alert thresholds should track a trailing 7-day baseline — hosted model performance drifts and static thresholds break.

• Rate limits are the #1 LLM failure mode in production, per Datadog's State of AI Engineering 2026 (1,000+ companies' telemetry): 2% of LLM spans errored in March 2026, with rate limits accounting for nearly one-third (8.4M errors). 69% of input tokens go to system prompts; only 28% of LLM spans use prompt caching despite cost and latency benefits; average token usage per request has more than doubled year-over-year.

• Jaeger v2 rebuilt on the OTel Collector framework and is adding MCP and Agent Client Protocol layers that translate natural-language queries ("500-level errors in the payment service with latency > 2s") into deterministic trace queries — the AI layer handles protocol translation only, not open-ended generation, to minimize hallucinations. Support for OTel GenAI semantic conventions (RAG pipelines, embedding latency, tool call spans) is in active development.

---

Observability Tooling

• Amazon CloudWatch now accepts metrics via OTLP (public preview), completing AWS's observability trifecta — traces and logs were already supported via OTLP endpoints. The high-cardinality store supports 150 labels per metric; teams can query with PromQL and build Managed Grafana dashboards, reusing existing Prometheus queries without migration.

• The OTel eBPF instrumentation project (formerly Grafana Beyla, donated to CNCF mid-2025) captures DNS traces at the kernel level — associating DNS lookup duration with specific workloads before SDK instrumentation runs, requiring no application restarts or code changes. Closes a production blind spot: application traces start after DNS resolution, so misconfigured DNS clients surface as unexplained application latency rather than as DNS failures.

---

SRE Practices

• DigitalOcean replaced incident-counting availability with SLI-based measurement: the old metric (100% minus declared incident minutes) masked partial degradations and caused availability to oscillate 99.5–99.9% without reflecting customer reality. The new framework separates control-plane (API request success rate) from data-plane SLIs (resource-minute or request-based), weighting by traffic volume, with a four-state error budget policy from Green (0–60%: operate normally) to Red (>100%: all-hands, no rollouts).

• incident.io draws a sharp line between AI automation in postmortems and genuine synthesis: AI handles compression well — assembling timelines from Slack/alerts/PRs, generating structured drafts, surfacing past patterns — but cannot determine why contributing factors aligned or name organizational issues a technical fix won't address. The risk: a postmortem that sounds exactly right but was produced without anyone doing the real thinking — formatted docs that change nothing.

• SRE Weekly #514 flags a critical divergence in incident response instincts: behaviors that work in outages (escalate fast, share status widely, publish updates) actively harm security incident response (restrict access, limit disclosure, preserve forensic state). Same issue: microservices sharing a database or coordinating on every request are a distributed monolith — "with extra latency and a much harder debugging story."

• PagerDuty's spring 2026 release positions the SRE Agent for autonomous detect-triage-diagnose before human paging, with agent-to-agent interoperability announced for AWS DevOps Agent and Azure AI SRE via enhanced MCP — forming a multi-agent response fabric targeting early access H2 2026 (AWS DevOps Agent GA previously covered April 23). PagerDuty is also deprecating its Postmortems feature in favor of Post-Incident Reviews by October 31, 2026.