Prove the system can detect, limit, and recover

Start with what the user needs: successful checkout rate, p95 response time, accepted-order rate, or oldest message age. This is the service-level indicator. Component signals such as CPU, healthy targets, errors, and database connections help explain it.

Metrics summarise numbers over time. Logs preserve events. Traces connect a sampled request across services and show where time or errors appear. Tracing can locate a failing payment call; it cannot create a healthy payment service.

Amazon CloudWatch supplies monitoring capabilities. AWS CloudTrail records supported API activity. AWS Config records and evaluates resource configuration. Use each for its actual question, and give every alarm an owner and action.

Put deployment and configuration changes on the same timeline as the business signal. A failure beginning immediately after a rollout is useful evidence, but rollback still needs a measured result. Correlation narrows the search; it does not replace verification.

Inventory service quotas in every active and standby account and Region. Check instances, network interfaces, IP addresses, targets, function concurrency, API throughput, database connections, snapshots, keys, and external dependencies.

A quota is a ceiling, not a reservation of physical capacity. Raise adjustable quotas early and load-test the recovery scale. An Auto Scaling maximum or infrastructure template cannot override a lower quota.

Use infrastructure as code and immutable artifacts to recreate a known version instead of repairing servers during an incident. Preserve templates, images, code, configuration, certificates, and secrets. Infrastructure code does not replace business-data backups.

Recovery dependencies need the same preparation. A standby can have enough Lambda concurrency yet fail because its subnets lack IP addresses or a third-party payment account has a lower limit. Test the smallest external ceiling as part of full-load readiness.

A game day starts with a measurable hypothesis: after losing AZ-a at peak, checkout remains above a stated rate and an alarm fires within two minutes. Inject the controlled failure and record detection, traffic, capacity, data loss, recovery time, and failback.

Quota readiness allows recovery resources to be created. Admission control decides how much work enters a fragile dependency. API Gateway throttling can reject excess synchronous requests with observable 429 responses; a queue or concurrency cap can pace work allowed to finish later.

Run the test under realistic load. A trace can identify the payment hop, but only failover or repair restores it. A successful backup job does not prove restore time. Keep identity, encryption, and audit controls active during the exercise, then fix findings and repeat.

Use authenticated client classes when allocating protected capacity. A caller-supplied priority label is not a trustworthy admission boundary. Measure accepted, rejected, queued, and completed work so throttling protects the dependency without silently breaking the business objective.

Repeat the exercise after each repair. One passing run is evidence for that tested version, load, dependency set, and failure—not a permanent guarantee.