Evidence-Backed Root-Cause Analysis for Failed ECS Deployments

A failed deployment is loud but unexplained. ECS reports a circuit breaker, the load balancer reports unhealthy targets, and the real reason sits in the application logs. The on-call engineer spends the first twenty minutes stitching those three views together before anyone decides what to do, and the rollback that follows is done by hand, under pressure, with no record of why the decision was right.

DeploySentinel AI is my hackathon build for that gap. It captures the failed deployment's own evidence, asks Amazon Bedrock for a structured root-cause analysis that is validated against that evidence, shows an operator one page they can act on, and runs recovery to the known-good revision only after a human approves it. The repository is public: github.com/mojahid0862/deploysentinel-ai.

Why this matters in production

Incident minutes are expensive in two currencies: user impact and operator attention. An analysis that arrives with its evidence attached shortens the first currency. A recovery that is approved rather than automatic protects the second, because nobody has to wonder what the machine did while they were reading. Keeping the two separate, explain first and act only on approval, is what makes AI useful in an incident instead of frightening.

Implementation approach

The stack is deliberately ordinary AWS. A demo service runs on ECS/Fargate behind an Application Load Balancer. Evidence collection reads ECS deployment and task state, ALB target health, CloudWatch application logs and the diff between task-definition revisions, then packages them into an immutable catalogue. Amazon Bedrock (Nova Pro, through the Converse API) returns a structured analysis against that catalogue. API Gateway and Lambda serve the incident, analysis and timeline from DynamoDB to a static dashboard on Amplify Hosting. Approval from the dashboard starts an AWS Step Functions execution that drives a least-privilege recovery Lambda and then verifies ECS, the load balancer and the application before anything is marked resolved. All 57 resources are managed by Terraform.

# The approval the operator sends: an empty body, no AWS identifiers
POST /api/incidents/INC-001/approve-recovery   {}

# What Step Functions does with it
1. start the recovery state machine for INC-001
2. recovery Lambda: update the ECS service to the known-good task definition
3. verify: ECS running count, ALB target health, GET /health returns 200
4. DynamoDB: incident -> resolved, timeline appended

# Guardrails
recovery execution disabled outside the demo window
temporary credentials only; no IAM users or long-lived keys

Real-world use case

The demo incident, INC-001, was a real failure. Revision 2 of the demo service shipped configuration that pointed it at a dependency it could not reach. Tasks reached RUNNING, the health endpoint returned HTTP 500 and the load balancer marked every target unhealthy. Evidence capture recorded the deployment events, the target health, the log lines and the configuration diff. Bedrock's analysis named the configuration change as the cause, cited the exact signals, and recommended returning to revision 1. A human approved, Step Functions rolled the service back, the verification steps passed and the incident was resolved with its timeline intact. Afterwards, Terraform plan reported no changes, so the recovered infrastructure and the declared infrastructure agreed.

Common mistakes and operating risks

The first mistake is handing the model raw log dumps and hoping for insight; without a catalogue to validate against, a confident wrong answer looks identical to a right one. The second is letting remediation run on its own, which turns an analysis error into an outage. Broad IAM on the recovery path is the third: the recovery Lambda should be able to do exactly one thing to exactly one service. Marking an incident resolved before ECS, the load balancer and the application have all been checked is the fourth. The fifth is quieter: fixing the live service while leaving Terraform describing the broken state, which guarantees the next apply redeploys the failure.

When this pattern fits best

This pattern fits container services behind a load balancer with real health checks, teams that want AI in the loop but a human on the trigger, and any environment where a rollback decision needs an audit trail. It fits less well where deployments cannot be expressed as revisions to return to, or where nobody is available to approve.

Checklist

• Capture deployment, target-health, log and configuration-diff evidence before asking any model anything.
• Validate every cited signal against the captured catalogue; reject analyses that cite what was never observed.
• Present confidence as an AI assessment, never as a measurement.
• Require a human approval with an empty body; keep cluster, service and revision fixed server-side.
• Verify ECS, load balancer and application independently before resolving, and keep Terraform in step with the recovered state.

How to roll this out safely

The safest rollout path is usually narrower than teams expect. Start with one service, one environment, or one clear platform boundary and baseline the metrics that matter before changing everything at once. Document ownership, define rollback or fallback behavior, and review the first few changes with the people who will support the system during real incidents. That approach prevents architecture optimism from outpacing operational reality. Mature patterns spread well because they are tested in small steps first, not because they looked complete in a design document.

What to measure after adoption

Success should be visible in operating outcomes, not only in implementation status. Good patterns reduce surprise, shorten diagnosis time, improve release confidence, or create a more predictable cost and performance profile. If the change only adds process, dashboards, or YAML without improving those outcomes, the design is probably too heavy. Measure the behaviors that matter to responders and service owners, then simplify aggressively anywhere the pattern creates ceremony without making production safer or easier to understand.

What teams usually learn after the first real test

The first serious deployment, spike, or incident almost always reveals something the design discussion missed. Maybe ownership was less clear than expected, maybe the observability path was too thin, or maybe the new process worked but took longer than planned because one dependency was not included in the original mental model. That is normal. Production patterns mature when teams capture that feedback immediately and adjust the defaults before the next rollout. In practice, the best patterns are not the most complicated ones. They are the ones that survive contact with real operations and become easier to use with every review.

Ownership and review cadence

Every useful platform practice needs a review loop. After the first few real uses, revisit the pattern with fresh evidence from deployments, incidents, and operator feedback. Ask what was confusing, what created noise, what saved time, and what controls were worth keeping. The strongest engineering patterns usually become smaller and clearer over time because teams trim the parts that do not change behavior. Review cadence turns a one-time implementation into a dependable operating habit.

That final review step is easy to skip when the initial rollout appears successful, but it is usually where the best long-term improvements are found. Small refinements in defaults, ownership, and observability often create more value than another wave of tooling.

A good rule is to treat the first month after adoption as part of the implementation rather than as an afterthought. Watch how the pattern behaves under normal changes, under stress, and during one real support event. If it remains understandable in all three cases, it is probably strong enough to become a team standard.

If the pattern is difficult to explain to a new engineer after that first month, it still needs refinement. Clarity is one of the most reliable indicators that a production practice is ready to scale across teams.

Documentation should evolve along with the pattern. Keep the shortest possible notes that explain ownership, the expected success signals, the rollback or fallback path, and the dashboards or logs responders should check first. Teams often over-document implementation detail and under-document the operational decisions that matter during a real event. A concise, current operating note is usually more valuable than a long design artifact nobody opens once the initial rollout is complete.

That knowledge-transfer step is especially important when more than one team or on-call rotation will depend on the pattern. A practice is not really finished until another engineer can use it confidently without needing the original author in the room.