DevOps Doctor: Read-Only AI Troubleshooting for Pipelines and Deployments

Most of DevOps troubleshooting is not thinking; it is fetching. Which job failed, what the runner printed, whether the task definition changed, whether the target group is healthy, what CloudTrail saw at 02:14. An engineer with the right CLIs open can answer each question in a minute, and the minutes add up to the hour that an incident actually costs.

DevOps Doctor is the open-source Codex plugin I built to do the fetching. It is a set of skills and helper scripts that gather evidence from locally authenticated CLIs and repository files, report blockers clearly when a tool or permission is missing, and pair any risky production change with validation and rollback guidance. It never writes to your environment. The repository is public: github.com/mojahid0862/devops-doctor.

Why this matters in production

An AI assistant is only as trustworthy as the evidence it is allowed to see and the actions it is allowed to take. Giving an agent write access to production to save a few minutes is a bad trade. Giving it read access through the operator's own credentials, with a strict habit of showing its evidence and naming what it could not check, turns it into a fast colleague rather than a risk. The read-only boundary is the feature, not a limitation.

Implementation approach

Install it as a Codex plugin marketplace, add the plugin, and start a new thread so the skills load. The included skills cover the AWS deployment doctor, ECS/Fargate doctor, GitLab pipeline doctor, GitLab MR review, Terraform plan review, security drift check, cost optimizer, observability doctor, rollback planner, and a general DevOps doctor and operator. Under them sit Python helpers: a fast MR scorer with changed-file risk buckets and a merge verdict, a failed-pipeline snapshot with root-cause candidates, a bounded AWS stack snapshot, an ECS/Fargate deploy correlation across tasks, target health, ECR, CloudTrail, alarms and logs, and a repository scanner for CI, Docker, IaC and secret-risk patterns.

# Add the repository as a Codex plugin marketplace
codex plugin marketplace add https://github.com/mojahid0862/devops-doctor.git --ref main

# Install the plugin, then start a new Codex thread so the skills load
codex plugin add devops-doctor@devops-doctor

# Authenticate your own tools first; the plugin never carries credentials
aws sts get-caller-identity
glab auth status

Real-world use case

A pipeline fails on the main branch after a routine merge. Instead of opening six tabs, the engineer asks the GitLab pipeline doctor what happened. The helper pulls the failed job evidence, surfaces root-cause candidates, and the skill reports what it found and what it could not reach. If the failure is a deployment, the ECS/Fargate doctor correlates the rollout with task state, target health, the image in ECR, CloudTrail events, alarms and logs, so the engineer sees one timeline instead of five consoles. The rollback planner then pairs the proposed change with the smallest validation command to run before and after. The engineer still presses every button; the plugin just made sure they were pressing the right one.

Common mistakes and operating risks

The obvious mistake is granting the agent broader credentials than the operator has, or any write access at all; the plugin is designed so that temptation never arises. The next is reading its conclusions without its evidence; every skill is built to show what it looked at, and a reader should expect that. Missing tools and permissions must be reported as blockers rather than papered over with guesses, and that is a rule, not a nice-to-have. Finally, a fast score on a merge request is a triage signal, not a substitute for a human review when the change is risky.

When this pattern fits best

DevOps Doctor fits teams on GitLab with AWS workloads, especially ECS/Fargate, who already use Codex and want faster first answers during reviews and incidents without widening the blast radius of automation. It fits less well where the tooling is not available locally or where the organisation forbids any AI access to operational data, even read-only through the operator's own session.

Checklist

• Keep the assistant read-only; it gathers evidence, you run the change.
• Authenticate aws, glab, kubectl and terraform yourself; never store credentials in a plugin.
• Expect evidence with every conclusion, and a clear blocker when something could not be checked.
• Use the fast MR path for triage and the deep root-cause path for failures and incidents.
• Pair every risky production change with a validation command and a rollback plan before rollout.

How to roll this out safely

The safest rollout path is usually narrower than teams expect. Start with one service, one environment, or one clear platform boundary and baseline the metrics that matter before changing everything at once. Document ownership, define rollback or fallback behavior, and review the first few changes with the people who will support the system during real incidents. That approach prevents architecture optimism from outpacing operational reality. Mature patterns spread well because they are tested in small steps first, not because they looked complete in a design document.

What to measure after adoption

Success should be visible in operating outcomes, not only in implementation status. Good patterns reduce surprise, shorten diagnosis time, improve release confidence, or create a more predictable cost and performance profile. If the change only adds process, dashboards, or YAML without improving those outcomes, the design is probably too heavy. Measure the behaviors that matter to responders and service owners, then simplify aggressively anywhere the pattern creates ceremony without making production safer or easier to understand.

What teams usually learn after the first real test

The first serious deployment, spike, or incident almost always reveals something the design discussion missed. Maybe ownership was less clear than expected, maybe the observability path was too thin, or maybe the new process worked but took longer than planned because one dependency was not included in the original mental model. That is normal. Production patterns mature when teams capture that feedback immediately and adjust the defaults before the next rollout. In practice, the best patterns are not the most complicated ones. They are the ones that survive contact with real operations and become easier to use with every review.

Ownership and review cadence

Every useful platform practice needs a review loop. After the first few real uses, revisit the pattern with fresh evidence from deployments, incidents, and operator feedback. Ask what was confusing, what created noise, what saved time, and what controls were worth keeping. The strongest engineering patterns usually become smaller and clearer over time because teams trim the parts that do not change behavior. Review cadence turns a one-time implementation into a dependable operating habit.

That final review step is easy to skip when the initial rollout appears successful, but it is usually where the best long-term improvements are found. Small refinements in defaults, ownership, and observability often create more value than another wave of tooling.

A good rule is to treat the first month after adoption as part of the implementation rather than as an afterthought. Watch how the pattern behaves under normal changes, under stress, and during one real support event. If it remains understandable in all three cases, it is probably strong enough to become a team standard.

If the pattern is difficult to explain to a new engineer after that first month, it still needs refinement. Clarity is one of the most reliable indicators that a production practice is ready to scale across teams.

Documentation should evolve along with the pattern. Keep the shortest possible notes that explain ownership, the expected success signals, the rollback or fallback path, and the dashboards or logs responders should check first. Teams often over-document implementation detail and under-document the operational decisions that matter during a real event. A concise, current operating note is usually more valuable than a long design artifact nobody opens once the initial rollout is complete.

That knowledge-transfer step is especially important when more than one team or on-call rotation will depend on the pattern. A practice is not really finished until another engineer can use it confidently without needing the original author in the room.