Three sources of data about what is happening answer different questions, and substituting one for another is pointless. Usage analytics answers who works with the tool and how much: it is built around people and volume rather than around security events. Audit logs answer which administrative and sign-in events occurred: a role change, a grant of access, a sign-in from a new device. Event handlers answer which actions passed through the agent's lifecycle: which tool was called, what was allowed, what was rejected. The full picture comes only from all three, and the absence of any one turns incident analysis into guesswork: without analytics the scale is invisible, without the audit log it is unclear who changed the rules, without agent events it is unclear what the agent actually did.
Rollout in an organization is sensibly run in phases rather than as a single switch-on. It helps to see those phases with a scope and an exit criterion once. Below is that map: inventory, a read-only pilot, scoped write access, shared content, the cloud and automations, controlled mutation. The order in it is not arbitrary: every next phase widens one thing only - either write rights, or the circle of people, or the number of places where code runs. Widening two dimensions at once is dangerous because at the first problem it is impossible to tell which of them caused it. Every phase has its own exit criterion - and that matters more than the sequence itself: without a criterion, phases turn into a formality passed by the calendar.
| Phase | Scope | Exit criterion |
|---|---|---|
| Inventory | Identity, repositories, tools, data classes | An agreed threat model and named owners |
| Read-only pilot | Non-sensitive repositories, modes without edits | A quality baseline and a support procedure |
| Scoped write | A review mode or an allowlist, the sandbox | Positive and negative tests passed |
| Shared content | Curated rules, skills, plugins | Versioning, review, rollback |
| Cloud and automations | A prepared environment, advisory output | Budget, audit, incident drills |
| Controlled mutation | Narrow tools and branches | Sustained metrics and an exception process |
The exit criterion of the second phase is especially useful - a quality baseline and a ready support procedure. Until the team knows what to expect from the tool and where to go with a problem, any widening of authority is premature: complaints will arrive in private messages rather than in a shared queue, and nobody will see how many there are. And the third phase requires both positive and negative tests: you have to prove not only that the allowed works but that the forbidden does not pass. The negative test costs more than the positive one and is therefore skipped more often - although it is precisely what separates a configured boundary from an assumed one.
How the three sources work together is visible in an ordinary investigation. A spike is noticed: over a week an unusual number of agent-authored edits appeared in the repository. Analytics shows that the volume grew for two people rather than for the whole team - so it is not a new version of the tool. The audit log shows that the day before, those two had their role widened. Agent events show which tools started being called and how many calls went through without approval. None of the three sources on its own yields that chain: the first sees volume without a cause, the second a cause without consequences, the third consequences without context.
The sequence of actions during an incident deserves to be described in advance rather than at the moment it is needed. First automation is stopped, affected keys are revoked and the integration or tool is disabled. The order inside that step is not accidental either: while the automation runs it keeps producing new events and smearing the picture, and a key that has not been revoked lets the same thing be repeated from another machine. Then evidence is preserved: agent and run identifiers, raw event logs, repository refs and artifacts. They must be preserved before any fixes - some data lives for a limited time and some is overwritten by the next run. Next it is determined which data and actions went beyond the expected scope.
After that recovery begins: rolling back or containing the consequences through version control and external systems. The difference between a rollback and containment is practical: an edit in the repository can be reverted, while a message sent into an external system or a task created there cannot, and has to be marked and compensated by hand. And only then comes the fix. The order of fixes is also set: first the deterministic control that should have stopped it is repaired, and only afterwards the steering instructions are amended. The reverse order leaves the hole open and the text updated - and creates the worst of states, in which the team is convinced the problem is solved.
The cycle ends with a negative regression test: reproduce the same attempt and make sure it is now blocked and recorded in the log. There are exactly two conditions here, and the second is forgotten more often: a block without a record leaves you without data at the next attempt. Without that step an incident is closed on faith in the fix. And only after it does it make sense to return the rollout to its previous state - gradually, starting from the phase where everything broke rather than from the one you had reached.
The phased approach has a price, and it is worth naming honestly: it is slow. A full pass takes months, and in a small team where every change is read by eye some phases can genuinely be compressed - compressed while keeping the exit criterion, not skipped. The sign by which real work reveals that a phase was passed too early is usually a single one: the share of approvals people click without reading grows. The second sign is subtler - agent runs exist while the corresponding events are absent from the logs. That means part of the work goes past observability, and the next investigation will be incomplete no matter how much data you collect afterwards.
The engineering conclusion is simple: observability and a response procedure are part of adoption rather than something taken up after the first mishap. Three data sources, phases with criteria and a written response order cost a day of work and save weeks at the moment when something goes wrong. Written is the key word: a procedure that exists only in the on-call engineer's memory does not work at three in the morning.
The typical failures are predictable. Substituting usage analytics for audit logs. Passing phases without exit criteria. Starting incident analysis with a fix without preserving evidence. Returning the rollout straight to its previous level, bypassing the phase that broke. And repairing wording instead of the deterministic control.
The incident sequence
1. stop the automation, revoke keys, disable the integration
2. preserve evidence: agent and run identifiers, logs, refs, artifacts
3. determine which data and actions went beyond the expected scope
4. roll back or contain the consequences through the repository and external systems
5. fix the deterministic control, then the steering instructions
6. run a negative regression test and restore the rollout gradually