Most operations teams do not think of themselves as sick. They just live with a quiet set of symptoms that have become so normal nobody calls them out: alerts nobody reads, decisions nobody can find, fixes nobody can trace. Then an incident happens, someone asks “who approved this?”, and the room goes silent. This piece works like a proper diagnostic — symptoms first, root cause second, then a treatment plan you can implement in stages. The treatment has a name, ChatOps, and a reference worth keeping open: the Ops Atlas page on ChatOps, which documents the pattern this diagnosis draws on.
The Symptoms: Four Signs Ops Communication Is Failing
- Nobody reads the alerts. Four hundred messages a week across a twelve-person team, ten seconds of attention each — around thirteen hours of collective scanning for content most people scroll past. The channel exists, so everyone believes the problem is handled.
- Decisions happen in direct messages. The work hides in private threads. When the person in the DM is on leave, the knowledge goes with them. Emojis get tapped as pseudo-approvals with no way to prove the reactor read anything.
- Action and awareness live apart. The alert appears in the chat tool; the restart happens in a terminal somewhere else; the explanation ends up in a third place, if it is written down at all.
- Audit means “search the chat”. Rooms get deleted, retention prunes history, and a channel search was never designed to answer evidentiary questions. The audit trail is a memory, not a record.
The Root Cause, and Why ChatOps Belongs in the Treatment Plan
Strip the symptoms down and one underlying condition remains: operational work is split across surfaces that do not talk, so visibility, permission and evidence all fail together. The remedy is to run the work through one shared room: a bot announces what happened, cleared humans type commands the bot is pre-authorised to run, and the outcome lands back in the thread beside the discussion that triggered it. GitHub open-sourced Hubot in 2011 to do exactly this, Jesse Newland named the practice in his February 2013 talk, and by 2015-16 it had documented form: deploy queues with guards and locks, Atlassian’s people-tools-process-automation framing, O’Reilly publishing Jason Hand’s 2016 report. This is not an experimental drug; it is established practice with fifteen years of case history.
The condition has two known complications, and any honest plan treats both. First, bot and command security — a command typed into chat is effectively a remote control pointed at production. Second, signal-to-noise: fixing visibility without fixing noise just moves the flood into one room.
Treatment Stage 1: Put Three Surfaces in One Room
Consolidate the traffic. Notifications announce events in the open. Commands let authorised people act from inside the same thread. Conversation stays attached to the action it discusses. Three rules keep the consolidation honest. Group related alerts so one underlying event produces one message — one published sample counted just 23% of notices as worth acting on, while the stated goal sat at half, which tells you how much content fails the “would anyone act on this?” test. Keep private messages for private matters; anything touching production belongs in the shared room. And grade every notification against a simple question: would a competent operator change behaviour after reading it?
Treatment Stage 2: Install a Gatekeeper, Not a Pass-Through
The architectural mistake behind most bot incidents is a straight wire from chat into systems. Durable setups split the job three ways instead. One component — call it the adapter — handles whatever dialect the chat product speaks. A second, the gateway, makes every decision before anything executes. The rest, executors, actually do the work. Inside that decision layer a fixed sequence plays out on each request: authenticate the request itself, work out which human stands behind it, decide whether that human may proceed, collect a second signature where the action demands one, then file the evidence somewhere durable. Each platform expresses “authenticate the request” in its own tongue. Slack’s messages carry an HMAC-SHA256 signature, stay replayable for five minutes, and must be acknowledged inside three seconds. On Teams the proof arrives as a signed JWT. Mattermost hands out a static token that never covers the body of the request, which means the contents could be swapped without tripping any lock. None of this belongs in a hurry — write it down, test it, then wire the first command. Along the way, two house rules: a request that fails authorisation gets refused and the refusal recorded with its reason, because that log is how you demonstrate the gate actually stops people; and channels get addressed by their internal ID, since names have an annoying habit of changing mid-incident.
Governance then runs to a calendar rather than a mood. Policy lives as code. The channel naming scheme is written and defended. Refusals get read through weekly, and the whole permission scope is re-audited once a quarter, when drift is still cheap to fix.
Treatment Stage 3: Grade the Permissions Like a Medical Ladder
Not every action carries the same risk, so not every action carries the same controls:
- Information only — post statuses, announce events. Grant freely.
- Read access — view queues and states. Grant per role.
- Reversible changes — restarts, re-sends, resubmissions. Grant per role and channel, logged every time.
- Consequential changes — destructive or high-impact moves. Two distinct people must agree, every time, with a documented break-glass route for genuine emergencies.
That top grade deserves its own paragraph, because it is where teams wobble. Two different people sign before anything moves — auditors call the control dual authorisation and file it under NIST AC-3(2) — and a documented bypass exists for genuine nights of fire, reviewed after every use. Chat adds one trap here: a thumbs-up is not consent, because no system on earth can verify that the person reacting actually read what they approved. Sign-offs therefore arrive as explicit commands issued by a separate, eligible account, never as reactions. AI assistants take the same medicine with an extra clause — they may suggest, a person decides, and the prompts plus tool calls land in the log. PromptArmor’s August 2024 prompt-injection work against Slack AI settled any remaining argument about whether that clause is optional; OWASP’s LLM01 and LLM06 catalogue the failure classes for posterity.
Treatment Stage 4: Move the Logbook Outside the Room
Every command, approval, refusal and override flows to an append-only store beyond the chat platform — NIST AU-3 describes the schema auditors expect. Bot identity deserves the same scepticism: tooling has shipped with defaults that execute chat-triggered work under a single shared account (StackStorm’s chatops_bot being the classic), which converts the whole bot into a confused deputy — one identity, blamed for everyone’s actions, responsible for none. Map actions back to people. Rotate webhook credentials through a secret manager rather than pasting them where channels can see, because those URLs are keys, and keys get copied the moment they sit in plain sight.
How You Know the Treatment Is Working
Recovery shows up in numbers, not vibes. Published figures from disciplined adopters: audited change coverage of 93.9%, command success rate of 98.1%. Track approval latency to make sure the two-person rule protects rather than paralyses. Watch notification load per shift trend down as grouping improves. Ignore raw command counts and message totals — the vanity numbers that rise while nothing improves. For teams working through the wider automation stack this pattern plugs into, the cloud-and-DevOps groundwork in accelerating business velocity with cloud and DevOps practices sets the foundation, the engineering standards behind such work are outlined in mastering next-generation cloud and automation engineering standards, and the prevention mindset pairs well with maintenance mistakes aircraft owners should avoid — both fields prove the same point: catching problems early is cheaper than explaining them later.
The prognosis is good because the treatment is behavioural as much as technical. One shared room, a gatekeeper in front of every action, permissions graded by risk, and a logbook nobody can delete. Symptom by symptom, that plan reverses the quiet failure most teams had simply stopped noticing.
