thenewstack.io · Saved

Running More Low-Severity Incidents Is Improving Our Culture

thenewstack.io/running-more-low-severity-incidents-is-improving-our-culture

What Jamie wrote

This is a thought provoking read. It made me think of one of my favorite DevOps sayings, "If it hurts, do it more until it stops." The point being that if releasing software is hard for your team, you need to do it more often not less. You can replace a lot of different actions in that statement, and if running incidents is hard for your team, maybe you should do it more?

We created a new severity type of “triage” with the simplest possible runbook condition: Create a Slack channel. This ensures that if something simply feels off, the engineer who spots the problem has a place to write down stream-of-consciousness or play-by-play notes and see what happens next.

As I think about my own use of incident management in teams we have gradually increased the scope of it to include more "types" or "severities" of incidents. If you have a good process, and it clearly helps you communicate, structure response, etc. there can be benefit to using it.