How to form a healthy on-call rotation?

2-3 engineers in an oncall is unhealthy IMO. The rule should be that the knowledge/context on the team should still survive if 2 people are not available (e.g. one person on vacation, and one person who quits). So having 2 people total in one rotation is definitely not good.

Of course, this depends on the nature of the oncall. Do you have a log of what issues the oncall deals with, and how much time it requires? If it's a low-stakes oncall, e.g. just updating documentation, it may be fine. But for something that directly impacts production, 8-10 people is much better.

Also see: https://www.jointaro.com/question/bzMluuZEOPJPKz9gzLj9/does-being-on-call-during-early-career-teach-you-things-or-allow-you-to-grow-technically/

I have complied a documentation for support run-book log for each corresponding issue/alert, so the the on-call team understand what's the level of severity of impacts to business. As the range of the issue can range from low priority to business critical.

However, The support run book doc is not complete as the ultimate source of truth since the production system support behaves like triage rather than debug system.

Then, the nature of on-call rotation can change from to acknowledge the alerts and follow the steps of documentation that support run book documentation to working with business owners. And, there's few time the issue is caused by another team or 3rd party vendor, and it's an issue that cannot be solved by the on-call person.

I am interested to learn more about what other people view about healthy on-call rotation.

It seems like there are several factors to form a healthy on-call.

• Team Size & number of engineers avaliable
• team maturity
• Well maintained Documentation

I see that there's few online post here:

• https://www.atlassian.com/incident-management/on-call/improving-on-call
• How to Design a Good On-call Process: https://refactoring.fm/p/on-call?utm_source=profile&utm_medium=reader2
• https://levelup.gitconnected.com/an-engineering-managers-guide-to-setting-up-on-call-in-a-startup-cbcb9df6ab74
• https://betterstack.com/community/guides/incident-management/on-call-scheduling/

Yes, 2-3 people is going to be overwhelming. I think you need at least 4 to make it sane (once a month) with a manager serving as a backstop. Will Larson endorses the notion of 8 people (https://lethain.com/sizing-engineering-teams/). I would also take a hard look at the seniority of your team to make sure you have engineers who are independent enough to deal with ambiguous questions and debug issues that cut across multiple domains/systems. Runbooks and documentation -- not to mention shadowing are important here too in making sure there is sufficient training and expertise to not have issues escalate to other engineers who are trying to get project work done.

The rotation of 2-3 engineers is hectic and overwhelming

2-3 engineers is way too few for a full, healthy oncall rotation. At this point, you should just do the "traditional" way where when an issue comes in, it just goes to whoever owns the code behind the issue.

I am surprised that a rotation of 2-3 engineers is hectic though: The amount of surface area owned across just 2-3 people should be low. From this, 1 of 2 things is probably true:

1. Despite not having a lot of code, there's a ton of technical debt and fragile system design already in what you own
2. There's a lot of engineers who are connected to the surface area of the oncall's issues, but they aren't on the oncall (so they're writing code that breaks which they don't need to own)

Is 8-10 engineer on-call rotation a healthy rotation?

Generally yes. I think 7-12 is the sweet spot for oncall size from my experience.

Here's some other resources on oncall, which may be helpful:

• "How can I maintain work-life balance with a hectic oncall?"
• "Does being on-call during early career teach you things or allow you to grow technically?"
• "How do I factor in oncall into my team selection?"

Over the past three months, my manager (Director) has repeatedly told me that we should have at least six full-time equivalent (FTE) engineers to handle on-call support. Unfortunately, he has been unable to deliver on this promise due to push-back from other teams. Currently, my director is still part of the on-call rotation and he told me that he would like to be relieved from on-call duties. I am considering what other actions I should take action to establish a more sustainable and healthy on-call rotation.

My manager told me that his ideal goal is to have on-call rotation for every 3 month, but the math currently does not add up. Any thoughts?

I am considering what other actions I should take action to establish a more sustainable and healthy on-call rotation.

If you aren't able to get the bodies, I just don't think you should have a formal oncall rotation. The ideal scenario here is to have a TPM who's decently good at routing fires and bugs to the proper owner.

In the meantime, focus on improving the system. Oncalls aren't inherently hectic - They become overwhelming when the system quality is poor. If you're getting a lot of issues, I would try to figure out the common root causes and fixing them. Maybe even dedicate an entire sprint towards "Better Engineering" to make the system more reliable and break less.

Hi Team,

• Currently working on revamping on-call but not enough staff
• Still feel like bad on-call is a problem though, so unsure how to move forward
• My new engineering manager is telling me to ignore/heavily deprioritize most bugs, focusing on feature work instead
• Winning over engineering leadership has been hard
• Working closely with my engineering director on this and he told me that he would like the on-call person to be focusing on bug fixes during on-call.

Noted: My Director is also part of the on-call, and he says it's expected to get alerts at odd hours due to the vendor outage, and he would re-route the fire back to the Vendor management over the weekend.

Props to your director for putting themselves on the line and being on the oncall rotation! That's pretty strange, but it's a nice gesture for sure.

At the end of the day, you don't need to fix every bug. It's great to fix most bugs to uphold system quality, but sometime this isn't possible and the org has to prioritize something else. Every org is different: As a software engineer, it's important to be open to every possible outcome.

So if you're clear to ignore 80%+ of bugs, then you don't need a super refined oncall system. You could probably just do the basic router system that I cover in the beginning of my Instagram oncall revamp case study and spend 90% of your time on feature work like your manager says.

Leaving a bunch of bugs open sucks for sure, but learning to effectively ignore many tasks and keep them out of your mental space is a really important skill. This is especially important for senior+ engineers, which I talk about in-depth here: "How does one effectively handle pressure especially when the stakes are high?"