How Slack streamlined their deployment process

This is a recap from an interview with Sean Mcllroy from Slack’s Release Engineering team for the Engineering Enablement podcast. Listen to the full episode here.

Challenge

At Slack, the manual deployment process for their massive monolithic codebase presented persistent challenges. Deploy Commanders, tasked with managing deployments, faced hundreds of graphs to monitor during each release. This level of complexity created stress, and the manual nature of deployment led to confusion, delays, and occasional errors, making an already high-pressure task even more difficult. Given the scale of the monolith, a deployment error could have widespread impact across multiple services, heightening the need for precision.

With hundreds of developers contributing to the monolithic codebase daily, the system was vulnerable to human error, which threatened both speed and reliability. Even a minor deployment error could impact numerous services simultaneously, leading to significant downtime and service interruptions across the entire platform. This heightened the overall risk and made it clear that a solution was needed to reduce manual workload, minimize errors, and streamline the complexity of deploying such a large and interconnected codebase.

Hypothesis

In the end, Slack’s Release Engineering team ended up fully automating the deployment process, but the idea of automating the process wasn’t on their radar when they first started this project. Instead, their original goal was to provide tools that would ease the manual workload of Deploy Commanders by simplifying deployment monitoring and troubleshooting. They hypothesized that by introducing a tool to improve issue detection, they could reduce stress and errors without replacing the human role entirely.

Metrics

To track the success of the new deployment system, the team focused on:

• Cycle time: Aiming to reduce how long developers spent managing deployments, streamlining workflows to improve productivity and minimize delays.
• Adoption rate: Monitoring how quickly developers embraced the automated system, ensuring ease of use and accessibility.
• Z-Scores: Using Z-scores to track anomaly detection, ensuring that potential issues were identified before they escalated into major problems.
• False positives reduction: Reducing unnecessary alerts compared to the manual threshold system, leading to smoother, less stressful deployments.

Solution

During the development process, the Release Engineering team initially built tools to assist Deploy Commanders, such as a manual process where developers would take shifts monitoring deployments. They implemented a dashboard filled with hundreds of graphs for engineers to watch during the process, and they also relied on Slack channels where developers could communicate about potential issues. These tools, however, placed heavy demands on human oversight, and the team quickly realized they needed more robust automation.

By researching solutions, they identified Z-scores as a highly effective tool for anomaly detection. Z-scores, a statistical measure of deviation, allowed the team to spot unusual behavior during deployments in real time, giving them early insights into potential issues. The use of Z-scores shifted the focus from providing support tools to automating key parts of the deployment process. This automated solution integrated directly into Slack, enabling the bot to monitor deployments, alert teams to any anomalies, and even initiate automatic rollbacks. This process freed developers from manual monitoring and allowed them to focus on building rather than troubleshooting. The system’s ability to detect even subtle deviations in real-time behavior led to more reliable and faster responses to issues, improving the overall deployment workflow.

As the system evolved, it began handling even more critical deployment tasks, such as automatically detecting problems without human intervention and rolling back problematic deployments when needed. This automation improved deployment speed and reliability, reduced the manual workload on developers, and increased their trust in the deployment process. The transition from manual oversight to full automation significantly enhanced Slack’s deployment efficiency and developer productivity.

Rollout

Slack’s automated deployment solution was rolled out in phases, starting with a small-scale internal test by the Release Engineering team. They worked alongside Deploy Commanders to gather feedback, ensuring the system could handle the real-world complexities of Slack’s deployment process. As the team fine-tuned the system, they gradually shifted to a fully automated process. The introduction of a bot, which utilized Z-scores for anomaly detection, was a key component in the automation.

As confidence grew, the bot’s role expanded beyond monitoring anomalies. It began managing critical tasks like executing automatic rollbacks, which significantly reduced manual intervention. Continuous improvements were made, including the implementation of dynamic thresholds that adjusted based on real-time data, refining the system’s ability to accurately detect issues and reducing false alarms. The enhanced alerting systems integrated directly with Slack channels, ensuring that the right teams received timely, relevant updates. This iterative rollout allowed Slack to gradually move from a labor-intensive, manual deployment process to a streamlined, efficient, fully automated one, ensuring better reliability and developer productivity.

Outcomes

Since the automated deployment solution was rolled out iteratively, the Slack Release Engineering team has observed several key improvements:

• Reduced manual workload: Developers no longer needed to monitor complex graphs for every deployment. The bot handled these tasks, allowing developers to focus on building and improving Slack’s core product.
• Earlier detection of issues: Z-scores provided early warnings of potential problems, enabling quicker responses and preventing incidents from escalating.
• Automated rollbacks: When the bot detected serious issues, it automatically triggered rollbacks, ensuring the integrity of the deployment without requiring manual intervention.
• Enhanced trust in the system: As the team and developers grew to trust the system’s reliability, deployment processes became faster, smoother, and more frequent, leading to better overall productivity and fewer delays.

Listen to Slack’s full story here → How Slack fully automates deploys and anomaly detection with Z-scores