A Foundational Guide to Site Reliability Engineering, Service Level Objectives, and Service Level Indicators

Transforming Devops from Build to Run

DevOps is a belief system, set of principles, and mindset that informs how software development and operations teams organize to deliver customer value. Close collaboration between development and operations teams, a belief in shared responsibility for outcomes, and commitment to quality and continuous improvement are all central DevOps themes. When that belief system and mindset are adopted across an organization or even a subset of teams, it begins to drive a larger cultural shift, and consequently, DevOps is often described as a cultural movement.

The challenge with DevOps, though, is that while it embraces a distinct way of thinking, and even has job titles, processes, practices, and underlying tools associated with it, it is not particularly prescriptive. DevOps is focused on how work gets done rather than prioritizing what work needs to be done. So as DevOps has continued to evolve and gain momentum across the enterprise, it is driving both the need and opportunity for organizations to become more focused on what work, exactly, their development and operations teams should be focused on to drive that customer value. Traditionally, that decision has been the domain of product development or business teams, with an emphasis on developing new features and enhancements.

While it’s true that new features and enhancements drive all-important customer value, a DevOps driven, breakneck pace of new releases can also create performance and reliability issues that negatively impact user experience, detracting from that value. Speedy delivery of new features can also put operations teams in a jam: they’re charged with maintaining and/or improving system performance, but instead are forced to spend time reacting to issues. Organizations need to balance innovation with site reliability: they need to make informed decisions about whether they should be focusing their efforts on building and releasing new features or running the system because both are essential to customer happiness.

Site reliability engineering (SRE), a practice that entails applying software engineering thinking to the operational aspects of site reliability, solves this problem. SRE brings both business and operations teams together around objective data about system performance so that they can make collective decisions about whether to prioritize building and releasing new features or proactively ensuring the system is running optimally. While the objective data part of that equation isn’t new—KPIs and metrics have been around for a long time—this collaboration between business and technology teams is.

Ultimately, SRE is a methodology for putting DevOps  into practice, or as Niall Murphy, author of Site Reliability Engineering: How Google Runs Production Systems describes it: “SRE is an opinionated  implementation of the DevOps philosophy.” 

Back to Top

Preventing Issues by Shifting Left/Shifting Right

It is important to understand that—while an ounce of prevention is always better than a pound of cure (e.g., it’s better to prevent issues than to resolve them)—issues, even critical ones, will undoubtedly arise because no system is inherently perfect. Even though monitoring SLIs helps organizations identify issues before they happen, they must invest effort into systems and processes that both minimize issues and simplify resolving them efficiently. Using an SRE approach, the reliability of a system should include both building reliability directly into the applications themselves early on, and evaluating backend metrics once the software is released. The concepts of shift left and shift right are critical to achieving this flexibility.

Shift left refers to the software development practice of focusing on quality in the initial development process in order to prevent issues, streamline testing, and ensure a better customer experience when the software is deployed. It is exactly how it sounds, in that it shifts the testing process to the left—or the beginning—of the development process. The overarching goal of shift left is to mitigate risk and reduce defects that might impact customer satisfaction by being proactive early on. 

Using the mantra, “Test early and often,” shift left embraces a culture of shared responsibility—it prioritizes building the development pipeline so that the software is as close to “perfect” as possible in the initial release. This approach moves away from focusing exclusively on speed, passing the code off to the testing team, and letting them sort out the issues. Shift left also prioritizes continuous integration and continuous delivery (CI/CD). In CI/CD, building, testing and deployment are automated, and manual effort is minimized so testing can be done quickly, early, and often.

The goal of shift left’s lesser-known counterpart, shift right, enables you to test in production, and prepare for the undefined, unknown, and unexpected. The term refers to the practice of doing more thorough testing after the software is released and continuing to test post-release. The reason for this is that, while staging environments can approximate production environments, they can never truly duplicate them, so post-release testing is necessary to truly understand what users are experiencing, to identify problems that might only exist in that real-life environment, and to get a better understanding of system dependencies.

A benefit of shifting right is that it creates a continuous feedback loop from actual user experience directly back into the development process. Teams are constantly shifting—meaning that they’re going through agile transformations and transitions all the time. Likewise, their feedback will shift and evolve, thus creating an ongoing channel for real-world feedback is imperative to the development process, and ultimately, the success of the software being deployed.

To boil it all down into one, cohesive sentence: Shift left emphasizes prevention and risk reduction as the key to an eventual positive customer experience, whereas shift right emphasizes the user experience as critical information in the feedback loop, leading to ongoing improvements and therefore leading to satisfied customers.

Back to Top

The Site Reliability Glossary

DevOps – DevOps is a cultural movement and set of principles and practices that informs how development and operations teams organize and collaborate in order to build, test, and deploy software faster and more reliably—objectives that drive customer value and result in a better user experience.

To achieve those objectives, DevOps practices incorporate automation and focus on continuous improvement, continuous integration (CI), and continuous delivery (CD).

Error Budget – Error budgets rely on the notion that no service is perfect, and users will tolerate some degree of error. An error budget represents the amount of error an aspect of a given service, like latency or availability, can experience before the user experience is affected to a degree that customers become unhappy. For example, if an organization has an SLA guaranteeing 99.99% availability, the error budget is .01%. Error budgets are used to help organizations determine whether they can continue to focus their efforts on new features and enhancements, or if they need to direct their efforts to work related to site reliability to remain within their error budget.

Service Level Agreement (SLA) – Service Level Agreements are commitments that organizations make to their customers around system performance, and are often tied to key metrics, such as uptime or how quickly issues will be resolved. SLAs are often written into contracts and are sometimes even financially backed.

Service Level Indicator (SLI) – If SLAs are commitments to customers for system performance, and SLOs are internal goals for system performance, then Service Level Indicators (SLIs) are actual measurements for how the system is performing. Organizations can compare SLIs— actual measurements—against SLOs—internal objectives—to determine if they need to focus work on improving performance and reliability or if they can continue to focus on new features and enhancements.

Service Level Objective (SLO) – While SLAs reflect a commitment to customers around a system’s performance, Service Level Objectives (SLOs), are internal thresholds that organizations set for system performance to make sure that they are able to meet parameters outlined in the SLA. In an organization that practices DevOps, SLOs serve as goals or commitments that development and operations agree to around system reliability. SLOs should be focused on the user experience and reflect the minimum performance necessary for a positive user experience.

Service Level Engineering (SRE) – Site Reliability Engineering is an outgrowth of DevOps that applies software engineering thinking to the operational aspects of site reliability. In practice, site reliability engineers collect data and use mathematical formulas to guide decisions about what to work on, in order to create a balance between releasing new features and maintaining and enhancing site reliability.

Shift Left – Shift left is a DevOps concept that refers to the software development practice of focusing on testing early in the development process in order to prevent issues and ensure a better customer experience when the software is initially deployed. Shift left also prioritizes continuous integration and continuous delivery (CI/CD). In CI/CD, building, testing, and deployment are automated so testing can be done quickly, early, and often

Shift Right – Shift right refers to the practice of testing thoroughly in the later stages (i.e post-production phase) of the development process. The goal of shift right is to focus on user experience and production scenarios as important metrics. Any of the issues found in this post-release testing obviously impact customer satisfaction, and serve to inform the developers on what types of changes need to be made to the software.

Back to Top