SLA, SLO, SLI

In distributed systems engineering, building a "100% reliable" service is physically impossible and financially non-viable. Instead of aiming for perfect availability, systems use service level frameworks to define realistic reliability goals. These goals are structured around three core concepts: Service Level Agreements (SLAs), Service Level Objectives (SLOs), and Service Level Indicators (SLIs).

1. Learning Objectives

• Differentiate between SLA (agreement), SLO (objective), and SLI (metric).
• Define and calculate standard SLIs for availability and latency.
• Calculate uptime availability percentages (the "number of nines").
• Analyze and manage error budgets to balance feature velocity and service stability.
• Construct burn rate alerts to detect rapid error budget depletion.
• Implement an SRE metrics dashboard and error budget calculator in Java, Python, and C++.

2. Prerequisites

Before learning about reliability metrics, ensure you understand:

• REST APIs: Standard request/response behaviors.
• Basic Probability: Calculating success ratios and percentages.
• Monitoring Systems: General monitoring tools (Prometheus, Grafana).

3. Why This Topic Matters

Setting reliability targets is a key operational concern for site reliability engineers (SREs).

Without clear metrics, product managers will push for rapid deployments (feature velocity), while operations engineers will resist changes to avoid outages (system stability).

SLAs, SLOs, and SLIs provide a mathematical framework to resolve this conflict:

• Shared Context: Provides developers and business stakeholders with a unified view of system health.
• Error Budgets: Quantifies the acceptable level of failures, allowing developers to take calculated risks.
• Contractual Safety: Protects businesses from financial liabilities by defining clear service terms for customers.

4. Real-world Analogy

Think of a Pizza Delivery Service:

The SLI (Service Level Indicator): Is the delivery stopwatch. It measures how long it takes for a pizza to arrive from the time it was ordered. (e.g. "Pizza delivered in 28 minutes").

The SLO (Service Level Objective): Is the shop's internal goal: "We want $95\%$ of our pizzas to be delivered in under 30 minutes every month."

The SLA (Service Level Agreement): Is the customer guarantee: "Get your pizza in 30 minutes or it's free!" If the shop misses this guarantee, they face a financial penalty (the cost of the free pizza).

5. Core Concepts

• Service Level Indicator (SLI): A specific, quantitative measure of service behavior. It answers: *What is the current performance metric?*
  $$\text{SLI} = \frac{\text{Good Events}}{\text{Total Events}} \times 100$$
• Service Level Objective (SLO): A target reliability level for an SLI. It answers: *How good must the performance be?* (e.g. availability $\ge 99.9\%$).
• Service Level Agreement (SLA): A legal contract promising a service level to customers, with penalties (refunds, service credits) if the target is missed.
  Rule: Internal SLOs are always stricter than external SLAs to create a safety margin (e.g. SLO $99.9\%$ vs. SLA $99.0\%$).
• The Error Budget: The allowable room for failure, calculated as: $$\text{Error Budget} = 100\% - \text{SLO}$$ A $99.9\%$ SLO provides a $0.1\%$ error budget. If this budget is exhausted, further feature releases are blocked until stability is restored.
• Availability (Number of Nines):
  • Three Nines (99.9%): Allows $\approx 8.76$ hours of downtime per year.
  • Four Nines (99.99%): Allows $\approx 52.6$ minutes of downtime per year.
  • Five Nines (99.999%): Allows $\approx 5.26$ minutes of downtime per year.
• Burn Rate: The rate at which a service consumes its error budget. A burn rate of 1 consumes the entire budget over the SLO period (e.g. 30 days). A burn rate of 30 consumes the entire budget in 1 day, triggering SRE pager alerts.

6. Visualizations

Service Level Hierarchy

Error Budget Consumption Balance

7. How It Works Step-by-Step

Managing Reliability Targets

1. Define the SLI: Choose a metric, such as HTTP latency: "Percentage of requests processed in under 100ms."
2. Set the SLO Target: Set an internal goal: "We want $99\%$ of requests to meet our latency SLI over a rolling 30-day period."
3. Set the SLA Contract: Define the customer guarantee: "We promise $95\%$ of requests will meet our latency target, or we will refund $10\%$ of your monthly bill."
4. Monitor Consumption: Track incoming requests. For every request that takes longer than 100ms, deduct tokens from the monthly error budget: $$\text{Remaining Budget} = (\text{Total Requests} \times 0.01) - \text{Slow Requests}$$
5. Enforce Policies: If the error budget hits $0\%$, pause feature releases. Redirect developers to focus on performance optimizations, memory leak fixes, and unit testing until the rolling budget recovers.

8. Internal Architecture

A metrics tracking pipeline processes logs in real-time:

• Metrics Scraper (Prometheus): Collects application metrics (latency, HTTP statuses) from app nodes at regular intervals.
• Time-Series Database: Stores metrics data to support rolling-window queries (e.g. 30-day availability calculations).
• Policy Engine: Calculates remaining error budgets and automatically blocks deployments in CI/CD tools (like Jenkins or GitHub Actions) if budgets are exhausted.
• Alerting Engine: Monitors burn rates and triggers pager alerts (e.g. via PagerDuty) if the budget is depleting too quickly.

9. Request Lifecycle

Let's trace a request from the perspective of an SRE monitoring pipeline:

1. Request Arrival: A user accesses the dashboard page.
2. Request Processing: The application server processes the request, taking 105ms, and returns an HTTP 200 response.
3. Log Generation: The server logs the event: { path: '/dashboard', status: 200, latency: 105 }.
4. Metric Aggregation: Prometheus reads the log:
   • Availability SLI: Success (recorded as Good Event).
   • Latency SLI: 105ms is over the 100ms threshold (recorded as Bad Event).
5. Budget Depletion: The error budget engine registers the latency failure, reducing the remaining budget.
6. Alerting Check: If the rolling latency rate drops below the $99\%$ SLO target, the engine sends an alert to the SRE team.

10. Deep Dive

A. The Mathematics of Nines (Downtime Budgets)

Availability %	Downtime / Year	Downtime / Month (30 Days)	Downtime / Day
99.0% (Two Nines)	3.65 days	7.20 hours	14.4 minutes
99.9% (Three Nines)	8.76 hours	43.8 minutes	1.44 minutes
99.99% (Four Nines)	52.56 minutes	4.38 minutes	8.64 seconds
99.999% (Five Nines)	5.26 minutes	25.9 seconds	0.86 seconds

B. Designing Burn Rate Paging Rules

Instead of flagging simple threshold spikes, SRE teams use Burn Rate Alerts to measure how fast the error budget is being consumed:

• 1x Burn Rate: Consumes the entire budget over 30 days. No urgent alert needed.
• 14.4x Burn Rate: Consumes $2\%$ of the budget in 1 hour. This indicates a minor issue that should alert developers via email or Slack.
• 14.4x Burn Rate over 6 hours: Consumes $12\%$ of the budget. This is a sustained issue that requires paging the on-call engineer.
• 36x Burn Rate over 1 hour: Consumes $5\%$ of the budget in 1 hour. This indicates a major outage that should trigger immediate high-priority pager alerts.

11. Production Examples

• Google SRE Teams: Pioneers of the error budget framework. Google mandates that if a service exhausts its error budget, product deployments are blocked, and developers must focus on stability tasks.
• AWS EC2 SLA: AWS guarantees a $99.99\%$ availability SLA for EC2 instances. If availability falls below this, AWS provides service credits (e.g. $10\%$ refund if availability is between $99.0\%$ and $99.9\%$).
• PagerDuty Alerting Pipelines: Uses burn-rate equations to filter noise, paging on-call engineers only when system outages consume significant error budgets.

12. Advantages

• Objective Risk Management: Quantifies the acceptable level of failure, resolving conflicts between developers and operations.
• Reduces Pager Noise: Burn rate alerting filters out temporary spikes, paging SREs only during critical outages.
• Ensures User Satisfaction: SLOs align development goals with user expectations (e.g. keeping latency low enough to keep users happy).

13. Limitations

• Metric Complexity: Measuring rolling indicators across millions of daily requests requires complex monitoring infrastructure.
• Goodhart's Law Risk: "When a measure becomes a target, it ceases to be a good measure." Teams might modify metric queries to artificially inflate SLO scores.
• Over-Engineering Costs: Building systems to meet $99.999\%$ availability targets is incredibly expensive and unnecessary for most applications.

14. Trade-offs

• High Availability vs. Cost: Aiming for $99.999\%$ availability requires redundant servers, multi-region database replication, and complex failover setups, which increases infrastructure costs. Set SLOs to match what users actually need, avoiding unnecessary over-engineering.
• Feature Velocity vs. System Stability: Deploying features quickly increases product value but consumes your error budget. Prioritizing stability protects availability but slows down product rollouts.

15. Performance Considerations

• Asynchronous Metric Aggregation: Track and collect metrics asynchronously to avoid adding latency to application request paths.
• Log Buffering: Buffer log events locally and push them to metrics scrapers in batches to conserve CPU and network resources.

16. Failure Scenarios

• Bad Metric Instrumentation (False Health): An app server fails to connect to the database, returning empty HTTP 200 responses to users. The monitor records these as successful requests, hiding the outage.
  Mitigation: Define availability SLIs to count empty responses or error pages as failed requests.
• Sudden Budget Depletion (GC freeze): A major garbage collection freeze slows down requests for several minutes, consuming the entire monthly error budget in one event.
  Mitigation: Configure burn-rate alerts to detect budget depletion early, allowing engineers to mitigate the outage before the entire budget is lost.

17. Best Practices

• Set internal SLO targets stricter than external SLAs to create a safety margin for operations.
• Use rolling 30-day windows to compute SLIs and track error budget consumption.
• Use burn-rate alerting rather than simple threshold alerts to reduce pager noise.

18. Common Mistakes

• Aiming for $100\%$ availability, which blocks deployment velocities and increases cost.
• Failing to define clear fallback policies when the error budget is exhausted.
• Measuring metrics from the server's perspective instead of tracking the actual user experience.

19. Implementation (SLA/SLO Monitor)

Below is a complete implementation of an SLA, SLO, and SLI monitor in Java, Python, and C++. The simulator tracks request latencies, calculates availability metrics, monitors remaining error budgets, and triggers alerts if targets are violated.

20. Interview Questions & Answers

Q1. Explain the difference between SLAs, SLOs, and SLIs. How do they relate?

Answer:

• SLI (Service Level Indicator): The measured metric showing how the service is performing (e.g. latency is 95ms).
• SLO (Service Level Objective): The target goal for the SLI over a rolling window (e.g. $99\%$ of requests must have latency under 100ms).
• SLA (Service Level Agreement): The legal commitment to customers, promising a target service level, often with financial penalties if missed.

Q2. What is an error budget and how is it used to manage release velocities?

Answer: An error budget is the allowable room for failure, calculated as $100\% - \text{SLO}$. It acts as a safety buffer that balances feature releases and system stability.

If the error budget is healthy, developers can deploy updates quickly. If the error budget is exhausted, the deployment pipeline is blocked, and developers must focus on stability tasks until the budget recovers.

Q3. Why are internal SLOs always stricter than external SLAs?

Answer: Stricter internal SLOs create a safety margin for operations. For example, if your internal SLO is $99.9\%$ availability and your external SLA is $99.0\%$, the SRE team receives alerts and can fix outages before the system violates the SLA, avoiding financial refunds to customers.

21. Practice Exercises

• Exercise 1 (Easy): Calculate the total allowed downtime per month (30 days) for a service with a $99.95\%$ availability SLO.
• Exercise 2 (Medium): Modify the Python SloTracker implementation to support a rolling 100-request window using a deque, tracking SLO compliance for only recent requests.
• Exercise 3 (Hard): Write a Python class that calculates the burn rate dynamically over a 1-hour window and triggers an alert if the burn rate is $> 14.4\text{x}$.

22. Challenge Problem

The Global Payment SLA Breach Challenge: You manage a payment system deployed across 3 global regions. The global availability SLO is $99.9\%$.

During a database migration, Region A experiences an outage, losing all availability for 4 hours. Regions B and C remain healthy.

• Calculate the impact of Region A's outage on the global monthly availability score, assuming traffic is distributed evenly.
• Draw a diagram showing the region availability logs, global load balancer routing, and the error budget depletion curve.
• Explain how you would design a Regional Error Budget policy to prevent regional outages from violating the global SLA.

23. Summary

SLA, SLO, and SLI metrics provide a mathematical framework for managing service reliability in distributed systems. SLIs measure performance, SLOs set internal targets, and SLAs define customer agreements. Using error budgets allows SRE teams to balance development velocity with service stability, ensuring a high-quality user experience.

24. Cheat Sheet

Concept	What it is	Target Audience	Penalty if missed
SLI	A measured metric (e.g. latency, error rate).	SREs, operations engineers.	None (it is just a measurement).
SLO	The internal target for the SLI.	Developers, product managers.	Blocked deployments (error budget freeze).
SLA	The customer contract promising a service level.	Customers, sales, legal teams.	Financial refunds or service credits.

25. Quiz

26. Further Reading

• Google SRE Book: Service Level Objectives.
• Google SRE Workbook: Alerting on SLOs.
• Site Reliability Engineering — Betsy Beyer et al.

27. Next Lesson Preview

Monitoring targets protects operations during healthy runs. To survive major site outages and server failures, we must implement Disaster Recovery plans—the core business continuity concern we will explore in the next lesson.