Distributed Transactions & Resilience
Resilience4j — Circuit Breaker, Retry, and Bulkhead
Decorating methods to degrade gracefully, retry failed requests, and isolate thread pools.
Interview: Fault tolerance architectures. Expect questions on circuit breaker states, sliding window configurations, bulkhead thread isolation, and writing robust fallback methods.
Introduction
When a downstream microservice experiences latency or outages, calling it continuously will exhaust your application thread pool. Tomcat threads will get stuck waiting for replies, causing cascading outages across your system.
**Resilience4j** is a lightweight, fault-tolerance library designed for Java 8+. It allows you to decorate methods with **Circuit Breaker** (fail-fast during outages), **Retry** (recover from transient errors), and **Bulkhead** (isolate execution thread pools) behaviors to keep your application resilient.
Why It Matters
Fault-tolerance patterns protect your application from cascading failures, isolate unstable dependencies, and enable graceful degradation (e.g. returning cached or default data when database queries fail).
Circuit Breaker States
The Circuit Breaker acts as an electrical switch, transitioning through three states:
• CLOSED (Normal): Requests flow to the downstream service. The circuit breaker monitors success and failure rates.
• OPEN (Failing): If the failure rate exceeds your threshold (e.g. 50%), the circuit opens. Downstream calls are blocked, and requests fail-fast immediately by executing a fallback method.
• HALF_OPEN (Testing): After a cooldown period, the circuit enters a testing state. It allows a limited number of requests to pass. If they succeed, it closes; if they fail, it reopens.
Practical Example
Let's write a Spring Service method decorated with Resilience4j annotations and configurations:
And here is the corresponding application.yml configuration:
Quick Quiz
Q1: In Resilience4j, what state does the Circuit Breaker transition to when its cooldown period expires, allowing test requests to pass?
A) CLOSED
B) OPEN
C) HALF_OPEN
D) SUSPENDED
Answer: C — The HALF_OPEN state allows a limited number of test calls to verify downstream recovery before closing the circuit.
Scenario-Based Challenge
Production Scenario:
Your application processes orders and calls a third-party shipping API. When the shipping API experiences latency spikes, your application thread pool gets saturated, causing order creation calls to time out. How do you isolate the shipping calls to protect order creation?
View Solution
To isolate dependencies and protect application resources:
1. **Implement a Bulkhead:** Decorate the shipping service method with @Bulkhead(name = "shippingService").
2. **Configure Thread Pool Isolation:** Set up a thread-pool bulkhead in properties. This allocates a dedicated thread pool for shipping calls, preventing them from consuming Tomcat's main execution threads.
3. **Define Concurrent Limits:** Limit max concurrent calls (e.g. maxConcurrentCalls=5) to fail-fast excess requests immediately once the bulkhead pool is full.
Interview Questions
1. What is the difference between a sliding window type of COUNT_BASED vs TIME_BASED in Resilience4j?
A COUNT_BASED sliding window calculates failure rates based on the last N requests (e.g. evaluate after the last 10 requests). A TIME_BASED sliding window evaluates failure rates based on requests within the last N seconds (e.g. evaluate using calls received in the last 10 seconds), regardless of the request count.
Production Considerations
Integrate Resilience4j metrics with Spring Actuator and Micrometer. Monitor circuit breaker states (closed, open, half-open) and failure rates in Grafana to detect downstream dependencies degradation before outages occur.