Service Discovery

In modern cloud architectures, microservice instances are dynamic. They scale up and down, restart after failures, and deploy on ephemeral IP addresses. Hardcoding IP addresses in configuration files is impossible. Service Discovery provides a dynamic network database where instances register their network locations, allowing other services to discover and call them.

1. Learning Objectives

• Explain the operational challenges of dynamic IPs in clustered environments.
• Differentiate between Client-Side and Server-Side discovery patterns.
• Understand the roles of the Service Registry, Service Provider, and Service Consumer.
• Evaluate registration mechanisms (Self-Registration vs. Third-Party Registration).
• Analyze how heartbeats and health checks prune dead instances from registries.
• Implement a thread-safe Service Registry and load balancer in Java, Python, and C++.

2. Prerequisites

Before learning about service discovery, ensure you understand:

• Load Balancing: Standard load distribution methods (Round-Robin, Random).
• DNS (Domain Name System): How hostname lookup resolves to IPs.
• REST APIs: Standard HTTP status checks and registration methods.

3. Why This Topic Matters

In a monolithic architecture, components communicate via in-memory function calls. In a microservices architecture, services communicate over the network.

In the cloud, container IPs are ephemeral: a node restart, auto-scaling event, or rollout changes instance IPs instantly.

Service Discovery solves this:

• Dynamic Lookup: Services can locate active nodes on demand.
• Automatic Failover: Dead nodes are automatically removed from routing tables.
• Seamless Deployment: Enables blue-green deployments by dynamically shifting traffic.

4. Real-world Analogy

Think of a Ride-Hailing Dispatcher:

If you need a ride, you do not call individual drivers directly on their personal phone numbers. Instead, you use the ride-sharing app.

The Service Registry (App Server): Keeps track of all online drivers, their current locations, and their status.

The Service Provider (Driver): Registers as online when they start their shift and sends periodic location updates (heartbeats) to the app server.

The Service Consumer (Rider): Requests a ride. The app server checks its active database, selects a driver, and connects the rider to them.

5. Core Concepts

• Service Registry: A centralized database containing the network locations of all active service instances (e.g. Consul, Netflix Eureka, etcd).
• Client-Side Discovery: The client queries the Service Registry to get a list of healthy instances, selects one using a load balancing algorithm (e.g. Round-Robin), and calls it directly.
  Note: Eliminates an extra network hop but requires routing logic in client code.
• Server-Side Discovery: The client calls a load balancer proxy (e.g. Nginx, AWS ALB). The load balancer queries the registry and routes the request to an available instance.
  Note: Simpler client code, but adds a network hop.
• Self-Registration: Service instances register themselves with the registry on startup and send periodic heartbeats to maintain active status.
• Third-Party Registration: A manager tool (e.g. Kubernetes Registrar) detects new instances, registers them, and handles health checks automatically.
• Eviction Policy: The process by which the registry automatically removes instances that fail to send heartbeats within a timeout window.

6. Visualizations

Client-Side vs. Server-Side Discovery

Registration & Heartbeat Lifecycle

7. How It Works Step-by-Step

1. Service Startup: An instance of OrderService boots up, assigned IP 10.0.0.12 and Port 8082.
2. Registration Call: The instance sends an HTTP POST request to the service registry: POST /registry/register { "name": "order-service", "id": "order-node-1", "address": "10.0.0.12", "port": 8082 }.
3. Heartbeat Scheduling: The instance schedules a background task to ping the registry every 30 seconds to maintain its active status.
4. Client Discovery: A consumer service (e.g. GatewayService) queries the registry: GET /registry/discover/order-service. The registry returns a list of healthy IP locations.
5. Load Balancing & Call: The consumer selects an IP from the list (using Round-Robin) and makes the API call.
6. Node Crash & Eviction: If order-node-1 crashes, it stops sending heartbeats. After a defined timeout window (e.g. 90 seconds), the registry marks the node as dead and evicts it from the active list.

8. Internal Architecture

A Service Registry acts as a highly available, read-heavy distributed database.

• Replicated Key-Value Store: Service registries use consensus algorithms (like Raft in Consul/etcd, or peer replication in Eureka) to synchronize data across nodes, ensuring high availability.
• Consistent vs. Available Registries:
  • Strong Consistency (CP): Registries like Consul or etcd prioritize consistency. A network partition blocks writes until a quorum is reached, ensuring clients never receive stale IPs.
  • High Availability (AP): Registries like Netflix Eureka prioritize availability. Nodes accept writes during partitions, returning cached (potentially stale) IPs, which clients handle using retries.
• Dynamic Cache Refresh: Clients cache service locations locally and query the registry periodically (e.g. every 30 seconds) to update their local lists, reducing load on the registry.

9. Request Lifecycle

Let's trace a client request calling a dynamically registered microservice:

1. API Gateway Routing: A user clicks "Order History". The API Gateway receives the request.
2. Registry Lookup: The gateway checks its local cache for order-service nodes. If empty, it queries the registry and caches the result.
3. Round-Robin Routing: The gateway selects node order-node-2 (10.0.0.14:8082) using Round-Robin.
4. Target API Call: The gateway calls http://10.0.0.14:8082/orders and returns the response to the user.

10. Deep Dive

A. Client-Side vs. Server-Side Discovery Comparison

Metric	Client-Side Discovery (e.g. Eureka/Ribbon)	Server-Side Discovery (e.g. Kubernetes/ALB)
Network Hops	Fewer (direct client-to-instance call).	Additional hop (must pass through proxy/load balancer).
Client Complexity	High (requires custom SDKs and routing logic).	Low (client simply calls a single URL endpoint).
Language Lock-in	Yes (SDKs must support the client language).	No (standard HTTP calls, language-agnostic).
Single Point of Failure	None (gateway outage does not affect routing).	Yes (the load balancer proxy is a potential SPOF).

B. DNS-based Service Discovery

Kubernetes uses CoreDNS to handle discovery inside clusters.

Instead of REST APIs, instances query internal DNS names: order-service.default.svc.cluster.local.

How it works: CoreDNS returns the IP of a virtual Kubernetes Service (ClusterIP). The request is routed to a healthy pod using iptables rules configured by kube-proxy, keeping client code simple.

11. Production Examples

• HashiCorp Consul: A CP-based distributed service registry that uses the Raft consensus algorithm. It supports health checks and key-value storage.
• Netflix Eureka: An AP-based service registry. Nodes replicate registration records asynchronously, prioritizing availability during network partitions.
• etcd: A strongly consistent key-value store used by Kubernetes to store cluster states, configurations, and service endpoints.

12. Advantages

• Dynamic Scaling: Instances register and deregister automatically, eliminating manual configuration updates.
• Self-Healing: Dead nodes are evicted automatically based on failed heartbeats, protecting traffic from routing failures.
• Decoupling: Clients do not need to know where downstream instances reside; they query the registry on demand.

13. Limitations

• Increased Complexity: Adding a service registry introduces another distributed system component to manage.
• Registry Overload: Millions of active instances sending heartbeats can overload registry network and CPU resources.
• Stale Cache Windows: If a node crashes, clients using local caches will continue calling the dead IP until their next cache sync.

14. Trade-offs

• AP vs. CP Registries: CP registries (Consul) guarantee all clients receive the same healthy list of IPs but will reject updates during network partitions. AP registries (Eureka) remain writable during partitions but return stale IPs, requiring clients to handle connection failures.
• Client-Side vs. Server-Side Routing: Client-side routing reduces network latency and avoids single-point-of-failure bottlenecks but requires complex SDKs. Server-side routing simplifies client code but adds network latency and load balancer costs.

15. Performance Considerations

• Optimize Heartbeat Intervals: Frequent heartbeats (e.g. 1 second) detect outages quickly but create high network overhead. 30-second intervals with eviction thresholds are standard in production.
• Local Cache Refresh Policies: Cache service lists on client machines to avoid querying the registry for every API call.

16. Failure Scenarios

• Registry Outage (Local Cache Fallback): If the service registry goes down:
  Mitigation: Configure clients to fallback to their cached list of instances to keep services running during registry outages.
• Split-Brain Partitions: During a network partition in a CP registry, nodes on the minor side cannot reach a consensus.
  Mitigation: The minor partition rejects registration updates but continues serving read queries using its last consistent state.

17. Best Practices

• Cache registry results locally on clients to protect against registry outages.
• Configure eviction thresholds to require multiple consecutive failed heartbeats before removing a node, preventing premature evictions during transient network drops.
• Secure registry communications using mutual TLS (mTLS) to prevent unauthorized services from registering or reading instance lists.

18. Common Mistakes

• Failing to configure local caching on clients, overloading the registry with lookups.
• Setting heartbeat timeout limits too low, causing healthy nodes to get evicted during brief network drops.
• Hardcoding registry IPs in microservice configs instead of resolving them using DNS.

19. Implementation (Service Registry and Load Balancer)

Below is a complete implementation of a Service Registry with client-side load balancing in Java, Python, and C++. The simulator handles service registration, heartbeat updates, dead instance eviction, and Round-Robin load balancing.

20. Interview Questions & Answers

Q1. Compare Client-Side and Server-Side Service Discovery. What are the key architectural trade-offs?

Answer:

• Client-Side Discovery (e.g. Netflix Eureka) requires the client to query the registry and select an instance locally. It avoids an extra network hop (improving performance) and eliminates a load balancer bottleneck. However, it couples routing logic to client code, requiring custom libraries (SDKs) for each programming language.
• Server-Side Discovery (e.g. Kubernetes ClusterIP) routes client calls through a load balancer proxy that queries the registry. It simplifies client code (decoupling routing) but adds a network hop and a single-point-of-failure load balancer proxy.

Q2. What is the distinction between AP and CP service registries during network partitions?

Answer:

• CP Registries (Consul, etcd) prioritize consistency. If a network partition occurs, registry nodes in the minority partition cannot reach a consensus and will reject registration updates, ensuring clients receive only consistent, verified IPs.
• AP Registries (Eureka) prioritize availability. During partitions, all nodes remain writable and accept heartbeats. This can lead to stale IPs being returned to clients, which clients must handle using connection retries.

Q3. How does etcd maintain strong consistency across registry nodes?

Answer: etcd uses the Raft consensus algorithm. Raft elects a single leader node that coordinates all write updates. Writes are replicated to follower nodes, and a write is only committed once a majority of nodes acknowledge it, guaranteeing strong consistency across the cluster.

21. Practice Exercises

• Exercise 1 (Easy): Trace a request path showing the network calls made under Server-Side Discovery compared to Client-Side Discovery.
• Exercise 2 (Medium): Modify the Python ServiceRegistry implementation to support weighted load balancing. Add a weight parameter to ServiceInstance and implement a weighted random selection.
• Exercise 3 (Hard): Write a Python simulation of Eureka's Peer-to-Peer Replication. Replicate registrations asynchronously between two registry instances and handle replication conflicts.

22. Challenge Problem

The Registry Network Partition Challenge: You operate a cluster of 5 service registry nodes using the Raft consensus algorithm. A network partition splits the cluster into two segments: Partition A (3 nodes) and Partition B (2 nodes).

A microservice instance attempts to register during the partition.

• Explain how Partition A and Partition B handle the registration request.
• Draw a diagram showing the node states, partition boundaries, and quorum boundaries.
• Describe the synchronization flow that occurs when the network partition heals and the cluster recovers.

23. Summary

Service Discovery provides a dynamic network database that replaces hardcoded configuration files in clustered cloud environments. Registries maintain live service instance addresses by tracking periodic heartbeats. Designing service discovery requires choosing between CP (strongly consistent) and AP (highly available) registries, and balancing the trade-offs of client-side load balancing versus server-side load balancer proxies.

24. Cheat Sheet

Feature	Consul (CP-Raft)	Netflix Eureka (AP-Peer)
Consistency Axis	Strong Consistency (CP).	Eventual Consistency (AP).
Partition Behavior	Rejects writes in minority partition.	Accepts registrations on all active nodes.
Health Checks	Active checks (HTTP, TCP, Script).	Passive checks (heartbeat pings).
DNS Interface	Supported natively.	Requires REST query wrappers.

25. Quiz

26. Further Reading

• Netflix Eureka GitHub Repository.
• HashiCorp Consul Documentation.
• Microservices Patterns — Chris Richardson (covers service discovery patterns).

27. Next Lesson Preview

Service registries rely on periodic health indicators to prune crashed nodes. In the next lesson, we will look at the Heartbeats pattern—the core mechanism that detects node failures across distributed clusters.