# Heartbeats, Timeouts and Leases: Detecting Failed Nodes
Every distributed system needs to answer “is that node still alive?” The honest answer is that you can never know for sure: a node that stops responding might be dead, overloaded, paused for garbage collection, or cut off by the network. Good designs work with that uncertainty.
Contents
# Heartbeats
Each node periodically sends a small “I’m alive” message, say every second, to a coordinator or to its peers. If no heartbeat arrives within a timeout, for example 5 seconds, the node is suspected dead.
Choosing the timeout is a trade-off:
- Too short: a GC pause or brief network blip triggers a false failover, which can be worse than the original problem (two leaders, data movement storms).
- Too long: real failures take a long time to detect and requests keep going to a dead node.
Adaptive detectors such as the phi accrual failure detector (used by Cassandra and Akka) track the history of heartbeat arrival times and output a suspicion level instead of a yes/no answer.
# Push vs pull
- Push: nodes send heartbeats to a monitor. Simple, but the monitor must handle all the traffic.
- Pull: the monitor probes nodes (health-check endpoints behind a load balancer).
- Gossip: nodes exchange membership information with random peers, which scales to thousands of nodes without a central monitor.
# Leases
A lease is a heartbeat with a promise: “you are the leader (or lock holder) until time T unless you renew”. The holder must stop acting as leader once its lease expires, even if it has not heard otherwise. This bounds how long two nodes can believe they are leader at the same time, as long as clocks do not drift too far apart. Add a safety margin and use fencing tokens on writes so a stale leader that wakes up late cannot corrupt data.
# What happens after detection
Detecting a failure is only half the job. The system must also reassign the dead node’s work: elect a new leader, move partitions to replicas, or remove the node from load balancing, and do it in a way that tolerates the “dead” node coming back.
# Interview tip
When you say “we’ll detect failures with heartbeats”, add the timeout trade-off and how you prevent split brain (leases plus fencing tokens). That turns a one-liner into a strong answer.
Disclosure: I’m part of the Linkjob AI team. Linkjob AI is a real-time AI interview assistant: https://www.linkjob.ai/