Notesdistributed🌱 seedling

Unreliable Network

Also: Asynchronous Network · Network Faults · Fallacies of Distributed Computing · Unbounded Delay

Data-centre and internet networks are asynchronous packet networks: a message may be lost, delayed without bound, duplicated or reordered, and the sender can't tell a lost request from a dead or slow receiver or a lost reply.

Why it exists

Shared networks give high utilization by queueing packets instead of reserving capacity per connection (like a phone circuit would). Queues mean variable delay; full queues mean drops. That bargain buys cheap bandwidth at the price of no timing guarantees.

How it works

  • Where delay comes from: a switch port queue when many senders target one receiver; the receiving OS when every core is busy; a hypervisor running another VM; TCP retransmitting a lost packet after its timeout (at least 200 ms on Linux), which turns loss into latency.
  • What the sender sees: silence, whatever went wrong. The only tool is a Timeout, which is a guess about a delay distribution.
  • What else can happen: duplicates (a retry after a lost reply), reordering (a retransmitted packet arriving after a later one), and Network Partitions.
  • TCP gives ordering and retransmission within one connection, not delivery guarantees across reconnects or processes.

Trade-offs

Network design Gains Costs
packet switching with queues (Ethernet, IP) high utilization, cheap bursts unbounded delay and loss
circuit-like reserved capacity bounded latency idle capacity is wasted

In practice

  • Microsoft data centres (2011 study): on average 5.2 device failures and 40.8 link failures per day; redundant paths cut the median impact of a failure by only 40%.
  • GitHub, December 2012: a switch fault blocked traffic for 90 seconds; delayed fencing messages left some file-server pairs both active, and on recovery each shot the other.

Pitfalls

  • Assuming "the network is reliable" and "latency is zero" (the first two Fallacies of Distributed Computing).
  • Redundant switches that share software and configuration fail together.

Recall

These cards are also in your review deck, scheduled with spaced repetition.