Phase 4 of 5  ·  Modern Engineering
Week 16 / 20   ·   Ch 17

Distributed
Systems

"CAP theorem in 30 seconds — and the 8 fallacies every dev must know"

📚 Ch 17 — Distributed SE📐 CAP Theorem💀 8 Fallacies⏱ ~20 min read

🔍Concept Deep Dives

Click each concept to expand — real examples, diagrams, pros & cons.

📐

CAP Theorem

You can only guarantee 2 of 3: Consistency, Availability, Partition Tolerance. You must choose.
⌄

When to Use

Designing any distributed database or service — you must choose your trade-off.

Real-World Example

DynamoDB: AP (available + partition-tolerant). PostgreSQL: CP (consistent + partition-tolerant). No real CA system exists at scale.

✓ Advantages

  • Forces explicit trade-off decisions
  • Explains why distributed DBs differ

⚠ Watch Out

  • Oversimplified — PACELC extends it
CAP theorem — pick two of three
ConsistencyEvery read sees the latest write
AvailabilityEvery request gets a response
Partition toleranceSurvives network splits
CP: PostgreSQL, Zookeeper  ·  AP: DynamoDB  ·  a true CA system can't exist at scale.
💀

8 Fallacies of Distributed Computing

False assumptions developers make about distributed systems. By Peter Deutsch, Sun Microsystems.
⌄

When to Use

Every time you design a distributed system — check against these.

Real-World Example

Every distributed systems outage is caused by someone violating one of these. The network is NOT reliable.

✓ Advantages

  • Prevents architectural mistakes
  • Exposes hidden assumptions

⚠ Watch Out

  • Easy to forget under deadline pressure
8 fallacies of distributed computing
1The network is reliable
2Latency is zero
3Bandwidth is infinite
4The network is secure
5Topology doesn't change
6There is one administrator
7Transport cost is zero
8The network is homogeneous
All eight are false in production — design as if they are.
🌐

Middleware

Software layer between distributed components — handles communication, transaction management, security.
⌄

When to Use

Any distributed system with multiple services that need to communicate reliably.

Real-World Example

Kafka (message queue), gRPC (RPC framework), Redis (distributed cache), Consul (service discovery).

✓ Advantages

  • Hides network complexity
  • Standard communication patterns
  • Enables loose coupling

⚠ Watch Out

  • Another layer to fail
  • Learning curve
  • Cost and operational overhead
Middleware — the layer between services
KafkaMessage queue between services
RedisShared distributed cache
gRPCDirect service-to-service RPC
Middleware hides network complexity and handles delivery guarantees.
🏗️

Distributed Architectures

Client-server, multi-tier, peer-to-peer, service-oriented — different models for different needs.
⌄

When to Use

Choosing the right distributed model for your scale and requirements.

Real-World Example

Spotify: microservices. BitTorrent: P2P. AWS: multi-tier (UI tier → app tier → data tier).

✓ Advantages

  • Match architecture to problem
  • Well-understood patterns

⚠ Watch Out

  • Wrong model = wrong trade-offs
Distributed architecture models
22-tier — client ↔ server
33-tier — UI ↔ app server ↔ database
NN-tier — add caching, queues and more layers
PP2P — every node is client + server
µMicroservices — many small independent services

📋Quick Reference

θ Ch 17 Cheat Sheet — Distributed Systems
CAP Theorem
Consistency, Availability, Partition Tolerance — pick 2. Real choice is CP vs AP (P is mandatory).
Consistency
Every read gets the most recent write. No stale reads.
Availability
Every request gets a response (possibly stale). Always responsive.
Partition Tolerance
System continues despite network splits. ALWAYS required in real distributed systems.
8 Fallacies
Network is NOT reliable, zero latency, infinite bandwidth, secure, static, one admin, free, homogeneous.
Eventual Consistency
Nodes will eventually agree on the same value — but not immediately. DynamoDB default.
Idempotency
An operation can be applied multiple times without changing result beyond first application. Critical for retries.
θ
Sommerville's Key Points — Ch 17
Author's own summary from the end of the chapter.
  • 1Distributed systems: software that runs on multiple computers communicating over a network.
  • 2CAP theorem: can only guarantee 2 of: Consistency, Availability, Partition Tolerance.
  • 38 Fallacies: network is NOT reliable, zero-latency, infinite bandwidth, secure, homogeneous.
  • 4Middleware: software layer handling communication, transactions, security between components.
  • 5Client-server, multi-tier, P2P, service-oriented — different distributed architectures.
  • 6Design for failure: network partitions WILL happen. Plan timeouts, retries, circuit breakers.

🌍From the Book & Beyond

📖

Case Study — The City Traffic Control System (Fig 17.7)

Sommerville's Figure 17.7 shows a city traffic control system built on a master–slave architecture: three logical processes running on separate processors. The slaves do the grunt work — a sensor control process polls distributed traffic-flow sensors and cameras, while a light control process converts commands into signals for the traffic light hardware. The master control-room process polls the sensor processor, computes traffic light sequences, displays status to operators, and accepts their commands. The lesson: use master–slave when you can predict the distributed processing in advance and pin it to dedicated slave processors — the classic recipe for real-time systems with hard deadlines.

⚡

2026 Perspective — Distributed Is the Default

Distributed is no longer a design choice — it's the default. Every serverless function, managed database, and edge node is part of someone's distributed system. Yet the classic fallacies ("the network is reliable," "latency is zero") still bite: recent large cloud outages took down thousands of "resilient" apps that assumed their managed services couldn't fail. In 2026, engineers design for failure explicitly — timeouts, retries, circuit breakers, multi-region failover — because abstractions like serverless hide the network without eliminating it. Sommerville's transparency-versus-honesty trade-off (should users see the distribution?) is now a daily engineering decision, not a textbook footnote.

🧠Quiz — Test Yourself

Think through your answer first, then reveal.

Q1
Recall
Explain the CAP theorem. Which two properties do most modern web-scale databases choose?
CAP: in a distributed system, during a network partition, you must choose between Consistency (all nodes return the same data) or Availability (all requests get a response). Partition Tolerance is mandatory in real networks. Most web-scale databases choose AP (DynamoDB, Cassandra, CouchDB) — they prioritize availability with eventual consistency. Strongly consistent systems (PostgreSQL, ZooKeeper, etcd) choose CP.
Q2
Apply
Name 3 of the 8 Fallacies and explain why each matters in practice.
1. 'Network is reliable' — it's not; design with timeouts, retries, circuit breakers. 2. 'Latency is zero' — network calls take time; avoid synchronous calls in hot paths. 3. 'The network is secure' — it's not; encrypt all inter-service communication (mTLS, TLS).
Q3
Analyze
What is eventual consistency and when is it acceptable?
Eventual consistency: nodes will eventually converge to the same value, but reads may return stale data temporarily. Acceptable for: social media likes/views counts (slightly stale is fine), product catalog prices (few seconds delay OK), user profiles (not mission-critical). Not acceptable for: bank balances, inventory reservation, medical records.
Up Next → Week 17
Microservices Architecture
When to choose microservices (and when NOT to)
Continue → Week 17