Challenges

• update multi-item state in a way that's safe even if coord crashes
• new coord needs to be able to recover/repair state
• elect MR coord (one at a time!)
• what if old coord doesn't realize it's been replaced
  • can it still read/write state in ZK?
  • or in other storage?
• read performance

Features

1. asynchronous
2. read by any server

Linearizablity ⇒ behaves like a single machine

1. Can construct a total order of ops (even for concurrent ones)
2. Order matches ‘real-time’
3. Read returns values of last write

ZooKeeper: change correctness definition

1. Linearizable writes
2. FIFO client order
   1. Read: observe last write from same client