Airbnb Rebuilt Its Key Value Store

Welcome to Engineering Unpacked, the series where I read a very smart engineering blog, open twelve tabs to understand it, and then explain it without making you do the same side quest.

Today we are unpacking Airbnb's post about building the next generation of Mussel, its internal key-value store for derived data. The original was written by Shravan Gaonkar, Chandramouli Rangarajan, and Yanhan Zhang. This is my plain-English breakdown, plus the system-design lessons I am stealing for later.

Nobody at Airbnb asked me to explain their storage system. I simply saw “more than a petabyte” and walked toward it like a moth approaching an extremely distributed lamp.

<!-- Add an image anywhere in the article like this:  -->

First, what is a key-value store?

A key-value store is the database version of a coat-check desk.

You hand it a key, such as listing:123:price, and it gives you the value attached to that key. No dramatic table joins. No detective work. Just: “Hello, I have this key. Please return my coat before the request times out.”

That simple model is useful when an application needs fast lookups at a ridiculous scale. Airbnb uses Mussel for derived data: information produced from other data and prepared for quick online access. Think fraud signals, personalization features, pricing information, or anything else that may be calculated elsewhere but must be available while a live request is waiting.

The awkward part is that this data arrives in two very different ways:

• huge bulk loads from offline data warehouses;
• constant streaming updates from live systems.

Mussel has to accept both while still serving low-latency reads. It is basically unloading a moving truck and accepting food-delivery orders through the same front door.

Meet Mussel V1

Mussel v1 worked for years. This is important because engineering rewrites are often introduced as if the old system was discovered in a cave beside a floppy disk.

The old version was not useless. Airbnb's needs simply grew beyond what it was comfortable handling.

Scaling or replacing its EC2 nodes required multi-step operational work. Static hash partitioning could create overloaded nodes and latency spikes. Teams had limited control over consistency, and it was difficult to see exactly which users were consuming which resources.

Meanwhile, the workloads kept getting louder: real-time fraud checks, instant personalization, dynamic pricing, enormous tables, and both streaming and batch ingestion.

At some point, “the system still works” becomes “the system works because several engineers know the sacred restart dance.” That is usually a sign.

What changed in Mussel v2?

Airbnb kept the friendly key-value interface but rebuilt what lived behind it.

Mussel v2 uses a distributed NewSQL backend and a Kubernetes-native control plane. The goal was not merely to choose a newer database. It was to make the whole platform easier to scale, automate, observe, and migrate.

The main pieces are:

• a stateless Dispatcher that accepts client requests;
• a distributed database that stores the actual records;
• Kafka as a durable write log;
• Replayers and Write Dispatchers that apply events to storage;
• an Airflow-and-S3 pipeline for enormous bulk loads;
• a distributed TTL service that deletes expired data;
• migration controls for dual writes, shadow reads, throttling, and fallback.

Yes, there are several dispatchers. Distributed systems apparently become safer when every component sounds like it owns a tiny radio.

The Dispatcher Hides Complexity

The Dispatcher is a stateless service that can scale horizontally in Kubernetes. Clients talk to it instead of knowing the details of the storage backend.

It translates API calls into database queries and mutations, handles retries and rate limits, and connects to Airbnb's service mesh for security and discovery. It also became the control point for the migration: the same layer could dual-write, shadow-read, redirect traffic, or fall back to v1.

This is a useful system-design pattern. Put a stable interface in front of infrastructure that will change. Your callers get a boring API while you replace the engine underneath without sending the entire company a “please update your database client by Friday” message.

Reads are intentionally boring

Each Mussel “dataname” maps to a logical table. The Dispatcher can perform point lookups, prefix queries, and range queries.

Callers can also choose stale reads from local replicas when the latest possible value is less important than lower latency or cost. Other workloads can read from the primary when freshness matters more.

That choice is the interesting bit. “Consistency versus performance” is not one decision made for the entire company. Mussel lets different namespaces choose the tradeoff that matches their service-level needs.

One caller may say, “I need the newest fraud signal immediately.” Another may say, “Yesterday's recommendation feature is fine; nobody is calling the police.” Same platform, different dial.

Writes take the Kafka scenic route

Writes are persisted to Kafka first. Replayers then consume those events and apply them to the backend in order.

Why add a queue instead of writing directly to the database and going home early?

Because Kafka gives the system a durable buffer. It can absorb bursts, preserve ordering, replay data, and let storage workers catch up without making every producer wait for the entire downstream journey. It was also extremely useful during migration because v1 and v2 could consume the same stream.

The tradeoff is more moving parts and eventual consistency in parts of the pipeline. Kafka is not decorative parsley sprinkled over an architecture diagram. If you add it, you now own consumers, lag, retries, duplicates, ordering rules, and several dashboards that will become emotionally important at 2 a.m.

For Mussel, those costs were worth paying because replayability was central to ingestion and migration.

Bulk Loading Without Fires

Airbnb still needed to move huge offline datasets into a system serving live traffic.

Mussel v2 preserves two useful bulk-load behaviours:

• merge adds data to what is already present;
• replace swaps the existing dataset for a new one.

Airflow coordinates the workflow. Data is transformed into a standard format and uploaded to S3. A stateless controller assigns work, while stateful workers ingest partitions in parallel and checkpoint progress.

The checkpointing matters because a load may run for hours or days. When worker number 47 has a small existential crisis, the job should resume instead of restarting the transfer of planet Earth.

Airbnb also uses deduplication, delta merges, and insert-on-duplicate-key-ignore behaviour to reduce unnecessary writes. At this scale, “we will just load it again” is not a recovery strategy. It is a budget proposal.

TTL Cleans Up Expired Data

TTL means time to live: once a record becomes old enough, the system can remove it automatically.

In v1, expiration depended on the storage engine's compaction process, which struggled as datasets grew. Mussel v2 instead runs a topology-aware expiration service.

It divides namespaces into range-based tasks and lets multiple workers scan and delete expired records in parallel. The scheduler limits how much this cleanup can interfere with live queries. Write-heavy tables get targeted deletion and version limits so expired data does not pile up forever like cables in my drawer.

The broader lesson: deletion is a workload. At small scale, TTL feels like a checkbox. At petabyte scale, it needs partitioning, scheduling, observability, and a sincere apology to the storage nodes.

How to Move a Petabyte

Very carefully. Then even more carefully than that.

Airbnb wanted zero data loss and no availability impact while moving more than a petabyte across thousands of tables. Mussel v1 did not offer convenient table snapshots or change-data-capture streams, so the team built a custom, reversible pipeline and migrated one table at a time.

For each table, they roughly did this:

1. Download and sample v1 backup data to understand its distribution.
2. Create a pre-split v2 table so the initial load would not concentrate traffic on a few shards.
3. Bootstrap the historical data, checkpointing progress along the way.
4. Compare checksums to prove the copy arrived correctly.
5. Consume the Kafka backlog created while the bootstrap was running.
6. Have both stores consume the same Kafka topic so v1 and v2 stayed close together.

Then came the read migration:

• Blue: v1 served everything.
• Shadow: v1 answered users while v2 secretly handled the same requests for comparison.
• Reverse: v2 answered users while v1 stayed warm as the fallback.
• Cutover: v2 became the real home after the checks stayed healthy.

Circuit breakers could move traffic back when error rates or replication lag increased. Every table had its own migration stage, so one suspicious workload did not hold the entire fleet hostage.

This is the opposite of a heroic midnight cutover. It is intentionally boring, gradual, measurable, and reversible. That is exactly how I want a database migration to behave.

For a smaller example of keeping incoming writes correct throughout this process, read Move Data Without Downtime. It covers the snapshot boundary, replay, deletes, and the moment write ownership changes.

The sneaky hard parts

The new backend was strongly consistent, but the old one had eventually consistent behaviour. Stronger consistency sounds like a free upgrade until old write patterns begin producing conflicts.

Airbnb needed write deduplication, hot-key protection, and lazy repair for rare inconsistencies. Some fixes traded storage cost for correctness; others traded read performance. The new database changed how range filters behaved, so some pagination logic moved to the client side.

Range sharding also made presplitting critical. If a large load contains consecutive keys, sending them into an unprepared range can create a hotspot. Sampling the old data first allowed Airbnb to create balanced shards before the migration arrived with all its luggage.

The lesson is rude but useful: changing databases does not remove tradeoffs. It gives you different tradeoffs wearing a new company hoodie.

The numbers that made me sit up

Airbnb reports that Mussel v2 can, in the same cluster:

• ingest tens of terabytes through bulk uploads;
• sustain more than 100,000 streaming writes per second;
• keep p99 reads below 25 milliseconds.

It also supports tables larger than 100 TB and gives callers per-namespace control over stale reads.

Those numbers are impressive, but the migration design is the part worth copying. Most of us do not have a petabyte lying around. We do have risky schema changes, old services, and somebody saying, “Can we switch it all on Friday afternoon?”

Please do not switch it all on Friday afternoon.

System-design lessons I am stealing

Here is my aggressively condensed notebook page:

1. Hide infrastructure behind a stable API. Clients should not care which storage engine is currently paying rent.
2. Use a durable log when replay matters. Kafka made catch-up, dual writes, and recovery possible.
3. Separate batch and streaming paths, then converge them at storage. They have different shapes and failure modes.
4. Migrate in tiny reversible steps. Table-level controls reduced the blast radius.
5. Shadow traffic before trusting a new system. Compare real behaviour without serving experimental answers.
6. Verify the data, not just the job status. A green pipeline is lovely; matching checksums are evidence.
7. Plan partitions around the data distribution. The wrong shard layout can turn a normal import into a hotspot festival.
8. Make tradeoffs configurable. Freshness, latency, and cost do not have one universal answer.
9. Treat operations as a feature. Automated rollouts, quotas, dashboards, and fallbacks are part of the architecture.

The five-line version

Airbnb had an internal key-value store called Mussel. It was fast, but the old backend became difficult to scale and operate for newer workloads.

Mussel v2 put a stateless Dispatcher in front of a distributed NewSQL backend, used Kafka as a durable write log, kept a separate bulk-ingestion path, and added distributed TTL cleanup.

The team moved more than a petabyte with a table-by-table blue/green migration: bootstrap, checksum, catch up, dual-write, shadow-read, reverse, and finally cut over.

The shiny database is interesting. The reversible migration is the masterpiece.

And that is Mussel v2, decoded with zero petabytes harmed on this blog.