How Spotify Serves Point Queries from an Exabyte Data Lake

Spotify (like all big companies) keeps most of its data in cheap object storage built to scan millions of rows at once. For decades companies have realized that data is valuable, so they almost never want to throw it away. This leads to a pattern of dumping data into inexpensive storage (a "data lake") where it's always accessible ... but maybe not in the way that you want.

Data lakes are great for a weekly report or a training run. They're terrible at a single question like "what did this person listen to?" That's exactly the kind of question an online feature, or an AI agent, needs answered before the page finishes loading. The usual fix is copying whatever needs to be fast into a database built for lookups or a cache like Redis. This is so common that if you've worked at a mature company, they probably have a few clicks or a config that does just this. But databases and caches cost real money per gigabyte, so teams end up rationing what's worth serving online.

Spotify built a way around that trade they call RAP, short for Random Access Parquet which is an index living outside the data files that records exactly which file and rows hold a given key's data. This makes it so a lookup that used to mean hunting through files one clue at a time becomes a single precise read. They paired it with changes to how new files get written, so one user's data sits together instead of being scattered across the file. The innovation is that none of this needs a second copy of the data or a rewritten pipeline. The same files BigQuery scans for a weekly report also answer one person's question, at roughly the cost of a single storage read.

This blog is relevant for a lot of data-heavy systems. Engineers are constantly navigating the tradeoff between massive storage and fast queries, and AI agents are making this tradeoff even more acute as they start asking way more questions of the data than humans ever did. Spotify's solution is a great example of a real third option between scanning the whole lake and paying to copy everything into a key-value store.

Before we get started, let's take a step back to describe this data lake problem in case you're unfamiliar with the idea. Feel free to skip ahead if this isn't new to you.

Every song a Spotify user plays lands as an event somewhere, and the working copy everyone downstream uses lives in what's called a data lake, which is just files sitting in object storage like Google Cloud Storage or S3 rather than rows in a database. Think of it like a big, cheap archive.

An analyst might query those files through a query engine like Trino or BigQuery, which read the files where they sit rather than loading them into a database first. A machine learning pipeline might read the same files for training sets. Neither is touching a database in the traditional sense. These are big sweeping scans, not the lookups and point queries an RDBMS like Postgres is built for.

There's a ton of benefits here! Object storage is cheap enough per gigabyte that Spotify keeps exabytes in GCS, the full history rather than a recent window, since nobody has to ration what gets kept. And every tool reading the same files also means one copy of the truth instead of several scattered across teams.

But, there are tradeoffs.

Picture an ordinary question: "what is the average listening time by country over the last week?"

Logically, this means touching two columns (country and listening time) across every row in that window. Spotify's lake files are Parquet files and they're made for queries just like this. Parquet is organized column by column rather than row by row. Every value from one column is packed together, then the next, split into row groups and pages, with a footer describing where everything lives. Values from the same column compress well because they're similar, and an engine can skip whole pages it doesn't need without reading them.

To answer our question, we just have to grab these 2 columns from every file in the window and do some math. It's a lot of data to scan, but it's exactly the data that we need. Data lakes work pretty well for this kind of query!

Now flip the question and ask what one specific person listened to. That's a handful of rows scattered across whatever files hold that user's data, and the layout that made our average-listening-time query fast does nothing for this one. If you can't pin down exactly where that row lives, you end up dragging gigantic pages of data off storage to get at a few hundred bytes. On top of that you pay for a full query plan, because Trino and BigQuery schedule every query as a distributed analytical job, even one that wants a single row.

Object storage itself isn't the culprit here. A GCS request comes back in 30 to 100 milliseconds, and newer tiers like S3 Express One Zone and GCS Rapid Storage are down in single digits. It's everything wrapped around the read that costs you.

That gap is why online features copy their data out. Whatever needs to be fast goes into Bigtable or DynamoDB, a second copy priced per gigabyte. If Spotify is like other companies they'll either have budgets or goals to reduce the size of this tier. So everything else stays in the lake, still around but too slow to serve a page from.

Got it? Data lake: great for scanning, terrible for lookups.

Pretty well-known approaches get you most of the way there. The parts that aren't as covered are the most interesting.

Across Spotify's billions of users, a single day's listening events land in thousands of files. So take that same question about one person's listening, the one we just asked, and put a window on it: a roughly 90-day summer, asked either by the user opening their profile or by an agent answering on their behalf. At 1,000 files a day, that's 90,000 candidate Parquet files. This is gigabytes to terabytes of data to scan which is just not practical in any online setting (and can get prohibitively expensive offline).

What many teams do is organize their data along the dimensions their queries actually use. If lookups by user are common, then within each day the files get bucketed by user ID: hash the ID, mod it by the number of buckets, and write the result into the path.

date=2026-07-14/bucket=0427/part-00031.parquet

With 1,000 buckets, hash(user_42) % 1000 = 427 means user 42's rows for that day can only be in bucket=0427. Not "probably" — the hash is deterministic, so every other bucket is ruled out without opening a single file. One candidate file per day instead of a thousand, and the 90,000 drops to 90.

Bloom filters can cut it down again. A Bloom filter is a compact structure that can tell you with certainty that a key is absent, though never that it's present. Spotify keeps one per file's user-ID column cached in a metadata store, so a file can be ruled out without ever opening it. That narrows the 90 candidates to roughly 12, the days that user was actually active. We're just gradually pruning away the files that can't possibly contain the data we need.

From here, we're a bit stuck and have to start reading the files. And that step is where most of the time goes. Reading a match out of a Parquet file not built for this means a chain of dependent reads. The reader grabs the footer first to learn where the row groups sit, parses their metadata, scans through the key column for rows that match, and only then can it follow the column and page indexes down to the pages holding each value it actually wants.

Basically, bucketing and Bloom filters get you to the right files, but they still leave you doing dependent disk reads that can't be parallelized.

Spotify's answer is RAP: an index living on disk alongside the data files but entirely outside them. A builder job reads the footers and page locations of existing Parquet files, scans the key columns, and writes a mapping from key to exactly where that data sits, without touching the original files. Each pipeline run appends a new fragment to the index, and old fragments are never modified, so the index is cheap to maintain incrementally rather than rebuilt from scratch.

The index is a multimap. One user ID shows up across many files and many partitions, so a single key maps to a list of entries. Each entry carries:

Indexing a terabyte of data produces gigabytes of index; a petabyte produces terabytes. Large indexes distribute by hashing keys into buckets, the same trick the data files use.

We talked about how we can prune down the scan space earlier: Parquet's own page index and Bloom filters are probabilistic, narrowing a scan by ruling things out without ever saying exactly where a row is, only where it can't be. This is where RAP is different. Since RAP's external index is definitive, rather than narrowing a search it replaces one, resolving directly to row numbers in effectively constant time. Once cached file metadata resolves those to byte ranges, the read becomes a handful of targeted ranged reads issued in parallel, since none depend on each other. A lookup that meant walking twelve files link by link becomes one index hit and one round of reads, dramatically reducing latency.

Note the parallelism in the graphic below:

There's a real limit, though. Against untouched files, the reader still fetches the whole page containing the target row, since a page is the smallest addressable unit and can run large, up to around 4 megabytes, even when the row wanted holds maybe 100 bytes. Getting the read smaller than the page meant changing how the files are written in the first place.

Good wins here already, but can we do more? Spotify proposes a couple more sets of optimizations.

An index solves finding a key. It doesn't shrink what you read once you get there. You're stuck on that because the layout was fixed back when the file was written. So Spotify changed the write side too.

The first is putting one key's rows physically together. Sorting by key is the simplest version and it's often close to free, since plenty of pipelines already emit sorted output anyway. On top of that, Spotify's writer can start a new page every time the key changes.

The page is still the smallest thing you can read. That doesn't change and you can't ask for half of one. What changes is what's sitting inside it. A page used to be a 4 MB slab holding your 100 bytes plus a few thousand other users' rows, and you paid to fetch and decompress all of it. Break on every key change and the page holds one key and nothing else, so it's small, and every byte in it is a byte you came for.

That costs about 20 bytes of page header per boundary, which is nothing against real per-key volume, and the files stay standard Parquet, so every other reader carries on none the wiser.

The second trick they outline is skipping the read altogether. The index builder already walks every row while it's building the mapping, so it may as well copy small values straight into the index entry while it's there. That makes it a covering index (see the connection back to databases?). The answer is sitting in the index, and there's no storage read at all. Spotify pushes the same trick to precomputed aggregates like a key's event count or its total play time.

Between the two, a point query comes down to one ranged read of a few kilobytes, or nothing at all.

Not every dataset has one natural key. A transactions table might need lookups by buyer_id one day and seller_id the next.

Because RAP is a structure sitting over files it never modifies, adding a second dimension is just a decision made at the serving layer. Build another access structure over the same entries and you can look up by seller_id too. The pipelines don't change and the data files are never rewritten, which is the whole pitch: a new index costs you an index, not a migration. DynamoDB secondary indexes answer the same need, though there the index is a managed copy of the data rather than a map over it.

The catch is that the files are still sorted and bucketed by the original key. A seller_id lookup scatters across far more files than a buyer_id one, and RAP can only soften that by merging adjacent byte ranges into fewer requests. A secondary lookup is never as cheap as the primary.

Data lives on a continuum between cheap and accessible for scans and expensive but useful for point lookups and queries. RAP adds one more point on that continuum. It's an index built over files that already exist, cheap enough to maintain that it doesn't force the same rationing.

The write-up doesn't give an outcome, though. There's no end-to-end latency number for a RAP lookup in production or account of how many features have adopted it. What Spotify describes is architecture, and the financial argument is a claim rather than something they try to measure or make the case for in the blog post. So treat it with a degree of skepticism.

The real takeaway here is this solution bolts on to their existing infrastructure which means they're not managing a gigantic migration or fragmenting their data infrastructure. You'll see this again and again in our In the Wild series. Companies make big bets and adapt around them rather than trying to change everything at once. This "organic system design" is why real-world systems look the way they do and why our Delivery Framework advocates a similar process for system design interviews.