Columnar Storage and Collection Columns: Where Your Arrays and Maps Really Live

Arif KHAN

— 3 min read
Share

Columnar storage formats like Parquet and ORC, and columnar execution engines such as BigQuery, are widely used in analytics systems because they store data column by column on disk instead of row by row. When a query reads only the columns it needs, only those columns are read from disk, reducing the amount of data that must be scanned compared to row-based storage.

At Zinzu, we use these formats in our pipelines and also work extensively with BigQuery.

This post focuses on a commonly misunderstood area of columnar storage: how collection columns like arrays and maps actually work.

Let’s start with simple types (briefly)

Consider the following table:

USER_ID   SUBSCRIPTION_PLAN
u1        premium_monthly
u2        premium_monthly
u3        free_trial
u4        premium_monthly
u5        free_trial

This is the same data regardless of how it is stored. 
What changes is the layout on disk.

In row-based storage, data is stored row by row:
[u1, premium_monthly]
[u2, premium_monthly]
[u3, free_trial]
[u4, premium_monthly]
[u5, free_trial]


In columnar storage, each column is stored separately:

USER_ID column:
[u1,
 u2,
 u3,
 u4,
 u5]


SUBSCRIPTION_PLAN column:
[premium_monthly,
 premium_monthly,
 free_trial,
 premium_monthly,
 free_trial]

Now let’s move to collection columns

So far, everything has been simple because each row contributes exactly one value per column.

But real-world data often looks like this instead:

interests ARRAY<STRING>
attributes MAP<STRING, STRING>

Where & how are array or map values actually stored in columnar systems?

Understanding map columns in columnar storage

attributes MAP<STRING, STRING>

sample data

| USER_ID | attributes                          |
| ------- | ----------------------------------- |
| u1      | {"city": "Seattle", "os": "iOS"}    |
| u2      | {"city": "Seattle"}                 |
| u3      | {"city": "Austin", "os": "Android"} |

At first glance, a map feels like a single value per row, just more complex. This is where many people assume: “Each row stores its own map.” That assumption is not correct in columnar storage.

In most columnar formats, a map is internally represented as:

ARRAY<STRUCT<key, value>>

so conceptually, the attributes column becomes three separate pieces:
column for keys
column for values
structure that links them back to rows
All map entries across all rows are flattened into separate columns as global key column & gloabal value column as follows:
keys takes its column as 
attributes.key:
[city, os, city, city, os]

values takes its column as
attributes.value:
[Seattle, iOS, Seattle, Austin, Android]


There is no per-row map object stored anywhere.
Every key and every value lives in separate two global lists.

How rows are linked back to map entries

Each row only stores offset information that says where its map entries live inside the global lists.

Row u1 → entries [0..1]   → (city=Seattle, os=iOS)
Row u2 → entries [2..2]   → (city=Seattle)
Row u3 → entries [3..4]   → (city=Austin, os=Android

So a row is effectively saying:

My map entries start here and end here in the global key/value lists.

That’s the only connection between rows and maps.

Why this design exists

This design keeps columnar properties intact:

  • Keys are stored together
  • Values are stored together
  • Each column can be scanned independently

Even though maps look row-oriented, their storage is still column-first.

What changes compared to scalar columns

With scalar columns:

  • One value per row
  • One-to-one mapping between rows and values

With map columns:

  • A row can contribute many values
  • Values from different rows are mixed together
  • Row boundaries are reconstructed later using offsets

This difference is subtle but critical.

Arrays: maps without keys, and why they behave the same way internally

Now that we understand how maps are stored, arrays become much easier to explain. An array column is essentially a map without keys.

Let’s start with a simple array column.

interests ARRAY<STRING>

sample data

| USER_ID | interests           |
| ------- | ------------------- |
| u1      | ["music", "sports"] |
| u2      | ["music"]           |
| u3      | ["travel", "music"] |

How arrays are actually stored

All array values across all rows are flattened into a single global column.

interests.values:
[music, sports, music, travel, music]


There is no separate array stored for each row.
Every element from every row lives in this one list.

How rows are linked back to array values

Each row only stores offset information that says which part of the global list belongs to it.

Row u1 → values[0..1] → ["music", "sports"]
Row u2 → values[2..2] → ["music"]
Row u3 → values[3..4] → ["travel", "music"]

So a row is effectively saying:

My array elements start here and end here in the global list.

That’s the only connection between rows and arrays.

What’s different from maps

Maps had:

  • A global list of keys
  • A global list of values

Arrays have:

  • Only a global list of values

Otherwise, the storage model is the same:

  • Flatten everything
  • Store offsets per row
  • Reconstruct rows later

Why Columnar Storage Scans Are Efficient

In columnar storage, applying a filter usually means scanning the entire column. That’s expected.

What makes this efficient is compression.

Most analytical columns contain repeated values, which compress extremely well. As a result, even though the engine scans the column, the actual data read from disk is much smaller than the raw size.

So yes, the column is scanned—but it’s a cheap scan, because compression drastically reduces I/O.

The worst-case example is a UUID column. When every row contains a unique value, there’s little to no repetition, and compression provides minimal benefit. Scanning such a column is close to reading raw data.

This is why columnar systems excel at analytics and aggregations, but are not designed for single-record lookups by ID.

Read more