How Block-Based File Formats Enable Parallel Processing
Conversations about data formats often focus on Parquet versus Avro, or columnar versus row-oriented storage. What is discussed less often is the block-based structure that actually enables files to be split and processed in parallel across distributed systems. This article explains that underlying mechanism.
A block (e.g., Parquet Row Group, Avro Data Block) is a contiguous, self-contained data partition inside a single file. Formats like Parquet and Avro are serialization frameworks that leverage this block-based architecture to enable distributed data processing.
Crucially, the file stores an index (footer) that maps each block to its exact byte range e.g., Block 1: bytes 0–128 MB, Block 2: bytes 128–256 MB.
It is the block structure that enables splittability, not compression. Compression is simply applied within each block independently. The compression dictionary resets at every block boundary, ensuring no block shares compression state with another.
During read, the engine reads this index first. It now knows exactly where every block lives. It creates one processing task per block and assigns each task a specific byte range. Each task reads only its assigned slice from disk/object storage, decompresses it independently (using that block's own compression context), and processes it. No task depends on data from another block, neither the raw bytes nor the decompression state.
Without blocks, the file is just a continuous stream, there is no index, so the engine cannot safely split it. It must read from byte 0 to the end using a single thread. If you apply compression to that entire stream, it becomes even worse: the compression dictionary spans the whole file, making it truly unsplittable. Blocks solve this by containing compression within themselves. Splittability comes from the blocks; compression is just a feature they safely encapsulate.
The Real-World Math: 10 TB, 256 MB Blocks, and 1,000 Cores
Consider a 10 TB Parquet file stored in S3 or HDFS. You configure your write job with parquet.block.size = 256 MB. The resulting file contains roughly 40,000 independent blocks (10 TB / 256 MB ≈ 40,000). Each block has its own byte-range offset recorded in the footer, its own compression dictionary, and its own column statistics.
Now imagine you have a Spark cluster with 1,000 executor cores. When you submit a query, the driver reads the footer, extracts the 40,000 block offsets, and generates 40,000 file splits. The scheduler distributes these splits across the 1,000 cores, each core processes one block at a time, decompressing its 256 MB slice indepedently. With 40,000 blocks and 1,000 cores, you achieve full utilization for 40 sequential waves of parallel processing. The 10 TB file is processed in minutes, not hours.
Now contrast this with a CSV file of the same 10 TB, compressed with GZIP at the file level. There is no block index. The engine sees one continuous byte stream with a single compression dictionary spanning the entire file. It generates exactly one split. One core reads from byte 0 to the end, decompressing sequentially. The other 999 cores sit idle. A 10 TB file that could have been processed in parallel across the cluster now bottlenecks to the throughput of a single thread, taking days instead of minutes.
The block structure is the difference. It's not about compression ratios, schema evolution, or columnar storage (though those help too). It's about giving the engine a map: a set of independent, addressable partitions that translate directly into parallel tasks. When you choose a serialization format, look for the blocks. That is what truly unlocks your cluster's horsepower.
You may argue about reducer, cardinality & so on...that's a different story :)
At Zinzu, we’re solving some of the hardest problems in data. Learn more: https://zinzu.io