Fwd: Parquet footer thrift definitions

[Jiayi Wang]

---------- Forwarded message ---------
From: Will Edwards <wedw...@spotify.com>
Date: Thu, Jun 25, 2026 at 8:47 AM
Subject: Re: Parquet footer thrift definitions
To: Jiayi Wang <jiayi...@databricks.com>
Cc: <rok.m...@gmail.com>, <apache-parquet-fo...@google.com>, Divjot Arora <div....@databricks.com>, Alkis Evlogimenos <alkis.ev...@databricks.com>

Hi folks,

Attached are the schema and a test file for the jump table I prototyped.

It uses packed byte arrays for O(1) access, but they reside in the optional footer index. All readers unfamiliar with or not using the index will simply ignore it.

There are some non-obvious elements learned from actually implementing a fast reader; e.g. the address of the end of a row group is also stored, as is the position of all the top-level elements.  With this schema a fast parser with the same public API shape as parquet-mr can be constructed.

This jump table solves the super-fast O(1) access to a projection within a thrift footer.

Other changes we suggest as part of this solution include:

1. Make path_in_schema optional
2. Tweak the implementation of mainstream Parquet writers to omit stats on columns that do not obviously benefit from them

These changes will shrink the footer, resulting in much less data to download etc.

Together with a jump table these changes would constitute Option 2 (maximum compatibility with existing readers) that we discussed.

best,
Will

On Wed, 24 Jun 2026 at 18:47, Jiayi Wang <jiayi...@databricks.com> wrote:

Hi all,

As discussed yesterday, I want to share the 3 thrift definitions of footer:
1. the current footer
2. the current footer without duplicate fields and with the jump table
3. the new footer

Thanks again for helping with the benchmark. Let me know if you have any questions.

Best,
Jiayi

[Jiayi Wang]

Hi Rok,

Thanks for putting together this microbenchmark. It looks great!

One thing I found slightly misleading is that the core part of the modular footer does not include column chunk statistics yet. To make the comparison fair, I removed the statistics from all three footer definitions.

The benchmark also includes a stat bytes dimension, which I do not think has any effect on any of the three footer formats in the current setup.

Other than that, it looks good to me!

Best,
Jiayi

On Sat, Jun 27, 2026 at 2:38 AM Rok Mihevc <rok.m...@gmail.com> wrote:

Hey all,

I ran a set of benchmark sweeps with soa, jumptable and proposed footer formats Jiayi provided. Deserializaion and serialization. Linked is a self contained notebook written with help of codex. All benchmarks are lies, so take this with a grain of salt! : https://gist.github.com/rok/fb11b94902b56448c84510c139c12d81

tldr: soa wins, jumptable is good too.

Rok

[Rok Mihevc]

> To make the comparison fair, I removed the statistics from all three footer definitions.

Perfect! Could you send the files please and I'll rerun the notebooks.

Also to comment on the results a little bit - benchmarks were produced using a completely different thrift library without parquet-related optimizations and we should only think about results in terms of directional improvement different data structures give us.

So asymptotic analysis yes, but when comparing different footers designs we should be more careful.

Rok

[Will Edwards]

Yeah I checked the numbers and code and the approach and results look sound  :thumbs up:

[Alkis Evlogimenos]

The reduction in bytes from SoA seems quite high. Are we sure we didn't put some really small offsets in there that favor uleb/varint encoding too much?

[Rok Mihevc]

> The reduction in bytes from SoA seems quite high. Are we sure we didn't put some really small offsets in there that favor uleb/varint encoding too much?

That could be, I'll check.

Also this is measured using https://github.com/Thriftpy/thriftpy2 thrift library directly in Python and any number of things could be going wrong. We really want instrument a real parquet footer writer to get the simulated workload as close to the real problem as possible. Let's not read too far into benchmarks just yet!

Rok

[Jiayi Wang]

The original three Thrift definitions I sent already had the statistics removed. So, if you are testing with those Thrift files, the results should already be without statistics.

[Rok Mihevc]

Thrift definitions from this thread were used.

[Jiayi Wang]

Hi Rok,

It looks like the notebook does not show the change in footer size. The only comparison displayed is the write cost in terms of time.

It would be interesting to see how much space the SoA encoding specifically saves compared with the optimized jump-table format after redundant fields are removed.

Best,
Jiayi

[Rok Mihevc]

> It would be interesting to see how much space the SoA encoding specifically saves compared with the optimized jump-table format after redundant fields are removed.

I can add metric, I'll take a look early next week.

[Jiayi Wang]

Hi everyone,

I made a few small updates to the notebook:

1. Removed the statistics section, since it is not part of the core footer.
2. Updated the notebook to use more realistic values for page header and column chunk sizes, instead of small fixed sizes.
3. Added a footer size section showing the footer size for all three options.

The write cost benchmark is still a bit questionable because we currently call serialize on each ColumnChunkMetadata to get its size, although this could be implemented in a nicer way. In general, we are not very concerned about writer speed unless there is a significant change.

Best,
Jiayi

[Rok Mihevc]

This looks great and about what we would expect.

Rok