PME in Modular footer

[Jiayi Wang]

Hi everyone,

I'd like to raise a question for the next sync.

PME has two encryption modes: footer encryption, where a single footer key encrypts every module, and per-column encryption, where each column can use its own key.

Footer encryption is straightforward: one key encrypts everything, as before.

Per-column encryption is trickier. A column's ColumnMetaData bundles two kinds of information with very different sensitivity: location (data/dictionary page offsets and sizes) and statistics (min/max/null count). Today PME encrypts the entire ColumnMetaData as a single module under one key.

In the modular footer, though, location and statistics are separate modules, and we want to store location as a struct-of-arrays. That makes per-column encryption impossible for location: a single column's offset is just one element inside a shared array, and we can't encrypt an individual element of a Thrift list under a column-specific key.

I think the reasonable resolution is to keep location in plaintext and only encrypt statistics. Location leaks little on its own, for example, if only one column is encrypted, its offset and size can already be reconstructed from the surrounding columns' metadata, so encrypting it buys almost nothing. Statistics (min/max) are what actually reveal data content, so that's where encryption matters.

In Databricks, there is almost no usage of PME, we have very little understanding of the real PME user cases, so let me know if you see it differently.

Best,
Jiayi

[Rok Mihevc]

# ccing Adam Reeve who might have a more nuanced opinion

I think location should be fine to exclude from PME.

Rok

[Adam Reeve]

Yes I agree the location data doesn't need to be encrypted with the column keys. Although when you say the location would be in plaintext, it would actually be encrypted by the footer key right? Unless a plaintext footer is enabled.

Thanks,

Adam

[Jiayi Wang]

Yes, the location is also encrypted when a footer key is used. It is only stored in plaintext when a column key is used and not footer key.

[Alkis Evlogimenos]

I think the reasonable resolution is to keep location in plaintext and only encrypt statistics. Location leaks little on its own, for example, if only one column is encrypted, its offset and size can already be reconstructed from the surrounding columns' metadata, so encrypting it buys almost nothing. Statistics (min/max) are what actually reveal data content, so that's where encryption matters.

I agree with this assessment.

On Wed, Jul 1, 2026 at 10:32 PM Jiayi Wang <jiayi...@databricks.com> wrote: