What makes WT need reconciliation compared to Innodb ?

[bai...@gmail.com]

Hi.

I try to find the original source about `reconciliation` in database fields, but no result (Obviously [1] is not what WT means). So,

1. Who was the first one that coined the term ?

2. Compared to Innodb, why does WT need reconciliation ? (I guess Innodb page is fixed and just synced to disk pages, whereas WT's page is not)

[1] https://docs.oracle.com/cd/E64763_01/html/Upgrade_Tool_Kit/UTKRCN07.htm

[Keith Bostic]

On Wednesday, February 3, 2021 at 3:42:41 AM UTC-8 bai...@gmail.com wrote:

Hi.

I try to find the original source about `reconciliation` in database fields, but no result (Obviously [1] is not what WT means). So,

1. Who was the first one that coined the term ?

Not really sure, and I'm not sure if it's unique to WiredTiger or not.

 

2. Compared to Innodb, why does WT need reconciliation ? (I guess Innodb page is fixed and just synced to disk pages, whereas WT's page is not)

Database engines have more-or-less differences in the on-disk representation vs. the in-memory representation of the data. Some need make only small changes when writing cached data to disk, others (like WiredTiger), have in-memory representations that very different from their on-disk representations. What WiredTiger calls "reconciliation" is simply the process of converting the in-memory representation to the on-disk representation.

 

[1] https://docs.oracle.com/cd/E64763_01/html/Upgrade_Tool_Kit/UTKRCN07.htm

[bai...@gmail.com]

Thanks for replying. 

Then what's the benefit of making in-memory representation differ a lot from on-disk representation ?  I guess reducing storage overhead may not be its goal, since compression does the job.

I search the git commit history of wiredtiger project,  and find reconciliation could be dated back to 2010. (https://github.com/wiredtiger/wiredtiger/commit/6782f0ba0)

So it seems to be an intrinsic thing in developer's mind since its beginning. What fundamental thing that make reconciliation an essential part of wiredtiger ?

[baoshang young]

I think it maybe gets better concurrency to make in-memory representation differ from on-disk representation. In a memory page, it has linkedlists and skiplists. Writer will not block reader for the same key.

[Keith Bostic]

On Wednesday, February 3, 2021 at 11:17:09 PM UTC-8 zaor...@gmail.com wrote:

I think it maybe gets better concurrency to make in-memory representation differ from on-disk representation. In a memory page, it has linkedlists and skiplists. Writer will not block reader for the same key.

Yes, exactly right.