checkpoints deletion & extents lists merging

[Matan Tennenhaus]

Hi,

A question about the example described in the documentation in 'Checkpoint deletion and merging' here: https://source.wiredtiger.com/develop/arch-block.html

After merging checkpoint A lists to B i see 2 things:

1) block M disappeared form Avail list.

2) block L appeared on both the Avail and alloc, and removed from Avail.

both are not stated and explained in the example. I would much appreciate the explanation behind it.

[Matan Tennenhaus]

Another question in the matter, after reading here:
https://github.com/wiredtiger/wiredtiger/wiki/Block-Manager-Overview

• When deleting a checkpoint it's extent lists are merged into the next most recent checkpoints extent lists. Any blocks that are no longer needed (i.e those that are on the allocate list of the earlier checkpoint and the discard list of the newer checkpoint) are moved onto a special available list in the live checkpoint (called ckpt_avail). Once a checkpoint is really deleted, the ckpt_avail list is merged into the live checkpoints available list. The merge is implemented in the __wt_block_checkpoint_resolve method.

Assume like the example in the previous comment, that I have checkpoints A & B, and now create checkpoint C and remove checkpoint A.

1) In the example the "most recent" checkpoint is checkpoint B?

2) How the example corresponds with the second part explained here that the ckpt_avail list is merged into the live checkpoint and not to checkpoint B? I would also assume that this would happen.

3) what is the logic behind merging A and B lists after deleting A?

Sorry if i made a confusion. many thanks.

ב-יום רביעי, 10 ביולי 2024 בשעה 18:20:12 UTC+3, Matan Tennenhaus כתב/ה:

[Marc Butler]

Hi Matan,

I'd like to assure you that your question is being worked on. This question itself prompted me to look into the documentation and now code and try to come up with a diagrammatic representation of the scenario described in arch-block.html, with the hope of making the example clearer. 

Using feedback from the team to ensure it feels logical and accessible, it's taking a few iterations to find a suitable form for the diagram. So if I could ask your patience, a response will be forthcoming as soon as practical.

Regards,

Marc

[Matan Tennenhaus]

Many thanks.

One final question:

from the code documentation:

"Find blocks for re-use: wherever the "to" checkpoint's allocate and discard lists overlap, move the range to the live system's checkpoint available list."

Why do I need this overlapping? Isn't the fact that the block is in the deleted checkpoint's discard list enough to re-use it? Or even if maybe the overlapping must have happened, because it was allocated before and merged to the alloc list, why would I need this overlapping check still not continue without the check with the discards only?

ב-יום שישי, 12 ביולי 2024 בשעה 08:42:58 UTC+3, Marc Butler כתב/ה:

[Marc Butler]

Hi Matan,

The diagram is taking a little longer than expected as I iterate over it.

I think the term overlap only describes the notional relationship between the entries in the respective Alloc and Discard extent lists.

As a single extent entry will represent a contiguous region of blocks, extents in different lists describe overlapping regions of the file.

From the example:

Merge A + B

Alloc [IJK] + [L] => [IJKL]

Discard [] + [J] => [J]

So the extent [J] in the Discard list overlaps with extent [IJKL] in the Alloc list.

Regards,

Marc

--
You received this message because you are subscribed to a topic in the Google Groups "wiredtiger-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wiredtiger-users/x5eTvXOpBQs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wiredtiger-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wiredtiger-users/c6735779-5765-4ed5-bf95-2b91e7f7fc8bn%40googlegroups.com.

[Matan Tennenhaus]

Thanks for the answer.

First of all again appreciate the fact that you are preparing a detailed response.
About my last question, i did understand the overlapping definition, not the logic behind searching for overlapping, isn't just the fact the extent is in the discard list enough to move it to ckpt_avail for reuse after the checkpoint is resolved?

BTW, i did manage to get some answers for my questions from the notes inside the code, they are great (mainly in __ckpt_process). But, at this opportunity, one final thing that I did not understand at all from anywhere in the code, is how the extends are used in recovery. (If its too much unrelated to the rest of my questions, maybe just pointing out for me to the relevant functions in the code and a simple explanation is good).

Many Thanks,

Matan.

ב-יום שלישי, 16 ביולי 2024 בשעה 09:57:59 UTC+3, Marc Butler כתב/ה:

[Marc Butler]

Hi Matan,

Existence of an extent in the discard list is sufficient to mark the region as available, but *if* a Discard extent overlaps with an Alloc extent, the Alloc extent list will need to be updated.

My reading of the code and arch document: the extent [IJKL] from the merge will be split into 2 extents: [I] and [KL], corresponding to case #5 in __block_ext_overlap().

Regards, Marc

To view this discussion on the web visit https://groups.google.com/d/msgid/wiredtiger-users/778ffc82-8cf3-4eee-96df-4dde570f744dn%40googlegroups.com.

[Matan Tennenhaus]

Hi Marc,

Is there an update with the graphs?

Another question: in reconciliation wrapup, we clear and replace the page reconciliation information. As part of that, earlier blocks are freed from the block manager. Inside the block manager there are 2 options for the block: reuse it (merge to live avail) immediately, if it exists in the live alloc list, else merge it to the live discard list.

For my own use-case experiment, I am willing to know, in reconciliation wrapup, what is/will be the fate of the page previous blocks.

I this information accessible somehow / can be interpreted in someway from reconciliation context? Can I maybe know if a page was reconciled multiple times for the same live checkpoint?

Thanks,

Matan.

ב-יום רביעי, 17 ביולי 2024 בשעה 05:53:58 UTC+3, Marc Butler כתב/ה:

[Marc Butler]

Hi Matan,

Thank you for your patience. I don't think you'd believe how many revisions this diagram has gone through. :) 

I hope it is of help to you.

It will be integrated into the architecture guide in the future. Questions and feedback are welcome. 

This diagram depicts the example under "Checkpoint deletion and merging" in the "Block Manager Section" Version 11.3.0

The diagram attempts to clarify implicit details not captured in the example text.

Regards,

Marc

[Matan Tennenhaus]

Much appreciate! looks great and really made things clear :)

ב-יום חמישי, 15 באוגוסט 2024 בשעה 11:37:01 UTC+3, Marc Butler כתב/ה:

[Matan Tennenhaus]

BTW, I think you have a mistake. block J is not suppose to be in checkpoint B' avail list. (?)

The avail lists are static and do not change as far as I understand.

ב-יום ראשון, 18 באוגוסט 2024 בשעה 00:44:22 UTC+3, Matan Tennenhaus כתב/ה:

[Marc Butler]

Hi Matan,

Yes, that is a mistake.  I've attached the updated diagram.

Best regards,

Marc

You received this message because you are subscribed to the Google Groups "wiredtiger-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wiredtiger-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wiredtiger-users/79c21c19-5adc-4ad3-9e0a-8f5910460c09n%40googlegroups.com.