Undo/Redo log internals - deletion/invalidation process

[Dimitris Stavrakakis]

Hello everyone,

I would like to ask one question regarding libpmemobj internals. I am investigating its logging system.

I have understood the way the undo/redo logs are being constructed and when each one of these is applied.

My question has to do with the deletion/invalidation of these logs.

From my point of understanding, the redo long is no longer needed when it has been published.

Therefore, in operation_process_persistent_redo function, the last step is to clobber the first entry by filling it with zeros apart from the next field. 

Then, in the operation_finish function, the ulog_free_next function is called to zero out the checksum of each saved redo log header in order to invalidate it.

Almost the same applies for the undo log apart from the fact that the generation number is incremented in the header in ulog_clobber_data to invalidate the log. 

After this step, the ulog_free_next is called once again to invalidate the rest of the connected logs.

The valid undo logs are also being freed during the recovery in the operation_finish function that is called for each one anyway.

I would like to ask you, in case of a crash during the ulog_free_next execution for the redo log,

will this interrupted action take place on recovery or we do not really care to actually free the "next" redo 

logs as their space will be reused afterwards no matter what we do?

Correct me if I have misunderstood any of the aforementioned steps.

Thank you a lot for your help.

I am looking forward to your reply.

Kind regards,

Dimitris

[Dimitris Stavrakakis]

Correction: in the ulog_free_next function, the "next" field is set to 0, not the checksum in order to break the "chain" of the logs till there.

[ppbb...@gmail.com]

I truly admire your resolve in examining this mechanism in libpmemobj :) ulog handling is one of those cases where readability took a hit due to performance optimizations. It's one of the abstraction layers we'd like to revisit eventually.

You describe it all correctly. The redo logs are not freed in recovery because a) we assume they are small, and b) they will be reused anyway.

It doesn't save us *that* much time during startup, but that's the tradeoff in the current code. But overall, I don't think it would be noticeable either way. Let me know if you think otherwise.

[Dimitris Stavrakakis]

Hello,

thanks for the quick response.

I see your point. It sounds quite rational. 

And it's also error-free since in every ulog extension, the new ulog_header is being initialiased prior to updating the next field of the to-be-extended log 

(pmalloc_construct calls the log constructor and updates the next field with the prerequisite that the constructor has been executed successfully).

That's also correct I suppose.

If I have further questions regarding this topic I will keep updating this post.

Thank you once again.

Dimitris

[Dimitris Stavrakakis]

Hello once again,

I have one more question for the undo log this time.

If we invalidate only the first undo log header by incrementing the generation number by one and right after this there is a system crash,

on recovery, the undo logs are not going to be freed at all and will remain there as the ctx->total_logged calculated in operation_resume function is 0.

Thus, we do not reach the OPERATION_CLEANUP state and do not free up the remaining undo logs.

Is this the intended behaviour?

The reason I am asking is because, in contrary to the redo log, the undo one can consume a bigger amount of PM.

I am looking forward to your response,

Thank you very much.

Dimitris

[ppbb...@gmail.com]

No, that's not intentional, we should fix that. Thanks for being diligent :)

[Dimitris Stavrakakis]

I am glad it helps.
For any further findings-weird behaviours-question, I will you updated.

Thanks.