Mnesia files

[Frank Muller]

Hi Mnesia users/experts,

Could someone pls explain me the roles of these files: LATEST.LOG, DAT, DCT and DCL?

Read the doc multiple times but couldn’t fully grasp their roles:

https://erlang.org/doc/apps/mnesia/Mnesia_chap7.html#log-file

What are they for?

What happens if in the middle of many transactions I unplug the electrical cable, put it back and restart Mnesia. How these files will help to recover the state.

Finally which ones are important to keep and which aren’t?

Thanks

[Mikael Pettersson]

On Wed, Apr 14, 2021 at 6:46 PM Frank Muller <frank.mu...@gmail.com> wrote:
>
> Hi Mnesia users/experts,
>
> Could someone pls explain me the roles of these files: LATEST.LOG, DAT, DCT and DCL?

DCT? Did you mean DCD?

> Read the doc multiple times but couldn’t fully grasp their roles:
> https://erlang.org/doc/apps/mnesia/Mnesia_chap7.html#log-file
>
> What are they for?

DCD and DCL files represent disc_copies tables. The DCD is an image of
the contents from the latest time the table was "dumped", while the
DCL contains a log of the side-effects made to that table since it was
dumped. A dump creates a new DCD and removes the DCL.

DAT files are DETS:es which contain disc_only_copies tables.

SCHEMA.DAT is a special DETS that contains the schema for that Mnesia instance.

LATEST.LOG is Mnesia's transaction log, which is periodically flushed
to the updated tables.

> What happens if in the middle of many transactions I unplug the electrical cable, put it back and restart Mnesia. How these files will help to recover the state.

See above.

> Finally which ones are important to keep and which aren’t?

They are all important.

[Frank Muller]

Many thanks Mikael. Each file role is clear now. 

But I still don’t get how the system recover from a failure. What Mnesia does in that case?

I suppose it will check the transaction logs first. 

Please elaborate. 

[Jesper Louis Andersen]

On Wed, Apr 14, 2021 at 11:14 PM Frank Muller <frank.mu...@gmail.com> wrote:

Many thanks Mikael. Each file role is clear now. 

But I still don’t get how the system recover from a failure. What Mnesia does in that case?

I suppose it will check the transaction logs first. 

Please elaborate. 

To be precise: we want the database to have ACID properties around transactions. You can drill this even more down and ask what kind of consistency level you want, see e.g., Kyle Kingsbury: https://jepsen.io/consistency - so ACID is a bit hand-wavy but it'll do.

If we lose power in the middle of a transaction, we want the database to come back either with the transaction comitted or not. We don't want a halfway transaction. We could achieve this by writing all our intentions down in a log, together with commit points. When the system then boots again, we replay the log from the genesis point up until the last good transaction commit point. Everything after that point, we discard and regard as not having happened.

However, that log might grow very large. So we want a way to circumvent indefinite growth. We can do this by keeping a snapshot of our tables on disk. Then we only need to keep a global log from the last snapshot point and forward rather than having to keep the full log available for replay. But taking snapshots is a relatively expensive operation too. So we keep a "local" log for each table. The current state is the snapshot + a replay of the local log. New transactions enter the global log first, and are then flushed out to the local logs. The next dump folds the local log into a new snapshot, so we can delete the local log and create a new local log for future changes.

Getting this to work is quite complicated, and it isn't made simpler by the fact that mnesia can also run in a distributed mode, where data is replicated among several nodes, and you have to handle several impossibility results from distributed computing.

[Frank Muller]

Hello Jesper,

[..] If we lose power in the middle of a transaction, we want the database to come back either with the transaction comitted or not. We don't want a halfway transaction. We could achieve this by writing all our intentions down in a log, together with commit points. When the system then boots again, we replay the log from the genesis point up until the last good transaction commit point. Everything after that point, we discard and regard as not having happened.

By log here you refer to LATEST.LOG, right?

However, that log might grow very large. So we want a way to circumvent indefinite growth. We can do this by keeping a snapshot of our tables on disk. Then we only need to keep a global log from the last snapshot point and forward rather than having to keep the full log available for replay. But taking snapshots is a relatively expensive operation too. So we keep a "local" log for each table.

How these local logs are named?

The current state is the snapshot + a replay of the local log. New transactions enter the global log first, and are then flushed out to the local logs. The next dump folds the local log into a new snapshot, so we can delete the local log and create a new local log for future changes.

Got it. 

Getting this to work is quite complicated, and it isn't made simpler by the fact that mnesia can also run in a distributed mode, where data is replicated among several nodes, and you have to handle several impossibility results from distributed computing.

Where can I find more info on the Mnesia internals besides reading the code?

[Mikael Pettersson]

On Thu, Apr 15, 2021 at 7:23 PM Frank Muller <frank.mu...@gmail.com> wrote:
>> However, that log might grow very large. So we want a way to circumvent indefinite growth. We can do this by keeping a snapshot of our tables on disk. Then we only need to keep a global log from the last snapshot point and forward rather than having to keep the full log available for replay. But taking snapshots is a relatively expensive operation too. So we keep a "local" log for each table.
>
>
> How these local logs are named?

For disc_copies tables those are the DCL files that were mentioned
before. AFAIK there's no such local log for disc_only_copies tables.

[Ulf Wiger]

On Thu, Apr 15, 2021 at 7:24 PM Frank Muller <frank.mu...@gmail.com> wrote:

Where can I find more info on the Mnesia internals besides reading the code?

There is a text doc included in the mnesia/doc directory:

https://github.com/erlang/otp/blob/master/lib/mnesia/doc/misc/implementation.txt

BR,

Ulf W 

[Frank Muller]

Thanks, I'll check it out.

<u...@wiger.net> a écrit :