Is it normal that journal files get larger an larger until the environment close() is called?

[Michael Möllney]

Hi!

Thank you for your nice small database library. It looks compact and easy to understand/learn.

I'm looking at hamsterdb for logging a time series.
So I have a key that is a 64bit int (coding the time stamp) and record/value is arbirary in size (e.g. around 1.4 MByte each record).

I have one thread writing into a db with all increasing time stamps at a rate of 2 records per second.
And one thread reading from the db looking for a time interval (the last seconds) to deliver every key/record pair within this time interval.
So actually I append at the end and read from the end at nearly the same time .....

When looking at the db file and the jounal file I find the db file constantly growing and the journal file seems to nearly about twice as big growing, too.
I have let it run over some hours and I ended up in a db file with 75GByte of data and a journal file with a about 150GB of data.

I relized that when i call the close() method of the environment, the journal file is erased/shrunk.

Is this normal/intential behaviour, that the journal is about twice as large and not shrinking?
Is there is way beside closing the environment to have the journal file shrink again?

Would be glad to hear any idea from you...

Michael

[Christoph Rupp]

Hi Michael,

thanks for your mail, and thanks for asking before giving up :)

I was able to reproduce this bug with anonymous transactions - i.e. when you call ham_db_insert, the second parameter ("txn") is set to NULL. Is that the case in your application?

It looks like the problem does not exist if you explicitly use transactions, i.e. like this:
ham_txn_t *txn;
ham_txn_begin(&txn, env, 0, 0, 0);

ham_db_insert(db, txn, &key, &record, 0); // now use the txn pointer!

ham_txn_commit(txn, 0);

You can use this workaround till i've fixed the issue (hopefully early next week).

best regards
Christoph

--
You received this message because you are subscribed to the Google Groups "hamsterdb User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hamsterdb-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Möllney]

Am Mittwoch, 10. September 2014 22:05:16 UTC+2 schrieb Christoph Rupp:
> Hi Michael,
>
> thanks for your mail, and thanks for asking before giving up :)
>

If you get an answer that fast, it would be stupid to give up that early :-)

>
> I was able to reproduce this bug with anonymous transactions - i.e. when you call ham_db_insert, the second parameter ("txn") is set to NULL. Is that the case in your application?
>

Yes in deed, I used the version with automatic transaction....

>
> It looks like the problem does not exist if you explicitly use transactions, i.e. like this:
> ham_txn_t *txn;
> ham_txn_begin(&txn, env, 0, 0, 0);
>
> ham_db_insert(db, txn, &key, &record, 0); // now use the txn pointer!
>
> ham_txn_commit(txn, 0);
>

I will give this a try tomorrow and will report the outcome.

>
> You can use this workaround till i've fixed the issue (hopefully early next week).
>

This would be great!

> best regards
> Christoph
>
>

Kind regards,
Michael

[Michael Möllney]

Hi Christoph,

I introduce the manual transaction as you described.

I now can see two growing journal files that seem to have a maximum size of about 45306 kByte before they "collapse". These journal files alternate in growing: one file grows, while the other one is at 45306 kByte. And then the other one starts at 0 bytes growing up to this max value ...

So in worst case there is ~90MB used for journal files.

This is what seems to be intended, right?

Is there a way to tune this size of the journal files? There is no problem with the current size, but are there some aspects about responsiveness of the db regarding the journal file size?

Thanks again for your help,
Michael

[Christoph Rupp]

Hi Michael,

yes, this is the correct behavior. The two files switch. The size of the files is depending on the number of transactions; currently the switch is made every 16 transactions. If your transactions are very large then the files will be larger.

There is no parameter to adjust this size; it's hard coded. But it would not be much effort to change it. If you feel the need then just drop me a mail.

best regards
Christoph

[Michael Möllney]

Hi Christoph,

my thought are around the time needed to switch from one journal to the other.

Is there some delay in the responsiveness of database, when this switching occours. E.g. is there some reorganization that takes significant time.
I would prefer small high frequent interruptions over significant long not so frequent interruptions.

The background is, that I want to update a GUI with the newest/latest results from the time series in the database. If there is a long interruption due to reorganization the view will get stuck for some significant/recognizable time.

So if it would be possible to e.g.  set: switch after each transaction, and have only small impact on the database performance this might be preferable.....

Can you give me some idea, if there is any impact on the performance in this aspect...

Thank you,
Michael

Am Donnerstag, 11. September 2014 12:17:45 UTC+2 schrieb Christoph Rupp:

Hi Michael,

yes, this is the correct behavior. The two files switch. The size of the files is depending on the number of transactions; currently the switch is made every 16 transactions. If your transactions are very large then the files will be larger.

There is no parameter to adjust this size; it's hard coded. But it would not be much effort to change it. If you feel the need then just drop me a mail.

best regards
Christoph

[Christoph Rupp]

Hi Michael,

the switch does not cause performance issues.

If you start writing (i.e. begin a transaction, insert/erase, then commit), the first 16 transactions will be logged to .jrn0. The second 16 transactions are logged to .jrn1. etc.

Whenever the log file is switched, the new log file's size is truncated. There is no additional logic involved.

Best regards
Christoph

--
You received this message because you are subscribed to the Google Groups "hamsterdb User" group.

[Michael Möllney]

Hi Christopher,

Thank you for the additional notes.

I had a look at the journal code. It looks as if you changed the number of transactions per journal to 32 with the fix for issue 38.

If I understand you right this would double the size of my journal files?

Could there be a problem with performance, when the journal files are bigger than the cache of the hard disk where the journal files live on?

So maybe setting a minimum number of transactions and a maximum size of the journal files as environment parameter would be something that might have to be set to tweak the performance?

Looking at the code I wonder, why you have favored to use enums for class const int instead of a static const int construct. I have something in mind that enum is an integer type, but the size of the int is up to the compiler. And 1024*1024 is beyond 16bit int.... many this might be a problem on some embedded device compilers...

Anyway, thanks again for your quick reaction on this issue.

[Christoph Rupp]

Hi Michael,

just before i start: the journal/recovery code is currently in flux because of the issue you found. It's still going through some changes.

I did change the threshold for the log file switching to 32, but for a single transaction the counter is now decremented twice. In the end you should have the same size again. But i will introduce a new parameter for the threshold and then it's anyway configurable.

The basic journalling/logging process is like this:

All logical operations (ham_txn_*, ham_db_insert, ham_db_erase) are appended to the journal.

As soon as a transaction is committed, its operations are flushed to the btree. First, this happens in-memory only, and results in a list of modified database pages. This list is then appended to the journal (this is called a "Changeset"). As soon as the Changeset is written, the modified pages are written to disk.

In case of recovery, hamsterdb reads the log from end to beginning. It first retrieves the last operation in the log. If this is a Changeset then it is reapplied (because Changesets are idempotent). Then all Transactions which were not yet flushed as part of this Changeset are recreated. Transactions which were not committed are discarded, committed transactions are flushed to disk.

In the bug that you found the last Changeset is incomplete, and the trailer is missing. Therefore the recovery function fails to properly read the data.

When fixing this, i found a few other places which were not robust enough or need more test coverage. I'll take care of that for the next release.

Regarding enums: you are right, the compilers are guaranteed to use int, everything else is compiler specific. hamsterdb won't run on 16bit, but i'll watch out for other values which might exceed 32bit.

Have a nice weekend,

Christoph