FB replication creates lots of files

[Florian Hector]

Hi,

I have set up some Windows (7, 10, 11) computer with the newest FB 5 version to asynchronously
replicate to a remote server.
All have the same settings regarding Journal directory, Archive directory, segment size and timeout,
only the journal_file_prefix is different for each installation.

Now, all of them except one behave as expected.
On one installation, the replicator creates a lot files in the archive directory whenever the
journal file gets moved from journal to archive. After that, the name for the next file in the
journal directory jumps from XXX.journal-000000001 to XXX.journal-000000131, then
XXX.journal-000000262 and so on. There are always 130 additional files with the exact same
timestamp, each one 1 MB in size whereas the amount of data to be replicated calls for a few KB at
most. This "small" file can also be found in the archive directory.
When all those files are moved to the replica side, the replicator incorporates the one small file
which is then deleted, the additional files are left in the the directory with the database file.

The log file on the master side does not show anything and the replication itself works properly.

Does anyone have any insight?

Florian

[Dimitry Sibiryakov]

Florian Hector wrote 21.05.2026 17:12:
> The log file on the master side does not show anything and the replication
> itself works properly.

Did you try to set "verbose_logging = true" in replication.conf?

--
WBR, SD.

[Florian Hector]

Dimitry,

yes I did...
This is the setting for the master side (same for all installations):
#
# Replication configuration
#
database
{
    ### PRIMARY SIDE SETTINGS

    journal_directory = c:\Utilities\Firebird5\DB\Journal\
    journal_file_prefix = XXX
    journal_segment_size = 524288 # 0,5MB
    journal_archive_directory = c:\Utilities\Firebird5\DB\Archive\
    journal_archive_timeout = 1500


    # Plugin used to perform replication.
    # Leave it empty to use built-in replication.
    #
.....

[Florian Hector]

Dimitry,

I added that line but have to check tomorrow for any entries

F.

[Florian Hector]

Dimitry,

could not get it to work, only after uninstalling Firebird and reinstall it, it seems to work now.
On a side note: Under Windows, which user needs write access for the Firebird installation directory?

Florian

[Dmitry Yemanov]

21.05.2026 18:17, 'Dimitry Sibiryakov' via firebird-support пишет:

>
>> The log file on the master side does not show anything and the
>> replication itself works properly.
>
>   Did you try to set "verbose_logging = true" in replication.conf?

It works on the replica side only.


Dmitry

[Florian Hector]

I thought the problem was solved after reinstalling FB5, but this only
lasted until FB was restarted.
Even when setting journal_segment_size to a very low value like 504288,
FB still creates about 130 additional files whenever the journal files
are transfered from journal to archive, each file 1 MB in size.
The daily growth of the database is only about 0.1 MB

It does not seem to make a difference whatever the settings in
replication.conf.

What can I do to find the culprit?

Florian

[Dimitry Sibiryakov]

Florian Hector wrote 05.06.2026 09:12:
> whenever the journal files
> are transfered from journal to archive, each file 1 MB in size.
> The daily growth of the database is only about 0.1 MB
>

> What can I do to find the culprit?

If you still wonder what is inside of these files, try this utility:

https://github.com/aafemt/fbdump/releases/tag/Initial

--
WBR, SD.

[Florian Hector]

Dimitry,

don't know what the utility should display, it behaves erratically: On some calls it only displays
some kind of header like this:

Replication journal d:\Temp\CPH.journal-000002044
        Signature: FBCHANGELOG
        Version: 1
        State: archived
        UUID: {19486DE8-6816-4BE6-85B2-4E76F72ACCA0}
        Sequence: 2044
        Length: 764581
Transaction number 13109, protocol 1, flags 0000 (), length: 764517
        Release savepoint
        Release savepoint
        Start savepoint
        Start savepoint
        Atom: TABOPCVALUES

Sometimes something like this:

Replication journal d:\Temp\CPH.journal-000002044
        Signature: FBCHANGELOG
        Version: 1
        State: archived
        UUID: {19486DE8-6816-4BE6-85B2-4E76F72ACCA0}
        Sequence: 2044
        Length: 764581
Transaction number 13109, protocol 1, flags 0000 (), length: 764517
        Release savepoint
        Release savepoint
        Start savepoint
        Start savepoint
        Atom: TABOPCVALUES
        Update record in TABOPCVALUES
                Old data length 1880
                * FC FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF * ................ *
                * FF FD FF FF 3F FF 20 00 E0 BF 9E FF FF 57 01 24 * ....?. ......W.$ *
                * 56 BF 07 C6 03 52 FC FC FF FD FF FF FF FF FF FF * V....R.......... *
                * FF FF FF FF F1 FF FB FF FF FF FF 0F 00 00 98 66 * ...............f *
                * 00 00 00 30 00 00 00 00 00 00 00 00 00 00 00 00 * ...0............ *
                * 00 00 00 00 00 00 00 00 B9 A9 05 00 00 00 00 00 * ................ *
                * 2A EF 00 00 C0 A1 AB 2E 00 00 00 00 00 00 00 00 * *............... *
                * 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 * ................ *
                * 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 * ................ *
                * 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 * ................ *
                * 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 * ................ *
                * 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 * ................ *
                * 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 * ................ *

But it always ends with this line:

Unexpected end of file: iostream error

By now, I have spent so much time on finding a solution w/o getting anywhere that I am ready to give
up, what prevents me from just living with it, is that I noticed that whenever the server does not
accept connections, it only happens after one of the clients has sent not one or two journal files
but something like 30 - 50.
Whenever that happens, the FB service must be restarted to get it working again.

I also tried the trace as suggested by Peter Bas Hofstede but the trace does not show what is
actually updated/inserted.

Any more insight is greatly appreciated

Florian

[Dimitry Sibiryakov]

Florian Hector wrote 07.07.2026 12:44:
> But it always ends with this line:
>
> Unexpected end of file: iostream error

Check that the file has length not less than one you see in the header
("Length: 764581"). If it is smaller - the file was truncated somehow.
If it is not the case, I would like to have one of such files for dissection.

--
WBR, SD.

[Florian Hector]

Her we go..

[Florian Hector]

Dimitry Sibiryakov schrieb am 2026-07-07 um 13:46:

...and another, smaller one

[Dimitry Sibiryakov]

Florian Hector wrote 07.07.2026 14:35:
>>
> Her we go..

Thanks.
Fixed version: https://github.com/aafemt/fbdump/releases/tag/v20260707

--
WBR, SD.

[Dimitry Sibiryakov]

Florian Hector wrote 07.07.2026 12:44:

> By now, I have spent so much time on finding a solution w/o getting anywhere
> that I am ready to give up, what prevents me from just living with it, is that I
> noticed that whenever the server does not accept connections, it only happens
> after one of the clients has sent not one or two journal files but something
> like 30 - 50.
> Whenever that happens, the FB service must be restarted to get it working again.

The only thing is clear: these journal files contain really a lot of small
updates in table TABOPCVALUES.
Data is stored in the journal unpacked, while in database it is packed.
Besides, perhaps the same record is updated multiple times. Because of this the
journal is much bigger than the database itself.

--
WBR, SD.

[Florian Hector]

'Dimitry Sibiryakov' via firebird-support schrieb am 2026-07-07 um 15:54:
>
>   The only thing is clear: these journal files contain really a lot of small updates in table
> TABOPCVALUES.
>   Data is stored in the journal unpacked, while in database it is packed. Besides, perhaps the
> same record is updated multiple times. Because of this the journal is much bigger than the
> database itself.
>

Dimitry,

yes, among the once read and written values from a PLC, there is an import from a text file which
grows throughout the day, so that many values are read multiple times and handled with an update or
insert statement.
Don't know if the replicator handles those updates regardless, despite the fact that those old
values are still the same.
I had assumed, that the replicator only handles "real" changes and new values.

Florian

[Dimitry Sibiryakov]

Florian Hector wrote 08.07.2026 20:42:
> I had assumed, that the replicator only handles "real" changes and new values.

Yes, the replicator ignores dummy updates, but in your case these updates are
real though small.
As far as I can see in the journal a single integer field is changed every time.

Nevertheless even a big amount of small changes in single transaction must
not hang Firebird server. This issue must be investigated separately.

--
WBR, SD.

[Florian Hector]

Dimitry,

how can you tell?
The only output I get is not readable for me in any way, something like this:

Replication journal d:\Temp\CPH.journal-000002044, length 764581

Do I need some additional app/program?

Florian

[Dimitry Sibiryakov]

Florian Hector wrote 08.07.2026 21:59:
> how can you tell?

I picked up several dumps of old value/new value into separated files and
compare them.
In the sample you quoted the difference is this:

2c2

< * FF FD FF FF 3F FF 20 00 E0 BF 9E FF FF 57 01 24 * ....?. ......W.$ *

---
> * FF FD FF FF 3F FF 20 00 E0 BF 1E FF FF 57 01 24 * ....?. ......W.$ *
60c60
< * 00 00 00 00 BF 08 00 00 7D 0C 00 00 00 00 00 00 * ........}....... *
---
> * 00 00 00 00 BF 08 00 00 7D 0C 00 00 28 36 00 00 * ........}...(6.. *

As you see, only three bytes are different between old and new record data.

You can even determine which exactly fields were changed if you look up in
RDB$FORMATS table the format for TABOPCVALUES
(https://firebirdsql.org/file/documentation/chunk/en/refdocs/fblangref50/fblangref-appx04-formats.html#fblangref-appx04-formats)
but I guess for you would be easier to find the stored procedure (or is it a
trigger?) that performs this update.

--
WBR, SD.