OOM killer

[Tomasz Dubiel]

Hello.

We migrated one customer from FB 3.0 to FB 5.0. We use java application to make backups and to validate them.

Application used Jaybird 5.0.6 (we will update it, but we don't know whether it changes something). Application maked backups and test restores successfully, no problem.

When we migrated to FB 5.0.3, application causes OOM killer when restoring any database, even as small as 300 MB (firebird ate all 60 GB of RAM). We even updated FB to the latest snapshot 5.0.4, but nothing changed. Were there any changes in Jaybird since 5.0.6 which could cause such problem?

What's important, during migration I of course had to restore databases and I did it by Delphi application which uses Service Manager. No problem with that.

What is wrong?

Best regards,

Tomek.

[Mark Rotteveel]

On 04/03/2026 11:13, Tomasz Dubiel wrote:
> Hello.
> We migrated one customer from FB 3.0 to FB 5.0. We use java application
> to make backups and to validate them.
> Application used Jaybird 5.0.6 (we will update it, but we don't know
> whether it changes something). Application maked backups and test
> restores successfully, no problem.
> When we migrated to FB 5.0.3, application causes OOM killer when
> restoring any database, even as small as 300 MB (firebird ate all 60 GB
> of RAM). We even updated FB to the latest snapshot 5.0.4, but nothing
> changed. Were there any changes in Jaybird since 5.0.6 which could cause
> such problem?

Given you're still on Jaybird 5.0.6, how would changes since 5.0.6 be
relevant as a cause? Or do you mean if there are fixes that addressed
similar problems? In any case, not that I can see.

> What's important, during migration I of course had to restore databases
> and I did it by Delphi application which uses Service Manager. No
> problem with that.
> What is wrong?

Jaybird uses the services manager as well. Which form of backup/restore
do you use, remote (FBBackupManager) or streaming
(FBStreamingBackupManager)? And does your Delphi application use the
same form of restore and other settings?

In any case, beyond support for parallel workers (not used by default)
and isc_spb_expected_db, nothing much has changed in either class in the
past 8 or so years, and even then most of the work is handled by the
services manager, especially in the case of remote (FBBackupManager).

I'm not sure what could be different between Jaybird and Delphi use of
the services manager that would blow up memory in one case and not the
other.

If you have a reproducible case, I can take a look, but I guess the core
developers on firebird-devel might be of more help here.

Mark

--
Mark Rotteveel

[Tomasz Dubiel]

Yeah, I asked whether there are some potential fixes in next releases.

It's a first such case for us. We moved already some customers to FB 5.0 and this application works correctly, but there it uses Jaybird at least from 2023 year or older. It would be some FB problem, but why would restore from Delphi work ok and from Java wouldn't? I didn't test normal gbak, but for now I don't want to "blow up" customers' server again :-)

After consulting with customer I will test with the newest Jaybird and with this old version of application and I will report you back.

Best regards.

[Tomasz Dubiel]

I tested already with customers' databases on my server (Windows) with old, new and the newest application and everything works as expected. The problem is present on Ubuntu 24.04.4

[Tomasz Dubiel]

I tested on customers server with the newest Jaybird and problem is still present...

[Mark Rotteveel]

On 04/03/2026 16:07, Tomasz Dubiel wrote:
> I tested on customers server with the newest Jaybird and problem is
> still present...

That's not unexpected, because nothing has really changed in the way
Jaybird's backup managers work. I'm not sure how Jaybird could cause the
memory to blow up server-side (I mean, I know of some theoretical ones,
but those involve info requests, and not backup and restore).

But you haven't answered by question: are you using FBBackupManager or
FBStreamingBackupManager?

Mark
--
Mark Rotteveel

[Tomasz Dubiel]

I asked developers and they said they use  FIBBackupService from FibPlus library which is a wrapper for FireDAC. (or something like that :-))

[Mark Rotteveel]

On 04/03/2026 18:04, Tomasz Dubiel wrote:
> I asked developers and they said they use FIBBackupService from FibPlus
> library which is a wrapper for FireDAC. (or something like that :-))

I meant with Jaybird.

Mark
--
Mark Rotteveel

[Tomasz Dubiel]

I use org.firebirdsql.management.FBBackupManager.

[Mark Rotteveel]

On 04/03/2026 18:52, Tomasz Dubiel wrote:
> I use org.firebirdsql.management.FBBackupManager.

That does hardly more than telling Firebird Services Manager where to
find the files and to do the restore, and reading (verbose?) gbak
output. I'm not sure how Jaybird could have any effect on Firebird
memory compared to the Delphi application doing that same thing.

I'll try to check if maybe Jaybird request excessive buffer sizes
somewhere, but that will have to wait, as I'm in the middle of migrating
my main computer away from Windows.

Mark
--
Mark Rotteveel

[Tomasz Dubiel]

We set verbose to false. No output.

[Tomasz Dubiel]

Well, there was a bug in the code. In case of missing parameter, the application set parallel workers with value of pagesize (16384 :-D).

Jaybird is probably clean.

I ran gbak with value of parallel 12 while max in firebird.conf is 8:

gbak:use up to 12 parallel workers
gbak:transportable backup -- data in XDR format
gbak: backup file is compressed
gbak:backup version is 10
gbak: WARNING:Wrong parallel workers value 12, valid range are from 1 to 8

It looks like Firebird doesn't use maxparallel workers value, but the one provided in parameter.

[Tomasz Dubiel]

Yeah, it's definitely a bug in Firebird. I reproduce this with Java application and with simple gbak. I think that's some really scary bug. You can quite easily blow up the server.

[Tomasz Dubiel]

Do you agree with Vlad's answer?

https://github.com/FirebirdSQL/firebird/issues/8927

Do you consider it normal that gbak knows no limits and when provided irrational parameter values (on purpose or by mistake) it can blow up the server?

[Mark Rotteveel]

On 3/5/26 11:53, Tomasz Dubiel wrote:
> Well, there was a bug in the code. In case of missing parameter, the
> application set parallel workers with value of pagesize (16384 :-D).

Ouch :) In any case good to hear the cause was found.

> Jaybird is probably clean.
> I ran gbak with value of parallel 12 while max in firebird.conf is 8:
> gbak:use up to 12 parallel workers
> gbak:transportable backup -- data in XDR format
> gbak:backup file is compressed
> gbak:backup version is 10
> gbak: WARNING:Wrong parallel workers value 12, valid range are from 1 to 8
>
> It looks like Firebird doesn't use maxparallel workers value, but the
> one provided in parameter.

Gbak (standalone, or within the services manager) itself can use
multiple connections to restore the database in parallel, and AFAIK,
that part is not subject to the maximum parallel workers setting, but
yes, gbak probably should apply some upper limit.

Only the parameter (isc_dpb_parallel_workers) that such a connection
itself passes to the server applies that setting when worker connections
are requested by a connection (e.g. when gbak creates indexes).

That is, the MaxParallelWorkers setting only affects how many worker
connections a process will create for connections to a specific database
within that process, and not how many connections gbak itself will
create to restore the DB.

Mark

[Mark Rotteveel]

On 3/5/26 13:16, Tomasz Dubiel wrote:
> Do you agree with Vlad's answer?
> https://github.com/FirebirdSQL/firebird/issues/8927
> Do you consider it normal that gbak knows no limits and when provided
> irrational parameter values (on purpose or by mistake) it can blow up
> the server?

Well, from a formal perspective, it's not a bug: it is documented
behaviour. From a usability perspective, having some upper limit
(hardcoded, or maybe derived from e.g. (CPU count + 1) of the machine
running gbak) would be desirable.

I agree with Vlad that using MaxParallelWorkers to configure gbak when
run from the Services Manager would overload the meaning, but more
importantly, it would be inconsistent if the standalone binary doesn't
read it. But if standalone gbak would also read it, you would get the
problem that the -PARALLEL switch then wouldn't have the desired effect
(running the restore in parallel) if the binary is placed outside the
Firebird directory: without access to firebird.conf, the default of
MaxParallelWorkers is 1 (i.e. no parallelism), or you'd need to have
multiple defaults, which could be confusing (overloading the meaning).

And although it is a footgun, it is not a security issue: restoring a
database requires administrator privileges or at least additional system
privileges like - IIRC - CREATE_DATABASE and USE_GBAK_UTILITY. Besides,
even a "normal" user would be able to blow up your server in a similar
way by creating a lot of connections (say 16384 ;-)) and maybe selecting
a few blobs with casts to a different subtype or character set, or doing
something else that consumes some additional memory.

Mark