Libtommath error code -3 in function mp_mod(&t, const_cast<mp_int*>(&val.t), &t)

[Sergey Khalyutin]

After own build Firebird (on 19.02.2026 3 commits behind FirebirdSQL/firebird:master.) with CMake+Clang under windows (no this error onder Linux) we have error  with a 90% probability (sometimes there is no error and create user ok)  :

isql -user sysdba employee
Database: employee, User: SYSDBA
SQL> create user SYSDBA password 'masterkey';
Statement failed, SQLSTATE = 2F000
Libtommath error code -3 in function mp_mod(&t, const_cast<mp_int*>(&val.t), &t)
SQL> exit;

Can anybody suggest a way to determine the cause of the error?  Please help.

  /* is divisor zero ? */
  if (mp_iszero (b) == 1) {
    return MP_VAL;             <--------
  }

Srp.dll!mp_div(mp_int * a, mp_int * b, mp_int * c, mp_int * d) Line 110 (extern\libtommath\bn_mp_div.c:110)

Srp.dll!mp_mod(mp_int * a, mp_int * b, mp_int * c) Line 29 (extern\libtommath\bn_mp_mod.c:29)

Srp.dll!Firebird::BigInteger::operator%=(const Firebird::BigInteger & val) Line 169 (src\common\BigInteger.cpp:169)

Srp.dll!Auth::RemotePassword::RemotePassword() Line 77 (src\auth\SecureRemotePassword\srp.cpp:77)

Srp.dll!Auth::RemotePasswordImpl<Firebird::Sha1>::RemotePasswordImpl() Line 132 (src\auth\SecureRemotePassword\srp.h:132)

Srp.dll!Auth::SrpManagement::SrpManagement(Firebird::IPluginConfig * par) Line 60 (src\auth\SecureRemotePassword\manage\SrpManagement.cpp:60)

Srp.dll!Firebird::SimpleFactoryBase<Auth::SrpManagement>::createPlugin(Firebird::CheckStatusWrapper * status, Firebird::IPluginConfig * factoryParameter) Line 174 (src\common\classes\ImplementHelper.h:174)

Srp.dll!Firebird::IPluginFactoryBaseImpl<Firebird::SimpleFactoryBase<Auth::SrpManagement>,Firebird::CheckStatusWrapper,Firebird::IVersionedImpl<Firebird::SimpleFactoryBase<Auth::SrpManagement>,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IPluginFactory>>>::cloopcreatePluginDispatcher(Firebird::IPluginFactory * self, Firebird::IStatus * status, Firebird::IPluginConfig * factoryParameter) Line 9055 (src\include\firebird\IdlFbInterfaces.h:9055)

isql.exe!Firebird::IPluginFactory::createPlugin<Firebird::CheckStatusWrapper>(Firebird::CheckStatusWrapper * status, Firebird::IPluginConfig * factoryParameter) Line 796 (src\include\firebird\IdlFbInterfaces.h:796)

isql.exe!`anonymous namespace'::ConfiguredPlugin::factory(Firebird::IFirebirdConf * firebirdConf) Line 610 (src\yvalve\PluginManager.cpp:610)

isql.exe!`anonymous namespace'::PluginSet::getPlugin(Firebird::CheckStatusWrapper * status) Line 1009 (src\yvalve\PluginManager.cpp:1009)

isql.exe!Firebird::IPluginSetBaseImpl<(anonymous namespace)::PluginSet,Firebird::CheckStatusWrapper,Firebird::IReferenceCountedImpl<(anonymous namespace)::PluginSet,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IVersionedImpl<(anonymous namespace)::PluginSet,Firebird::CheckStatusWrapper,Firebird::Inherit<Firebird::IPluginSet>>>>>::cloopgetPluginDispatcher(Firebird::IPluginSet * self, Firebird::IStatus * status) Line 8428 (src\include\firebird\IdlFbInterfaces.h:8428)

Engine14.dll!Firebird::IPluginSet::getPlugin<Firebird::CheckStatusWrapper>(Firebird::CheckStatusWrapper * status) Line 518 (src\include\firebird\IdlFbInterfaces.h:518)

Engine14.dll!Firebird::GetPlugins<Firebird::IManagement>::getPlugin() Line 166 (src\common\classes\GetPlugins.h:166)

Engine14.dll!Firebird::GetPlugins<Firebird::IManagement>::GetPlugins(unsigned int iType, const Firebird::Config * conf, const char * namesList) Line 73 (src\common\classes\GetPlugins.h:73)

Engine14.dll!Auth::Get::Get(const Firebird::Config * firebirdConf, const char * plugName) Line 57 (src\common\security.cpp:57)

Engine14.dll!Jrd::UserManagement::getManager(const char * name) Line 239 (src\jrd\UserManagement.cpp:239)

Engine14.dll!Jrd::UserManagement::execute(unsigned short id) Line 370 (src\jrd\UserManagement.cpp:370)

Engine14.dll!user_management(Jrd::thread_db *, short phase, Jrd::DeferredWork * work, Jrd::jrd_tra * transaction) Line 6492 (\build\code\src\jrd\dfw.epp.master.cpp:6492)

Engine14.dll!DFW_perform_work(Jrd::thread_db * tdbb, Jrd::jrd_tra * transaction) Line 5352 (\build\code\src\jrd\dfw.epp.master.cpp:5352)

[marius adrian popa]

The error code -3 from the Libtommath library, as seen in the function call mp_mod, typically corresponds to an Out of Memory error, denoted by the constant MP_MEM.

The stack trace in the email indicates the error is occurring during user creation (create user SYSDBA...) within the Secure Remote Password (SRP) authentication mechanism in Firebird. The specific function call is deep within the SRP module, which uses Libtommath for cryptographic operations.

Here are the most likely causes and suggested solutions:

• Out of Memory (MP_MEM): The cryptographic operation in mp_mod requires temporary memory allocation. The -3 error code suggests this allocation failed.
  • Solution: Given the intermittent nature of the error (90% probability), this could be due to memory fragmentation or a temporary lack of available memory in the build environment, particularly when the process is running. Ensure your system has sufficient RAM and that no other memory-intensive processes are running during the test.
• Build/Configuration Issue with Clang on Windows: The original email notes that the error does not occur under Linux. This suggests the issue is specific to the CMake/Clang/Windows build configuration.
  • Solution: Since the error is specific to your custom build, consider testing with an official or verified build environment to confirm if the issue is with the Firebird source or your local build process.
• Bug in Firebird's SRP implementation: The error occurs in the Auth::RemotePassword::RemotePassword() constructor, which means the issue is within the code that initializes the SRP module.

--
Support the ongoing development of Firebird! Consider donating to the Firebird Foundation and help ensure its future. Every contribution makes a difference. Learn more and donate here:
https://www.firebirdsql.org/donate
---
You received this message because you are subscribed to the Google Groups "firebird-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebird-deve...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/firebird-devel/b5084769-824d-42bd-bbed-4fc5692c530fn%40googlegroups.com.

[Mark Rotteveel]

This sounds like AI slop without real knowledge or information. Please
don't just dump the output of AI as an answer.

Mark

On 19/02/2026 09:28, marius adrian popa wrote:
> The error code |-3| from the Libtommath library, as seen in the function

> call |mp_mod|, typically corresponds to an *Out of Memory* error,

> denoted by the constant |MP_MEM|.
>
> The stack trace in the email indicates the error is occurring during

> user creation (|create user SYSDBA...|) within the Secure Remote

> Password (SRP) authentication mechanism in Firebird. The specific
> function call is deep within the SRP module, which uses |Libtommath| for
> cryptographic operations.
>
> Here are the most likely causes and suggested solutions:
>

> * *Out of Memory (|MP_MEM|):* The cryptographic operation in |mp_mod|

>  requires temporary memory allocation. The |-3| error code suggests
> this allocation failed.

> o *Solution:* Given the intermittent nature of the error (90%

> probability), this could be due to memory fragmentation or a
> temporary lack of available memory in the build environment,
> particularly when the process is running. Ensure your system has
> sufficient RAM and that no other memory-intensive processes are
> running during the test.

> * *Build/Configuration Issue with Clang on Windows:* The original

> email notes that the error does not occur under Linux. This suggests
> the issue is specific to the CMake/Clang/Windows build configuration.

> o *Solution:* Since the error is specific to your custom build,

> consider testing with an official or verified build environment
> to confirm if the issue is with the Firebird source or your
> local build process.

> * *Bug in Firebird's SRP implementation:* The error occurs in the |

> Auth::RemotePassword::RemotePassword()| constructor, which means the
> issue is within the code that initializes the SRP module.

--
Mark Rotteveel

[Dimitry Sibiryakov]

'Mark Rotteveel' via firebird-devel wrote 19.02.2026 9:31:
> This sounds like AI slop without real knowledge or information.

And it is. Typical buzz from GitHub Copilot. Error -3 is MP_VAL (as indicated
in the starting post as well), not MP_MEM (-2).

--
WBR, SD.

[Sergey Khalyutin]

About build host - all latest ...

Processor Intel(R) Core(TM) i7-14700F (2.10 GHz)  20 core

Installed RAM 64.0 GB (63.8 GB usable)
System type 64-bit operating system, x64-based processor

Edition Windows 11 Pro
Version 25H2
Installed on ‎19/‎12/‎2025
OS build 26200.7840
Experience Windows Feature Experience Pack 1000.26100.291.0

MS SDK  Microsoft SDKs\Windows Kits\10\ExtensionSDKs\Microsoft.Midi.GmDls\10.0.26100.0

Clang  21

[Dimitry Sibiryakov]

Sergey Khalyutin wrote 19.02.2026 10:01:
> About build host - all latest ...

> ProcessorIntel(R) Core(TM) i7-14700F (2.10 GHz)  20 core
> Installed RAM64.0 GB (63.8 GB usable)

> System type64-bit operating system, x64-based processor
> EditionWindows 11 Pro
> Version25H2
> Installed on‎19/‎12/‎2025
> OS build26200.7840

> ExperienceWindows Feature Experience Pack 1000.26100.291.0

>
> MS SDK  Microsoft SDKs\Windows
> Kits\10\ExtensionSDKs\Microsoft.Midi.GmDls\10.0.26100.0
> Clang  21

Didn't you say that it was Linux?..
In any case you have call stack, so you can use debugger. Look if prime value
is really zero or compiler optimization played a dirty trick around const_cast.

--
WBR, SD.

[Sergey Khalyutin]

In Linux the command is executed correctly, the problem under windows

RemotePassword::RemotePassword()

    : group(RemoteGroup::getGroup())

{

#if SRP_DEBUG > 1

    privateKey = BigInteger("XXXXXXX hidden for publish XXXXXX");

#else

    privateKey.random(RemotePassword::SRP_KEY_SIZE);

#endif

    privateKey %= group->prime; <-- probably zero appears here

}

...

class RemoteGroup

{

public:

    BigInteger  prime, generator, k;

    explicit RemoteGroup(Firebird::MemoryPool&)

        : prime(primeStr), generator(genStr), k() <-- init here

    {

        Auth::SecureHash<Firebird::Sha1> hash;

I don't understand  next  ...

-> "is really zero or compiler optimization played a dirty trick around const_cast." 

That's an idea, thank you

[Dimitry Sibiryakov]

Sergey Khalyutin wrote 19.02.2026 10:50:
>         : prime(primeStr), generator(genStr), k() *<-- init here*

Yes, the only problem that this initialization is on demand and can be not
called in your case.

--
WBR, SD.

[Alex Peshkoff]

On 2/19/26 12:50, Sergey Khalyutin wrote:

    privateKey = BigInteger("XXXXXXX hidden for publish XXXXXX");

No danger publishing this constant :-)
It is anyway:
- present in open source file,
- used only for debugging.


Did you try Debug build? Almost all optimizations turned off in it, i.e. it's a simple way to determine is it compiler related or not.

[Sergey Khalyutin]

Unfortunately no positive effect with add_compile_options(-O0 -Xclang -disable-llvm-optzns); Error is unstabile.

after full rebuild with  -O0 -Xclang -disable-llvm-optzns:

D:\source\COMPONENTS\SQLfb\build\code\firebird>isql -user sysdba employee

Database: employee, User: SYSDBA
SQL> create user SYSDBA password 'masterkey';
Statement failed, SQLSTATE = 2F000
Libtommath error code -3 in function mp_mod(&t, const_cast<mp_int*>(&val.t), &t)
SQL> exit;

 

after full rebuild without it:

D:\source\COMPONENTS\SQLfb\build\code\firebird>isql -user sysdba employee

Database: employee, User: SYSDBA
SQL> create user SYSDBA password 'masterkey';

SQL> commit;
SQL> select cast(SEC$USER_NAME as varchar(10)), cast(SEC$PLUGIN as varchar(10)) from SEC$USERS;

CAST       CAST
========== ==========
SYSDBA     Srp

SQL> create user myUser password 'myPass';
SQL> select cast(SEC$USER_NAME as varchar(10)), cast(SEC$PLUGIN as varchar(10)) from SEC$USERS;

CAST       CAST
========== ==========
SYSDBA     Srp

SQL> commit;
SQL> select cast(SEC$USER_NAME as varchar(10)), cast(SEC$PLUGIN as varchar(10)) from SEC$USERS;

CAST       CAST
========== ==========
SYSDBA     Srp
MYUSER     Srp

SQL> exit;

D:\source\COMPONENTS\SQLfb\build\code\firebird>isql -user sysdba employee
Database: employee, User: SYSDBA
SQL> select cast(SEC$USER_NAME as varchar(10)), cast(SEC$PLUGIN as varchar(10)) from SEC$USERS;

Statement failed, SQLSTATE = 2F000
Libtommath error code -3 in function mp_mod(&t, const_cast<mp_int*>(&val.t), &t)
SQL>

:(

[Dimitry Sibiryakov]

Sergey Khalyutin wrote 20.02.2026 11:03:
> Error is unstabile

So, did you check in a debugger actual value of the prime?

--
WBR, SD.

[Alex Peshkoff]

Also please check - did constructor
    explicit RemoteGroup(Firebird::MemoryPool&)
(in srp.cpp) execute or not?

[Sergey Khalyutin]

    explicit RemoteGroup(Firebird::MemoryPool&)

        : prime(primeStr), generator(genStr), k()

    {

... RemoteGroup execute - YES. but sometime prime = 0x000002b3690d4fa0 {0}, sometime = 0x0000019150c13b90 {16} sometime = {48}

If it's what it seems to me, then the large integer is not prime but division by 0 - when checking of it

(Of couse it only my guess, since I see this code for the first time)

Additionally, the ISQL freezes periodically. Entering an infinite loop here:

code\extern\libtommath\bn_mp_reduce_2k_l.c

int mp_reduce_2k_l(mp_int *a, mp_int *n, mp_int *d)

{

   mp_int q;

   int    p, res;

   

   if ((res = mp_init(&q)) != MP_OKAY) {

      return res;

   }

   

   p = mp_count_bits(n);    

top:

   /* q = a/2**p, a = a mod 2**p */

   if ((res = mp_div_2d(a, p, &q, a)) != MP_OKAY) {

      goto ERR;

   }

   

   /* q = q * d */

   if ((res = mp_mul(&q, d, &q)) != MP_OKAY) {

      goto ERR;

   }

   

   /* a = a + q */

   if ((res = s_mp_add(a, &q, a)) != MP_OKAY) {

      goto ERR;

   }

   

   if (mp_cmp_mag(a, n) != MP_LT) {

      s_mp_sub(a, n, a);

      goto top; <-- enter to infinity loop

   }

   

ERR:

   mp_clear(&q);

   return res;

}

[Alex Peshkoff]

On 2/20/26 20:04, Sergey Khalyutin wrote:

    explicit RemoteGroup(Firebird::MemoryPool&)

        : prime(primeStr), generator(genStr), k()

    {

... RemoteGroup execute - YES. but sometime prime = 0x000002b3690d4fa0 {0}, sometime = 0x0000019150c13b90 {16} sometime = {48}

Must say it's not clear to me what you've shown here, struct mp_int is not what can be easily checked using debugger - sooner of all it's debugger output?

struct  {
    int used, alloc, sign;
    mp_digit *dp;
} mp_int;

Members used, alloc, sign should always remain the same for prime, pointer dp is dynamically allocated and may have any value.

If it's what it seems to me, then the large integer is not prime but division by 0 - when checking of it

(Of couse it only my guess, since I see this code for the first time)

prime should remain prime wih value from primeStr for lifetime - it never changes (if I write BigInt code today prime and generator to be marked as const in the class).

Additionally, the ISQL freezes periodically. Entering an infinite loop here:

code\extern\libtommath\bn_mp_reduce_2k_l.c

int mp_reduce_2k_l(mp_int *a, mp_int *n, mp_int *d)

{

   mp_int q;

   int    p, res;

   

   if ((res = mp_init(&q)) != MP_OKAY) {

      return res;

   }

   

   p = mp_count_bits(n);    

top:

   /* q = a/2**p, a = a mod 2**p */

   if ((res = mp_div_2d(a, p, &q, a)) != MP_OKAY) {

      goto ERR;

   }

   

   /* q = q * d */

   if ((res = mp_mul(&q, d, &q)) != MP_OKAY) {

      goto ERR;

   }

   

   /* a = a + q */

   if ((res = s_mp_add(a, &q, a)) != MP_OKAY) {

      goto ERR;

   }

   

   if (mp_cmp_mag(a, n) != MP_LT) {

      s_mp_sub(a, n, a);

      goto top; <-- enter to infinity loop

   }

   

ERR:

   mp_clear(&q);

   return res;

}

Something goes VERY wrong with tommath library on your box. No more ideas what happened - we use >15 years and never-never used to see something bad from it.

A.

[Sergey Khalyutin]

I localized the problem in:

int mp_read_radix (mp_int * a, const char *str, int radix)

{

...

...

    if (y < radix) {

      if ((res = mp_mul_d (a, (mp_digit) radix, a)) != MP_OKAY) {

         return res;

      }

      if ((res = mp_add_d (a, (mp_digit) y, a)) != MP_OKAY) {

         return res;

      }

    } else {

...

...

  return MP_OKAY;

}

In Linux The debugger shows that the value  of  a->dp  increases with each processing iteration and reaches a large number. as expected

and

    Firebird::string r;

       prime.getText(r);

    std::cout << r.c_str() << std::endl;

print long text string "ABAC...."

In Windows, the value   of   a->dp    remains zero, as if it wasn't returned from the function   mp_mul_d()  and   mp_add_d().

    Firebird::string r;

       prime.getText(r);

    std::cout << r.c_str() << std::endl;

print "0" or sometimes "10".

I can't figure out why.   (mix c/c++ code?,  length of  Long type? ... )

I`m  do not find  in  builds\win32\make_boot.bat  special compiler flags for  libtommath and

I`m add   libtommath as:

########################################

# LIBRARY libtommath

########################################

file(GLOB libtommath_src "extern/libtommath/*.c")

add_library               (${libtommath_ID} ${libtommath_src})

project_group             (${libtommath_ID} Extern)

:(?

[Vlad Khorsun]

21.02.2026 16:58, Sergey Khalyutin:

> I`m  do not find  in  builds\win32\make_boot.bat  special compiler flags for libtommath and

In make_boot you could see that LibTomMath is build using extern\libtommath\libtommath_MSVC15.sln.
thus you should look for compiler and linker settings in extern\libtommath\libtommath_MSVC15.vcxproj.

Regards,
Vlad

[Sergey Khalyutin]

We found that the macro MP_MASK

in https://github.com/FirebirdSQL/firebird/blob/master/extern/libtommath/bn_mp_add_d.c

#define MP_MASK  ((((mp_digit)1)<<((mp_digit)DIGIT_BIT))-((mp_digit)1))

...

line 68      *tmpc++ &= MP_MASK;

...

which works fine in Linux (under VM VirtualBox)

but under Windows change value of variable 'c->dp'

1-iteration before line 68 c->dp=14 after c->dp=4

2-iteration before line 68 c->dp=10 after c->dp=0

end so on.

after all iteration in Linux I`m see expected result like "AABABAABAAAA...FDFDF"

but under Windows I`m see result "0" or "10" or "40"

We have applied the definitions and file list

from extern\libtommath\libtommath_MSVC15.vcxproj

but unfortunately it didn't help.

However, thanks to this, in the Linux dockerfile we noticed downloads

and compilations on the fly libtommath v1.2.0.

We found that the function mp_add_d() there is very different...

We are replace source code

from  https://github.com/FirebirdSQL/firebird/blob/master/extern/libtommath

to https://github.com/libtom/libtommath/releases/tag/v1.2.0

under Windows I`m see result like "AABABAABAAAA...FDFDF"

as it need, SYSDBA was created and remote connection with Flameroin was successful.

The computer used for the assembly was purchased recently and has the latest software and hardware.

int

mp_add_d (mp_int * a, mp_digit b, mp_int * c)

{

  int     res, ix, oldused;

  mp_digit *tmpa, *tmpc, mu;

   ...

  /* source alias */

  tmpa    = a->dp;

  /* destination alias */

  tmpc    = c->dp;

  /* if a is positive */

  if (a->sign == MP_ZPOS) {

     /* add digit, after this we're propagating

      * the carry.

      */

     *tmpc   = *tmpa++ + b;

     mu      = *tmpc >> DIGIT_BIT;

     *tmpc++ &= MP_MASK;

     /* now handle rest of the digits */

     for (ix = 1; ix < a->used; ix++) {

        *tmpc   = *tmpa++ + mu;

        mu      = *tmpc >> DIGIT_BIT;

        *tmpc++ &= MP_MASK;       <-------------

     }

     /* set final carry */

     ix++;

     *tmpc++  = mu;

  }

  ...

  return MP_OKAY;

}

[Alex Peshkoff]

However, thanks to this, in the Linux dockerfile we noticed downloads

and compilations on the fly libtommath v1.2.0.

It was initially designed in linux to prefer when possible use of system library, not builtin. 

We found that the function mp_add_d() there is very different...

We are replace source code

from  https://github.com/FirebirdSQL/firebird/blob/master/extern/libtommath

to https://github.com/libtom/libtommath/releases/tag/v1.2.0

under Windows I`m see result like "AABABAABAAAA...FDFDF"

as it need, SYSDBA was created and remote connection with Flameroin was successful.

I.e. libtommath in firebird tree to be replaced with fresh one?