Package constants. Weird message "HY000 / Statement format outdated" can raise

[Pavel Zotov]

Consider script:

set bail on;
set autoterm on;
set list on;
set echo on;
shell if exist r:\temp\tmp4test.fdb del r:\temp\tmp4test.fdb;
create database 'localhost:r:\temp\tmp4test.fdb' user sysdba password 'masterkey';

create package pg_test as
begin
    constant s_head smallint = 1;
    procedure sp_test returns(sp_outcome smallint);
    function fn_test returns smallint;
end
;
recreate package body pg_test as
begin
    constant s_body smallint = -1; ------------------------------------------- [ 1 ]
    procedure sp_test returns(sp_outcome smallint) as
    begin
        sp_outcome = s_head;
        suspend;
    end

    function fn_test returns smallint as
    begin
        return 1;
    end
end
;

commit;
connect 'localhost:r:\temp\tmp4test.fdb' user sysdba password 'masterkey'; --- [ 2 ]
set bail OFF;

select sp_outcome as result_1 from pg_test.sp_test;

select pg_test.fn_test() as result_2  from rdb$database;

select pg_test.fn_test() as result_3 from pg_test.sp_test rows 1;

NOTE on statements marked as [ 1 ] and [ 2 ].

Run this script.

On 6.0.0.1965 output will be:

select o2 from pg_test.sp_test;
O2                              2
select pg_test.fn_test() from rdb$database;
FN_TEST                         1
select o2, pg_test.fn_test() as fn from pg_test.sp_test rows 1;
Statement failed, SQLSTATE = HY000
Statement format outdated, need to be reprepared

Now let's comment out  [ 1 ] and/or [ 2 ].

Script will complete fine (will display "RESULT_3      1").

PS.   constant s_body - just declared and not used at all.

[Pavel Zotov]

[ UPD after "On 6.0.0.1965 output will be: "]
actual output will be:

select sp_outcome as result_1 from pg_test.sp_test;

RESULT_1                        1

select pg_test.fn_test() as result_2  from rdb$database;

RESULT_2                        1

select pg_test.fn_test() as result_3 from pg_test.sp_test rows 1;

Statement failed, SQLSTATE = HY000
Statement format outdated, need to be reprepared

(oririnal msg contains result of copy-paste from another terminal, sorry)

среда, 27 мая 2026 г. в 01:00:32 UTC+3, Pavel Zotov:

[Artyom Abakumov]

This is a known issue: https://github.com/FirebirdSQL/firebird/pull/9022#issuecomment-4431995459

I spent a couple of days trying to fix this, but to no avail. The problem connects to the new MetaCache system, the code of which I'm not yet very familiar with.

среда, 27 мая 2026 г. в 01:05:02 UTC+3, Pavel Zotov:

[Alex Peshkoff]

On 5/27/26 11:19, Artyom Abakumov wrote:
> This is a known issue:
> https://github.com/FirebirdSQL/firebird/pull/9022#issuecomment-4431995459
> I spent a couple of days trying to fix this, but to no avail. The
> problem connects to the new MetaCache system, the code of which I'm
> not yet very familiar with.
>

I will take a look at it.

[Artyom Abakumov]

Alex Peshkoff:

Thank you. Here is a simple way to reproduce it:

set term ^;

CREATE PACKAGE TEST
AS
BEGIN
    CONSTANT C1 INTEGER = 10;
END^

CREATE PROCEDURE P1(I INT) RETURNS (PROCEDURE_OUTPUT INT)
AS
BEGIN
    PROCEDURE_OUTPUT = TEST.C1;
    SUSPEND;
END^

set term ;^
commit;

-- Test Public
select TEST.C1 from rdb$database ;
select * from P1(0);
commit;

 

Interestingly, if you swap the queries, no error will occur.

select * from P1(0);

select TEST.C1 from rdb$database ;

[Alex Peshkoff]

On 5/27/26 14:23, Artyom Abakumov wrote:

set term ^;

CREATE PACKAGE TEST
AS
BEGIN
    CONSTANT C1 INTEGER = 10;
END^

CREATE PROCEDURE P1(I INT) RETURNS (PROCEDURE_OUTPUT INT)
AS
BEGIN
    PROCEDURE_OUTPUT = TEST.C1;
    SUSPEND;
END^

set term ;^
commit;

-- Test Public
select TEST.C1 from rdb$database ;
select * from P1(0);
commit;

 

You calculate hash of constant as 
    digest.process(sizeof(m_value.vlu_desc), &m_value.vlu_desc)
i.e. all fields of descriptor are hashed as whole binary object. That's OK when in dsc_address stored not address but offset in a message. That's typical for objects in metadata cache but in your case it's definitely address of variable.

select TEST.C1 from rdb$database ;

Thread 1 "isql" hit Breakpoint 3, Jrd::ConstantValue::hash (this=0x7fffef2dfde0, tdbb=0x7fffffffbc98, digest=...)
   at /usr/home/firebird/HEAD/temp/Debug/jrd/Package.cpp:407
407             digest.process(sizeof(m_value.vlu_desc), &m_value.vlu_desc);
(gdb) p m_value.vlu_desc  
$11 = {
 dsc_dtype = 9 '\t',
 dsc_scale = 0 '\000',
 dsc_length = 4,
 dsc_sub_type = 0,
 dsc_flags = 0,
 dsc_address = 0x7fffef2dfeb8 "\n"
}
(gdb) c
Continuing.

C1                              10


Moreover, it changes sometimes.


select * from P1(0);

Thread 1 "isql" hit Breakpoint 3, Jrd::ConstantValue::hash (this=0x7fffef2e01d0, tdbb=0x7fffffffbc98, digest=...)
   at /usr/home/firebird/HEAD/temp/Debug/jrd/Package.cpp:407
407             digest.process(sizeof(m_value.vlu_desc), &m_value.vlu_desc);
(gdb) p m_value.vlu_desc  
$12 = {
 dsc_dtype = 9 '\t',
 dsc_scale = 0 '\000',
 dsc_length = 4,
 dsc_sub_type = 0,
 dsc_flags = 0,
 dsc_address = 0x7fffef2e02a8 "\n"
}


I think that for format of single constant it makes absolutely no difference where is it's data stored. Moreover, actual value also does not matter - it does not affect format unlike probably offset in a message. 
I.e. I suggest to hash all fields except dsc_address. But may be I'm missing something re constants.

PS. Is change of dsc_address in constant descriptor normal behavior? Looks strange at the first glance.

[Artyom Abakumov]

Alex Peshkoff:

On 5/27/26 14:23, Artyom Abakumov wrote:

PS. Is change of dsc_address in constant descriptor normal behavior? Looks strange at the first glance.

Hm, seem like it is the root problem. the Package::reload was called twice, causing dsc address to change and leading to different hash.

 Alex Peshkoff:

Moreover, actual value also does not matter - it does not affect format unlike probably offset in a message.

I am not sure. For a function or procedure, only the format is important. But literal value is a part of the Constant. A value can be changed without changing the type (i.e. dsc fields will remain the same). Will it be safe from the Metacache point of view?

[Dmitry Yemanov]

01.06.2026 10:01, Artyom Abakumov wrote:
>
> I am not sure. For a function or procedure, only the format is
> important. But literal value is a part of the Constant. A value can be
> changed without changing the type (i.e. dsc fields will remain the
> same). Will it be safe from the Metacache point of view?

Is there any problem to calculate and combine two hashes - (1) &dsc
before dsc_address and (2) the value itself, i.e. the bytes in the
{dsc_address, dsc_length} range?


Dmitry

[Dimitry Sibiryakov]

Dmitry Yemanov wrote 01.06.2026 9:13:
> Is there any problem to calculate and combine two hashes - (1) &dsc before
> dsc_address and (2) the value itself, i.e. the bytes in the {dsc_address,
> dsc_length} range?

And, BTW, what is the purpose of this hash?
Usage of a hash as a format id is not reliable anyway, so, perhaps, hash of
the value alone should be enough.

--
WBR, SD.

[Artyom Abakumov]

Dimitry Sibiryakov:

But what if two values ​​of different types have the same binary representation?

[Dimitry Sibiryakov]

Artyom Abakumov wrote 01.06.2026 9:24:
> But what if two values ​​of different types have the same binary representation?

Then the situation is the same as if a hash collision happen. That's why I
ask about the purpose.

--
WBR, SD.

[Alex Peshkoff]

Let me answer both questions at once. The purpose of a hash is to avoid various errors when combining statement compiled using old metadata with new metadata. The first step done at any request start - version of metacache is compared with version of statement, if they match (almost always) nothing else is needed. (Version of metacache incremented each time when any world-visible change in metadata takes place.)

If versioned changed we should check does it affect us or not. The only idea of how objects directly affect each other that came to my mind is message formats, both incoming and outgoing. In theory it's possible to keep full format copies - but I dislike that due to memory usage. What about collisions in hashes of format - we use same approach (hash comparison) in Srp, should it also be called not reliable?

Calculation of hash for both dsc before address and value itself is certainly possible. But on my mind adding value does not make sense - if we changed value of constant but kept it's format all statements using it remain correct, why take value into an account?

[Dimitry Sibiryakov]

'Alex Peshkoff' via firebird-devel wrote 01.06.2026 12:45:
> The only idea of how objects directly affect each other that came to my mind is
> message formats, both incoming and outgoing. In theory it's possible to keep
> full format copies - but I dislike that due to memory usage.

Objects themself have to keep their formats, the cache doesn't need to keep
copies of the formats, it always can retrieve a reference from the cached objects.

> What about collisions in hashes of format - we use same approach (hash comparison) in Srp, should it also be called not reliable?

Generally - yes, but crypto-usage of hashes is completely different from
search-usage of hashes.
Srp is operating with small number of objects using big and slow
cryptohashes. It reduces collision probability to acceptable low level. Besides,
consequences of the collision is unpleasant but not critical.
Hash tables (and MetaCache in this case) has to use small and fast hashes for
performance reason and even smallest probability of false equivalence may be
devastating. Besides, hash-and-compare in this case is most likely slower than
simple full direct comparison.
I see it this way.

--
WBR, SD.

[Alex Peshkoff]

On 6/1/26 14:00, 'Dimitry Sibiryakov' via firebird-devel wrote:
> 'Alex Peshkoff' via firebird-devel wrote 01.06.2026 12:45:
>> The only idea of how objects directly affect each other that came to
>> my mind is message formats, both incoming and outgoing. In theory
>> it's possible to keep full format copies - but I dislike that due to
>> memory usage.
>
>   Objects themself have to keep their formats, the cache doesn't need
> to keep copies of the formats, it always can retrieve a reference from
> the cached objects.
>

That's also possible in theory but required version of object may be
already gone.

>> What about collisions in hashes of format - we use same approach
>> (hash comparison) in Srp, should it also be called not reliable?
>
>   Generally - yes, but crypto-usage of hashes is completely different
> from search-usage of hashes.
>   Srp is operating with small number of objects using big and slow
> cryptohashes. It reduces collision probability to acceptable low
> level. Besides, consequences of the collision is unpleasant but not
> critical.

Logon without correct password is not critical? Are you kidding?

> Hash tables (and MetaCache in this case) has to use small and fast
> hashes for performance reason and even smallest probability of false
> equivalence may be devastating. Besides, hash-and-compare in this case
> is most likely slower than simple full direct comparison.
>   I see it this way.
>

SHA256, hash has obviously size of 256 bits, i.e. 4 pointers. IMO not
critical from RAM usage POV but collision hardly possible.

What about performance - I did not check it. But hash calculation is
needed only after statement compilation (IMO incompatible re performance
thing) and in a case when DDL was done on database, which happens
compared with ML not too often.

[Dimitry Sibiryakov]

'Alex Peshkoff' via firebird-devel wrote 01.06.2026 13:52:
> That's also possible in theory but required version of object may be already gone.

In this case comparison in guaranteed to fail anyway, right? If no, the
version must be kept in the cache until is surely unneeded.

> Logon without correct password is not critical? Are you kidding?

Not "logon without correct password", but "more than one correct password
with very low probability", i.e. the system is slightly more vulnerable to a
brute force attack. Not critical.

> SHA256, hash has obviously size of 256 bits, i.e. 4 pointers. IMO not critical from RAM usage POV but collision hardly possible.

Still right algorithms should perform full comparison after hash match.
Rule of the thumb: different hashes - surely different data, same hashes -
probably different data.

--
WBR, SD.

[Alex Peshkoff]

On 6/1/26 15:02, 'Dimitry Sibiryakov' via firebird-devel wrote:
> 'Alex Peshkoff' via firebird-devel wrote 01.06.2026 13:52:
>> That's also possible in theory but required version of object may be
>> already gone.
>
>   In this case comparison in guaranteed to fail anyway, right?

Not completely. Imagine table T1 has fields F1, F2. Some user statement
has among other

select f1, f2, from T1 ..................

It's prepared but runs rarely. Once upon a time field is added to T1. It
certainly does not affect format of message used by user request. Time
goes (interesting transaction numbers grow), statement has no active
requests, and old copy of T1 is removed from cache. Now user wants to
execute request based on that statement - and what? Yes, we can treat it
as definitely wrong, if that old copy of T1 is missing. That's possible
approach, I've chosen another one.

> If no, the version must be kept in the cache until is surely unneeded.
>
>> Logon without correct password is not critical? Are you kidding?
>
>   Not "logon without correct password", but "more than one correct
> password with very low probability", i.e. the system is slightly more
> vulnerable to a brute force attack. Not critical.

Pay attention that with skipped old format you will in the worst case
return to pre-6 behavior - get strange error message about BLR, offset,
etc. Average user goes mad - but nothing critical.

>
>> SHA256, hash has obviously size of 256 bits, i.e. 4 pointers. IMO not
>> critical from RAM usage POV but collision hardly possible.
>
>   Still right algorithms should perform full comparison after hash match.
>   Rule of the thumb: different hashes - surely different data, same
> hashes - probably different data.
>

Yes, and that will be done by BLR execution code. But possibility is
very low...

[Dimitry Sibiryakov]

'Alex Peshkoff' via firebird-devel wrote 01.06.2026 16:04:
>
> Not completely. Imagine table T1 has fields F1, F2. Some user statement has
> among other
>
> select f1, f2, from T1 ..................
>
> It's prepared but runs rarely. Once upon a time field is added to T1. It
> certainly does not affect format of message used by user request. Time goes
> (interesting transaction numbers grow), statement has no active requests, and
> old copy of T1 is removed from cache.

Isn't statement(s) that depends on it purged from statement cache in this case?

> Now user wants to execute request based on that statement - and what?

The statement check metadata cache, see that it is outdated and commit
suicide. New statement is prepared from scratch using new versions of metadata,
no?..

>> Rule of the thumb: different hashes - surely different data, same hashes - probably different data.
>
> Yes, and that will be done by BLR execution code. But possibility is very low...

Too late, at this point re-preparation of the statement with new metadata is
impossible.

--
WBR, SD.

[Alex Peshkoff]

On 6/1/26 17:31, 'Dimitry Sibiryakov' via firebird-devel wrote:
> 'Alex Peshkoff' via firebird-devel wrote 01.06.2026 16:04:
>>
>> Not completely. Imagine table T1 has fields F1, F2. Some user
>> statement has among other
>>
>> select f1, f2, from T1 ..................
>>
>> It's prepared but runs rarely. Once upon a time field is added to T1.
>> It certainly does not affect format of message used by user request.
>> Time goes (interesting transaction numbers grow), statement has no
>> active requests, and old copy of T1 is removed from cache.
>
>   Isn't statement(s) that depends on it purged from statement cache in
> this case?

Statement does not depend on version of object, active request - does
depend on it. And we do not need statement cache for this to take place
- user which prepared statement and drinks coffee is enough.

>
>> Now user wants to execute request based on that statement - and what?
>
>   The statement check metadata cache, see that it is outdated and
> commit suicide. New statement is prepared from scratch using new
> versions of metadata, no?..
>

Yes, such approach is possible.

>>>   Rule of the thumb: different hashes - surely different data, same
>>> hashes - probably different data.
>>
>> Yes, and that will be done by BLR execution code. But possibility is
>> very low...
>   Too late, at this point re-preparation of the statement with new
> metadata is impossible.
>

If user once per million years gets strange message - no criminal.
Saving a lot of re-prepares when message format actually unchanged
appears more interesting.