BLR versions

[Dmitry Yemanov]

All,

So far we were extending BLR without touching its version. I believe
there are two reasons for that -- SQL dialects (blr_version5 is
generated for Dialect 3) and downgrade support (BLR not using new
features should work in prior versions too).

While Dialect 1 is deprecated many years ago, it's still supported and
may disappear only when all the existing migration issues are resolved.
However, maybe a new BLR version could be introduced for Dialect 3 only.

The downgrade support is also somewhat desirable but generally not
guaranteed, as we tend to generate some new BLR verbs for the old
commands too, so they will not be parsed by the older engines. But given
that all stored objects may be altered to regenerate their BLR, maybe
this is not so important either.

So let's decide whether we're OK with bumping BLR version for some
global changes that affect many verbs or rather use the new blr_flags
instead. If the latter, how we're going to address the single-byte code
space limit that we're going to reach soon? Or maybe use blr_flags now
(before the codes are exhausted) and switch to the new BLR version
afterwards?

I'm currently extending the context limit and having some global
modifier flag looks better than introducing a dozen of new BLR verbs.
Any of the aforementioned options should work for me, let's just agree
on the one or another.


Dmitry

[Dimitry Sibiryakov]

Dmitry Yemanov wrote 31.12.2025 10:04:
> So let's decide whether we're OK with bumping BLR version for some global
> changes that affect many verbs or rather use the new blr_flags instead. If the
> latter, how we're going to address the single-byte code space limit that we're
> going to reach soon? Or maybe use blr_flags now (before the codes are exhausted)
> and switch to the new BLR version afterwards?

Currently translation DSQL->nodes->BLR->nodes is done literally in a single
routine several lines away from each other. It was discussed many times that at
some point the path should be changed into SQL->nodes, leaving BLR for backward
compatibility/ESQL only.
May be the time has come for a massive cleanup and final merge of Scratch,
Statement and Request?
I can do it.

--
WBR, SD.

[Dmitry Yemanov]

31.12.2025 13:33, 'Dimitry Sibiryakov' via firebird-devel wrote:
>
>   Currently translation DSQL->nodes->BLR->nodes is done literally in a
> single routine several lines away from each other. It was discussed many
> times that at some point the path should be changed into SQL->nodes,
> leaving BLR for backward compatibility/ESQL only.

Yes, this was discussed. And it shouldn't be hard for DSQL. However,
IIRC we never decided what to do with those users who prefer to hide
(nullify) their PSQL sources.

>   May be the time has come for a massive cleanup and final merge of
> Scratch, Statement and Request?

I don't want to see new big changes appearing in the v6 pipeline, this
will delay the release even more.


Dmitry

[Dimitry Sibiryakov]

Dmitry Yemanov wrote 31.12.2025 11:41:
> However, IIRC we never decided what to do with those users who prefer to hide (nullify) their PSQL sources.

For this use case a brand-new BLR version would be more suitable, IMHO.
May be we can use encryption for the sources as Oracle does. It is a placebo
of course, but still a lot of people find it enough to calm down.

> I don't want to see new big changes appearing in the v6 pipeline, this will
> delay the release even more.

You declared shortened release cycle many times. Firebird 5 was released
almost a year ago. May be it a good time to merge feature PRs, fork v6 branch
and declare feature freeze in it letting cleanup PRs freely destabilise HEAD?..

--
WBR, SD.

[Mark Rotteveel]

On 31/12/2025 10:04, Dmitry Yemanov wrote:
> So far we were extending BLR without touching its version. I believe
> there are two reasons for that -- SQL dialects (blr_version5 is
> generated for Dialect 3) and downgrade support (BLR not using new
> features should work in prior versions too).
>
> While Dialect 1 is deprecated many years ago, it's still supported and
> may disappear only when all the existing migration issues are resolved.
> However, maybe a new BLR version could be introduced for Dialect 3 only.

[..]

> I'm currently extending the context limit and having some global
> modifier flag looks better than introducing a dozen of new BLR verbs.
> Any of the aforementioned options should work for me, let's just agree
> on the one or another.

I can't oversee the impact of BLR version changes, but intuitively I'd
say that this asks for a new BLR version. But if we're going to change
the version, maybe it is a good idea to see if there are other
fundamental things that need to change or improve in BLR.

Also, what is the impact for the client-side, which also send a limited
form of BLR for statement execution/fetches?

Mark
--
Mark Rotteveel

[Dmitry Yemanov]

31.12.2025 14:10, 'Dimitry Sibiryakov' via firebird-devel wrote:
>
>   You declared shortened release cycle many times. Firebird 5 was
> released almost a year ago.

Almost two years ago, actually ;-)

> May be it a good time to merge feature PRs,
> fork v6 branch and declare feature freeze in it letting cleanup PRs
> freely destabilise HEAD?..

"Merging feature PRs" gonna take the next 3-6 months, I'm afraid. And
stabilizing features in two branches is not going to be funny. But
generally I agree that making the master available for the new
development ASAP would be a good thing.


Dmitry

[Dimitry Sibiryakov]

Dmitry Yemanov wrote 31.12.2025 12:28:
> And stabilizing features in two branches is not going to be funny.

It may be simpler if no new features can get into stable branches.
Frontporting of bugfixes is not that hard.

--
WBR, SD.

[Dmitry Yemanov]

31.12.2025 14:27, 'Mark Rotteveel' via firebird-devel wrote:
>
> I can't oversee the impact of BLR version changes, but intuitively I'd
> say that this asks for a new BLR version. But if we're going to change
> the version, maybe it is a good idea to see if there are other
> fundamental things that need to change or improve in BLR.

So far only the limited code space (we're at 235 now, and a few holes
are also available in the middle) looks being an issue, but it looks
enough for at least a couple of major versions, likely even more. I
don't remember other fundamental problems, but maybe other guys have
something in mind.

> Also, what is the impact for the client-side, which also send a limited
> form of BLR for statement execution/fetches?

I believe BLR v4/v5 is still to be supported internally, so the client
may send any of the v4/v5/v6 BLR and it will be handled gracefully. But
to send BLR v6 it needs to check whether the server supports it. Maybe
it would be easier to just leave BLR v4/v5 in the client-side
communication forever.


Dmitry

[Dimitry Sibiryakov]

Dmitry Yemanov wrote 31.12.2025 12:44:
>
> So far only the limited code space (we're at 235 now, and a few holes are also
> available in the middle) looks being an issue, but it looks enough for at least
> a couple of major versions, likely even more. I don't remember other fundamental
> problems, but maybe other guys have something in mind.

I wonder if parser tokens can be used as "a new BLR" and the parser can
produce/eat a stream containing them. It could be a base for "Firebird wrap
utility" effectively obfuscate PSQL sources.

--
WBR, SD.

[Vlad Khorsun]

31.12.2025 11:04, Dmitry Yemanov:

> All,
>
> So far we were extending BLR without touching its version. I believe there are two reasons for that -- SQL dialects (blr_version5 is
> generated for Dialect 3) and downgrade support (BLR not using new features should work in prior versions too).
>
> While Dialect 1 is deprecated many years ago, it's still supported and may disappear only when all the existing migration issues are
> resolved. However, maybe a new BLR version could be introduced for Dialect 3 only.
>
> The downgrade support is also somewhat desirable but generally not guaranteed, as we tend to generate some new BLR verbs for the old
> commands too, so they will not be parsed by the older engines. But given that all stored objects may be altered to regenerate their
> BLR, maybe this is not so important either.
>
> So let's decide whether we're OK with bumping BLR version for some global changes that affect many verbs or rather use the new
> blr_flags instead. If the latter, how we're going to address the single-byte code space limit that we're going to reach soon? Or
> maybe use blr_flags now (before the codes are exhausted) and switch to the new BLR version afterwards?

If bump BLR version: I suppose new BLR version will work by Dialect 3 rules only, correct ?
I.e. it will effectively throw out Dialect 1 users ?

> I'm currently extending the context limit and having some global modifier flag looks better than introducing a dozen of new BLR
> verbs. Any of the aforementioned options should work for me, let's just agree on the one or another.

About extending the context limit. I hope you not going to create huge number of RPB's - for every context
number ? Most of contexts have limited scope and its RPB's can be reused to save memory. Note, the context
limit is actual for SP and triggers that live in metadata cache and its memory doesn't released when user
statement frees. Honestly, even current 255 contexts is too much from memory usage POV.

Regards,
Vlad

[Dmitry Yemanov]

03.01.2026 19:02, Vlad Khorsun wrote:
>
> If bump BLR version: I suppose new BLR version will work by Dialect 3
> rules only, correct ?

Yes.

> I.e. it will effectively throw out Dialect 1 users ?

All three BLR versions are to be supported. BLR v4/v5 will be generated
for the existing code (and accepted from the clients). BLR v6 will be
generated for Dialect 3 queries with more than 256 contexts. Dialect 1
queries with more than 256 contexts will throw an error, as now. This is
how I see it.

> About extending the context limit. I hope you not going to create
> huge number of RPB's - for every context
> number ? Most of contexts have limited scope and its RPB's can be reused
> to save memory. Note, the context
> limit is actual for SP and triggers that live in metadata cache and its
> memory doesn't released when user
> statement frees. Honestly, even current 255 contexts is too much from
> memory usage POV.

I'd prefer to go the following route:

1) Introduce longer (two-byte or maybe even four-byte) context numbers
in the BLR (they're already ULONG inside the engine).
2) Keep USHORT for context numbers inside DSQL and keep the existing
MAX_STREAMS = 4096 limit.
3) Optimize memory usage for csb_repeat and record_param.

Then some another day reconsider/optimize the RPBs usage in general and
only then possibly raise the MAX_STREAMS limit.


Dmitry

[Adriano dos Santos Fernandes]

On 12/31/25 07:33, 'Dimitry Sibiryakov' via firebird-devel wrote:

>   May be the time has come for a massive cleanup and final merge of
> Scratch, Statement and Request?

You misunderstood these things completely to think in merge them.


Adriano

[Dimitry Sibiryakov]

Adriano dos Santos Fernandes wrote 05.01.2026 12:30:
>>   May be the time has come for a massive cleanup and final merge of
>> Scratch, Statement and Request?
> You misunderstood these things completely to think in merge them.

It looks like you misread my message.

I explain:

1) Merge Scratch with DsqlScratch
2) Merge Statement with DsqlStatement
3) Merge Request with DsqlRequest.

--
WBR, SD.

[Adriano dos Santos Fernandes]

On 12/31/25 06:04, Dmitry Yemanov wrote:
> All,
>
> So far we were extending BLR without touching its version. I believe
> there are two reasons for that -- SQL dialects (blr_version5 is
> generated for Dialect 3) and downgrade support (BLR not using new
> features should work in prior versions too).
>
> While Dialect 1 is deprecated many years ago, it's still supported and
> may disappear only when all the existing migration issues are resolved.
> However, maybe a new BLR version could be introduced for Dialect 3 only.
>
> The downgrade support is also somewhat desirable but generally not
> guaranteed, as we tend to generate some new BLR verbs for the old
> commands too, so they will not be parsed by the older engines.

I consider these cases as bugs and when this is impossible to do,
backport compatibility layer.

v3 release was shown us that automatic database upgrade,which later may
need a manual revert, is required.

We don't even have an official dump utility, so this should continue to
be done.


Adriano

[Adriano dos Santos Fernandes]

IMO BLR (or better, binary representation of compiled sources) should
not be removed from storage (it could from ad-hoc DSQL).

JVM is well with its bytecode and do not want people to deploy
Java/Kotlin/etc source code and require complete recompilation every time.

Same for BLR. It's faster to compile than sources.

We have already discussed instruction-based execution and I'm even
playing with it. We may store binary instruction (completely different
than current BLR) in the future. Of course, Firebird instruction-set
would not be so low level, so the same aspects we think today (size of
context) would still be needed to be thought from scratch.


Adriano

[Jim Starkey]

Lest there be any confusion, bumping the blr version number is
absolutely required when either extending or making an incompatible
change to blr.  The purpose, of course, is to enable forward and
backward compatibility (or at least catch errors is compatibility is
inconvenient).

For the record, initially I stupidly neglected to include a version
number in the blr definition.  When I realized the screw up during the
process to make it a DEC standard, I recognized that all existing blr
generators started with blr_begin (2), so the first actual blr version
number was 3.

Blr, which, incidentally, I would drop in a shot, was intended to be
dense, explicit, next to free to parse, and capable of expressing SQL,
Quell, and DEC's family of data management products semantics.  I never
expected it to live for 42 years, but then I would have killed it
decades ago.

--
Jim Starkey