Verifying understanding of protocol 19

[Mark Rotteveel]

I'm currently looking at protocol 19 to implement it in Jaybird, and I'd
like to double check my understanding, so far I've seen the following:

* The upper limit for inline blob length is 65535 bytes (0xFFFF)
* `op_execute`/`op_execute2` has an extra (unsigned) 32-bit int
`p_sqldata_inline_blob_size` with the requested inline blob size sent
immediately after `p_sqldata_cursor_flags`
* The response to `op_execute2` will have zero or more `op_inline_blob`
(114) responses before the `op_sql_response` packet
* The response to `op_fetch` will have zero or more `op_inline_blob`
(114) responses before each `op_fetch_response` with the blobs of the
following row
* The `op_line_blob` response is the `P_INLINE_BLOB` message with the
following payload:
** `p_tran_id` 32-bit int (formally 16-bit, but 32-bit due to XDR):
transaction handle
** `p_blob_id` 64-bit int: blob id
** `p_blob_info` buffer (32-bit length, data, padding): blob info with
(`isc_info_blob_num_segments`, `isc_info_blob_max_segment`,
`isc_info_blob_total_length`, `isc_info_blob_type`, `isc_info_end`)
** `p_blob_data` buffer (32-bit length, data, padding): blob data
(including segment lengths!)
* The client is the one responsible for tracking maximum blob cache size
(i.e. the server will just send blobs, and it is up to the client to
store them or throw away)

Is my understanding correct, did I miss anything important?

FYI, I don't cover the changes for `op_exec_immediate2` as Jaybird
doesn't implement it.

Mark
--
Mark Rotteveel

[Vlad Khorsun]

14.03.2025 12:08, 'Mark Rotteveel' via firebird-devel:

> I'm currently looking at protocol 19 to implement it in Jaybird, and I'd like to double check my understanding, so far I've seen the
> following:
>
> * The upper limit for inline blob length is 65535 bytes (0xFFFF)

To be more precise, this is limit of client's buffer size, i.e. it includes blob's segments
and its lengths.

> * `op_execute`/`op_execute2` has an extra (unsigned) 32-bit int `p_sqldata_inline_blob_size` with the requested inline blob size
> sent immediately after `p_sqldata_cursor_flags`

Yes

> * The response to `op_execute2` will have zero or more `op_inline_blob` (114) responses before the `op_sql_response` packet

Yes

> * The response to `op_fetch` will have zero or more `op_inline_blob` (114) responses before each `op_fetch_response` with the blobs
> of the following row

Yes

> * The `op_line_blob` response is the `P_INLINE_BLOB` message with the following payload:
> ** `p_tran_id` 32-bit int (formally 16-bit, but 32-bit due to XDR): transaction handle

Yes

> ** `p_blob_id` 64-bit int: blob id

Or ISC_QUAD - two 32-bit integers

> ** `p_blob_info` buffer (32-bit length, data, padding): blob info with (`isc_info_blob_num_segments`, `isc_info_blob_max_segment`,
> `isc_info_blob_total_length`, `isc_info_blob_type`, `isc_info_end`)
> ** `p_blob_data` buffer (32-bit length, data, padding): blob data (including segment lengths!)

Yes

> * The client is the one responsible for tracking maximum blob cache size (i.e. the server will just send blobs, and it is up to the
> client to store them or throw away)

Yes. I considered a way to ask server to stop\pause sending blobs when client cache is
full, but decided to not over complicate code for now.

> Is my understanding correct, did I miss anything important?

Looks pretty correct.

> FYI, I don't cover the changes for `op_exec_immediate2` as Jaybird doesn't implement it.

Ok.

Regards,
Vlad

[Tomasz Tyrakowski]

Hi

I suspect the Android ICU build shipped with Firebird is broken. It
affects both embedded and server packages. In the current master branch,
this part of the ICU building docker file:

https://github.com/FirebirdSQL/firebird/blob/master/extern/icu/android/Dockerfile#L157-L171

simply renames the .so.77.1 library files to .77.1.so files. However,
the libraries are linked against each other (e.g. libicui18n depends on
libicuuc and libicudata) and internally they still point to .so.77.
Simply renaming the resulting libraries doesn't make them point to the
new ones internally and the dynamic library loader on the target system
still looks for the original library names.
In the 5.0 Firebird source tree, there's a script:
extern/icu/android/renameLibs.sh, which does the same for ICU 63.1.
Trying to dlopen libicui18n.63.1.so from the 5.0.3 embedded package on
Android results in an error (missing libicuuc.so.63) - I've posted the
details on firebird-support.

That's why I suspect both Android server and embedded simply don't work.
I've encountered the problem in a Flutter application, but it boils down
to this very issue: the ICU libraries for Android are built as .so.XX,
and then file-renamed to .XX.so.
I know there are problems with including files other than .so as native
libraries on Android (using the .so.63 or .so.77 files as native
libraries may be troublesome), so the solution would require building
the ICU in a way it produces proper library names (63.1.so, 77.1.so,
etc.) from the start (and links one library against the other with the
proper name).

I decided I'd post here before creating an issue on Github, because
maybe there's something I simply don't get and a team member more
familiar with the process of cross-compiling ICU for Android can simply
point out my mistake. The hard fact is, however, that it's not possible
to dlopen the ICU library provided in the embedded package in its
current state on Android (and since the server package contains exactly
the same libraries, IMO it affects the server packages the same way).

Best regards
Tomasz

[Adriano dos Santos Fernandes]

What is the actual problem you see? Test case, please.

Your report here looks for me more like suppositions that I cannot confirm:

- https://github.com/asfernandes/firebird/commit/1ba3f5a6418e360dfdc097a5af2292c747f7aa99

- https://github.com/asfernandes/firebird/actions/runs/19620344906/job/56179685269#step:8:489

- https://github.com/FirebirdSQL/firebird/blob/v5.0-release/src/common/unicode_util.cpp#L71

It does not mean we can't improve it.

Adriano

--
Support the ongoing development of Firebird! Consider donating to the Firebird Foundation and help ensure its future. Every contribution makes a difference. Learn more and donate here:
https://www.firebirdsql.org/donate
---
You received this message because you are subscribed to the Google Groups "firebird-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebird-deve...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/firebird-devel/85316176-b7dc-453d-8b23-29fb398f75b3%40sol-system.pl.

[Adriano dos Santos Fernandes]

BTW, there is this example
https://github.com/FirebirdSQL/examples/blob/main/android-cpp/README.md
which use this class
https://github.com/FirebirdSQL/firebird/blob/master/android/embedded/src/main/java/org/firebirdsql/android/embedded/FirebirdConf.kt

I don't remember details of this now, if it's there because Android does
not extract the things automatically. And the decision of rename the
shared libraries predates a lot the aar package.

If you have small test case flutter app with easy and clear instructions
on how to test, I would like to check.


Adriano

[Adriano dos Santos Fernandes]

On 11/24/25 08:13, Adriano dos Santos Fernandes wrote:
>
> I don't remember details of this now, if it's there because Android does
> not extract the things automatically. And the decision of rename the
> shared libraries predates a lot the aar package.
>

Just remembered, Android loads the libraries inside the bundle, it has a
syntax something like bundle_file!library_file.so. Firebird can dlopen
others bundled files.

But when Firebird detect that path it would try to load assets (for
example .conf files) from that location and that do not work. So this
FirebirdConf class was created to extract the assets and set environment
variables. One of these assets is the icu data file.


Adriano

[Tomasz Tyrakowski]

On 24.11.2025 at 03:11, Adriano dos Santos Fernandes wrote:
> What is the actual problem you see? Test case, please.
> > Your report here looks for me more like suppositions that I cannot
confirm:

My suspicion came from the fact, that when I use embedded aar and in the
application code I do:

DynamicLibrary.open("libicui18n.63.1.so") (a Dart FFI wrapper around
dlopen), I get the following error:

Failed to load dynamic library 'libicui18n.63.1.so': dlopen failed:
library "libicuuc.so.63" not found

which suggests the Android loader tries to load libicuuc.so.63, which
libicui18n.63.1.so depends on (and the lib was renamed to
libicuuc.63.1.so after building ICU for Android).

At the same time, DynamicLibrary.open("libfbclient.so") succeedes, but
later fails with 335545167 ("Could not find acceptable ICU library") on
embedded attachment.

When I preload the libraries in the proper order:
DynamicLibrary.open("libicudata.63.1.so")
DynamicLibrary.open("libicuuc.63.1.so")
DynamicLibrary.open("libicui18n.63.1.so")

dlopen reports no errors, but when trying to attach I get the same error
335545167 ("Could not find acceptable ICU library").

Also, in the logs (adb logcat) I can see:

I/FIREBIRD(22398): ConfigStorage: Cannot initialize the shared memory region
I/FIREBIRD(22398): Can not access lock files directory
/data/local/tmp/firebird.tmp.ebJGLX

That in itself may be a problem, but probably not related to the ICU
issue (but I simply don't know - maybe it is).

So, yes, all I reported are my suspicions, that's why I haven't created
an issue for it, but instead asked for more information.

I confirm that when I install the 5.0.3 server on Android (in the way
similar to how the test suite does it), I can then connect to
employee.fdb from within isql on Android and I get no errors.

The example application code with embedded Firebird is available at
https://github.com/hipercompl/fbdbmobdemo/tree/embedded

The preloading is here:

https://github.com/hipercompl/fbdbmobdemo/blob/embedded/lib/login_screen.dart#L106-L118

If you change the order and try to load libicui18n.63.1.so first, it
fails with the error I mentioned above.

However, no matter whether you preload ICU or not, the app fails on
attachment with 335545167, for the reason described below.

The actual attachment takes place in a separate isolate (thread /
process), which loads libfbclient.so and uses the FB interfaces.
Preloading ICU in the main application isolate doesn't do much, as the
libraries need to be loaded again (by the embedded engine library) in
the background isolate, in which libfbclient is loaded.
I'll try to preload ICU in the background isolate and see if it helps.

The same code works fine when using TCP/IP connection (not embedded) -
the main branch of the repo is a client/server version and it works fine
(but it doesn't need ICU, so the problem obviously doesn't occur).

As a finishing thought: I didn't mean to sound so alarmist, maybe I
indeed overreacted, sorry for that.

regards
Tomasz

[Adriano dos Santos Fernandes]

First do what
https://github.com/FirebirdSQL/examples/blob/main/android-cpp/README.md
do in relation to
https://github.com/FirebirdSQL/firebird/blob/master/android/embedded/src/main/java/org/firebirdsql/android/embedded/FirebirdConf.kt


Adriano

[Tomasz Tyrakowski]

1. Clone / download the "embedded" branch of this repo:
https://github.com/hipercompl/fbdbmobdemo
The 5.0.3 embedded aar is already there (in android/app/libs),
build.gradle is adjusted to include it as a runtime dependency.

2. Install Android SDK and flutter (from flutter.dev).
3. Go to the cloned repo and execute "flutter pub get" to download
dependencies.
4. Open emulator.
5. Execute "flutter run".

Alternatively, you can do "flutter build apk" and push the apk to the
emulator (the apk will be in build/app/outputs/flutter-apk/).
Flutter puts code for arm32, arm64 and x86_64 in the same bundle. The
.so libraries from Firebird are placed in architecture-specific
subdirectories in the apk (as usual).

I'll provide any assistance required, I can freely change the code of
both fbdbmobdemo and the fbdb driver to do additional testing.

In the meantime, I added preloading of all ICU libraries in the
background isolate, but it didn't change anything (still the same
error). The reference of DynamicLibrary.open says, that "Calling this
function multiple times with the same path, even across different
isolates, only loads the library into the DartVM process once", so it
shouldn't even be necessary to preload ICU in the background isolate.

I started to think that maybe the error is misleading, maybe it actually
has something to do with the tmp permission denied error reported in the
logs (I described it in the previous post), so i double-checked if I
clear the status vector before calling attach (to avoid reporting a
stale error), and as far as I can see, I do.

regards
Tomasz

[Tomasz Tyrakowski]

On 24.11.2025 at 12:31, Adriano dos Santos Fernandes wrote:
> First do what
> https://github.com/FirebirdSQL/examples/blob/main/android-cpp/README.md
> do in relation to
> https://github.com/FirebirdSQL/firebird/blob/master/android/embedded/src/main/java/org/firebirdsql/android/embedded/FirebirdConf.kt

OK, I'll set the env vars before loading libfbclient and see if it
helps. Do firebird files from assets really need to be copied to a
writable location? I see what extractAssets does, but I wonder why it
does it. I understand that lock and tmp have to be writable, but why the
asset files? Oh, is it because the assets cannot be simply opened as
standard files, but need to be read from the bundle with different API?
Haven't thought about it, but now it seems obvious (Firebird would need
to load e.g. config files differently on Android). Makes sense.
Anyway, thanks for the suggestion, it may improve things (I need some
time to implement it, though ;) ).

regards
Tomasz

[Adriano dos Santos Fernandes]

On 11/24/25 08:47, Tomasz Tyrakowski wrote:
> On 24.11.2025 at 12:31, Adriano dos Santos Fernandes wrote:
>> First do what
>> https://github.com/FirebirdSQL/examples/blob/main/android-cpp/README.md
>> do in relation to
>> https://github.com/FirebirdSQL/firebird/blob/master/android/embedded/
>> src/main/java/org/firebirdsql/android/embedded/FirebirdConf.kt
> OK, I'll set the env vars before loading libfbclient and see if it
> helps. Do firebird files from assets really need to be copied to a
> writable location? I see what extractAssets does, but I wonder why it
> does it. I understand that lock and tmp have to be writable, but why the
> asset files?

It should be files, not bundled files. fopen and similar low level APIs
do not open bundled files.

BTW, FirebirdConf class is present in the aar bundle, so you may call it
at least for testing.


Adriano

[Tomasz Tyrakowski]

On 24.11.2025 at 12:51, Adriano dos Santos Fernandes wrote:
> On 11/24/25 08:47, Tomasz Tyrakowski wrote:
>> OK, I'll set the env vars before loading libfbclient and see if it
>> helps. Do firebird files from assets really need to be copied to a
>> writable location? I see what extractAssets does, but I wonder why it
>> does it. I understand that lock and tmp have to be writable, but why the
>> asset files?
> It should be files, not bundled files. fopen and similar low level APIs
> do not open bundled files.

Yes, so I figured, but thanks for confirmation.

> BTW, FirebirdConf class is present in the aar bundle, so you may call it
> at least for testing.

I'm not sure it will be easier than copying the assets ;). Flutter
compiles to native code. Loading JVM class and using it requires a lot
of indirection. For something so simple as copying bundled resources to
a new location I'd rather just write Dart code.
Thanks again Adriano for pointing me in the right direction.
I wonder how that might be related to the ICU error, but maybe the fact
that fbintl.conf could not be loaded caused all the problems.

regards
Tomasz