Jaybird 4.0.3 jvm crash

[sdnetwork]

Hello,

i have a problem on an spring application, in some condition it crashs the jvm, it occurs only if i set the connection in local or embedded type.

Env :

Debian 10

Tomcat + Spring

Java 8

Jaybird 4.0.3

Firebird 4

i can reproduce easily, but i don(t know how i can help to find where is the problem, it genrates a core dump but i'm not an expert to analyse it efficiently.

maybe somebody can help me to find where is the problem ? 

thanks in advance.

[Mark Rotteveel]

With Firebird 3 and higher, using local or embedded pretty much amount
to the same thing. However, I would strongly recommend to use the
pure-java protocol instead of using the native libraries. Personally, I
would avoid the hassle of using the native bindings unless you really
want to use Firebird Embedded.

In any case, as to your problem, I would need to have the JVM crash
information and the core dump (though to be honest, I have no experience
with analyzing coredumps myself, so I'm not sure how helpful they would
be).

Given you yesterday also posted this issue
https://github.com/mrotteveel/jaybird-fbclient/issues/1, I'm guessing
that you're also using the org.firebirdsql.jdbc:jaybird-fbclient
library. If that is the case, the problem might also be that you're
trying to use a 3.0.x fbclient to connect to 4.0.x using embedded. After
you posted your issue, I noticed that I hadn't yet released the 4.0.x
libraries, so I released org.firebirdsql.jdbc:jaybird-fbclient:4.0.0.0
yesterday. However, if you're running on a system that has Firebird
installed, you shouldn't need to use the
org.firebirdsql.jdbc:jaybird-fbclient library, as you already should
have the appropriate libfbclient installed locally.

Could you provide me with:

1. JVM crash report and coredump
2. Confirmation whether or not you are indeed using
org.firebirdsql.jdbc:jaybird-fbclient, and if so, which version.

Mark

[sdnetwork]

Hi,

I continued the tests and I finally understood in which case the crash occurred, it occurs if there is a call to a UDF (Freeadhoc in my case).

only linux plattform are concerned with embedded or local protocol (FB3 & FB4), the native connection works fine. 

On windows no problem.

When the crash occurs there is no jvm crash report, the dump is pretty big, how do you want to share to you ?)

I have try with org.firebirdsql.jdbc:jaybird-fbclient (3 & 4) on firebird server 3 & 4, same behaviour.

I think it's not a problem with jaybird, but i like understand this kind of problem often the tree that hides the forest ;)

[Mark Rotteveel]

On 2021-07-15 07:33, sdnetwork wrote:
> I continued the tests and I finally understood in which case the crash
> occurred, it occurs if there is a call to a UDF (Freeadhoc in my
> case).
> only linux plattform are concerned with embedded or local protocol
> (FB3 & FB4), the native connection works fine.
> On windows no problem.
>
> When the crash occurs there is no jvm crash report, the dump is pretty
> big, how do you want to share to you ?)
> I have try with org.firebirdsql.jdbc:jaybird-fbclient (3 & 4) on
> firebird server 3 & 4, same behaviour.
>
> I think it's not a problem with jaybird, but i like understand this
> kind of problem often the tree that hides the forest ;)

I think this happens **because** you're using
org.firebirdsql.jdbc:jaybird-fbclient, you should use the libfbclient of
your local Firebird installation instead. I guess this is due to some
incompatibility between how the libfbclient.so of
org.firebirdsql.jdbc:jaybird-fbclient is compiled and the local
libremote13.so (and other dependencies) of your local Firebird
installation. It could also be some kind of incompatibility with loading
Firebird embedded and/or the UDF into the Java process.

Using the libfbclient.so of your local installation will likely fix
that. If Jaybird cannot automatically find libfbclient.so, make sure
that is available (e.g. as a symlink) on the LD_LIBRARY_PATH, or specify
the directory containing libfbclient.so in the system property
jna.library.path.

As to sharing the dump, you could use something like wetransfer and send
me the link privately.

But I really have to ask: why do you want to use the native, local or
embedded protocols?

Using the pure Java protocol (the default) would probably do away with
all of these problems (and would simplify your deployment).

Mark

[Arioch The]

Core devs promised to totally remove old skool UDF support in FB4, or didn't them yet? 

[Jiří Činčura]

> Core devs promised to totally remove old skool UDF support in FB4, or
> didn't them yet?

It's deprecated, but still available.

--
Mgr. Jiří Činčura
https://www.tabsoverspaces.com/

[sdnetwork]

Hi,

unfornutately i have also try without the org.firebirdsql.jdbc:jaybird-fbclient and i obtain the same behaviour with FB3 and FB4.

the beginning of my story with embedded and local is this part of documentation : 

> The NATIVE and LOCAL types (JDBC Type 2) use a JNA proxy to access the Firebird client library and requires installation of the Firebird client. The NATIVE driver type is used to access the remote database server, the LOCAL type accesses the database server running on the same host by means of IPC (Inter-Process Communication). Performance of NATIVE driver is approximately 10% lower compared to the PURE_JAVA driver, but LOCAL type has up to 30% higher performance compared to the PURE_JAVA driver when connecting the server on the same host. This is mostly due to the fact that TCP/IP stack is not involved in this mode.

it talks about a 30% improvment with local protocol ;)

i will prepare a wetransfert link for the crash dump

[Arioch The]

> it talks about a 30% improvment with local protocol ;)

Sure, XNET has much less cost than TCP stack, but then network bytes
joggling is not the only thing DB does.

This 30% sounds like an extreme edge case when an application was
explicitly designed to put most work on the transport layer and almost
none upon the very database.
Say, let's take "select 1 from rdb$database" query and would in a
loop prepare/unprepare it without ever executing it. This would show
us huge latency costs in TCP path - but real applications don't act
like that. So you better do your tests w.r.t. YOUR real patterns of
database usage. Maybe your real app would score 29% out of those "up
to 30" or maybe just 1%.

Plus on top of that there was one modestly recent change in Firebird.

https://github.com/FirebirdSQL/firebird/issues/4880

https://sourceforge.net/p/firebird/mailman/firebird-devel/thread/5425915...@yandex.ru/

[sdnetwork]

. Maybe your real app would score 29% out of those "up
to 30" or maybe just 1%.

i totaly agree with you, that is why i want to test on my real app but it crashs ;) 

[Arioch The]

>> . Maybe your real app would score 29% out of those "up
>> to 30" or maybe just 1%.
>>
>
> i totaly agree with you, that is why i want to test on my real app but it crashs ;)

But you said before:

>>>> On windows no problem.

...so go and test? On Windows?

TCP stack on Linux is knowingly faster than on Windows, so if Windows
would be good enough then Linux too.

------

Thing is, traditionally most Interbase UDFs were developed for Win32
with Delphi.

Win64, FreePascal / VirtualPascal - could come later as afterthoughts
and niche cases.

Linux? What is it?

And this: Last published version of the FreeadhocUDF is version adhoc
2010106 from 2010-12-06 (St.Nicholas edition)

And then you can not even download sources any more - their FTP is
dead as a doorknob.

Here is some old sources -
https://packages.altlinux.org/ru/p8/srpms/FreeAdhocUDF/rpms
...and to me they shout "don't touch it with a flagpole in Linux, or
own it and totally refactor it and make them your own at fullest"

Even on Windows it seems barely usable any more...
...https://github.com/FirebirdSQL/firebird/issues/5583

So i believe you have a "simple" choice.

- you became the new FreeadhocUDF maintainer, master plain C shared
libraries development on Linux, master Firebird plain C interface on
Linux, find FreeadhocUDF recent sources and modernize them to match
the C compiler of your distro's Firebird and so forth.
- you took it as a sign of times and you get rid of this UDF library
altogether (yes, cleaning up tech debt is easier said than done)
- - finding plain SQL replacement for them, and
- - moving non-replaced parts into application itself, and
- - the rest few functions you re-implement as your own, from
scratch, clean small C UDF library, fully under your control.
- you do not move to Linux and keep praying that a next build of
Windows or Firebird for Windows won't kill compatibility with this
ancient library for you.


Personally i like UDF interface as a relatively no-hassle simple tool
for simple things, compared to overengineered UDR that is quite hard
to grasp and would be next to impossible to debug on binary level if
auto-tooling would fail. That said, even me the proponent did not have
real use for UDFs for ages.

[Mark Rotteveel]

On 2021-07-15 15:01, sdnetwork wrote:
> unfornutately i have also try without the
> org.firebirdsql.jdbc:jaybird-fbclient and i obtain the same behaviour
> with FB3 and FB4.
>
> the beginning of my story with embedded and local is this part of
> documentation :
>
>> The NATIVE and LOCAL types (JDBC Type 2) use a JNA proxy to access
> the Firebird client library and requires installation of the Firebird
> client. The NATIVE driver type is used to access the remote database
> server, the LOCAL type accesses the database server running on the
> same host by means of IPC (Inter-Process Communication). Performance
> of NATIVE driver is approximately 10% lower compared to the PURE_JAVA
> driver, but LOCAL type has up to 30% higher performance compared to
> the PURE_JAVA driver when connecting the server on the same host. This
> is mostly due to the fact that TCP/IP stack is not involved in this
> mode.
>
> it talks about a 30% improvment with local protocol ;)

That number is only on Windows (and with Firebird 3 and 4 that would
only work when overriding the providers option, or maybe explicitly
including XNET:// before the filename, though I'm not sure the URL
parser doesn't choke on that). On Firebird 2.5 on Linux, LOCAL would be
identical to NATIVE (using TCP/IP to localhost). On Firebird 3.0 and
higher - by default - LOCAL will use Firebird Embedded (if available) on
any platform, so basically behaves the same as EMBEDDED. If the embedded
engine cannot be loaded, it will - same as Firebird 2.5 - behave
identical to NATIVE on Linux.

And I would take those performance numbers with a grain of salt. To be
honest, I can't recall if I benchmarked that myself, or if I just reused
information by one of my predecessors based on the previous JNI-based
implementation. In addition, something that is faster, but crashes and
burns when you try to use it, is really not better than something that
might be slightly slower but doesn't crash.

The fact UDFs are involved complicates things a lot. It is entirely
possible that the UDF is buggy, or does something (or causes the
Firebird embedded engine to do something) that conflicts with something
else inside the Java process, and therefor causes a JVM crash.

> i will prepare a wetransfert link for the crash dump

Thanks. However, let me be clear that I'm not sure what - if anything -
I'll be able to do with that. Do not get your hopes up.

[Mark Rotteveel]

Microsoft now recommends not using TCP Loopback Fast Path (which BTW is
a Windows only feature, and the OP is using Linux), because it can
actually introduce issues when certain software-based firewalls or other
things intercepting socket creation are used. They now consider this
option to be a mistake.

Mark

[Arioch The]

For what i know (by rumours) it is because Linux had that "Fast path"
at the core of their TCP stack. You always had it on Linux for
granted, you did not have to "opt in". After all LOTS of Linux local
applications used TCP for purely local communications, starting with
almost all X GUI software. Windows apps instead prefered using
inherently local IPCs based on shared memory. Until portable cross-OS
applications became a tad Microsoft had no intention to optimize
non-networked TCP. But at Windows 8 they were catching up...

I did not know Microsoft backtracked though. Well, it always sounded
weird to me they were making it opt-in rather than just bringing it to
their stack's core for all local-to-local connections. And if some
firewall is not compatible to it, then perhaps ball was in firewall's
park to fix it?

it seems that is not about 3rd party actually at all...
https://github.com/mgravell/Pipelines.Sockets.Unofficial/issues/58

чт, 15 июл. 2021 г. в 19:56, Mark Rotteveel <ma...@lawinegevaar.nl>:

> --
> You received this message because you are subscribed to the Google Groups "firebird-java" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to firebird-jav...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/firebird-java/34a010a3d71e17a57beaf3ce82eea3ee%40lawinegevaar.nl.

[sdnetwork]

this is the link of dump 

https://we.tl/t-YSF15NTKuB

[Jiří Činčura]

> Microsoft now recommends not using TCP Loopback Fast Path (which BTW is
> a Windows only feature, and the OP is using Linux), because it can
> actually introduce issues when certain software-based firewalls or other
> things intercepting socket creation are used. They now consider this
> option to be a mistake.

In fact, it's even marked as deprecated. Even in WDK.

[Arioch The]

> In fact, it's even marked as deprecated. Even in WDK.

Well, Jiří, what is the proper way to connect to MS SQL ?

It is ODBC, right?
Oh no more, it is OLE DB now, right?
Oh no, it is ODBC again! OLE DB is deprecated!
Oh no, OLE DB is no more deprecated and was it ever?

Well, you get my drift? :-)

[Jiří Činčura]

Not sure what MS SQL has to do with it (and with Jaybird)…

--
Mgr. Jiří Činčura
https://www.tabsoverspaces.com/

> --
> You received this message because you are subscribed to the Google
> Groups "firebird-java" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to firebird-jav...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/firebird-java/CAB_oP34hp2goZ6T_JcXLpMvpwO3qRhY88qt%3DY_7B8ARVovrpRQ%40mail.gmail.com.
>

[Arioch The]

Nothing, but it has to do with Microsoft's claims of revolutionism and
obsoleteness in technologies

пт, 16 июл. 2021 г. в 12:53, Jiří Činčura <ji...@cincura.net>:

> To view this discussion on the web visit https://groups.google.com/d/msgid/firebird-java/1ad27147-1c95-42c1-a494-b30888adb64c%40www.fastmail.com.

[sdnetwork]

I have done a last test 

1) i have create the more simple UDF that is possible

2) do a simple java to launch the UDF

and .... it crashs, i send a link to try if you want, for me i think that fbembed + udf + java don't seems to be compatible ;)

this is the link https://we.tl/t-Qk6F90ihEb.

don't forget to shutdown the server to be sure that use the fbembed ;)

[Mark Rotteveel]

On 16-07-2021 17:36, sdnetwork wrote:
> I have done a last test
>
> 1) i have create the more simple UDF that is possible
> 2) do a simple java to launch the UDF
>
> and .... it crashs, i send a link to try if you want, for me i think
> that fbembed + udf + java don't seems to be compatible ;)
>
> this is the link https://we.tl/t-Qk6F90ihEb.
>
> don't forget to shutdown the server to be sure that use the fbembed ;)

Thanks, I'll try to reproduce. However, it would still be helpful to
also have the JVM crash report. Normally when the JVM crashes, it
creates a hs_err_<pid>.log in the current working directory, or if that
is not writable, in the temp directory.

Also, what is the exact Java version you're using?

Mark
--
Mark Rotteveel

[sdnetwork]

Thanks, I'll try to reproduce. However, it would still be helpful to
also have the JVM crash report. Normally when the JVM crashes, it
creates a hs_err_<pid>.log in the current working directory, or if that
is not writable, in the temp directory.

Unfortunately in this case the hs_err file is not generated ...

 

Also, what is the exact Java version you're using?

 

i have try with adoptopenjdk 8 & 11

openjdk version "11.0.11" 2021-04-20

OpenJDK Runtime Environment AdoptOpenJDK-11.0.11+9 (build 11.0.11+9)

OpenJDK 64-Bit Server VM AdoptOpenJDK-11.0.11+9 (build 11.0.11+9, mixed mode)

openjdk version "1.8.0_292"

OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_292-b10)

OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.292-b10, mixed mode)

[Mark Rotteveel]

I tried on Ubuntu 20.04.2 LTS with openjdk 1.8.0.292 (Ubuntu build) and
Firebird 3.0.5 (as that is apparently the latest in the repo).

I also had to rename the library to FreeAdhocUDF.so and put it in the
right UDF directory (and modify the firebird.conf to allow UDF usage),
so make sure you were actually trying with your reduced version of the
library.

The application worked as expected, no crash.

Mark
--
Mark Rotteveel

[sdnetwork]

humm there are some differences with my environment but it really strange ...

Env : debian 10, adoptopenjdk, firebird install from binary.

> so make sure you were actually trying with your reduced version of the library.

yes i'm sure that i use the reduced version (the output is different).

in your test are you sure that the fbserver is down ?, because i have remarked that if fallback easily to a native connection if for example your user don't have r/w access of database file.

An other thing, it's rare but sometimes it doesn't crash so in the little java example you can add a while(true) :

    while (true) {

            try (Statement stmt = connection.createStatement();

                 ResultSet rs = stmt.executeQuery("select * from sys_get_uuid")) {

                    if (rs.next()) {

                      System.out.println("Database name: " + rs.getString(1));

                    }

            }

     }

now, i will replace step by step each part of my environment to see where the problem disapear.

can you just confirm the program works as expected even the fbserver down and with the wile please.

Thx

[Arioch The]

Now this makes me wonder if there are warranties rs and stmt objects are freed in every loop iteration, or they keep living in thousands until late and lazy GC kicks in. 

Does modern Java has that C#/Scala using(obj expression) {.....} pattern with timely resources releasing ? 

I wonder if TraceAPI (FBProfiler, IBExpert, etc) would show your server has those queries closed timely and pooled to some reasonable finite amount, or nor. 

Additionally I wonder if that function can get called simultaneously from different threads... 

In a sense, is there prohibition for some experimental JVM to parallelise exit-less while loop?.. 

сб, 17 июл. 2021 г., 13:59 sdnetwork <sdne...@gmail.com>:

--

You received this message because you are subscribed to the Google Groups "firebird-java" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebird-jav...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebird-java/fc941d29-40e2-4c61-983e-90dc20b0bddbn%40googlegroups.com.

[Mark Rotteveel]

On 17-07-2021 13:17, Arioch The wrote:
> Now this makes me wonder if there are warranties rs and stmt objects are
> freed in every loop iteration, or they keep living in thousands until
> late and lazy GC kicks in.
>
> Does modern Java has that C#/Scala using(obj expression) {.....} pattern
> with timely resources releasing ?

The code shown is using try-with-resources, which does exactly that.

> I wonder if TraceAPI (FBProfiler, IBExpert, etc) would show your server
> has those queries closed timely and pooled to some reasonable finite
> amount, or nor.
>
> Additionally I wonder if that function can get called simultaneously
> from different threads...
> In a sense, is there prohibition for some experimental JVM to
> parallelise exit-less while loop?..

No, that is simply not how Java works.

Mark

--
Mark Rotteveel

[sdnetwork]

I have just finish my comparaison between the Mark env and mine

and the result that is it works on ubuntu 20 (as Mark) but not on debian 10 .... amazing !!!! i have start with clean distrib only java and firebird added.

the firebird version is the same, i have even copied the ubuntu libfbclient on debian but no success ....

so to reproduce we need a debian 10

[sdnetwork]

unfortunately the problem occurs also on ubuntu but need more iteration than with debian. 

[Mark Rotteveel]

On 18-07-2021 07:00, sdnetwork wrote:
> unfortunately the problem occurs also on ubuntu but need more iteration
> than with debian.

I actually just ran the same application I ran yesterday, and it crashed
immediately. So it seems the reproduction is possible, but at least
inconsistent. I also didn't have a hs_err<pid>.log file as far as I can
tell.

I'll investigate further, but it might take a while.

Mark

--
Mark Rotteveel

[Mark Rotteveel]

On 17-07-2021 12:59, sdnetwork wrote:
> in your test are you sure that the fbserver is down ?, because i have
> remarked that if fallback easily to a native connection if for example
> your user don't have r/w access of database file.
> An other thing, it's rare but sometimes it doesn't crash so in the
> little java example you can add a while(true) :
>     while (true) {
>             try (Statement stmt = connection.createStatement();
>                  ResultSet rs = stmt.executeQuery("select * from
> sys_get_uuid")) {
>                     if (rs.next()) {
>                       System.out.println("Database name: " +
> rs.getString(1));
>                     }
>             }
>      }
>
> now, i will replace step by step each part of my environment to see
> where the problem disapear.
>
> can you just confirm the program works as expected even the fbserver
> down and with the wile please.

Does it crash for you during the loop, or only when you terminate the
application? So far I've only been able to replicate it when terminating
the application, which suggests the problem is with the shutdown or
cleanup of the embedded engine.

Mark
--
Mark Rotteveel

[sdnetwork]

> Does it crash for you during the loop, or only when you terminate the application?

it crashs during the loop

[sdnetwork]

maybe a clue, I have compiled firebird in debug and the problem does not occur anymore ...

With gdb I got the stack before the call to the UDF function and I noticed that there was an optimization (<optimized out>) done probably due to a compilation directive like -O3

this is the stack in debug : 

#2  0x00007fffa27a2f34 in invoke (tdbb=0x7ffff5bb3190, function=0x7fff9e326d40, return_ptr=0x7fff9e00f0f0, value=0x7fff9dfac510, args=0x7ffff5bb17b0, return_blob_struct=0x0, result_is_null=@0x7ffff5bb193f: false, 

    udfError=@0x7ffff5bb1938: UeNone) at /home/java/firebird/temp/Debug/jrd/fun.cpp:1079

in release :

#2  invoke (tdbb=<optimized out>, function=0x7fff9e5c2750, value=0x7fff9e5ae480, args=0x7ffff5bb1710, return_blob_struct=0x0, result_is_null=@0x7ffff5bb1673: false, udfError=@0x7ffff5bb1674: UeNone, return_ptr=<optimized out>)

    at /home/java/firebird/temp/Release/jrd/fun.cpp:1079

we can see two variables with optimized out , tdbb and return_ptr

maybe an optimization too much agressive from gcc ? 

the library that seems call the invoke function is libengine12.so

[Adrian Marius Popa]

libengine12.so is from Firebird 3.0 

[Mark Rotteveel]

On 18-07-2021 13:17, sdnetwork wrote:
> maybe a clue, I have compiled firebird in debug and the problem does not
> occur anymore ...
> With gdb I got the stack before the call to the UDF function and I
> noticed that there was an optimization (<optimized out>) done probably
> due to a compilation directive like -O3
>
> this is the stack in debug :
>
> #2  0x00007fffa27a2f34 in invoke (tdbb=0x7ffff5bb3190,
> function=0x7fff9e326d40, return_ptr=0x7fff9e00f0f0,
> value=0x7fff9dfac510, args=0x7ffff5bb17b0, return_blob_struct=0x0,
> result_is_null=@0x7ffff5bb193f: false,
>     udfError=@0x7ffff5bb1938: UeNone) at
> /home/java/firebird/temp/Debug/jrd/fun.cpp:1079
>
> in release :
>
> #2  invoke (tdbb=<optimized out>, function=0x7fff9e5c2750,
> value=0x7fff9e5ae480, args=0x7ffff5bb1710, return_blob_struct=0x0,
> result_is_null=@0x7ffff5bb1673: false, udfError=@0x7ffff5bb1674: UeNone,
> return_ptr=<optimized out>)
>     at /home/java/firebird/temp/Release/jrd/fun.cpp:1079
>
> we can see two variables with optimized out , tdbb and return_ptr
>
> maybe an optimization too much agressive from gcc ?
> the library that seems call the invoke function is libengine12.so

Unfortunately, I have to say I don't know what to do about this. Linux
isn't really my forte, and my attempts to get more information out of
the crash I can reproduce locally has gotten me nowhere.

Maybe you can get more help on firebird-devel.

Mark
--
Mark Rotteveel