INET/inet_error: send errno = 10054

[Casey Ketelsen]

Hello firebird support.

My Assessor's Office uses CAMAvision along with firebird. They are running version 2.1.1.1.17910-0. 

What is happening when randomly trying to save in CAMA through firebird they are receiving the following errors in the firebird.log-

COMPUTERNAME Wed Jun 16 18:13:33 2021

INET/inet_error: send errno = 10054, server host = xxxx, address = IP/3050

Any help on troubleshooting this error would be greatly appreciated.

Thanks,

Casey Ketelsen

IT Director - Mitchell County, IA

[Dimitry Sibiryakov]

17.06.2021 16:20, Casey Ketelsen wrote:
> What is happening when randomly trying to save in CAMA through firebird they are receiving
> the following errors in the firebird.log-
>

> COMPUTERNAMEWed Jun 16 18:13:33 2021

> INET/inet_error: send errno = 10054, server host = xxxx, address = IP/3050
>
>
> Any help on troubleshooting this error would be greatly appreciated.

It used to be a network issue such as tight idle TCP connections timeout on some
router/firewall.

--
WBR, SD.

[Mark Rotteveel]

On 17-06-2021 16:20, Casey Ketelsen wrote:
> Hello firebird support.
>
> My Assessor's Office uses CAMAvision along with firebird. They are
> running version 2.1.1.1.17910-0.
>
> What is happening when randomly trying to save in CAMA through firebird
> they are receiving the following errors in the firebird.log-
>

> COMPUTERNAMEWed Jun 16 18:13:33 2021

> INET/inet_error: send errno = 10054, server host = xxxx, address = IP/3050
>
>
> Any help on troubleshooting this error would be greatly appreciated.

I would recommend that you contact the support of Vanguard Appraisals,
Inc for support.

However, in short, error 10054 is connection reset by peer, which means
the other side unexpectedly closed/reset the connection (or potentially,
network hardware between the client application and the Firebird
server). This can range from issues in Firebird itself, or network
hardware/firewalls closing idle or long-running connections.

You may need to check the configuration of your network for such issues,
or otherwise mitigate them by decreasing the TCP keepalive delay of your
OS, or by enabling the Firebird DummyPacketInterval.

Given Firebird 2.1.1 is positively ancient, and has known security
vulnerabilities and other stability problems, I would highly recommended
to upgrade (if only to 2.1.7, which is still ancient and vulnerable, but
more secure and stable than 2.1.1). I assume that Vanguard Appraisals
support will be able to inform you if newer versions of Firebird (i.e.
2.5, 3.0 or 4.0) are supported with their application.

Mark
--
Mark Rotteveel

[Casey Ketelsen]

I appreciate the feedback. Will see about upgrading firebird version and looking at the keepAlive settings to prevent the dropped connections.

[Tim Crawford]

I see this all the time at customers, using Firebird 2.5.7 on Windows,
and not always related to actually lost connections that I can tell.
Does the client program actually get disconnected in our case?
Is your database Windows or Linux?

http://www.firebirdfaq.org/faq119/ says

WSAECONNRESET
(10054)
Connection reset by peer.
An existing connection was forcibly closed by the remote host.
This normally results if the peer application on the remote host is suddenly stopped,
the host is rebooted, or the remote host uses a hard close (see setsockopt for more
information on the SO_LINGER option on the remote socket.)
This error may also result if a connection was broken due to keep-alive activity detecting
a failure while one or more operations are in progress. Operations that were in progress
fail with WSAENETRESET. Subsequent operations fail with WSAECONNRESET.

I have never quite figured it out, I am not seeing failures by the client programs, so abandoned investigation.
We have had cases where users complain of lost connections, but do NOT see this error in the log...
One time the database server (Windows) had a network card driver with a power saving
option "disable when not in use to save battery" or some such turned on (!?!?)
and I seem to recall the 10054 errors showing up in that case, even though the description
above indicates it is the remote host that closed the connection

Since usually both our client programs and the database server are running on virtual machines,
I am suspicious of that, the underlying hardware resources get allocated for balancing.
I have no idea the ratio of virtual resources to physical resources the customer is using.
And the virtual machines do not have a 'real' network card (usually) but share the
one on the physical hardware.

Sorry for the long winded brain dump, thinking out loud...

  It used to be a network issue such as tight idle TCP connections timeout on some router/firewall. .

[Casey Ketelsen]

Sorry, I actually was wrong with the firebird version. They are running version  3.0.5.33220 on a windows 2012 server.

[Mark Rotteveel]

On 17-06-2021 16:48, Tim Crawford wrote:
> I have never quite figured it out, I am not seeing failures by the
> client programs, so abandoned investigation.
> We have had cases where users complain of lost connections, but do NOT
> see this error in the log...
> One time the database server (Windows) had a network card driver with a
> power saving
> option "disable when not in use to save battery" or some such turned on
> (!?!?)
> and I seem to recall the 10054 errors showing up in that case, even
> though the description
> above indicates it is the remote host that closed the connection

The basic case of a connection reset by peer is that the client or
server receives a TCP/IP RST packet. If a network interface goes AWOL on
the server like in your example, my guess is -- but I don't know for
sure -- that the TCP/IP stack will generate a RST (reset) packet for all
open TCP/IP connections on that server that used that network interface.
In other words, the network stack on the server notifies the
applications on that server; it is not actually the remote host, but it
looks as if the remote host sent the reset.

Same with some network hardware or routers. If they close idle or
long-running connections, they generate a RST packet and send it to both
sides of the connection. Again, it is not the remote host sending the
reset, but for both sides it looks as if the (other) remote host does it.

Mark
--
Mark Rotteveel

[Mark Rotteveel]

On 17-06-2021 16:56, Casey Ketelsen wrote:
> Sorry, I actually was wrong with the firebird version. They are running
> version 3.0.5.33220 on a windows 2012 server.

That makes stability problems with Firebird less likely (though it may
be worthwhile to upgrade to 3.0.7), and more likely network issues or
network hardware terminating idle connections.

Mark
--
Mark Rotteveel

[Dimitry Sibiryakov]

17.06.2021 16:58, Mark Rotteveel wrote:
> The basic case of a connection reset by peer is that the client or server receives a
> TCP/IP RST packet. If a network interface goes AWOL on the server like in your example, my
> guess is -- but I don't know for sure -- that the TCP/IP stack will generate a RST (reset)
> packet for all open TCP/IP connections on that server that used that network interface. In
> other words, the network stack on the server notifies the applications on that server; it
> is not actually the remote host, but it looks as if the remote host sent the reset.
>
> Same with some network hardware or routers. If they close idle or long-running
> connections, they generate a RST packet and send it to both sides of the connection.
> Again, it is not the remote host sending the reset, but for both sides it looks as if the
> (other) remote host does it.

Google gave me a very interesting paper:
https://portal.nutanix.com/page/documents/kbs/details?targetId=kA00e000000LTA6CAO

--
WBR, SD.

[Tim Crawford]

Thanks Mark and Dimitry for the excellent information!

[Tim Crawford]

Maybe of interest.... this happened to me on my laptop yesterday,
for the first time I have ever noticed. I had 2 different client side programs
running against a local database on my laptop. One and only one of them
lost its connection to the database and it was logged into he firebird.log as 10054
I'm almost positive it happened when a Cisco Netscaler Gateway connection
I had left going timed out and force disconnected me. That could explain
the lost connection as per Marks info below, but it's odd only one
of my two programs seems to have been affected... the one that didn't
an interactive program and the other was running without the user
interface active as a scheduled task. (Of course the program that
did failed a POS and does no logging, already needs a complete rewrite)
Anyway I thought it was interesting.