Restart works, errno = 104 Firebird 2.5<LI-V2.5.2.26540
Abdul Muqtadir
localhost.localdomain (Server) Mon Sep 9 07:07:55 2024
INET/inet_error: read errno = 104
localhost.localdomain (Server) Mon Sep 9 07:07:55 2024
INET/inet_error: read errno = 104
localhost.localdomain (Server) Mon Sep 9 07:07:55 2024
INET/inet_error: read errno = 104
localhost.localdomain (Server) Mon Sep 9 07:07:55 2024
INET/inet_error: send errno = 32
and
localhost.localdomain (Server) Mon Sep 9 07:48:44 2024
INET/inet_error: read errno = 104
localhost.localdomain (Server) Mon Sep 9 07:49:05 2024
INET/inet_error: read errno = 9
localhost.localdomain (Server) Mon Sep 9 08:01:30 2024
INET/inet_error: read errno = 9
localhost.localdomain (Server) Mon Sep 9 08:10:49 2024
INET/inet_error: read errno = 110
localhost.localdomain (Server) Mon Sep 9 08:11:23 2024
Shutting down the server with 51 active connection(s) to 2 database(s), 0 active service(s)
localhost.localdomain (Client) Mon Sep 9 08:11:27 2024
/opt/firebird/bin/fbguard: /opt/firebird/bin/fbserver terminated abnormally (-1)
localhost.localdomain (Client) Mon Sep 9 08:11:27 2024
/opt/firebird/bin/fbguard: guardian starting /opt/firebird/bin/fbserver
localhost.localdomain (Client) Mon Sep 9 08:11:42 2024
/opt/firebird/bin/fbguard: guardian starting /opt/firebird/bin/fbserver
localhost.localdomain (Server) Mon Sep 9 08:12:32 2024
INET/inet_error: bind errno = 98
localhost.localdomain (Client) Mon Sep 9 08:12:32 2024
/opt/firebird/bin/fbguard: /opt/firebird/bin/fbserver terminated due to startup error (2)
localhost.localdomain (Client) Mon Sep 9 08:12:32 2024
/opt/firebird/bin/fbguard: /opt/firebird/bin/fbserver terminated due to startup error (2)
localhost.localdomain (Server) Mon Sep 9 08:13:12 2024
INET/inet_error: read errno = 9
localhost.localdomain (Server) Mon Sep 9 08:17:58 2024
Shutting down the server with 13 active connection(s) to 1 database(s), 0 active service(s)
localhost.localdomain (Client) Mon Sep 9 08:17:59 2024
/opt/firebird/bin/fbguard: /opt/firebird/bin/fbserver normal shutdown.
localhost.localdomain (Client) Mon Sep 9 08:18:14 2024
/opt/firebird/bin/fbguard: guardian starting /opt/firebird/bin/fbserver
localhost.localdomain (Server) Mon Sep 9 08:34:21 2024
INET/inet_error: read errno = 9
localhost.localdomain (Server) Mon Sep 9 09:07:26 2024
INET/inet_error: read errno = 9
Clueless at the moment and I have exhausted chatgpt as I am not a senior but still could not find the root cause. These errors are regular and when the application freezes, the firebird service is still running.
Can only think of one error that is consistent on how the application is usually freezing on more than 50 connections but there have been cases it has done so on way less than 50 connections but only happens on the Monday morning.
Karol Bieniaszewski
Hi
Look at output of gstat -h
And check if your application handle transaction correctly.
There are infos like:
Oldest transaction 166
Oldest active 167
Oldest snapshot 167
Next transaction 170
For details you can run some select on MON$ tables to see where the exact problem is.
PS. Which Firebird version?
Regards,
Karol Bieniaszewski
--
You received this message because you are subscribed to the Google Groups "firebird-support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebird-suppo...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/firebird-support/6de2ceea-a34d-4d4e-be58-34c977e9bd52n%40googlegroups.com.
Abdul Muqtadir
Flags 0
Checksum 12345
Generation 19744
Page size 4096
ODS version 11.1
Oldest transaction 17138
Oldest active 17139
Oldest snapshot 17139
Next transaction 17140
Bumped transaction 1
Sequence number 0
Next attachment ID 2614
Implementation ID 16
Shadow count 0
Page buffers 0
Next header page 0
Database dialect 3
Creation date May 24, 2016 14:24:40
Attributes force write
Variable header data:
Sweep interval: 20000
*END*
Karol Bieniaszewski
Hi
Few things.
- You are using discontinued version of Firebird and also 2.5.2 when the latest is 2.5.9 in the series.
- Use gstat -h when you have problems with db, not after restore 😉.
- You are using really low page size (4096).
- Page buffers are 0 in db, what is DefaultDBCachePages in Firebird.conf?
- What is server type, classic, superserver?
To view this discussion on the web, visit https://groups.google.com/d/msgid/firebird-support/edc894a5-ee62-408f-8ab5-39538630d010n%40googlegroups.com.
Abdul Muqtadir
SuperServer is running, based on the presence of the fbserver process.
Database header page information:
Flags 0
Checksum 12345
Page size 4096
ODS version 11.2
Oldest transaction 6164270
Oldest active 6164271
Oldest snapshot 6164271
Next transaction 6164272
Bumped transaction 1
Sequence number 0
Implementation ID 19
Shadow count 0
Page buffers 0
Next header page 0
Database dialect 3
Attributes force write
Variable header data:
Sweep interval: 20000
*END*
Flags 0
Checksum 12345
Page size 4096
ODS version 11.1
Oldest active 67315
Oldest snapshot 67315
Next transaction 67316
Bumped transaction 1
Sequence number 0
Implementation ID 16
Shadow count 0
Page buffers 0
Next header page 0
Database dialect 3
Attributes force write
Variable header data:
Sweep interval: 20000
*END*
Database header page information:
Flags 0
Checksum 12345
Page size 4096
ODS version 11.1
Oldest active 65425
Oldest snapshot 65425
Next transaction 65426
Bumped transaction 1
Sequence number 0
Implementation ID 24
Shadow count 0
Page buffers 0
Next header page 0
Database dialect 3
Attributes force write
Variable header data:
Sweep interval: 20000
*END*
Dimitry Sibiryakov
> Still stuck on the same issue, why do I have to restart firebird on Monday and
> why does it usually logs it like "Shutting down the server with 67 active
> connection(s) to 2 database(s), 0 active service(s)", to 2 databases is
> consistent on the firebird.log with a lots of INET/inet_error: read errno = 104
> when I end up having to restart.
databases at the moment when you decided to restart your server.
If your server is not locked for new attachments completely you'd better to
find out what it is busy with using monitoring tables. Perhaps you or someone
else scheduled a long process that uses a transaction with unusually high
isolation level.
Audit service can help you to track such situation.
--
WBR, SD.
Karol Bieniaszewski
I do not know how big is your database but
pageSize * DefaultDbCachePages = 4096 * 2048 = 8 MB cache memory used for support your database 😉
To view this discussion on the web, visit https://groups.google.com/d/msgid/firebird-support/e64128bb-275c-45e8-8aae-71b0fb9eba6fn%40googlegroups.com.
Abdul Muqtadir
Dimitry Sibiryakov
> Sorry about the incompetence, I am not a senior and just taking this opportunity
> at work to learn.
investigation.
But at least you can start from finding out what exactly "application stops
loading data" mean. Does it return an error? Does it wait for something?
If it waits - find out exactly what it is waiting for with precision up to
Firebird API call. Then you can proceed to finding out what can cause this wait.
Probably, the application developer was furious about the firing order and
planted a time bomb into the app in retaliation...
--
WBR, SD.
Abdul Muqtadir
If it waits - find out exactly what it is waiting for with precision up to
Firebird API call. Then you can proceed to finding out what can cause this wait.
Probably, the application developer was furious about the firing order and
planted a time bomb into the app in retaliation..."
find out what it is busy with using monitoring tables. Perhaps you or someone
else scheduled a long process that uses a transaction with unusually high
Mark Rotteveel
> Hi SD & Karol,
>
> Really appreciate your replies.
>
> memory_ used for support your database 😉"
> I am leaning towards this being the root cause as cannot think of
> anything else.
bugfix versions, which, IIRC, also were crashes and hangs.
Ultimately, you should also upgrade to a supported version, like
Firebird 5, but that will be more involved than a point release upgrade.
Mark
--
Mark Rotteveel
Dimitry Sibiryakov
> for now the app just keeps loading forever during the outage and is usually fast when it works.
> Please find the attached screenshot, maybe you can deduce something out of it.
> Also attaching firebird.log.
control over transactions lifetime.
Also perhaps there are issues with firewall or antivirus that blocks
secondary events connections.
Output of `top` shows high CPU load by Firebird which means that the server
is busy with something. Audit is a good way to find out what makes it busy.
--
WBR, SD.
Karol Bieniaszewski
Dimitry - about „Timeout occurred while waiting for a secondary connection for event processing” this issue occurred for us too.
And starting clearly new app even on different computer was not possible as Firebird stopped responding for events connections.
It respond for normal connections without events, but events stopped working.
Only restart of the Firebird service helped, no changes in the firewall was required at all. So looks like some issue with FB itself is possible here too.
Regards,
Karol Bieniaszewski
Od: 'Dimitry Sibiryakov' via firebird-support
Wysłano: środa, 11 września 2024 11:22
Do: firebird...@googlegroups.com
Temat: Re: [firebird-support] Restart works, errno = 104 Firebird 2.5<LI-V2.5.2.26540
Abdul Muqtadir wrote 11.09.2024 10:25:
--
You received this message because you are subscribed to the Google Groups "firebird-support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebird-suppo...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/firebird-support/dc5a9e80-127e-4b4d-ab8a-c35d21033855%40ibphoenix.com.
Dimitry Sibiryakov
> Dimitry - about „Timeout occurred while waiting for a secondary connection for
> event processing” this issue occurred for us too.
>
> And starting clearly new app even on different computer was not possible as
> Firebird stopped responding for events connections.
>
> It respond for normal connections without events, but events stopped working.
>
> Only restart of the Firebird service helped, no changes in the firewall was
> required at all. So looks like some issue with FB itself is possible here too.
--
WBR, SD.
Karol Bieniaszewski
Yes, it looks exactly the same.
Regards,
Karol Bieniaszewski
Od: 'Dimitry Sibiryakov' via firebird-support
Wysłano: środa, 11 września 2024 13:55
Do: firebird...@googlegroups.com
Temat: Re: [firebird-support] Restart works, errno = 104 Firebird 2.5<LI-V2.5.2.26540
'Karol Bieniaszewski' via firebird-support wrote 11.09.2024 13:43:
--
You received this message because you are subscribed to the Google Groups "firebird-support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebird-suppo...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/firebird-support/9f87e66f-4bfd-4739-9348-f351a110481d%40ibphoenix.com.
Abdul Muqtadir
[root@localhost bin]# ulimit -n
1024
[root@localhost bin]# sysctl net.core.somaxconn
net.core.somaxconn = 128
[root@localhost firebird]# pstree -p | grep fbserver
|-fbguard(26664)---fbserver(26665)-+-{fbserver}(26669)
| |-{fbserver}(26671)
| |-{fbserver}(26710)
| |-{fbserver}(26711)
| |-{fbserver}(26712)
| |-{fbserver}(26713)
| |-{fbserver}(26714)
| |-{fbserver}(3249)
| |-{fbserver}(3503)
| |-{fbserver}(3505)
| |-{fbserver}(6098)
| |-{fbserver}(8503)
| |-{fbserver}(11447)
| |-{fbserver}(11764)
| |-{fbserver}(11770)
| `-{fbserver}(12297)
[root@localhost firebird]# /usr/sbin/service firebird status
Firebird server [default] is running (pid 26665)
[root@localhost firebird]# free -h
total used free shared buff/cache available
Mem: 5.5G 1.5G 285M 148M 3.8G 3.6G
Swap: 15G 294M 15G
[root@localhost firebird]# netstat -an | grep 3050
tcp 0 0 0.0.0.0:3050 0.0.0.0:* LISTEN
tcp 0
0 *.*.*.* :3050 ********** :57204
ESTABLISHED
*more than 50 established user connections here below*
===================
[root@localhost firebird]# lsof -i :3050
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
fbserver 26665 firebird 0u IPv4 300649456 0t0 TCP *:gds_db (LISTEN)
fbserver 26665 firebird 3u IPv4 300768648 0t0 TCP localhost.localdomain:gds_db->************************:56512 (ESTABLISHED)
*more than 50 established user connections here below*
===============================
Abdul Muqtadir
Karol Bieniaszewski
Hi
Looking at the logs, nothing stands out. However, there are multiple updates recorded on the same table.
You can check the full gstat to see if there aren't too many record versions on the updated table.
There is a small gap in transactions, but it isn't significant (Oldest active vs. Next transaction).
However, without knowing the exact operation that is slow for you and causing you to stop the service, it's difficult for us to pinpoint the cause.
Regards,
Karol Bieniaszewski
To view this discussion on the web, visit https://groups.google.com/d/msgid/firebird-support/ea17b61b-9496-4046-b189-2929e0b6a914n%40googlegroups.com.
Abdul Muqtadir
Dimitry Sibiryakov
> What about this sql text, is this normal?
>
> MON$SQL_TEXT:
> update jigs set jig_job = ? where jig_id = ?
you must ask database and application developer.
> Where would you have me look next? I am just curious as to why I am unable to
> find the root cause. Are there any tools I can use to provide more information?
You are unable to find the root of the problem because you don't know how the
application really works. Examining trace logs can help you to find it out.
--
WBR, SD.
Abdul Muqtadir
Dimitry Sibiryakov
> I have enabled fbtrace and have confirmed that it is logging.
>
> But it has logged 1MB on the weekend with no activity to 1MB in one hour.
>
> Is the logging I enabled too extensive?
read such volume even with naked eyes.
--
WBR, SD.
Abdul Muqtadir
2024-09-23T09:04:36.3710 (27247:0xebbb28f8) ERROR AT jrd8_execute
/home/storage/network/files/ProdDatabase/PM-ARCHIVE.GDB (ATT_13147, SYSDBA:NONE, NONE, TCPv4:10.3.88.127)
335544466 : violation of FOREIGN KEY constraint "TRUSSDATA_CAMBER_FK" on table "TRUSSDATA_CAMBER"
335544838 : Foreign key reference target does not exist
2024-09-23T09:04:36.3720 (27247:0xebbb28f8) EXECUTE_STATEMENT_START
/home/storage/network/files/ProdDatabase/PM-ARCHIVE.GDB (ATT_13147, SYSDBA:NONE, NONE, TCPv4:10.3.88.127)
(TRA_68092, READ_COMMITTED | REC_VERSION | WAIT | READ_WRITE)
- first time violation appears here
Statement 48:
-------------------------------------------------------------------------------
INSERT INTO COUNTRY(COUNTRY."COUNTRY_ABBR", COUNTRY."COUNTRY_CAPITAL", COUNTRY."COUNTRY_CODE", COUNTRY."COUNTRY_FLAG_ICON", COUNTRY."COUNTRY_NAME", COUNTRY."COUNTRY_NUMBER") VALUES (?, ?, ?, ?, ?, ?)
param0 = char(3), "AND"
param1 = varchar(80), "Andorra"
param2 = char(2), "AD"
param3 = blob, "<NULL>"
param4 = varchar(80), "ANDORRA"
param5 = smallint, "20"
2024-09-23T07:00:05.9630 (27247:0xebbb28f8) ERROR AT jrd8_execute
/home/storage/network/files/ProdDatabase/PM-ARCHIVE.GDB (ATT_13147, SYSDBA:NONE, NONE, TCPv4:10.3.88.127)
335544665 : violation of PRIMARY or UNIQUE KEY constraint "PK_COUNTRY" on table "COUNTRY"
2024-09-23T07:00:05.9690 (27247:0xebbb28f8) EXECUTE_STATEMENT_START
/home/storage/network/files/ProdDatabase/PM-ARCHIVE.GDB (ATT_13147, SYSDBA:NONE, NONE, TCPv4:10.3.88.127)
(TRA_68092, READ_COMMITTED | REC_VERSION | WAIT | READ_WRITE)
Please advise, I'll dig deeper in the logs.
Abdul
Dimitry Sibiryakov
> Please advise, I'll dig deeper in the logs.
"time_treshold" setting can reduce log size limiting it to slow queries only.
--
WBR, SD.
Abdul Muqtadir
Mark Rotteveel
> Only question I have is from the above attached fbtrace.conf file, why
> is it not logging anything regarding services? It logs everything
> regarding databases to the audittrace.log 512MB file but
> the audittraceservices.log 0KB is completely empty I used the same
> config file that I have uploaded here to test.
using the trace facility. Maybe there are simply no service API calls done?
Mark
--
Mark Rotteveel
Abdul Muqtadir
Mark Rotteveel
> Yeah, you are right but I had enabled logging for 3 days which also
> included firebird restarts with creation of .GBK files every weekday at 2AM.
>
> We have a cronjob running to create those .GBK files every weekday. What
> kind of database backups will get logged in that file?
use the -se option with gbak.
Mark
--
Mark Rotteveel