Timeout too short when connecting to MySQL server resulting in crash

[Gil Zigelman]

Hi,

I am using Orthanc as a docker image (19.6.2) and MySql docker image (8.0).

Sometimes, after starting docker-compose, I see that Orthanc has crashed with error that the DB is not available. This happens about 20 seconds after I start the docker-compose. If I then start the crashed container later using 'docker start' then it works fine.

Is there a solution for this? Can I set the timeout for DB connection to be longer?

Following is an image of what I see in the logs:

Thank you very much,

Gil

[Sébastien Jodogne]

Hello,

Your init/systemd script to start the Orthanc service should depend on the MySQL service to have started. Please have a look at the examples provided on the following page:

https://book.orthanc-server.com/faq/debian-daemon.html

For init flavor, your daemon should have lines similar to:

# Should-Start:      postgresql mysql
# Should-Stop:       postgresql mysql

And for systemd flavor:

[Unit]

Requires=mysql.service postgresql.service

HTH,

Sébastien-

On Wednesday, September 25, 2019 at 9:16:48 PM UTC+2, Gil Zigelman wrote:

Hi,

I am using Orthanc as a docker image (19.6.2) and MySql docker image (8.0).

Sometimes, after starting docker-compose, I see that Orthanc has crashed with error that the DB is not available. This happens about 20 seconds after I start the docker-compose. If I then start the crashed container later using 'docker start' then it works fine.

Is there a solution for this? Can I set the timeout for DB connection to be longer?

Following is an image of what I see in the logs:

[Gil Zigelman]

Thank you for your quick answer.

I am using dockers, so the MySql is run in a different container, therefore from what I understand I cannot wait in the Orthanc container while monitoring for a local MySql service to be started (MySql is not local...)

Am I missing something?

Thank you very much,

Gil

בתאריך יום חמישי, 26 בספטמבר 2019 בשעה 09:56:10 UTC+3, מאת Sébastien Jodogne:

[Alain Mazy]

Hi Gil,

In your docker-compose, you should configure the restart policy of Orthanc such that it restarts after a failure so, at the second startup, it should be able to connect to the mysql container.

https://bitbucket.org/osimis/orthanc-setup-samples/src/master/docker/mysql-easy/docker-compose.yml 

 

BTW, this problem is not specific to Orthanc but to docker-compose in general and there might be other solutions over the web.

Best regards,

Alain.

--
You received this message because you are subscribed to the Google Groups "Orthanc Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orthanc-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/orthanc-users/e184f320-e362-4efa-afe5-2aed829d01c8%40googlegroups.com.

--

Alain Mazy / Software Developer
a...@osimis.io / +32 494 31 67 27

	OSIMIS S.A.  Quai Banning 6BE-4000 Liège  www.osimis.io

[Gil Zigelman]

Hi Alain,

I know about the option to restart the service, but I still think Orthanc should not crash with an exception if there is no DB connection. 

When working in a distributed environment there could be many cases where another remote service may not be available, for many reasons. I think this should be handled internally without crashing the system.

what do you think?

Thanks,

Gil

בתאריך יום חמישי, 26 בספטמבר 2019 בשעה 15:22:58 UTC+3, מאת Alain Mazy:

To unsubscribe from this group and stop receiving emails from it, send an email to orthan...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/orthanc-users/e184f320-e362-4efa-afe5-2aed829d01c8%40googlegroups.com.

[Alain Mazy]

I think this "crash/exit" only happens when the DB is not available at startup.  To me, it makes sense not to retry a billion times at that time.  However, I agree we could implement a smarter retry policy.

During operation, Orthanc shall not crash if the DB is not available although there are still some improvements to be implemented regarding that topic: https://bitbucket.org/sjodogne/orthanc/issues/83/serverindex-shall-implement-retries-for-db

To unsubscribe from this group and stop receiving emails from it, send an email to orthanc-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/orthanc-users/65be2f13-e18b-4682-91cc-a71266f9019e%40googlegroups.com.

[Gil Zigelman]

Thanks!

When working with docker and containers, it usually takes a lot of time for DB services to start. For example, for SqlServer it could be a few minutes until it enables clients connection.

This will ALWAYS happen on docker-compose environment startup, and Orthanc will repeatedly crash during that time...

I agree it could be handled better :-)

Thanks for all your opinions,

Gil

בתאריך יום חמישי, 26 בספטמבר 2019 בשעה 17:03:44 UTC+3, מאת Alain Mazy:

I think this "crash/exit" only happens when the DB is not available at startup.  To me, it makes sense not to retry a billion times at that time.  However, I agree we could implement a smarter retry policy.

During operation, Orthanc shall not crash if the DB is not available although there are still some improvements to be implemented regarding that topic: https://bitbucket.org/sjodogne/orthanc/issues/83/serverindex-shall-implement-retries-for-db

To view this discussion on the web visit https://groups.google.com/d/msgid/orthanc-users/65be2f13-e18b-4682-91cc-a71266f9019e%40googlegroups.com.

[Sébastien Jodogne]

Hello,

Please have a look at this part of the documentation of docker-compose that discusses how to control the order of startup of the services:

https://docs.docker.com/compose/startup-order/

You will find examples to wait for PostgreSQL being started before launching Orthanc.

HTH,

Sébastien-

[Gil Zigelman]

Hi Sébastien,

Thank you for the answer.

I agree this can solve the issue, but it requires generating a new customized Orthanc docker image for the deployment.

Your link declares that the recommended solution is to:

design your application to attempt to re-establish a connection to the database after a failure. If the application retries the connection, it can eventually connect to the database.
The best solution is to perform this check in your application code, both at startup and whenever a connection is lost for any reason

The script solution is suggested as an alternative if this is not possible ;-)

In any case, its good to have a solution, even if not optimal :-)

Best regards,

Gil

בתאריך יום ראשון, 6 באוקטובר 2019 בשעה 12:49:43 UTC+3, מאת Sébastien Jodogne:

[Sébastien Jodogne]

Hello,

What would you concretely suggest as a complete solution?

Having a configuration option to make Orthanc endlessly retry to connect to the database (instead of 10 times as currently implemented)?

Sébastien-

--

You received this message because you are subscribed to the Google Groups "Orthanc Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to orthanc-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/orthanc-users/e21e932b-c1bd-4818-ab40-0a9f442a1915%40googlegroups.com.

[Gil Zigelman]

Hi,

I think having the ability to control the timeout for DB access failure is a good idea. The developer can choose whether to crash Orthanc after the timeout, or to continue retries indefinite.

Until connection to the DB is finalized, all DB related API to Orthanc should fail, or even the rest API should be blocked.

But this is just my though :-)

Thanks for the great tool!!

Gil

בתאריך יום שני, 7 באוקטובר 2019 בשעה 09:29:42 UTC+3, מאת Sébastien Jodogne:

Hello,

What would you concretely suggest as a complete solution?

Having a configuration option to make Orthanc endlessly retry to connect to the database (instead of 10 times as currently implemented)?

Sébastien-

To unsubscribe from this group and stop receiving emails from it, send an email to orthan...@googlegroups.com.

[Sébastien Jodogne]

Hello,

Back to this old thread, the MySQL plugin 3.0 for Orthanc introduced the "MaximumConnectionRetries" and "ConnectionRetryInterval" (in seconds) configuration options.

You could for instance set "MaximumConnectionRetries" to 10 (default value) and "ConnectionRetryInterval" to 10 in order to wait for about one and a half minute.

HTH,

Sébastien-