Error: read ECONNRESET

[Runnymede College]

Hi, 

Every night at 3:45am I get the following error and Etherpad shuts down:

[2017-11-20 03:45:13.519] [ERROR] console - Error: read ECONNRESET

    at exports._errnoException (util.js:1020:11)

    at TCP.onread (net.js:568:26)

    --------------------

    at Protocol._enqueue (/Users/server3/etherpad-lite/src/node_modules/mysql/lib/protocol/Protocol.js:135:48)

    at Connection.query (/Users/server3/etherpad-lite/src/node_modules/mysql/lib/Connection.js:185:25)

    at Timeout._onTimeout (/Users/server3/etherpad-lite/src/node_modules/ueberdb2/mysql_db.js:57:13)

    at ontimeout (timers.js:386:11)

    at tryOnTimeout (timers.js:250:5)

    at Timer.listOnTimeout (timers.js:214:5)

[2017-11-20 03:45:13.519] [INFO] console - graceful shutdown...

[2017-11-20 03:45:13.565] [INFO] console - db sucessfully closed.

I have to restart it manually every morning.

This is running on a Mac server under OS X 10.11.6

Any help would be appreciated.

[Luiza Figueiredo Pagliari]

Never heard of something like this... Some questions for you:

1. Is the DB still up at this time? The code that is being run when the exception is raised is supposed to keep MySQL running (see this PR here: https://github.com/Pita/ueberDB/pull/71/files)

2. Do you have the same issue when running on dirty DB, instead of MySQL? John gave some clues about what might be happening on this issue here: https://github.com/ether/etherpad-lite/issues/2148#issuecomment-42336415. Maybe some config on your MySQL instance?

Regards,

Luiza

--
You received this message because you are subscribed to the Google Groups "Etherpad Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to etherpad-lite-dev+unsubscribe@googlegroups.com.
To post to this group, send email to etherpad-lite-dev@googlegroups.com.
Visit this group at https://groups.google.com/group/etherpad-lite-dev.
For more options, visit https://groups.google.com/d/optout.

[Runnymede College]

Hi Luiza,

Thanks for your reply.

In answer to your questions:

1. Yes the DB is still up. I can reconnect normally every morning.

2. I have just now switched to dirtyDB and will see if it survive the night. 

This happens every night at exactly the same time (03:45), regardless of when the last user was logged in.

I had read that perhaps one of the plugins was attempting to communicate home and closed everything down if it didn't get through. However, I switched off all the plugins and the problem persists.

I'll check the patch you have sent me and see if it helps to keep the connection alive.

Best 

Luis

[Runnymede College]

dirtyDB seems to hold up OK.

I use the MySQL server for a number of DBs and I don't have a problem with connections. From what I have read, this is Etherpad requesting something from the DB or another server and not getting an answer. It should establish the connection but doesn't. 

At the time it disconnects there are no users and it disconnects at exactly the same time each night. There must be some process that gets activated at that time and as it can't connect home, it closes the whole thing down.

On Monday, 20 November 2017 09:15:44 UTC+1, Runnymede College wrote:

[Alexandros Kosiaris]

On Thursday, November 30, 2017 at 1:32:38 PM UTC+2, Runnymede College wrote:

dirtyDB seems to hold up OK.

I use the MySQL server for a number of DBs and I don't have a problem with connections. From what I have read, this is Etherpad requesting something from the DB or another server and not getting an answer. It should establish the connection but doesn't. 

At the time it disconnects there are no users and it disconnects at exactly the same time each night. There must be some process that gets activated at that time and as it can't connect home, it closes the whole thing down.

If it's at the same time each night, then there's a good chance this is a scheduled task. You should look at the various scheduled commands infrastructure's configuration for your operating system, e.g. cron, and try to see what matches.

[Runnymede College]

Thanks Alexandros,

Yup, I have checked the logs and cannot find anything strange, nor any process that kicks in at that time.

[2017-11-23 03:45:12.264] console - graceful shutdown...

[2017-11-23 03:45:12.268] console - db sucessfully closed.

[2017-11-24 03:45:15.512] console - graceful shutdown...

[2017-11-24 03:45:15.535] console - db sucessfully closed.

[2017-11-27 03:45:09.406] console - graceful shutdown...

[2017-11-27 03:45:09.433] console - db sucessfully closed.

[2017-11-28 03:45:11.528] console - graceful shutdown...

[2017-11-28 03:45:11.589] console - db sucessfully closed.

[2017-11-30 03:45:13.338] console - graceful shutdown...

[2017-11-30 03:45:13.400] console - db sucessfully closed.

[2017-12-01 03:45:09.598] console - graceful shutdown...

[2017-12-01 03:45:09.684] console - db sucessfully closed.

On the 28th I tried dirtyDB and it didn't quit on the 29th.

The server does not have much activity as there are no users working at that time. 

[Alexandros Kosiaris]

On Friday, December 1, 2017 at 4:44:07 PM UTC+2, Runnymede College wrote:

Thanks Alexandros,

Yup, I have checked the logs and cannot find anything strange, nor any process that kicks in at that time.

[2017-11-23 03:45:12.264] console - graceful shutdown...

[2017-11-23 03:45:12.268] console - db sucessfully closed.

[2017-11-24 03:45:15.512] console - graceful shutdown...

[2017-11-24 03:45:15.535] console - db sucessfully closed.

[2017-11-27 03:45:09.406] console - graceful shutdown...

[2017-11-27 03:45:09.433] console - db sucessfully closed.

[2017-11-28 03:45:11.528] console - graceful shutdown...

[2017-11-28 03:45:11.589] console - db sucessfully closed.

[2017-11-30 03:45:13.338] console - graceful shutdown...

[2017-11-30 03:45:13.400] console - db sucessfully closed.

[2017-12-01 03:45:09.598] console - graceful shutdown...

[2017-12-01 03:45:09.684] console - db sucessfully closed.

On the 28th I tried dirtyDB and it didn't quit on the 29th.

From the exception it's quite clear that the connection to mysql is abruptly being closed so it makes perfect sense that dirtyDB does not exhibit problems. In fact ECONNRESET happens when a TCP RST packet is sent which can happen when mysql is closing down (for a restart perhaps?) IIRC or some intermittent firewall decides to close the connection in its own. However I don't think a firewall would be doing this on 03:45 every day unless it was really weirdly configured. I would stop searching the issue on the host that runs etherpad and check the host that runs mysql and/or intermittent devices.

[Alexandros Kosiaris]

Btw, when I said intermittent, I meant intermediate.

[Runnymede College]

Thanks.

I've now installed MySQL on a local server and will test with that.