Gearman server connection failed/refused

[Yao Li]

My app server adds tasks (indexing data) to Gearman server which is hosted on Google Cloud,  but it complains the following error (keep popping out the similar logs) and slows down the performance (over 10 hours to finish the indexing data from MySQL to Elasticsearch with 2.5 million rows data in MySQL), is there anyone have similar issue before? Thanks in advance for any idea or suggestion.

11:41:42.662 PHP Warning: fsockopen(): unable to connect to 10.138.0.12:4730 (Connection refused)

11:41:47.549 PHP Warning: GearmanWorker::work(): flush(GEARMAN_COULD_NOT_CONNECT) Connection to 10.138.0.12:4730 failed -> libgearman/connection.cc:712: pid(237)

11:41:47.549 PHP Warning: GearmanJob::sendStatus(): flush(GEARMAN_COULD_NOT_CONNECT) Connection to 10.138.0.12:4730 failed -> libgearman/connection.cc:712: pid(170)

11:41:47.549 PHP Warning: GearmanJob::sendStatus(): flush(GEARMAN_COULD_NOT_CONNECT) Connection to 10.138.0.12:4730 failed -> libgearman/connection.cc:712: pid(340)

11:41:47.670 Socket Error 111: Connection refused 10.138.0.12:4730

[Yao Li]

Gearman Server version 0.33

PHP 7.1

[Clint Byrum]

Can you connect to that ip/port with other tools, like netcat or telnet?

Also you mentioned you're running gearmand 0.33. Quite a few releases
have been made since 0.33. I don't think that's the problem, but, you
may want to consider upgrading.

Excerpts from Yao Li's message of 2017-08-21 11:23:13 -0700:

[Yao Li]

I can connect to the ip/port with telnet.

Is there any documents for upgrading?

[Clint Byrum]

Excerpts from Yao Li's message of 2017-08-22 08:44:10 -0700:

> I can connect to the ip/port with telnet.
>

Ok, doesn't make much sense why your app code can't then, unless you
have a local firewall which is specifically disallowing that executable
or user or host to connect. Check all of those things (like, login to
the box that has the errors, and telnet from the user your code runs as).

> Is there any documents for upgrading?
>

Just build and install, or find binary packages. But don't let that
distract you: this doesn't appear to be an issue with 0.33 that is fixed
in later versions.

[Yao Li]

Actually it can get connected in the very beginning, after running for a while, it complains connection failure, then it is re-connected after a period time because I used supervisor.

Usually there are 20 to 30 workers, it should be the acceptable amount, isn't it?

[Clint Byrum]

Yes. Is it possible you are restarting gearmand periodically for log
rotation or some other purpose?

Excerpts from Yao Li's message of 2017-08-22 11:24:39 -0700:

[Yao Li]

I set autorestart=true in supervisord.conf, it should only restart when the program exits as the supervisor config doc.

Workers get the job done in the average speed in staging environment (gearman server is hosted on Google Compute Engine, slower than local - my mac, but still acceptable fast), but slower and slower after a while, then keep a slow speed to finish jobs around 15 hours to index 2 million mysql rows.

(It seems workers are waiting for something, then log complains connection with gearman server failed/refuse, then supervisor make it reconnect, finally keeps an almost constant speed to go. )

[neil....@crispthinking.com]

Did you get to the bottom of this? I am having the same issues and seem to have a similar setup. When running on my Mac, not an issue, but on a google cloud vm with docker installed containerising everything its fine, then I get bursts of these errors, then works for a bit and repeats. Very annoying.

[Yao Li]

I limited concurrent thread number and it works fine.