I'm using Gearman to import/parse an xml feed in PHP.
A job will parse the xml(400.000 items) and for each item will create a new job to have it imported.
But if I just call "doBackground" in a loop the first jobs are reaching the gearman server and after some thousands of jobs, the gearman server seems to not be able to process the icoming requests as quickly as needed and just reject the request for some time after accepting these again a bit later.
Error I get:
"PHP Warning: GearmanClient::doLowBackground(): send_packet(GEARMAN_COULD_NOT_CONNECT) Failed to send server-options packet -> libgearman/connection.cc:433"
I tried to add some sleeping between the calls "usleep(5000);" but this seems to just postpone a bit the issue.
- Is this the expected behaviour?
- How could I improve the Gearman server to accept more jobs as fast as possible?
- Should I add another Gearman server?
- Should I increase the wait time between queuing requests?
Thanks for your help.