Re: Beanstalkd for large non-blocking file uploads on drupal

[Gordon Heydon]

Hi,

You can do this 2 ways.

1. You can implement a Drupal worker which will run this for you, using hook_cron_queue_info() http://drupalfunctions.com/api/drupal/drupal--modules--system--system.api.php/function/hook_cron_queue_info/7.x and then you have full access to the Drupal system, and then run the runqueue.sh which is in the beanstalkd directory.

or 2. You can write something else to follow the queue from a shell script or something to process the queue.

Hope this helps

Gordon.

On 29/10/2012, at 8:46 PM, galo <ga...@jpresult.nl> wrote:

Hi,

I'm very new with beanstalkd and work queues in general.

I am looking to setup a queue system to upload large files (~150MB). For specific reasons we decided to do this using SFTP (phpseclib). After some research I came to the conclusion that I would probably need something like a work queue. I decided to go with Beanstalkd since there was a drupal module for it.

I have managed to get a Beanstalkd server running and add items to the queue via the drupal module integration. But now I'm kind of confused as to how I would need to execute a specific script (SFTP upload).

Can anyone help with this particular part of getting beanstalkd queue items to run in a non-blocking way? The users of the site are already experiencing file upload to the web server, and I wouldn't want the to experience it again while uploading the file from the web server to a remote server. 

Do I need to maybe write a shell script to run a php cli or am I looking at it all wrong?

--
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To view this discussion on the web visit https://groups.google.com/d/msg/beanstalk-talk/-/UuKMRzuyRAgJ.
To post to this group, send email to beansta...@googlegroups.com.
To unsubscribe from this group, send email to beanstalk-tal...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.

[galo]

Hi,

Thanks for the reply. 

I setup the hook_cron_queue_info() in my module. But I have one gripe with that method. Even though the queue item isn't processed correctly it still deletes it from the queue.

I still can't get the runqueue.sh to work properly. Not sure how I should be running it. Do I need to create a cron job to run the runqueue.sh periodically?

[galo]

Hi again,

I now understand how the runqueue.sh works. It listens to beanstalkd and if there are ready jobs it executes the related function.

It's working like a charm. Thanks for the help. Much appreciated.

I did notice that the process memory keeps increasing. I saw some posts in the mailing-list that state that "Supervisor" is needed to take care of this. Will look into it.