Re: Can I use beanstalkd to queue up tasks that can be executed at a specific time during the day?

[Keith Rarick]

On Mon, May 20, 2013 at 5:05 AM, Abishek.R Srikaanth
<abishekr...@gmail.com> wrote:
> Can I use beanstalkd to queue up tasks that can be executed at a specific
> time during the day?

Yes, but I wouldn't recommend it.

You can approximate it by subtracting the current time
from the time you want and adding a delay for that duration.
But there's no guarantee the job will start on time; you must
make sure there are workers working at the right time and
that no other jobs are ahead in line keeping them busy.

The general meaning of beanstalkd's delay is to wait *at
least* the specified amount of time.

> Would I be able to replace beanstalkd in performing jobs that a cron can
> perform?

If you need cron, use cron.

[Chad Kouse]

what he said.

-- chad

--

You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.

To unsubscribe from this group and stop receiving emails from it, send an email to beanstalk-tal...@googlegroups.com.

To post to this group, send email to beansta...@googlegroups.com.

Visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.

For more options, visit https://groups.google.com/groups/opt_out.

[Jesús Gabriel y Galán]

On Tue, May 21, 2013 at 2:38 AM, Keith Rarick <k...@xph.us> wrote:
> On Mon, May 20, 2013 at 5:05 AM, Abishek.R Srikaanth
> <abishekr...@gmail.com> wrote:
>> Can I use beanstalkd to queue up tasks that can be executed at a specific
>> time during the day?
>
> Yes, but I wouldn't recommend it.
>
> You can approximate it by subtracting the current time
> from the time you want and adding a delay for that duration.
> But there's no guarantee the job will start on time; you must
> make sure there are workers working at the right time and
> that no other jobs are ahead in line keeping them busy.
>
> The general meaning of beanstalkd's delay is to wait *at
> least* the specified amount of time.

Forgive me for jumping in this thread, but I had a similar use case in
mind that could be solved by Beanstalkd and I would like to ask your
opinion.

I have to perform a task for each user (of which I have thousands) in
a weekly basis, but at a different time for each user, based on each
user's subscription time for example. Also each task can fail and
should be retried with a policy that can be dynamic. So I made a proof
of concept in which when a user subscribed I insert a job in
beanstalkd with a week delay. Then n workers will listen for jobs, get
one, make a try and if it fails return it to the queue with a
calculated delay (5 minutes or 1 day, for example).

Do you think this fits correctly? In my case I'm not really concerned
with being accurate to the second in processing the job, I just need
that it's run approximately one week after the previous one.

There's also a concern in that I would like to be able to check every
once in a while that all users have a corresponding job in the queue,
that I'm not sure how to do. Is there a way to do this in beanstalkd,
or should I use an external database and run checks against this
external data. I wouldn't want this since the only way to do that
would be to check that a user wasn't updated for more than a week,
which could mean that I lose an event for some potentially relevant
period of time.

Thanks,

Jesus.

[Keith Rarick]

On Tue, May 21, 2013 at 12:37 AM, Jesús Gabriel y Galán
<jgabrie...@gmail.com> wrote:
> So I made a proof
> of concept in which when a user subscribed I insert a job in
> beanstalkd with a week delay. Then n workers will listen for jobs, get
> one, make a try and if it fails return it to the queue with a
> calculated delay (5 minutes or 1 day, for example).

It's hard to say whether this is good without knowing some
more details about what you're doing. But if you've already
tried it and it works well, then I'd say it's a good approach.

> should I use an external database and run checks against this
> external data

That will probably be more reliable in the long run. Beanstalkd
isn't a database. You might consider a periodic cron job that
scans the database looking at timestamps, and generates
beanstalkd jobs to do whatever work is necessary. But again,
evaluating these approaches is hard to do in the abstract.

[Jesús Gabriel y Galán]

On Wed, May 22, 2013 at 7:17 AM, Keith Rarick <k...@xph.us> wrote:
> On Tue, May 21, 2013 at 12:37 AM, Jesús Gabriel y Galán
> <jgabrie...@gmail.com> wrote:
>> So I made a proof
>> of concept in which when a user subscribed I insert a job in
>> beanstalkd with a week delay. Then n workers will listen for jobs, get
>> one, make a try and if it fails return it to the queue with a
>> calculated delay (5 minutes or 1 day, for example).
>
> It's hard to say whether this is good without knowing some
> more details about what you're doing. But if you've already
> tried it and it works well, then I'd say it's a good approach.

Thanks for your answer.
What I am doing is managing many paid subscription services. Users
subscribe to the service, and every week I have to bill them. If the
billing fails, I have a retry policy within the week, that depends on
each service. I don't mind to-the-second accuracy, but I wouldn't like
to skip any billing try for several hours, so it's fine.

>> should I use an external database and run checks against this
>> external data
>
> That will probably be more reliable in the long run. Beanstalkd
> isn't a database. You might consider a periodic cron job that
> scans the database looking at timestamps, and generates
> beanstalkd jobs to do whatever work is necessary. But again,
> evaluating these approaches is hard to do in the abstract.

In my proof of concept, I never delete the jobs, I only release with
delay. This means that even if the client fails, the job is always in
the queue, correct?
Anyway, what I wanted to understand was if there was a way to inspect
the jobs that are in the queue, to check that every user has at least
one job in the queue. If there isn't, the only way is to check that a
job has executed for the user in the last week, or to insert jobs for
all users every once in a while, but this might be overkill.

Thanks,

Jesus.