Frequency interval based scheduling and maximum number of instances reached

[jay_t]

Hi,

I'm using APscheduler in a project I'm working on and I'm very happy with it.

During testing however I ran into the situation that the scheduler logs this error: 

...skipped: maximum number of running instances reached (1)

I understand that it means there's still a previous job running while the following one has to be executed.

I try to understand what could cause this in my situation.

I have 10 jobs running with a frequency of 1 second.

I have measured the time it takes to execute the function that the scheduler has to execute, which floats by average around 0.05 seconds peaking to max 0.08 seconds.

I try to understand how I can end up in this situation, and what I can do about it to find/understand the cause and possibly do something about it.

Thanks,

Jay

[jay_t]

I figured out that occasionally the function being called was taking longer than it should due to an unreliable connection to a message broker resulting into jobs running longer than the interval.

Thanks,

Jay,

[Sudeep Nesakumar]

Hi Jay,

 Even I've been facing the same problem . And in my code i run a cron job which is run for every five seconds, and I've been getting the same WARNING ever since.

WARNING:apscheduler.scheduler:Execution of job "test_3 (trigger: cron[second='*/5'], next run at: 2013-11-28 15:56:30)" skipped: maximum number of running instances reached (1)
I tried giving time gap of 2minutes it doesn't solve the issue.....
Help me in overcoming this issue..

[Alex Grönholm]

28.11.2013 12:44, Sudeep Nesakumar kirjoitti:

Hi Jay,

 Even I've been facing the same problem . And in my code i run a cron job which is run for every five seconds, and I've been getting the same WARNING ever since.

WARNING:apscheduler.scheduler:Execution of job "test_3 (trigger: cron[second='*/5'], next run at: 2013-11-28 15:56:30)" skipped: maximum number of running instances reached (1)
I tried giving time gap of 2minutes it doesn't solve the issue.....
Help me in overcoming this issue..

This would happen if your job takes too long to run. Can you provide code that reproduces the issue?

On Sunday, February 5, 2012 8:10:40 AM UTC+5:30, jay_t wrote:

Hi,

I'm using APscheduler in a project I'm working on and I'm very happy with it.

During testing however I ran into the situation that the scheduler logs this error: 

...skipped: maximum number of running instances reached (1)

I understand that it means there's still a previous job running while the following one has to be executed.

I try to understand what could cause this in my situation.

I have 10 jobs running with a frequency of 1 second.

I have measured the time it takes to execute the function that the scheduler has to execute, which floats by average around 0.05 seconds peaking to max 0.08 seconds.

I try to understand how I can end up in this situation, and what I can do about it to find/understand the cause and possibly do something about it.

Thanks,

Jay

--
You received this message because you are subscribed to the Google Groups "APScheduler" group.
To unsubscribe from this group and stop receiving emails from it, send an email to apscheduler...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Sudeep Nesakumar]

1) This is code that.. I have been working on to capture the incoming tcp packets from the network... And this code skips some of the packets without capturing by issuing the warning message. So kindly help me out with this problem.
2) The second thing is that , how do i stop this entire process after a specified amount of time. I am currently stopping this process by pressing Ctrl+c twice to stop the ongoing execution.

import os
from apscheduler.scheduler import Scheduler
import logging

logging.basicConfig()
sched = Scheduler()

sched = Scheduler()

@sched.interval_schedule(seconds=5)
def test_dump1():
    command_str="sudo tcpdump -i eth0 -n tcp -w traffic_data.pcap"
    os.system(command_str)
  
sched.start()

while True:
    pass

[Alex Grönholm]

29.11.2013 08:53, Sudeep Nesakumar kirjoitti:

1) This is code that.. I have been working on to capture the incoming tcp packets from the network... And this code skips some of the packets without capturing by issuing the warning message. So kindly help me out with this problem.
2) The second thing is that , how do i stop this entire process after a specified amount of time. I am currently stopping this process by pressing Ctrl+c twice to stop the ongoing execution.

First off, why do you create the scheduler twice? Not that it matters, just looks weird.
Second, that while True: pass is a big No-No because it's a busy-waiting loop that will slaughter your CPU performance. The standalone mode was created for this purpose.
Third, this whole schedule makes no sense. The command you are executing blocks forever, so it will still be executing after the five seconds are up.
What are you really trying to do here?

import os
from apscheduler.scheduler import Scheduler
import logging

logging.basicConfig()
sched = Scheduler()

sched = Scheduler()

@sched.interval_schedule(seconds=5)
def test_dump1():
    command_str="sudo tcpdump -i eth0 -n tcp -w traffic_data.pcap"
    os.system(command_str)
  
sched.start()

while True:
    pass

[scls]

Le vendredi 29 novembre 2013 21:37:35 UTC+1, Alex Grönholm a écrit :

29.11.2013 08:53, Sudeep Nesakumar kirjoitti:

 

Second, that while True: pass is a big No-No because it's a busy-waiting loop that will slaughter your CPU performance. The standalone mode was created for this purpose.

Hello,

I did  myself the same kind of error
maybe documentation should be improved to reflect the  examples / interval.py
http://pythonhosted.org/APScheduler/intervalschedule.html

"""
Basic example showing how to start the scheduler and schedule a job that executes on 3 second intervals.
"""

from datetime import datetime

from apscheduler.scheduler import Scheduler


def tick():
    print('Tick! The time is: %s' % datetime.now())


if __name__ == '__main__':
    scheduler = Scheduler(standalone=True)
    scheduler.add_job(tick, 'interval', {'seconds': 3})
    print('Press Ctrl+C to exit')
    try:
        scheduler.start()
    except (KeyboardInterrupt, SystemExit):
        pass

Because my first reflex was to look at documentation not at source code
available at https://bitbucket.org/agronholm/apscheduler/
which is quite difficult to find (maybe a link to bitbucket should also be present into
documentation) because in order to find it I went to PyPy !!!


Thanks for your scheduler that's a great piece of software.

[Alex Grönholm]

Perhaps the examples should have a better presence in the documentation.
That said, I don't think there are any busy waiting loops anywhere in the APScheduler documentation or examples.

Thanks for your scheduler that's a great piece of software.

[Sudeep Nesakumar]

Hi Alex,

Thanks for your analysis. But, this scheduling is mainly to work with the tcpdump inorder to capture all the incoming tcp packets every 5seconds and to create file where in I dump all the contents into this pcap file. And then I use the packet analzer tools on these files to do the analysis of the packets. This is why I am trying to schedule. The schedule is needed to be done in such a way that it starts every morning and stops in the evening automatically.

I am a beginner in python and I am teaching myself from the python.org documentations.... Could you suggest me some more documentation or even the examples on how to use the scheduler, so that i can get better understanding...

sorry, it was my mistake that i had copied twice the scheduler initialization. its actually done only once in my code.
 Thanks in advance.

[Alex Grönholm]

02.12.2013 07:31, Sudeep Nesakumar kirjoitti:
> Hi Alex,
>
> Thanks for your analysis. But, this scheduling is mainly to work with
> the tcpdump inorder to capture all the incoming tcp packets every
> 5seconds and to create file where in I dump all the contents into this
> pcap file. And then I use the packet analzer tools on these files to
> do the analysis of the packets. This is why I am trying to schedule.
> The schedule is needed to be done in such a way that it starts every
> morning and stops in the evening automatically.
>

What you were attempting to do in your code is start a new packet
listener every 5 seconds. So after 20 seconds you would have had 3
listeners running concurrently (if it weren't for the default
concurrency limit of 1). If all you wanted to do was to listen to
packets, what did you need the scheduler for? Please help me understand.

> I am a beginner in python and I am teaching myself from the python.org
> documentations.... Could you suggest me some more documentation or
> even the examples on how to use the scheduler, so that i can get
> better understanding...

The example code is available both in the distributed tarball and the
bitbucket repository here:
https://bitbucket.org/agronholm/apscheduler/src/e517d9adc8a1c750adf0eecdc89cbb9fa64f553c/examples/?at=default

>
> sorry, it was my mistake that i had copied twice the scheduler
> initialization. its actually done only once in my code.
> Thanks in advance.

[Sudeep Nesakumar]

Hi Alex Grönholm,

I actually wanted this packet capture to be done from morning till evening and then stop. This is needed to be followed everyday. So I guess I should be using the cron-scheduler in python instead. I'll now change the api with the cron scheduler and try  it. Thanks  for the  insight. I will also go through  the example codes as well. Thanks again.
I'll get back again ......

[Sudeep Nesakumar]

Hi Alex Grönholm,

I was successful in using the scheduler, I now know its strength in terms of scheduling... its a great one, I used the cron style ap scheduler and it worked out for my needs.

i created a scheduling script, which has the apscheduler in it and this would call another script using the subprocess.Popen(), which will do the job of capturing the packets using the tcpdump command.

And the problem was still existing and the flaw was that there was no stop for the capturing of packets and once i had given a timing mechanism to only stop the operation of capturing it worked fine.
And it now does not show up with the warning messages; maximum instances reached.
Thanks for your help.

Best regards,
Sudeep