Is there a plan to make apscheduler distributed ?

[Kyle Sawyer]

Hello, all

I am using APScheduler as a billing engine, means using it to do a lot of different cron jobs, I like the APScheduler API very much, but I am suffering the pain of single point of failure, it can only run one apschduler process, or it will do the duplicated job, though I can use something like distributed lock to sync apscheduler jobs, but I think it is ugly and inefficient.

Another python project, celery, is distributed,  it can run multiple workers to do the task, but it doesn't have APIs like APScheduler, which makes me think it should be good if we have celery + apscheduler.

So, is it possible to make apscheduler distributed, or it is just targetd to be a python library?

Thanks.

[Alex Grönholm]

APScheduler was designed to be a scheduler, and nothing else. A distributed task queue is way beyond the scope of the original design, and would also require complexities that make celery hard to set up (according to nearly everyone I've asked). Adding distributed operation capabilities would considerably add to the maintenance burden as well.

--
You received this message because you are subscribed to the Google Groups "APScheduler" group.
To unsubscribe from this group and stop receiving emails from it, send an email to apscheduler...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[George Kappel]

I would use APScheduler to put messages into a queue (RabbitMQ, AWS SQS, Redis, or others) and then you can have as many processes as you want pulling those messages off the queue to actually run the task

[Shola Smith]

In this part of your statement: "then you can have as many processes as you want pulling those messages off the queue to actually run the task", would you do it with APScheduler? Please, I'd need and example to help me achieve something like this. I'm still trying my hands on APScheduler. Thanks.

[Anoop Khandelwal]

Hi,is there any update on it?

We need to make apscheduler distributed but facing problems of duplicatiing jobs?

Can it be possible from custom executors in apscheduler?

[Alex Grönholm]

There is no ongoing work to this end. I might be willing to accept a well prepared pull request that addresses this issue (perhaps using distributed locks) but I don't expect anyone to come forward with something like that.

--

[sankarap...@gmail.com]

Hello all,

 I'm using APScheduler. It works well in standalone python program. But when I embed it with my product and used, it throws an exception as 

Traceback (most recent call last): File "site-packages\PyInstaller\loader\rthooks\pyi_rth_win32comgenpy.py", line 47, in <module> File "c:\users\smackc~1\appdata\local\temp\pip-build-zxpdbt\PyInstaller\PyInst aller\loader\pyimod03_importers.py", line 389, in load_module File "lib\site-packages\win32com_init__.py", line 6, in <module> File "c:\users\smackc~1\appdata\local\temp\pip-build-zxpdbt\PyInstaller\PyInst aller\loader\pyimod03_importers.py", line 389, in load_module File "lib\site-packages\pythoncom.py", line 2, in <module> File "c:\users\smackc~1\appdata\local\temp\pip-build-zxpdbt\PyInstaller\PyInst aller\loader\pyimod03_importers.py", line 389, in load_module File "lib\site-packages\win32\lib\pywintypes.py", line 124, in <module> File "lib\site-packages\win32\lib\pywintypes.py", line 61, in _import_pywin32 _system_module ImportError: Module 'pywintypes' isn't in frozen sys.path ['C:\\Users\\SMACKC~1\ \AppData\\Local\\Temp\_MEI14~3'] Failed to execute script pyi_rth_win32comgenpy

Any help is appreciated. Thanks in advance

[Alex Grönholm]

As you can see, the traceback does not originate from APScheduler so this problem does not belong to this Google group.

[sankarap...@gmail.com]

Thank you Sir. But when I comment the import statement, I could not come across the error. And the exception is raised only when I use my product in Windows and it works fine in Ubuntu

[Alex Grönholm]

Regardless, this problem has nothing to do with APScheduler, so please seek help elsewhere. APScheduler does not involve either pywin32 or PyInstaller.

[sankarap...@gmail.com]

Thank you for your fast response Sir. Let me check my side. I've one more query. When I started running APScheduler in Ubuntu, I came across the exception    

 File "site-packages\apscheduler-3.3.1-py2.7.egg\apscheduler\__init__.py", line

 2, in <module>

  File "site-packages\pkg_resources\__init__.py", line 552, in get_distribution

  File "site-packages\pkg_resources\__init__.py", line 426, in get_provider

  File "site-packages\pkg_resources\__init__.py", line 968, in require

  File "site-packages\pkg_resources\__init__.py", line 854, in resolve

pkg_resources.DistributionNotFound: The 'APScheduler' distribution was not found

 and is required by the application

I've modified __init__.py release and version_info as 3.3.1cf. Is there any other method to rectify the error. 

Thank you in advance

[Alex Grönholm]

This happens because PyInstaller does not include APScheduler's .egg_info directory when packaging your application. Therefore its packaging metadata and entry points (which it relies for its correct operation) is not available. You should contact the PyInstaller developers and ask them to fix this bug.

[sankarap...@gmail.com]

Thank you very much Sir.