async watch directory for new files

Zoran

unread,

Apr 23, 2021, 5:31:52 PM4/23/21

to

Hi,

I need to watch for new files in directory, and when it shows up, I should create async task with file's full path for it, and wait for new file.

If anyone here used a library for such task, please share which one.

Regards.

Gene Heskett

unread,

Apr 24, 2021, 10:06:45 AM4/24/21

to

inotifywait contains several variations of a notifier, I use it rather
heavly here to automate such goings on as my incoming e-mail.

The variation I use most is launched by a bash script, when then waits
for its return with the name of the new file, that bash script then
takes an action determined by the name that was returned.

Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
- Louis D. Brandeis
Genes Web page <http://geneslinuxbox.net:6309/gene>

Dieter Maurer

unread,

Apr 24, 2021, 12:52:24 PM4/24/21

to

Zoran wrote at 2021-4-23 14:31 -0700:
>I need to watch for new files in directory, and when it shows up, I should create async task with file's full path for it, and wait for new file.
>
>If anyone here used a library for such task, please share which one.

The solution likely depends on the OS you are using.

For Linux, `inotify` informs applications about file system changes.
PyPI contains a binding for it -- with identical name.

Zoran

unread,

Apr 24, 2021, 4:13:07 PM4/24/21

to

As I can see https://pypi.org/project/inotify/ is not asyncio frendly.
Whatever I use, it should be asynchronous.
Final OS is linux (Centos 7), but I am doing development on Windows 10 machine, so it would be nice if I have something that works on both.

Zoran

unread,

Apr 24, 2021, 4:14:01 PM4/24/21

to

> inotifywait contains several variations of a notifier, I use it rather
> heavly here to automate such goings on as my incoming e-mail.

Can you be more specific about inotifywait?
Do you have an link or something?

Chris Angelico

unread,

Apr 24, 2021, 4:59:29 PM4/24/21

to

Bear in mind that asyncio is NOT the only way to be asynchronous. So
what you're asking for is, very specifically, an asyncio-compatible
inotify. Turns out, there's an ainotify on PyPI that might be what you
want.

ChrisA

Paul Rubin

unread,

Apr 24, 2021, 7:32:08 PM4/24/21

to

Chris Angelico <ros...@gmail.com> writes:
> So what you're asking for is, very specifically, an asyncio-compatible
> inotify. Turns out, there's an ainotify on PyPI that might be what you
> want.

I don't know if there is an io_uring event for inotify but if there
isn't, that ought to be fixed. I also don't know if there are good
Python bindings for io_uring yet, but if not, we need them. At that
point there shouldn't be a special asyncio binding just for inotify.

Zoran

unread,

Apr 25, 2021, 1:15:34 AM4/25/21

to

> Bear in mind that asyncio is NOT the only way to be asynchronous. So
> what you're asking for is, very specifically, an asyncio-compatible
> inotify. Turns out, there's an ainotify on PyPI that might be what you
> want.

Yes there is:
https://github.com/rbarrois/aionotify
https://asyncinotify.readthedocs.io/en/latest/
https://pypi.org/project/aionotify/
https://pypi.org/project/butter/
https://github.com/giannitedesco/minotaur
https://pypi.org/project/watchgod/
...

There are many of them.
I hoped that someone with experience here can suggest a library that he has been used on some project.

Julio Di Egidio

unread,

Apr 25, 2021, 4:16:09 AM4/25/21

to

Watching for files is usually supported by the operating system, so I'd
go in that direction:

<https://duckduckgo.com/?q=python+file+system+watcher>

Implementations are usually just callback-based. (Apologies for the
generic link, I haven't needed this in Python yet: anyway, those are the
keywords.)

Julio

Zoran

unread,

Apr 25, 2021, 2:50:55 PM4/25/21

to

> <https://duckduckgo.com/?q=python+file+system+watcher>
>
> Implementations are usually just callback-based. (Apologies for the
> generic link, I haven't needed this in Python yet: anyway, those are the
> keywords.)

:) before asking a question here I googled the subject a lot.
Anyway, the simplest solution is this one:

import asyncio
import pathlib
from watchgod import awatch, Change

watch_dir = pathlib.Path(r'C:\Users\User\watch_folder')

async def main():
async for changes in awatch(str(watch_dir)):
while changes:
change_type, path = changes.pop()
if change_type == Change.added:
print('processing:', path)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

watchgod library is quite young. I hope that there will be no suprises.

Chris Angelico

unread,

Apr 25, 2021, 2:59:50 PM4/25/21

to

On Mon, Apr 26, 2021 at 4:56 AM Zoran <zlju...@gmail.com> wrote:
>
> > <https://duckduckgo.com/?q=python+file+system+watcher>
> >
> > Implementations are usually just callback-based. (Apologies for the
> > generic link, I haven't needed this in Python yet: anyway, those are the
> > keywords.)
>
> :) before asking a question here I googled the subject a lot.

We couldn't tell that from your question.

ChrisA

Zoran

unread,

Apr 25, 2021, 3:44:28 PM4/25/21

to

Anyway, thanks for participating.

Gene Heskett

unread,

Apr 25, 2021, 5:28:56 PM4/25/21

to

No need for a link or URL, it should be available from the repo's of your
distro.

Julio Di Egidio

unread,

Apr 26, 2021, 4:41:40 AM4/26/21

to

On 25/04/2021 20:50, Zoran wrote:
>> <https://duckduckgo.com/?q=python+file+system+watcher>
>>
>> Implementations are usually just callback-based. (Apologies for the
>> generic link, I haven't needed this in Python yet: anyway, those are the
>> keywords.)
>
> :) before asking a question here I googled the subject a lot.

Maybe, still you have snipped and altogether missed the actual point of
my answer, which was that of indicating an alternative approach to the
async notifiers that you guys had been considering up-thread.

> Anyway, the simplest solution is this one:
>
> import asyncio
> import pathlib
> from watchgod import awatch, Change
>
> watch_dir = pathlib.Path(r'C:\Users\User\watch_folder')
>
>
> async def main():
> async for changes in awatch(str(watch_dir)):
> while changes:
> change_type, path = changes.pop()
> if change_type == Change.added:
> print('processing:', path)
>
>
> loop = asyncio.get_event_loop()
> loop.run_until_complete(main())
>
> watchgod library is quite young. I hope that there will be no suprises.

Notice also that this is a public forum, so, not just for your info, I
suppose you mean this one: <https://github.com/samuelcolvin/watchgod>
FWIW, at my cursory inspection, looks clean and alive...

HTH,

Julio

Barry Scott

unread,

Apr 26, 2021, 2:32:37 PM4/26/21

to

> On 24 Apr 2021, at 21:12, Zoran <zlju...@gmail.com> wrote:
>
> As I can see https://pypi.org/project/inotify/ <https://pypi.org/project/inotify/> is not asyncio frendly.

> Whatever I use, it should be asynchronous.
> Final OS is linux (Centos 7), but I am doing development on Windows 10 machine, so it would be nice if I have something that works on both.

In this case trying to develop the code on Windows will make your life very hard.

Are you using Windows because you do not have easy access to Centos 7?
If so why not run Centos 7 in a VM on your Windows machine as your dev environment?

Barry