[Boost-users] [ASIO] directory monitor ASIO extension

[Szymon Gatner]

Hi,

I am going through code of directory monitor ASIO extension by Boris Schäling (http://www.highscore.de/boost/dir_monitor.zip) and I am wondering: what is the benefit of making this piece of code / library an ASIO extension.

It just feels like it could be easier to use and implement if it wasn't? I hope I am missing something tho and looking here for enlightenment ;)

Regards,
Szymon Gatner

[Keith Bennett]

if you're trying to write platform independent code and already using boost, then it's a strong incentive right there, to avoid #if PLATFORM do this #else do something else, #endif, etc.

A previous employer of mine had used Boris's directory monitor service code. However there were some issues which we had to work around. I don't remember what they were. Unfortunately I no longer work with that employer so don't have access to the source code to jog my memory. It might have simply been the way in which we were using the code.

_______________________________________________
Boost-users mailing list
Boost...@lists.boost.org
http://lists.boost.org/mailman/listinfo.cgi/boost-users

--
Keith Bennett

[Szymon Gatner]

if you're trying to write platform independent code and already using boost, then it's a strong incentive right there, to avoid #if PLATFORM do this #else do something else, #endif, etc.

That is what I am not seeing, extensions will often require platform depended code anyway, like this directory monitor  - has native Windows and POSIX calls.

 

A previous employer of mine had used Boris's directory monitor service code. However there were some issues which we had to work around. I don't remember what they were. Unfortunately I no longer work with that employer so don't have access to the source code to jog my memory. It might have simply been the way in which we were using the code.

OK ;)

[Keith Bennett]

On Tue, Aug 5, 2014 at 10:46 AM, Szymon Gatner <szymon...@gmail.com> wrote:

That is what I am not seeing, extensions will often require platform depended code anyway, like this directory monitor  - has native Windows and POSIX calls.

True enough. But at the time there was a difference between us writing and maintaining platform dependent code verses that fact being hidden away in a third party library. So effectively our code would be write-once for all platforms and then the extension would hide away the details for each platform: theoretically reducing our time to market. Then if we had any particular issues, we had the option of either fixing it ourselves or contacting the original author for support. Or if it was egregious enough, we could look for another third party library. Only finally after exhausting those, would we look at writing platform dependent code ourselves.

--
Keith Bennett

[Boris Schäling]

On Tue, 05 Aug 2014 15:36:05 +0200, Szymon Gatner
<szymon...@gmail.com> wrote:

Hi Szymon,

> I am going through code of directory monitor ASIO extension by Boris
> Schäling (http://www.highscore.de/boost/dir_monitor.zip) and I am
> wondering: what is the benefit of making this piece of code / library an
> ASIO extension.

Keith has already answered your question. :) I just want to let you know
that others have continued working on the directory monitor. There is for
example this project at GitHub: <https://github.com/berkus/dir_monitor>

Please note that the directory monitor (it's from 2008) can be implemented
much better today. For example, have a look at this implementation (from
2011):
<https://github.com/boostcon/2011_presentations/blob/master/wed/asio_extensions/directory_monitor.cpp>

The directory monitor from 2008 uses a background thread. The
(Windows-only) implementation from 2011 is integrated into Boost.Asio's
event loop and uses no extra thread anymore.

Boris

[Szymon Gatner]

Hey Boris!

Please note that the directory monitor (it's from 2008) can be implemented much better today. For example, have a look at this implementation (from 2011): <https://github.com/boostcon/2011_presentations/blob/master/wed/asio_extensions/directory_monitor.cpp>

thanks for answering. I actually saw both implementations (2008 ans 2011) and my question arose from differences from them.

Version from 2011 is single class and NOT an ASIO extension - there is no deriving from basic_io_object or io_service::service. Hence my question: what is the benefit of deriving from those classes and making your class an ASIO extension vs 2011 version where io_service is just used "externally" for synchronization purposes. 

[Boris Schäling]

On Tue, 05 Aug 2014 21:05:30 +0200, Szymon Gatner
<szymon...@gmail.com> wrote:

> [...]Version from 2011 is single class and NOT an ASIO extension - there

> is no
> deriving from basic_io_object or io_service::service. Hence my question:
> what is the benefit of deriving from those classes and making your class
> an
> ASIO extension vs 2011 version where io_service is just used "externally"
> for synchronization purposes.

I think the short answer is that you shouldn't derive from basic_io_object
or create your own service unless you have to. :)

Before the I/O object windows::object_handle was added to Boost.Asio,
there was no way to use Windows event objects as there was no code in
Boost.Asio which would call Windows functions like WaitForSingleObject()
or RegisterWaitForSingleObject(). It wouldn't have been possible to reuse
an existing I/O object to add support for event objects. Thus,
windows::object_handle was created which is derived from basic_io_object,
comes with its own service (where those Windows API functions are called)
and integrated pending asynchronous operations into Boost.Asio's event
loop (so Boost.Asio doesn't have to create its own background threads).
It's a perfect fit into Boost.Asio's framework. Today if you ever need to
wait for events asynchronously, you can reuse windows::object_handle and
can ignore the entire framework Boost.Asio set up for I/O objects.

[Szymon Gatner]

2014-08-05 21:42 GMT+02:00 Boris Schäling <bo...@highscore.de>:

On Tue, 05 Aug 2014 21:05:30 +0200, Szymon Gatner <szymon...@gmail.com> wrote:

[...]Version from 2011 is single class and NOT an ASIO extension - there is no

deriving from basic_io_object or io_service::service. Hence my question:
what is the benefit of deriving from those classes and making your class an
ASIO extension vs 2011 version where io_service is just used "externally"
for synchronization purposes.

I think the short answer is that you shouldn't derive from basic_io_object or create your own service unless you have to. :)

[...]

 Today if you ever need to wait for events asynchronously, you can reuse windows::object_handle and can ignore the entire framework Boost.Asio set up for I/O objects.

Answer I was looking for, thanks a lot!

Regards,

Szymon Gatner 

[Bjorn Reese]

On 08/05/2014 03:36 PM, Szymon Gatner wrote:

> I am going through code of directory monitor ASIO extension by Boris

You may also be interested in AFIO (F for file.) It has an experimental
directory monitor living in some branch whose name I do not know.

https://github.com/BoostGSoC/boost.afio.git

[Niall Douglas]

On 6 Aug 2014 at 0:17, Bjorn Reese wrote:

> On 08/05/2014 03:36 PM, Szymon Gatner wrote:
>
> > I am going through code of directory monitor ASIO extension by Boris
>
> You may also be interested in AFIO (F for file.) It has an experimental
> directory monitor living in some branch whose name I do not know.
>
> https://github.com/BoostGSoC/boost.afio.git

Ah yes. The directory monitor isn't ready for production use. By
production use I mean:

* Can cope with 1m entry directories easily as the rest of AFIO can.

* Can handle 50,000 entries changing every second in a 1m entry
directory without losing change deltas and without racing on
individual stats on a 2Ghz single core, with approximate linear
scaling with additional cores i.e. four cores can monitor four 1m
entry directories seeing 5% change.

* Works as expected on remotely mounted filing systems like NFS.

* Uses native OS features to avoid polling when possible.

* Works reliably on Microsoft Windows (hard), Linux (slightly
easier), BSD and OS X (easiest by far of the lot).

Paul Kirth presented on this at C++ Now 2014. It is a surprisingly
tough engineering problem, and requires a design approaching ideal
for it to work at all.

tl;dr; it'll be a while longer. Paul is still kicking at it.

Niall

--
ned Productions Limited Consulting
http://www.nedproductions.biz/
http://ie.linkedin.com/in/nialldouglas/

[Niall Douglas]

On 6 Aug 2014 at 1:54, Niall Douglas wrote:

> > > I am going through code of directory monitor ASIO extension by Boris
> >
> > You may also be interested in AFIO (F for file.) It has an experimental
> > directory monitor living in some branch whose name I do not know.
> >
> > https://github.com/BoostGSoC/boost.afio.git
>
> Ah yes. The directory monitor isn't ready for production use. By
> production use I mean:

Sorry, I forgot (it's nearly 2am) to mention that AFIO itself is
production ready and has been in the review queue since October 2013.
You can see its CI test dashboard at the bottom of the github page
plus a link to its documentation. There is a slight correction to the
URL:

https://github.com/BoostGSoC13/boost.afio

You *will* find AFIO's directory enumeration facilities to be faster
than anything else, including the system APIs. AFIO goes direct to
the kernel and skips the intervening baggage, hence happily working
with 1m or even 10m entry directories on BSD, Linux and Windows with
ease.

[Szymon Gatner]

2014-08-06 2:54 GMT+02:00 Niall Douglas <s_sour...@nedprod.com>:

On 6 Aug 2014 at 0:17, Bjorn Reese wrote:

> On 08/05/2014 03:36 PM, Szymon Gatner wrote:
>
> > I am going through code of directory monitor ASIO extension by Boris
>
> You may also be interested in AFIO (F for file.) It has an experimental
> directory monitor living in some branch whose name I do not know.
>
>    https://github.com/BoostGSoC/boost.afio.git

Ah yes. The directory monitor isn't ready for production use. By
production use I mean:

* Can cope with 1m entry directories easily as the rest of AFIO can.

* Can handle 50,000 entries changing every second in a 1m entry
directory without losing change deltas and without racing on
individual stats on a 2Ghz single core, with approximate linear
scaling with additional cores i.e. four cores can monitor four 1m
entry directories seeing 5% change.

I should be OK :) What are the advantages over Boris' version tho?

 

Paul Kirth presented on this at C++ Now 2014. It is a surprisingly
tough engineering problem, and requires a design approaching ideal
for it to work at all.

I actually just watched that presentation yesterday. Pretty great piece of code.

Regards,

Szymon Gatner

[Szymon Gatner]

Sorry, I forgot (it's nearly 2am) to mention that AFIO itself is

production ready and has been in the review queue since October 2013.

Why isn't it under review yet? 

[Niall Douglas]

On 6 Aug 2014 at 8:58, Szymon Gatner wrote:

> > Ah yes. The directory monitor isn't ready for production use. By
> > production use I mean:
> >
> > * Can cope with 1m entry directories easily as the rest of AFIO can.
> >
> > * Can handle 50,000 entries changing every second in a 1m entry
> > directory without losing change deltas and without racing on
> > individual stats on a 2Ghz single core, with approximate linear
> > scaling with additional cores i.e. four cores can monitor four 1m
> > entry directories seeing 5% change.
>
> I should be OK :) What are the advantages over Boris' version tho?

Boris's won't deadlock as frequently as the AFIO one :). If code
isn't in the master or stable branch in AFIO, I wouldn't use it.

You could use AFIO's directory enumeration which is in stable
combined with Boris' directory monitoring. It depends on if you need
unracy metadata support on Windows as Filesystem is currently racy
there.

> > Sorry, I forgot (it's nearly 2am) to mention that AFIO itself is
> > production ready and has been in the review queue since October 2013.
> >
> Why isn't it under review yet?

There is a large backlog, some libraries have been waiting in the
queue for years. Part of this is due to sickness in the Boost culture
and the ongoing decline of Boost for two years now, part is simply
because no one wants nor needs a portable asynchronous file i/o
library and therefore no one is interested in volunteering as review
manager.

Besides, it wouldn't pass a review right now because its API would be
considered unusable as it was designed for maximum speed, and
therefore isn't "Boosty enough" - people would ask "where are the
iterators?" etc etc. This is also because the API is designed as a
stable binary ABI to be assembled by the forthcoming monadic
continuations framework being led out by Vicente. Still, if you need
maximum portable file i/o performance now, it works very well and
will continue to work into the future.

[Brian Wood]

Niall Douglas writes

> * Works reliably on Microsoft Windows (hard), Linux (slightly
> easier), BSD and OS X (easiest by far of the lot).
>

I believe you.  Having considered some of your previous
comments in a different thread, on kqueue, I ported my
code generator from Linux to BSD.  I only found relief
from a problem I was having by doing so. 

> Paul Kirth presented on this at C++ Now 2014. It is a surprisingly
> tough engineering problem, and requires a design approaching ideal
> for it to work at all.

Are videos from that conference available yet?

--

Brian
Ebenezer Enterprises - In G-d we trust.
http://webEbenezer.net

[Szymon Gatner]

Are videos from that conference available yet?

Yup: https://www.youtube.com/watch?v=QiSV4uXikjk 

[Niall Douglas]

On 6 Aug 2014 at 6:58, Brian Wood wrote:

> > * Works reliably on Microsoft Windows (hard), Linux (slightly
> > easier), BSD and OS X (easiest by far of the lot).
>
> I believe you. Having considered some of your previous
> comments in a different thread, on kqueue, I ported my
> code generator from Linux to BSD. I only found relief
> from a problem I was having by doing so.

kqueues are indeed the least worst solution available for watching
for change of anything, though they do consume fd's rapidly for
directory watching. I personally wish you could use the fd to
indicate what you want to watch, and then close the fd after
registering it with kqueue.

The huge problem with Microsoft's otherwise very good API is it can
spontaneously throw away data under load, and you can't recover and
retry. inotify under Linux has the same problem. That means you need
to checkpoint every success, and roll back to that last success and
run polling to calculate deltas until you can resync the watched
state with the ground state.

A real shame in that design :(