Noting EVERY File Write/Change/Delete in (near) Real Time ???

[25B.Z969]

There are a couple of monitoring functions in Linux, but
they usually require being directed at specific folders,
which is not practical if there are hundreds/thousands
or worse.

I look at softRAID ... and it's doing exactly what I'm
looking for as it mirrors changes super-promptly. There
is SOME tap into the lower-level kernel functions that
shows everything going on as it happens. I just don't
see a system function, or any GCC library, that can
provide the same utility. I'm not interested in duplication
though, just logging what's happened so I can do other
stuff with the info.

Yea, yea ... you can run rsync on everything over and
over and over forever and have it build a log of what's
changed. I've used that for other projects. Alas that's
WAY too slow and uses too much CPU. Scanning is NOT
the way, I want notice of EVERY file write/rewrite
as it happens.

Any ideas ? Words of wisdom ? SoftRAID tells me it CAN
be done, but I can't penetrate the code well enough to
see how it's doing the job.

[Robert Riches]

If by "softRAID" you mean mdadm-style kernel-supported RAID, that
is for mirroring disks on a local machine. It works really well
for that purpose.

Perchance, might you be looking for something closer to a
Distributed Replicated Block Device, aka DRBD?

https://en.wikipedia.org/wiki/Distributed_Replicated_Block_Device

HTH

--
Robert Riches
spamt...@jacob21819.net
(Yes, that is one of my email addresses.)

[David W. Hodgins]

On Thu, 08 Sep 2022 22:55:26 -0400, 25B.Z969 <25B....@noda.net> wrote:
> the way, I want notice of EVERY file write/rewrite
> as it happens.

See if "man inotifywatch" from the package inotify-tools works for your needs.

Regards, Dave Hodgins

[Richard Kettlewell]

"David W. Hodgins" <dwho...@nomail.afraid.org> writes:
> 25B.Z969 <25B....@noda.net> wrote:
>> the way, I want notice of EVERY file write/rewrite
>> as it happens.
>
> See if "man inotifywatch" from the package inotify-tools works for your needs.

He wants every IO operation, inotify makes you pick the files and
directories to monitor, making it impractical for whole-system
monitoring.

--
https://www.greenend.org.uk/rjk/

[Richard Kettlewell]

"25B.Z969" <25B....@noda.net> writes:
> There are a couple of monitoring functions in Linux, but
> they usually require being directed at specific folders,
> which is not practical if there are hundreds/thousands
> or worse.
>
> I look at softRAID ... and it's doing exactly what I'm
> looking for as it mirrors changes super-promptly. There
> is SOME tap into the lower-level kernel functions that
> shows everything going on as it happens. I just don't
> see a system function, or any GCC library, that can
> provide the same utility. I'm not interested in duplication
> though, just logging what's happened so I can do other
> stuff with the info.

RAID applies to block devices, it’s too low down the stack for what
you’re asking for.

> Yea, yea ... you can run rsync on everything over and
> over and over forever and have it build a log of what's
> changed. I've used that for other projects. Alas that's
> WAY too slow and uses too much CPU. Scanning is NOT
> the way, I want notice of EVERY file write/rewrite
> as it happens.
>
> Any ideas ? Words of wisdom ? SoftRAID tells me it CAN
> be done, but I can't penetrate the code well enough to
> see how it's doing the job.

You probably want fanotify; see
https://man7.org/linux/man-pages/man7/fanotify.7.html for details.

--
https://www.greenend.org.uk/rjk/

[25B.Z969]

Yes, it does. But at some level, in some way, it is
intercepting write/re-write/delete info on a per-file
basis so it can duplicate the action on the mirror
drive(s).

I want that info, in (near) real time.

> Perchance, might you be looking for something closer to a
> Distributed Replicated Block Device, aka DRBD?
>
> https://en.wikipedia.org/wiki/Distributed_Replicated_Block_Device

Um ... not exactly. I'm not looking to write to alternate
devices, I just need to know WHAT files are being written/deleted.

The OS knows this at some level - but, despite extensive searching,
I can't find it. I want to write a service that makes use of this
info, needs to be fairly quick, but a short queue of writes/deletes
that's not TOO deep is tolerable ... cope with them during 'slack'
CPU time. "Hello Dave, I just wrote "/A/B/C/D/E/F/G/junk.txt and
then deleted "/W/X/Y/Z/morejunk.txt".

Now there ARE a couple of system utilities, you generally
point them at a specific folder. I suspect they just keep
watching for datetime changes by scanning over and over
again quickly. I could write that in Python or 'C' during
lunch break. This is NOT efficient, NOT tenable for use
on "/" or multiple 12tb NAS drives - and you have to keep
potentially huge "previous state" tables.

Real-time as-its-done is what I'm looking for. Somehow I'm
sure it CAN be done, but I cannot find where to probe for
this info. Somwhere, just before drivers turn it into actual
disk sectors, is "I am committing to writing file
"/A/B/C/D/E/F/G/junk.txt".

This seems to be one of those places where Linux is as
opaque as Winders.

[25B.Z969]

Yep, that's the problem. SOUNDED good until I looked
into it.

SOMEWHERE in the kernel there's a "I now commit to writing
"/A/B/C/D/E/F/G/junk.txt" ... not far before the device
drivers turn that into actual disk sectors and such.

There's also going to be a place where the FATs are
updated with "/A/B/C/D/E/F/G/junk.txt is exactly HERE
on the drive". That'd do.

Assume multiple 12+tb NAS volumes being involved here.

It's not a "usual" kind of problem, but I need it for
a special project.

[25B.Z969]

On 9/9/22 3:22 AM, Richard Kettlewell wrote:
> "25B.Z969" <25B....@noda.net> writes:
>> There are a couple of monitoring functions in Linux, but
>> they usually require being directed at specific folders,
>> which is not practical if there are hundreds/thousands
>> or worse.
>>
>> I look at softRAID ... and it's doing exactly what I'm
>> looking for as it mirrors changes super-promptly. There
>> is SOME tap into the lower-level kernel functions that
>> shows everything going on as it happens. I just don't
>> see a system function, or any GCC library, that can
>> provide the same utility. I'm not interested in duplication
>> though, just logging what's happened so I can do other
>> stuff with the info.
>
> RAID applies to block devices, it’s too low down the stack for what
> you’re asking for.

I agree on the technical point ... I think it just mirrors
the lowest-level write instructions (simplest case, maybe
RAID-1) to one or more other devices.

But SOMEWHERE there's a "I commit to writing /A/B/C/junk.txt"
before it's passed on to those lower levels. The OS *knows*.

And I want to know too, every time, promptly.

>> Yea, yea ... you can run rsync on everything over and
>> over and over forever and have it build a log of what's
>> changed. I've used that for other projects. Alas that's
>> WAY too slow and uses too much CPU. Scanning is NOT
>> the way, I want notice of EVERY file write/rewrite
>> as it happens.
>>
>> Any ideas ? Words of wisdom ? SoftRAID tells me it CAN
>> be done, but I can't penetrate the code well enough to
>> see how it's doing the job.
>
> You probably want fanotify; see
> https://man7.org/linux/man-pages/man7/fanotify.7.html for details.
>

fanotify is closer than ionotify ... but I'm trying to
see if it SCALES well - assume multiple 12tb NAS drives
being involved. ionotify is generally pointed at one,
preferably small, folder. It's hopeless for "/" or
"/dev/sdc1/".

I'll have to evaluate. I can't have something that tries
to keep a state table for 20 million files (which, these
days, is real-world). Something in the kernal that says
"I now commit to writing /dev/sdc1/A/B/C/D/junk.txt" is
the ideal.

[Robert Riches]

Nope. The kernel-supported mdadm-style RAID works at the block
level, not at the file operation level.

As opaque? With Linux, the whole kernel source is there for you
to study. A decade or so ago, I was assigned to write a device
mapper block device that would do intecept block-level
operations. I studied several modules of kernel code to get an
idea of how to do that. Fortunately, I was reassigned to a
different task before I had to start debugging that code.

[The Natural Philosopher]

On 10/09/2022 05:09, 25B.Z969 wrote:
> But SOMEWHERE there's a "I commit to writing /A/B/C/junk.txt"
>   before it's passed on to those lower levels. The OS *knows*.
>
>   And I want to know too, every time, promptly.

Maybe patch the kernel/stdlib and compile from source?



--
“A leader is best When people barely know he exists. Of a good leader,
who talks little,When his work is done, his aim fulfilled,They will say,
“We did this ourselves.”

― Lao Tzu, Tao Te Ching

[25B.Z969]

I suspected. So, BUMMER !

But there's SOME final point in the kernel where the
file committed to be writ/deleted exists and can be
read.

'fanotify' is closest to what I want - but I'm gonna
have to see if it scales well to maybe 100tb worth -
multiple drives/volumes on a 'moderately busy' system.

Sure ... if I was going to live to be 199 ........ :-)

There are the competent, and then there are 'gurus'.

I am not a kernel guru.

I also don't want to try and modify the kernel, my
app needs to run on any 'vanilla' system.

> A decade or so ago, I was assigned to write a device
> mapper block device that would do intecept block-level
> operations. I studied several modules of kernel code to get an
> idea of how to do that. Fortunately, I was reassigned to a
> different task before I had to start debugging that code.
>

Heh, heh .... I've looked at some of that kernel code
and it's, well, 'dense'. That's a super-specialty job
best done by the guru class, people deep DEEP into
that stuff.

I just need something that'll blab when it's about to
write/delete a file.

[Grant Taylor]

On 9/8/22 8:55 PM, 25B.Z969 wrote:
> Any ideas ?

You might try researching some of the system accounting and / or system
auditing functions. One or both of them might have something that can
trigger when a write happens.



--
Grant. . . .
unix || die

[John-Paul Stewart]

On 2022-09-10 01:48, Grant Taylor wrote:
> On 9/8/22 8:55 PM, 25B.Z969 wrote:
>> Any ideas ?
>
> You might try researching some of the system accounting and / or system
> auditing functions.  One or both of them might have something that can
> trigger when a write happens.

In addition to those suggestions, I can't help but get the feeling that
what the OP is looking for is a lot like the filesystem journal concept.
It might be worth trying to read and/or preserve the journal with
debugfs (see its logdump command). There's also the jls command from
the sleuthkit package for examining the journal.

Or there's the option of custom code in the kernel to preserve
filesystem journal entries directly, if all else fails.

[25B.Z969]

On 9/10/22 9:14 AM, The Natural Philosopher wrote:
> On 10/09/2022 05:09, 25B.Z969 wrote:
>> But SOMEWHERE there's a "I commit to writing /A/B/C/junk.txt"
>>    before it's passed on to those lower levels. The OS *knows*.
>>
>>    And I want to know too, every time, promptly.
>
> Maybe patch the kernel/stdlib  and compile from source?

Nah ... I want it to run on any 'vanilla' kernel - it's
potentially not a one-off app.

'fanotify' comes closest so far - but it's a pain to
set up properly, esp if you want it to scan recursively
down the entire tree. Found a few fair examples I can
mod ... but the most coveted options to note Create
and Delete won't work using any common 'C' compiler.
"Modify" and "Write" work, but 'Delete' IS critical too.
"rm xyz.junk" doesn't trigger ANYTHING in the working
set of options.

'fanotify' is a thing where the MAN page really, badly,
SHOULD have had a good working well-commented example.
The "It should be perfectly clear to all how ..." premise
falls flat here.

[25B.Z969]

Pretty close ... I want to be informed of every file create/mod/delete
more or less AS IT HAPPENS across multiple large volumes so I can do
certain things with that info.

It's there somewhere. There's a point where the act is finally
committed, the path is known, and all that's left to do is
translate it to actual disk locations and write.

'fanotify' - a poorly-documented bit - can sort of do that by
polling something. However it's a bit like 'ionotify' in that
it expects to be focused on one folder non-recursively. Apparently
it IS possible have it check the entire tree - multiple trees -
but I'm trying to see what burden that imposes.

As other projects are pending it'll be a week or so before I
can fully explore fanotify. I know that a lot of the things
it's supposed to report don't work - crashes the program.
File DELETE is important to me and, of course, that's one of
the things that hangs it all up.

[26C.Z968]

Followup :

'fanotify' isn't much better than 'ionotify'. A lot
of its features don't even work ... even though
its authors say they OUGHT to work in newer kernels.

Like 'ionotify' it is sort of intended to monitor ONE
directory and maybe its subdirs. It is NOT an easy
means to be informed of ALL file i/o ops across
entire disks/volumes.

Which is annoying.

Hey, kernel jocks, we CAN think of uses for knowing
EVERY file create/mod/delete no matter where. If
there isn't a place to check for those, MAKE one !

[Fritz Wuehler]

2> kernel jocks, we CAN think of uses for knowing EVERY file
2> create/mod/delete no matter where. If there isn't a place to
2> check for those, MAKE one !

What you are asking for could easily lead to an infinite loop:
recording all filesystem changes on the filesystem being monitored
is a sure way to crash your system.


If you really want it now and can't wait until the kernel
developers add such a feature:

- get the sources of some userland filesystem like CFS, the
cryptographic file system by Matt Blaze (it's similar to NFS
with an extra encryption layer)

- add your desired hooks

- mount the directory/ies of interest on some other filesystem

- enjoy

[Rich]

26C.Z968 <26C....@noada.net> wrote:
> Hey, kernel jocks, we CAN think of uses for knowing EVERY file
> create/mod/delete no matter where. If there isn't a place to check
> for those, MAKE one !

The very point of open source is if there is a feature you want, then
either you can add it yourself, or you can pay someone to add it for
you.

But demanding that others do it is usually going to fall on deaf ears,
esp. when the "itch being scratched" is quite unique to you and your
usecase.

[Richard Kettlewell]

"26C.Z968" <26C....@noada.net> writes:
> Followup :
>
> 'fanotify' isn't much better than 'ionotify'. A lot
> of its features don't even work ... even though
> its authors say they OUGHT to work in newer kernels.
>
> Like 'ionotify' it is sort of intended to monitor ONE
> directory and maybe its subdirs. It is NOT an easy
> means to be informed of ALL file i/o ops across
> entire disks/volumes.

It sounds like you want FAN_MARK_FILESYSTEM.

--
https://www.greenend.org.uk/rjk/

[Richard Kettlewell]

It doesn’t seem like a niche use case to me, it’s the primitive used to
keep indexes up to date for fast whole-filesystem search (e.g Spotlight
in macOS, although the kernel API is different from Linux).

--
https://www.greenend.org.uk/rjk/

[26C.Z968]

On 9/30/22 8:39 AM, Fritz Wuehler wrote:
> 2> kernel jocks, we CAN think of uses for knowing EVERY file
> 2> create/mod/delete no matter where. If there isn't a place to
> 2> check for those, MAKE one !
>
> What you are asking for could easily lead to an infinite loop:
> recording all filesystem changes on the filesystem being monitored
> is a sure way to crash your system.

Either IQ, or programmed IQ, should solve that problem.

The OS is told to create/mod/delete a file. Don't really
care what or where. All I ask is a way to monitor every
such event. Full path + filename and action. Nothing
super-fancy beyond that.

> If you really want it now and can't wait until the kernel
> developers add such a feature:

Um ... no ... I have a few uses for such info but
I need it to work in 'vanilla' systems - Arch,RH,Deb-
based. No special mods, no distro-dependent variants,
plug-n-play. Just need *A* spot, *a* tempfile or short
log or var, that can be easily, quickly, efficiently
polled for the current file op.

As I said, 'fanotify' is the closest ... but the most
useful potential functions are NOT supported, at least
in the kernels used for Deb. Also it's really NOT meant
to monitor changes to whole volumes, more like 'ionitify'
in that respect. You, theoretically, CAN point it at
"/" but it looks very very inefficient there.

SEEMS like such a simple thing - yet it's unobtanium.

[Joe Beanfish]

On Sat, 01 Oct 2022 00:30:32 -0400, 26C.Z968 wrote:

> On 9/30/22 8:39 AM, Fritz Wuehler wrote:
>> 2> kernel jocks, we CAN think of uses for knowing EVERY file
>> 2> create/mod/delete no matter where. If there isn't a place to
>> 2> check for those, MAKE one !
>>
>> What you are asking for could easily lead to an infinite loop:
>> recording all filesystem changes on the filesystem being monitored
>> is a sure way to crash your system.
>
>
> Either IQ, or programmed IQ, should solve that problem.
>
> The OS is told to create/mod/delete a file. Don't really
> care what or where. All I ask is a way to monitor every
> such event. Full path + filename and action. Nothing
> super-fancy beyond that.

If you just want a log to examine as apposed to actually intercepting
such events, look at auditd which can log pretty much everything the
kernel does. Not sure if it will log writes (modifications) or not,
doing so might overwhelm the log on an active system.

[Joe Beanfish]

p.s.
See also systemtap

or a web search for "linux profiling" (you'll get hits about monitoring
individual programs as well as the whole system)

[26C.Z968]

Both interesting suggestions - however auditd lets you
put a 'watch' on *a* file ... but some of the aforementioned
utilities can do that and a little bit more. Systemtap is
basically a customized kernel - and I really really need
this app to run on nice safe bland trustworthy vanilla kernels.

It's a vexing problem. Things ARE being read/written/modded/deleted
but there's no way to SEE/LOG that across all disks/volumes in
anything approaching an efficient fashion. As said, I don't
want to "intercept" said ops, just know what they were so I can
do interesting things with that info a bit later with a fairly
'nice' process.

[The Natural Philosopher]

On 15/10/2022 04:30, 26C.Z968 wrote:
> there's no way to SEE/LOG that across all disks/volumes in
>   anything approaching an efficient fashion

Which is why no one has bothered to do it.
--
When plunder becomes a way of life for a group of men in a society, over
the course of time they create for themselves a legal system that
authorizes it and a moral code that glorifies it.

Frédéric Bastiat

[26C.Z968]

On 10/15/22 3:53 AM, The Natural Philosopher wrote:
> On 15/10/2022 04:30, 26C.Z968 wrote:
>> there's no way to SEE/LOG that across all disks/volumes in
>>    anything approaching an efficient fashion
>
> Which is why no one has bothered to do it.

But they SHOULD, there needs to be a a point, a var,
something, you can put a watch on and log all file
activities. There ARE uses ... but I'm not gonna tell :-)

I wonder if Winders can do that ... ?

[The Natural Philosopher]

Sorceror's apprentice. The log file logs all file access to...a log
file, including its won access to that log file...to file the logs on
file accesses...

Beware Recursion, my son...

[Richard Kettlewell]

The Natural Philosopher <t...@invalid.invalid> writes:
> On 15/10/2022 04:30, 26C.Z968 wrote:
>> there's no way to SEE/LOG that across all disks/volumes in
>>   anything approaching an efficient fashion
>
> Which is why no one has bothered to do it.

About 50 lines of C with fanotify will tell you about all modifications
within a selected filesystem. The intended use cases are filesytsem
indexing, virus scanners, etc. Apparently it’s not working out well for
the OP but he’s not made very clear why.

--
http://www.greenend.org.uk/rjk/

[26C.Z968]

What fanotify SAYS it will document is not really
what it WILL document. The authors complain that
even new kernels do not support the most useful
(esp to me) file events. They were "supposed to"
quite some time back, but still do not. Bullseye
certainly doesn't.

I need Create/Modify/Delete at minimum for my project
with Delete being just as important as the rest.

I'm still not entirely clear about the 'efficiency'
of fanotify (or ionotify) when it comes to documenting
said events across ALL disks/volumes. They are both
more oriented towards watching a SINGLE file, or at
most a single subdir, and I'm concerned that trying
to scale them up means the damned things would literally
try to separately watch EVERY file in there - of which
are hundreds of thousands.

So, I don't want to watch a file for changes, I'm
looking for a tap just a bit above there where the
system has decided it's gonna do something with
one/many files/folders/disks but maybe hasn't even
passed that along to the lower-level stuff quite yet -
where you only have to watch ONE var or buffer or
whatever. That would be efficient.

So far ... nothing quite like that.

[26C.Z968]

On 10/16/22 6:12 AM, The Natural Philosopher wrote:
> On 16/10/2022 03:34, 26C.Z968 wrote:
>> On 10/15/22 3:53 AM, The Natural Philosopher wrote:
>>> On 15/10/2022 04:30, 26C.Z968 wrote:
>>>> there's no way to SEE/LOG that across all disks/volumes in
>>>>    anything approaching an efficient fashion
>>>
>>> Which is why no one has bothered to do it.
>>
>>    But they SHOULD, there needs to be a a point, a var,
>>    something, you can put a watch on and log all file
>>    activities.  There ARE uses ... but I'm not gonna tell  :-)
>>
>>    I wonder if Winders can do that ... ?
> Sorceror's apprentice. The log file logs all file access to...a log
> file, including its won access to that log file...to file the logs on
> file accesses...
>
> Beware Recursion, my son...

Heh heh ... indeed ! :-)

But a little programming can spot that - and
prevent it. I'm most interested in "users files",
not system logs/caches/etc.

The OTHER, horrible, way is to run rsync on
EVERYTHING over and over, with the flags to
document 'changed' and/or 'disappeared' files.
That is NOT efficient. I use it for a species
of double-mirrored backups where the 2nd set
gets sent off for compression/encryption, but
those only run every day or two or seven.

I wonder if there's a general rule that can spot
where infinite recursion is likely to happen ?
Even worse, 'fractal/quantum recursion' where
the very act of monitoring very slightly changes
a lot of files/caches/etc, which, of course,
must be documented, which causes even more
small changes ..........

At least Capt. Kirk found an easy way to hang
up the overly-helpful androids. Interesting how
a HUMAN, even a child, can usually spot infinite
recursion happening or about to happen real
quick, but it's absolute hell to get a computer
to recognize such even in relatively narrow
controlled circumstances. What's our trick ?
How do we instinctively know that Zeno's Paradox
is bogus even though the formal proofs seem to
be very complex ?

[Anssi Saari]

"26C.Z968" <26C....@noada.net> writes:

> But they SHOULD, there needs to be a a point, a var,
> something, you can put a watch on and log all file
> activities. There ARE uses ... but I'm not gonna tell :-)
>
> I wonder if Winders can do that ... ?

As far as I know, Mac OS X does that. It was used by their Spotlight
search. It was a neat upgrade on the traditional updatedb+locate,
especially on spinning media. I don't know if that interface was usable
by user apps though.

I don't if Macs still have that, I haven't used OS X much since the
decade before last, it was called "Tiger" back then.

[Richard Kettlewell]

Anssi Saari <a...@sci.fi> writes:
> "26C.Z968" <26C....@noada.net> writes:
>
>> But they SHOULD, there needs to be a a point, a var,
>> something, you can put a watch on and log all file
>> activities. There ARE uses ... but I'm not gonna tell :-)
>>
>> I wonder if Winders can do that ... ?
>
> As far as I know, Mac OS X does that. It was used by their Spotlight
> search. It was a neat upgrade on the traditional updatedb+locate,
> especially on spinning media. I don't know if that interface was usable
> by user apps though.

Yes and yes. Windows has a similar API.

--
http://www.greenend.org.uk/rjk/

[Rich]

26C.Z968 <26C....@noada.net> wrote:

I have no affiiation with this project, I just saw it flow across HN:

https://github.com/e-dant/watcher

It might get you closer to your desire.

[26C.Z968]

Thank you, I will check it out. First glance
looks promising.