Linux inotify - how can I get process id of processing doing action?
bolta...@yahoo.co.uk
I'm currently learning to use the Linux inotify subsystem API which allows a
program to be notified of file system changes/accesses etc. However , while
it'll tell you what files or directories have had something done to them there
doesn't appear to be a way to tell you what process actually did it.
ie: its all very well knowing that file /dev/foo/bar has just been moved to
another location but it would be nice to know what process did the moving.
Is there a way to obtain this information or is it not provided because
a lot of the processes will be ephemeral (eg mv) and the process would already
had exited before my program gets its pid?
Thanks for any help
B2003
Barry Margolin
> Hi
>
> I'm currently learning to use the Linux inotify subsystem API which allows a
> program to be notified of file system changes/accesses etc. However , while
> it'll tell you what files or directories have had something done to them there
> doesn't appear to be a way to tell you what process actually did it.
>
> ie: its all very well knowing that file /dev/foo/bar has just been moved to
> another location but it would be nice to know what process did the moving.
You can't generally find out what user or process did anything to a
file, immediately or later.
>
> Is there a way to obtain this information or is it not provided because
> a lot of the processes will be ephemeral (eg mv) and the process would already
> had exited before my program gets its pid?
I suspect the designers of inotify didn't think that information would
be generally useful. There might also be security problems with
providing it -- if the writing process doesn't belong to the same userid
as the monitoring process, they probably shouldn't be aware of each
other.
The primary purpose of APIs like this is to replace loops that call
stat() periodically, as in "tail -f". These programs don't care what
process triggered it, they just want to know that it's time to do
something with the changed.
--
Barry Margolin, bar...@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
Gordon Burditt
If the process is or could be ephemeral, of what possible use is
its pid? About the only use I can think of is for a virus that has
managed to get to protect infected files and keep administrators
out. If anyone modifies the infected file, the virus restores it
and sprays -9 signals at the offender that fixed it, making sure not
to kill its own viral processes.
David Schwartz
It's not a meaningful concept. There is not a one-to-one
correspondence between actions a process can take and inotify
notifications, so there is no guarantee there is one and only one
process that can be reported. Rather than provide sort-of information,
the kernel developers opted not to provide it.
If you want to know what process caused a change, you need to trap the
operations themselves. You cannot tell from the operation's effects,
which is what inotify operates on.
DS
Rainer Weikusat
>>I'm currently learning to use the Linux inotify subsystem API which allows a
>>program to be notified of file system changes/accesses etc. However , while
>>it'll tell you what files or directories have had something done to them there
>>doesn't appear to be a way to tell you what process actually did it.
>>
>>ie: its all very well knowing that file /dev/foo/bar has just been moved to
>>another location but it would be nice to know what process did the moving.
>>
>>Is there a way to obtain this information or is it not provided because
>>a lot of the processes will be ephemeral (eg mv) and the process would already
>>had exited before my program gets its pid?
>
> If the process is or could be ephemeral, of what possible use is
> its pid?
It could be worse than 'of use': If the process was ephemeral, the pid
could meanwhile have been assigned to another process.
bolta...@yahoo.co.uk
gordon...@burditt.org (Gordon Burditt) wrote:
>>Is there a way to obtain this information or is it not provided because
>>a lot of the processes will be ephemeral (eg mv) and the process would already
>>had exited before my program gets its pid?
>
>If the process is or could be ephemeral, of what possible use is
>its pid? About the only use I can think of is for a virus that has
>managed to get to protect infected files and keep administrators
>out. If anyone modifies the infected file, the virus restores it
>and sprays -9 signals at the offender that fixed it, making sure not
>to kill its own viral processes.
Actually we need to monitor a world writable directory on a machine and the
pid would simply have been a stepping stone to finding the user id.
I don't see why it wouldn't be possible to provide the pid , after all , just
this sort of mechanism is used to provide information to a process about
who sent it a signal (process id, user id etc) using sigaction() and
siginfo_t. I don't see any fundamental difference between providing the pid
of someone who sent a signal and someone who modified the filesystem. If that
would require root privs to get the pid then fine , but it seems a bit
strange not to provide the functionality at all.
B2003
David Schwartz
> Actually we need to monitor a world writable directory on a machine and the
> pid would simply have been a stepping stone to finding the user id.
There is no user id associated with a file operation. Really.
For the simplest way to see why, consider:
1) Process opens file, memory maps it.
2) Process changes user ID.
4) Process writes a byte to the memory, no kernel call is involved.
5) Process changes user ID.
6) Process writes another byte to the memory, no kernel call is
involved.
7) Process closes/flushes file.
So when the single write is committed, what user ID should be
reported?
> I don't see why it wouldn't be possible to provide the pid , after all , just
> this sort of mechanism is used to provide information to a process about
> who sent it a signal (process id, user id etc) using sigaction() and
> siginfo_t. I don't see any fundamental difference between providing the pid
> of someone who sent a signal and someone who modified the filesystem.
But the difference is obvious, one and only one process with one exact
kernel calls sent the signal. On the other hand, a file change can be
the result of many operations in total done by many different
processes over time.
> If that
> would require root privs to get the pid then fine , but it seems a bit
> strange not to provide the functionality at all.
It makes no sense. A file change *report* is not associated with an
operation that creates a change in a file, and not all such operations
are even noticed by the kernel immediately.
DS
Rainer Weikusat
> On Mon, 15 Jun 2009 22:06:11 -0500
> gordon...@burditt.org (Gordon Burditt) wrote:
>>>Is there a way to obtain this information or is it not provided because
>>>a lot of the processes will be ephemeral (eg mv) and the process would already
>>>had exited before my program gets its pid?
>>
>>If the process is or could be ephemeral, of what possible use is
>>its pid? About the only use I can think of is for a virus that has
>>managed to get to protect infected files and keep administrators
>>out. If anyone modifies the infected file, the virus restores it
>>and sprays -9 signals at the offender that fixed it, making sure not
>>to kill its own viral processes.
>
> Actually we need to monitor a world writable directory on a machine and the
> pid would simply have been a stepping stone to finding the user id.
>
> I don't see why it wouldn't be possible to provide the pid , after all , just
> this sort of mechanism is used to provide information to a process about
> who sent it a signal (process id, user id etc) using sigaction() and
> siginfo_t. I don't see any fundamental difference between providing the pid
> of someone who sent a signal and someone who modified the
> filesystem.
There is at least one difference: The signal is targetted at a
particular process or set of processes. File system manipulations are
not. In particular, this implies that 'signals' are being used as an
IPC-mechanism among cooperating processes while the events which are
returned by inotify aren't. As an argument, this is - of course -
partially a red herring because in absence of a suitable API, they
cannot be used as such. But the interface has (probably) been designed
in order to enable process to monitor file system changes, not 'other
processes' and the person who did this design cannot possibly foresee
everything someone may want to do in case of a 'filesystem change
event'.
bolta...@yahoo.co.uk
David Schwartz <dav...@webmaster.com> wrote:
>So when the single write is committed, what user ID should be
>reported?
The real user id, not the effective one obviously.
>But the difference is obvious, one and only one process with one exact
>kernel calls sent the signal. On the other hand, a file change can be
>the result of many operations in total done by many different
>processes over time.
So? Only 1 specific inotify event will be associted with 1 process id however.
>It makes no sense. A file change *report* is not associated with an
>operation that creates a change in a file, and not all such operations
>are even noticed by the kernel immediately.
Of course they are otherwise how would the file be changed if the kernel
wasn't involved! Its the kernel that updates the cache and disk, the
programs don't write to the disk controller directly!
B2003
David Schwartz
> On Tue, 16 Jun 2009 03:23:30 -0700 (PDT)
>
> David Schwartz <dav...@webmaster.com> wrote:
> >So when the single write is committed, what user ID should be
> >reported?
>
> The real user id, not the effective one obviously.
The real user ID could be completely different in each of those steps.
Each of those steps could even be done by a different process and only
one notify event would be produced.
> >But the difference is obvious, one and only one process with one exact
> >kernel calls sent the signal. On the other hand, a file change can be
> >the result of many operations in total done by many different
> >processes over time.
> So? Only 1 specific inotify event will be associted with 1 process id however.
That is incorrect. One inotify event will be associated with one file
modification, which may be the result of numerous actions by numerous
processes.
> >It makes no sense. A file change *report* is not associated with an
> >operation that creates a change in a file, and not all such operations
> >are even noticed by the kernel immediately.
> Of course they are otherwise how would the file be changed if the kernel
> wasn't involved! Its the kernel that updates the cache and disk, the
> programs don't write to the disk controller directly!
The kernel does not update the cache. The kernel maps the cache into a
process' address space and the process can modify the cache without
kernel intervention. The kernel then writes the modified page back to
disk, but it has no idea which processes modified it.
You are operating based on numerous false assumptions about how the
kernel works.
Here's another example:
1) Process A maps a file writable. It is currently marked readable
only, though mapped writable.
2) Process B maps the file writable.
3) Process A modifies a byte in one page. The kernel notices the
change and marks the page writable/modified.
4) Process B now changes a byte in that page, the kernel has *no* way
to detect this.
5) The kernel later writes the page back.
How can it possibly know that process B modified the file?
DS
bolta...@yahoo.co.uk
David Schwartz <dav...@webmaster.com> wrote:
>> So? Only 1 specific inotify event will be associted with 1 process id how=
>ever.
>
>That is incorrect. One inotify event will be associated with one file
>modification, which may be the result of numerous actions by numerous
>processes.
Sorry , you're not making any sense. How could for example modifying the
file attributes be done by more than 1 process in a way that would only
generate 1 notify event?
>1) Process A maps a file writable. It is currently marked readable
>only, though mapped writable.
>
>2) Process B maps the file writable.
>
>3) Process A modifies a byte in one page. The kernel notices the
>change and marks the page writable/modified.
>
>4) Process B now changes a byte in that page, the kernel has *no* way
>to detect this.
>
>5) The kernel later writes the page back.
>
>How can it possibly know that process B modified the file?
Well in that case how can it ever send the IN_MODIFY event if its not
aware of a file being modified? After all, the file could be mapped write
by both processes then no changes made to it so it can't just be relying on
the file being opened to write - thats another event entirely called IN_OPEN -
or just relying on a writeable file being closed since that is covered
by the IN_CLOSE_WRITE event.
B2003
Rainer Weikusat
> David Schwartz <dav...@webmaster.com> wrote:
[...]
>>1) Process A maps a file writable. It is currently marked readable
>>only, though mapped writable.
>>
>>2) Process B maps the file writable.
>>
>>3) Process A modifies a byte in one page. The kernel notices the
>>change and marks the page writable/modified.
>>
>>4) Process B now changes a byte in that page, the kernel has *no* way
>>to detect this.
>>
>>5) The kernel later writes the page back.
>>
>>How can it possibly know that process B modified the file?
>
> Well in that case how can it ever send the IN_MODIFY event if its not
> aware of a file being modified?
IN_MODIFY events are only generated for modifications done using
write(v), as can be easily checked by searching for the corresponding
helper function (fsnotify_modify) in the kernel source tree (done for
2.6.30). Which implies that the PID of the process who cause the event
to happen is always known.
Gordon Burditt
>>>a lot of the processes will be ephemeral (eg mv) and the process would already
>>>had exited before my program gets its pid?
>>
>>If the process is or could be ephemeral, of what possible use is
>>its pid? About the only use I can think of is for a virus that has
>>managed to get to protect infected files and keep administrators
>>out. If anyone modifies the infected file, the virus restores it
>>and sprays -9 signals at the offender that fixed it, making sure not
>>to kill its own viral processes.
>
>Actually we need to monitor a world writable directory on a machine and the
>pid would simply have been a stepping stone to finding the user id.
It's *NOT* a stepping stone to finding the user id. It's a dead
end, most of the time, for processes that terminate quickly. And
sometimes it could give you a decidedly wrong answer (pids get re-used).
>I don't see why it wouldn't be possible to provide the pid , after all , just
There are plenty of things that *COULD* be provided. The question
is whether the information is of any use. For example, a locking
call like flock() or fcntl() could provide information on how many
locks there were on that section of the file exactly pi days ago.
Fine, now of what use is that information, especially given that
it can be out of date by the time the caller gets to use it?
>this sort of mechanism is used to provide information to a process about
>who sent it a signal (process id, user id etc) using sigaction() and
>siginfo_t.
Signals are not always sent by a process nor a user id. Anyone who
thinks otherwise shouldn't be using that information.
>I don't see any fundamental difference between providing the pid
>of someone who sent a signal and someone who modified the filesystem. If that
>would require root privs to get the pid then fine , but it seems a bit
>strange not to provide the functionality at all.
It's useless functionality that seems to advertise much more utility
than it actually has.
Scott Lurndal
>>I don't see any fundamental difference between providing the pid
>>of someone who sent a signal and someone who modified the filesystem. If that
>>would require root privs to get the pid then fine , but it seems a bit
>>strange not to provide the functionality at all.
>
>It's useless functionality that seems to advertise much more utility
>than it actually has.
Actually, it is a significant security issue to expose "who" modified
a file to a process executing as a different user. The signal analogy
is flawed because the process receiving the signal was the explicit target
of a "communication" from the source process, so exposing the PID to
the receiver is correct.
Exposing any information to 'inotify clients' other than that the file has changed
is a security hole, and even noting that it has changed could be a
security issue (i.e. a low-bitrate covert channel).
I would hope that the inotify implementation won't allow notifications
for files that aren't accessible to the caller.
Gordon, Please attempt to preserve the attribution lines.
scott
David Schwartz
> >That is incorrect. One inotify event will be associated with one file
> >modification, which may be the result of numerous actions by numerous
> >processes.
> Sorry , you're not making any sense. How could for example modifying the
> file attributes be done by more than 1 process in a way that would only
> generate 1 notify event?
See the example below:
> >1) Process A maps a file writable. It is currently marked readable
> >only, though mapped writable.
>
> >2) Process B maps the file writable.
>
> >3) Process A modifies a byte in one page. The kernel notices the
> >change and marks the page writable/modified.
>
> >4) Process B now changes a byte in that page, the kernel has *no* way
> >to detect this.
>
> >5) The kernel later writes the page back.
>
> >How can it possibly know that process B modified the file?
> Well in that case how can it ever send the IN_MODIFY event if its not
> aware of a file being modified?
One of two ways. It can send the event when Process A makes the change
or it can send the event when it does the write back. The one thing it
cannot do is send the event when Process B makes the change, as I
explained above.
> After all, the file could be mapped write
> by both processes then no changes made to it so it can't just be relying on
> the file being opened to write - thats another event entirely called IN_OPEN -
> or just relying on a writeable file being closed since that is covered
> by the IN_CLOSE_WRITE event.
Did you read my example? I explained this. Even though the file is
mapped write, the kernel internally maps it only read. When the
process does modify the page, that causes a fault (because the page is
mapped read only). At that point, the kernel maps the page read/write
and knows that it has to write the page back. When the kernel writes
the page back, if needed, it will change the internal mapping to read-
only and start the process over.
Thus the kernel need do *nothing* when process B modifies the file.
DS
David Schwartz
> IN_MODIFY events are only generated for modifications done using
> write(v), as can be easily checked by searching for the corresponding
> helper function (fsnotify_modify) in the kernel source tree (done for
> 2.6.30). Which implies that the PID of the process who cause the event
> to happen is always known.
Right, but you don't design interfaces based on how things just happen
to work. You design interfaces around how things are semantically. As
I explained, there is not a semantic one-to-one correspondence between
process events that modify files and file modification notifications.
DS
Maxim Yegorushkin
One way to do so, is to implement your own proxy filesystem (using fuse, for
example). This proxy filesystem will record all the information relevant to you
(pid, etc..) and delegate the operation to the real filesystem.
--
Max
bolta...@yahoo.co.uk
gordon...@burditt.org (Gordon Burditt) wrote:
>>Actually we need to monitor a world writable directory on a machine and the
>>pid would simply have been a stepping stone to finding the user id.
>
>It's *NOT* a stepping stone to finding the user id. It's a dead
>end, most of the time, for processes that terminate quickly. And
>sometimes it could give you a decidedly wrong answer (pids get re-used).
You could get a wrong answer when the check the pid that sent a signal.
>>this sort of mechanism is used to provide information to a process about
>>who sent it a signal (process id, user id etc) using sigaction() and
>>siginfo_t.
>
>Signals are not always sent by a process nor a user id. Anyone who
>thinks otherwise shouldn't be using that information.
Which is why you check whether the siginfo_t* passed to your signal handler
is NULL first or the si_pid field is zero. Your point is?
>It's useless functionality that seems to advertise much more utility
>than it actually has.
I'm not sure if you're talking about signals or inotify but I've had reason
in the past for a process to know what sent it a signal and I have reason
now for one to know who modified a file. Just because you can't think of
a use for something doesn't mean other people don't have a use for it.
B2003
bolta...@yahoo.co.uk
sc...@slp53.sl.home (Scott Lurndal) wrote:
>>It's useless functionality that seems to advertise much more utility
>>than it actually has.
>
>Actually, it is a significant security issue to expose "who" modified
>a file to a process executing as a different user. The signal analogy
Rubbish. You might as well say "ls" has security issues because it lets you
find out the file owner and group.
B2003
bolta...@yahoo.co.uk
David Schwartz <dav...@webmaster.com> wrote:
>One of two ways. It can send the event when Process A makes the change
>or it can send the event when it does the write back. The one thing it
>cannot do is send the event when Process B makes the change, as I
>explained above.
So you're saying that when a process does a write() that system call doesn't
actually go through the kernel then? Thats news to me.
B2003
Rainer Weikusat
Interface are designed in order to accomplish some purpose and the
purpose of 'inotify' is to monitor the file system and not to monitor
processes.
Independently of this, monitoring memory accesses of a processes is
generally considered to be 'an expensive operation' and therefore
often done by a hardware memory management unit. This implies that an
application should not be designed to require monitoring of memory
accessses done by other applications, or, for this specific example,
that IN_MODIFY-notifcations including a process-ID are unlikely to
ever be implemented for write accesses to memory-mapped files (because
this would render the TLB completely useless).
But this is 'just how things happen to work'.
Rainer Weikusat
No. A write access to a page of memory doesn't go through the kernel,
except if it causes a page fault as side effect, and because files can
be 'memory mapped', such write accesses can result in modifications to
the contents of the file system. But this is somewhat hypothetical in
the context of inotify which (presently) doesn't support any
modification notifications for this case.
David Schwartz
> David Schwartz <dav...@webmaster.com> wrote:
If you don't understand what a unified file/page cache is, then you'll
have to learn that before you'll be able to understand what I'm
saying. But the short answer is that the kernel often puts itself 'out
of the loop' as a massive performance optimization. As a result, it
would have to disable these optimizations to be able to notify you of
which process made all file writes.
I'll try one more time to explain it in simple terms.
Suppose two processes each map the same page of the same file into
memory. Then suppose one of those processes modifies that page. At
this point, the kernel knows that this page of memory contains the
authoritative copy of that part of the file, it knows it has been
modified, and it knows that at some point it must write it back to
disk. The kernel now has no reason to intercede in any process that
reads from or writes to this page. Eventually, the kernel will write
the page back to disk, assuming the file is not deleted.
The kernel simply has no way to alert you to each write to this file,
because the kernel does not trap them. The page is mapped writable in
each process that has mapped it semantically-writable.
The kernel can detect the initial modification because it maps the
file readable even though the process requested it to be writable. But
once the page is modified, the kernel allows the mapping to be
writable and has no need to trap or detect subsequent file
modifications. A transition to kernel space is not needed either, so
the kernel does not force one.
What you are suggesting would require a forced transition to kernel
space on every file modification, even if that part of the file were
already resident in memory and already modified. That would be a
horrible performance pessimization.
Suppose I have 'char *foo' which is a pointer to a memory-mapped page
of a file. My code does this:
while(*ptr!=0) *foo++=*ptr++;
Do you want a notification for each byte of the string I write to the
file? If yes, you're crazy. If no, what if the bytes of string come
from alternating processes. Still want it?
DS
Nate Eldredge
> The kernel can detect the initial modification because it maps the
> file readable even though the process requested it to be writable.
Perhaps Linux works this way, but in general I don't think there's any
reason why it should have to be. On x86, for example, there is a
"dirty bit" for each page that is set by the hardware when the page is
written. So the kernel could map the file read/write and clear the
dirty bit, and then just let the process run. If at some later time it
sees that the dirty bit is set, it would know that the page needs to be
written back to backing store.
> But once the page is modified, the kernel allows the mapping to be
> writable and has no need to trap or detect subsequent file
> modifications. A transition to kernel space is not needed either, so
> the kernel does not force one.
Yep.
> What you are suggesting would require a forced transition to kernel
> space on every file modification, even if that part of the file were
> already resident in memory and already modified.
That is, a transition to kernel space on every instruction that writes
to that memory. Essentially you would be single-stepping the program.
This might be useful for debugging but not otherwise.
> That would be a horrible performance pessimization.
No kidding.
Alex Fraser
[snip]
> IN_MODIFY events are only generated for modifications done using
> write(v), as can be easily checked by searching for the corresponding
> helper function (fsnotify_modify) in the kernel source tree (done for
> 2.6.30).
Never having looked at the implementation, I'm slightly surprised. I'd
have expected IN_MODIFY to be generated whenever the file's mtime is
"marked for update" - that being a side-effect of some calls, and also
presumably done at some point for mmap'ed files that are modifed (since
it is required by POSIX). Similar for IN_ACCESS.
Out of interest, is there a particular reason you know but I can't think
of for it not to work like this?
Alex
Scott Lurndal
Indeed, in a secure system, you would not be able to 'ls' a file with
an owner or group different from your own. And even unix/linux defaults
to preventing search access to other users files.
It's called a covert channel in the biz (low bitrate, to be sure).
scott
David Schwartz
> David Schwartz <dav...@webmaster.com> writes:
> > The kernel can detect the initial modification because it maps the
> > file readable even though the process requested it to be writable.
> Perhaps Linux works this way, but in general I don't think there's any
> reason why it should have to be. On x86, for example, there is a
> "dirty bit" for each page that is set by the hardware when the page is
> written. So the kernel could map the file read/write and clear the
> dirty bit, and then just let the process run. If at some later time it
> sees that the dirty bit is set, it would know that the page needs to be
> written back to backing store.
It can be done that way. I wasn't suggesting that it was done any
particular way on any particular operating system, just that there
doesn't have to be a one-to-one correspondence between process actions
and kernel notifications that a file has changed.
My intuition is that using the dirty bit would be inferior simply
because it would require scanning all writable pages periodically to
make sure their dirty bits weren't set. Otherwise, you could not
ensure timely commitment of changes to stable storage. Then again,
their probably aren't that may pages that are writable, unmodified,
and that need to be committed to stable storage, so maybe scanning
them all every 10 seconds or so is no big deal.
I don't actually know which way Linux does it. (And it may vary by
architecture.)
DS
Rainer Weikusat
> Rainer Weikusat wrote:
> [snip]
>> IN_MODIFY events are only generated for modifications done using
>> write(v), as can be easily checked by searching for the corresponding
>> helper function (fsnotify_modify) in the kernel source tree (done for
>> 2.6.30).
>
> Never having looked at the implementation, I'm slightly surprised. I'd
> have expected IN_MODIFY to be generated whenever the file's mtime is
> "marked for update" - that being a side-effect of some calls, and also
> presumably done at some point for mmap'ed files that are modifed
> (since it is required by POSIX).
Ok, I haven't had the time to check this in detail so far and it is
unlikely that I will in the near future, but from the few bits I have
looked at, I figure that I was wrong and you are right: There is a
subroutine I overlooked which is called 'fsnotify_change' and is
reponsible for sending notifications for i-node changes and 'content
modifications' and these may (and likely, is) be called at some
unspecified time after a writable mapping has been changed (it is
certainly called in case of a MS_SYNC msync).