Solaris equivelent of inotify?
Rick Ingham
really use this capability for when files are opened/closed/modified by
another process in a Solaris application of mine.
Anyone know of a similar function/facility in Solaris?
Rick.
David Schwartz
"Rick Ingham" <rdin...@comcast.net> wrote in message
news:YYudnQib47AZHdHe...@comcast.com...
I think FAM is what you want.
DS
William Ahern
That's just a wrapper, though. On Linux it uses dnotify, *BSD kqueue(2), and
Solaris likely falls back to a stat(2) loop.
So, in that case it might be easier just to use stat(2) yourself rather than
install and wrestle with FAM.
Rick Ingham
FAM doesn't support open/close notification. Appears to be only i-node
modifications.
What I need is to know when a file is opened and closed to synchronize
access between two processes. I need to know when a command line
background process gets a file open to proceed in the foreground
process, and when the background process has caught up and closes the
file. I'm doing this to coordinate with an external process I don't
have any control over.
Rick.
Rick Ingham
> In article <zaidnZBGRuq...@comcast.com>,
> availble.
>
If this were just a toy system, or just one server, I'd be intrigued by
the idea. But, there's no way the my employer will let me tinker at the
kernel level for over 160 production data centers, over 200 servers.
This program and process is no where near worth that kind of risk.
Rick.
Chris Ridd
Ingham" <rdin...@comcast.net> wrote:
I don't think anyone's suggested this yet - it *is* quite a hack - but you
could monitor the background process closing the file via truss, and then
have something monitor the truss output and signal the foreground process.
OK, I said it was a hack :-)
Maybe you could use dtrace instead of truss, but you get the idea.
Cheers,
Chris
Thomas Maier-Komor
>
> FAM doesn't support open/close notification. Appears to be only i-node
> modifications.
>
> What I need is to know when a file is opened and closed to synchronize
> access between two processes. I need to know when a command line
> background process gets a file open to proceed in the foreground
> process, and when the background process has caught up and closes the
> file. I'm doing this to coordinate with an external process I don't
> have any control over.
>
> Rick.
you can get more control over a process than you might think,
as long as you have all necessary rights to do this...
What about LD_PRELOAD'ing special versions of open(2) and close(2) and
trigger on those files that you care about and add standard file locking
using flock(3ucb) or lockf(3c)?
HTH,
Tom
Drazen Kacar
> Michael Vilain wrote:
> > In article <zaidnZBGRuq...@comcast.com>,
> > Rick Ingham <rdin...@comcast.net> wrote:
> >>What I need is to know when a file is opened and closed to synchronize
> >>access between two processes. I need to know when a command line
> >>background process gets a file open to proceed in the foreground
> >>process, and when the background process has caught up and closes the
> >>file. I'm doing this to coordinate with an external process I don't
> >>have any control over.
> >
> > Well, you could modify the kernel to add this feature. Source is now
> > availble.
>
> If this were just a toy system, or just one server, I'd be intrigued by
> the idea. But, there's no way the my employer will let me tinker at the
> kernel level for over 160 production data centers, over 200 servers.
> This program and process is no where near worth that kind of risk.
Do you have control over the file which is being opened and closed? You
could put a FIFO in its place, and you'd get the notifications in that
case. (Didn't work on FreeBSD 4.x last time I needed it, though.)
But FIFO doesn't work too well if there are multiple readers and writers,
so maybe something else would work better. On Solaris maybe you could use
fattach() to connect it with your streams module, for example.
If you tell us more about your constraints, somebody might be able to come
up with a better idea. Information about the OS you're using would also be
helpful; your post is crossposted to Solaris, Linux and general Unix
programming groups, so I can't tell the portability requirements.
--
.-. .-. Yes, I am an agent of Satan, but my duties are largely
(_ \ / _) ceremonial.
|
| da...@fly.srk.fer.hr
Chuck Dillon
>
> What I need is to know when a file is opened and closed to synchronize
> access between two processes. I need to know when a command line
> background process gets a file open to proceed in the foreground
> process, and when the background process has caught up and closes the
> file. I'm doing this to coordinate with an external process I don't
> have any control over.
>
> Rick.
Just a thought. Is there any way you could have this process you have
no control over access the data via fifos or domain sockets serviced by
processes you do have control over?
-- ced
--
Chuck Dillon
Senior Software Engineer
NimbleGen Systems Inc.
Darren Dunham
> Well, you could modify the kernel to add this feature. Source is now
> availble.
Well, yes, but that doesn't make it easy to add to production servers.
I'd imagine you'd need to be running Solaris 10.
If you have that anyway, then you could use dtrace. I wouldn't want to
instrument (or deal with the performance impact) of 'truss' as part of a
process, but 'dtrace' would work okay here.
It would of course require Solaris 10...
--
Darren Dunham ddu...@taos.com
Senior Technical Consultant TAOS http://www.taos.com/
Got some Dr Pepper? San Francisco, CA bay area
< This line left intentionally blank to confuse you. >
William Ahern
> you can get more control over a process than you might think,
> as long as you have all necessary rights to do this...
> What about LD_PRELOAD'ing special versions of open(2) and close(2) and
> trigger on those files that you care about and add standard file locking
> using flock(3ucb) or lockf(3c)?
I've vote for this method, too. I had to do this a couple of weeks ago to
use a MySQL server in a regression test. It had /etc/my.conf hardcoded into
the binary. Something like the following should work on Solaris, I think.
Haven't ever tried it, though. I would start MySQL like
`LD_PRELOAD="mysqlfix.so" /path/to/mysqld blah blah`.
#define _GNU_SOURCE
#include <stdio.h> /* FILE */
#include <stdlib.h> /* getenv(3) */
#include <stdarg.h> /* va_list va_start */
#include <string.h> /* strcmp(3) */
#include <sys/types.h> /* mode_t */
#include <fcntl.h> /* O_CREAT */
#include <dlfcn.h> /* RTLD_NEXT dlsym(3) */
/*
* Interpose open so we can actually specify which my.cnf to open, instead
* of always trying to open /etc/my.cnf. MySQL does this regardless of the
* commandline options you give.
*/
int open(const char *path, int flags, ...) {
static int (*real_open)(const char *, int, ...);
const char *real_path;
va_list ap;
mode_t mode;
if (!real_open && !(real_open = dlsym(RTLD_NEXT,"open")))
return -1;
if (flags & O_CREAT) {
va_start(ap,flags);
mode = va_arg(ap,mode_t);
va_end(ap);
}
if (0 == strcmp(path,"/etc/my.cnf") && (real_path = getenv("ETC_MY_CNF")))
path = real_path;
return real_open(path,flags,mode);
} /* open() */
int open64(const char *path, int flags, ...) {
static int (*real_open)(const char *, int, ...);
const char *real_path;
va_list ap;
mode_t mode;
if (!real_open && !(real_open = dlsym(RTLD_NEXT,"open64")))
return -1;
if (flags & O_CREAT) {
va_start(ap,flags);
mode = va_arg(ap,mode_t);
va_end(ap);
}
if (0 == strcmp(path,"/etc/my.cnf") && (real_path = getenv("ETC_MY_CNF")))
path = real_path;
return real_open(path,flags,mode);
} /* open64() */
FILE *fopen(const char *path, const char *mode) {
static FILE *(*real_fopen)(const char *, const char *);
const char *real_path;
if (!real_fopen && !(real_fopen = dlsym(RTLD_NEXT,"fopen")))
return 0;
if (0 == strcmp(path,"/etc/my.cnf") && (real_path = getenv("ETC_MY_CNF")))
path = real_path;
return real_fopen(path,mode);
} /* fopen() */
FILE *fopen64(const char *path, const char *mode) {
static FILE *(*real_fopen)(const char *, const char *);
const char *real_path;
if (!real_fopen && !(real_fopen = dlsym(RTLD_NEXT,"fopen64")))
return 0;
if (0 == strcmp(path,"/etc/my.cnf") && (real_path = getenv("ETC_MY_CNF")))
path = real_path;
return real_fopen(path,mode);
} /* fopen64() */
#if 0
#include <stdio.h>
int main(void) {
FILE *fp = fopen("/etc/my.cnf","r");
char *ln = 0;
size_t n = 0;
while (-1 != getline(&ln,&n,fp)) {
printf("line: %s",ln);
}
return 0;
}
#endif
Logan Shaw
> What I need is to know when a file is opened and closed to synchronize
> access between two processes. I need to know when a command line
> background process gets a file open to proceed in the foreground
> process, and when the background process has caught up and closes the
> file. I'm doing this to coordinate with an external process I don't
> have any control over.
Well, if you are forced to coordinate with an external process that
doesn't want to cooperate, you could always use poll with
popen() and /usr/sbin/fuser.
Or, have a look at the Open Solaris source code for fuser and see
what trick it is using to figure out which processes are using
what file, and use that trick yourself. /usr/sbin/fuser isn't
set-uid, so it should be possible to do it from your own code.
Actually, a quick truss indicates it seems to be using /dev/kstat,
so it's probably the kstat library ("man -a kstat"). Another
possible place to look is the (third-party) lsof source code.
- Logan
Rick Ingham
Unfortunately, my problem goes deaper than that. I won't know what the
pid of the process I'm waiting for is. Maybe it would help if I tell
you exactly what I'm doing.
I'm actuall trying to drive IBM DB2's command line program 'db2' to
initiate a "LOAD" operation from a message pipe. In one case, I'm
waiting for DB2 to open the named pipe (fifo) before I start feeding it
data. Int he second case, I need to monitor when DB2 has let go of the
log output file I told it to put the log messages in. Here's the
kicker: The actual process that will have hold of the file in both
cases is a 'db2bp' background process from a different instance of a
'db2' child process. The 'db2' command spawns a child in the background
that holds the connection to the agent. So I really need to monitor the
file access by any program, and I don't have a link to the PID of the
process I may need to pay attention to.
I have to do it this way as I already have vicariously a connection to
the database through a 3rd party library, and my process can no open
another connection to directly drive DB2.
Currently I run up to 6 separate parralel load processes on the same
machine, database, at a time. I currently use system("fuser {log}") to
detect when processes have the file open or have let go. fuser is
increadibly expensive, and doing every few seconds in multiple
processes, really suck down the machine.
inotify() would exactly fit the bill, but alas, that's not a Solaris
feature.
When I read aboud inotify, I was sure Solaris would have a similar
feature. So far, no dice.
Rick.
Logan Shaw
> Chris Ridd wrote:
>> I don't think anyone's suggested this yet - it *is* quite a hack - but
>> you
>> could monitor the background process closing the file via truss, and then
>> have something monitor the truss output and signal the foreground
>> process.
> Unfortunately, my problem goes deaper than that. I won't know what the
> pid of the process I'm waiting for is. Maybe it would help if I tell
> you exactly what I'm doing.
You can tell "truss" to continue to trace forked processes. It
would be horribly inefficient though.
> I'm actuall trying to drive IBM DB2's command line program 'db2' to
> initiate a "LOAD" operation from a message pipe. In one case, I'm
> waiting for DB2 to open the named pipe (fifo) before I start feeding it
> data.
Why? Are you on a platform where the named pipe implementation is
broken? You should be able to just write the data to the pipe,
and if nobody has opened the pipe for reading, the writes will
eventually block.
> Int he second case, I need to monitor when DB2 has let go of the
> log output file I told it to put the log messages in. Here's the
> kicker: The actual process that will have hold of the file in both
> cases is a 'db2bp' background process from a different instance of a
> 'db2' child process. The 'db2' command spawns a child in the background
> that holds the connection to the agent. So I really need to monitor the
> file access by any program, and I don't have a link to the PID of the
> process I may need to pay attention to.
I'm not sure why you can't just wait for the db2 (parent) process
to exit and then do something after that, although maybe the process
does several things and this is only one of them.
> Currently I run up to 6 separate parralel load processes on the same
> machine, database, at a time. I currently use system("fuser {log}") to
> detect when processes have the file open or have let go. fuser is
> increadibly expensive, and doing every few seconds in multiple
> processes, really suck down the machine.
If you can't find a better solution than fuser, you could at least
improve the performance by doing a stat() on the log file and checking
the modification date. Only if the modification date hasn't changed
in 5 seconds (or whatever interval) do you need to check if the other
process has closed the file, because if it keeps changing the modification
time, you can be pretty sure it hasn't stopped writing to it.
- Logan
Rick Ingham
This is on Solaris 9, running on 280r's up to e2900's.
FIFO's are tricky animals. open() for write will block until someone
opens it for read. FIFO's don't buffer any data, so a write will block
until the reader reads the record being written. I did a lot of work to
make sure the foreground process doesn't block waiting for DB2 when DB2
fails to even get to opening the file for some reason. The only way not
to get hung was to not do the open() until the reader had. For the log
file, the problem is I have to wait for DB2 to catch up and finish
processing and is finished writing the log file before I read and
analyze if for errors.
I was hoping to find a way to avoid fuser, but it's sounding a lot like
there isn't an approprate library or system API to do what I need in
Solaris.
This application gets used enough that I'd love to fix it, but not
enough to warrant a redesign.
Rick.
Logan Shaw
> FIFO's are tricky animals. open() for write will block until someone
> opens it for read. FIFO's don't buffer any data, so a write will block
> until the reader reads the record being written. I did a lot of work to
> make sure the foreground process doesn't block waiting for DB2 when DB2
> fails to even get to opening the file for some reason.
How about
db2 < /tmp/mypipe
Or it db2 insists on a being passed a pathname, then
db2 < /tmp/mypipe /dev/fd/0
Or if db2 insists on using stdin for something different, then
db2 4< /tmp/mypipe /dev/fd/4
Since in all the above you are causing the shell to open the pipe,
it should not fail to be opened.
Or heck, perhaps you need a named pipe only because db2 requires a
pathname and an unnamed pipe wouldn't work only for that reason.
In that case, you can easily do something like
data-producer | db2 /dev/fd/0
or if db2 insists on using stdin for other things, then
data-producer | 4<&0 db2 /dev/fd/4
Just some ideas, of course.
- Logan
Drazen Kacar
> FIFO's are tricky animals. open() for write will block until someone
> opens it for read. [...]
> The only way not to get hung was to not do the open() until the reader
> had.
You can open it with O_NONBLOCK and it will not hang. Then you should be
able to get notification via poll() when the reader attaches. Or SIGPOLL,
if that suits you better.
James Antill
> Rick Ingham wrote:
>
>> FIFO's are tricky animals. open() for write will block until someone
>> opens it for read. [...]
>
>> The only way not to get hung was to not do the open() until the reader
>> had.
>
> You can open it with O_NONBLOCK and it will not hang. Then you should be
> able to get notification via poll() when the reader attaches. Or SIGPOLL,
> if that suits you better.
Not true.
% man 4 fifo
A process can open a FIFO in non-blocking mode. In this case, opening
for read only will succeed even if noone has opened on the write side
yet; opening for write only will fail with ENXIO (no such device or
address) unless the other end has already been opened.
...as the above says, you can't open() for write only with O_NONBLOCK and
poll().
--
James Antill -- ja...@and.org
http://www.and.org/and-httpd
Rick Ingham
I do a popen("db2") and write a LOAD command to it that specifies both
an input file name (a name pipe fifo) and a file name to write the
output messages to. The 'db2' program/command will pass the request to
an agent handler (db2bp) process that isn't necessarily a child of this
'db2' process. The db2bp, if it is happy to start the LOAD will then
open the fifo. I have to watch the fifo polling with fuser for a
reasonable period of time before I give up and read the output file and
the pipe to 'db2' to extract the error messages why it wouldn't or
couldn't start.
If an inotify like more methods. I have one that works. It was tricky
to get the coordination just right the first time. inotify would make
it easier, anything else makes it harder.
I really have my answer. Such a facility/API doesn't exist in Solaris,
so there isn't much point to continuing this investication.
Thanks for all the input, everyone.
Rick.
Rick Ingham
> Rick Ingham wrote:
>
>
>> FIFO's are tricky animals. open() for write will block until someone
>> opens it for read. [...]
>
>
>> The only way not to get hung was to not do the open() until the reader
>> had.
>
>
> You can open it with O_NONBLOCK and it will not hang. Then you should be
> able to get notification via poll() when the reader attaches. Or SIGPOLL,
> if that suits you better.
>
It still doesn't address the primary requirement to know when the db2bp
has opend the file (signaling ready to read and startup was successful)
and knowing when it has closed the log (signaling end of processing) to
check for errors.
Rick.
Rick Ingham
Right, and it's the writer that I can't afford to block.
Rick.
Robert Milkowski
Basically you can do it with DTrace and syscall provider (or/and IO provider).
You will need to use Solaris 10 to get DTrace.
--
Robert Milkowski
rmilkow...@wp-sa.pl
Geoff Lane
If you switch on BSM Audit you get a complete trace of file open/close
actions with very low overhead. You would have to write a program that
tracked the BSM logs but that is relatively simple.
--
Geoff Lane
The Eternal Triangle is usually right tangled.
Drazen Kacar
You're right. So I've tried to create a pipe, called fattach() with one
end and closed the other end[1]. That created a FIFO on the file system and
my process has a file descriptor (O_RDWR).
Another process can open the FIFO with O_RDONLY, but read() returns 0, as
if no other process has the write end opened. Weird.
[1] Reversing the operations, ie. close() before fattach() results in
ENXIO for the reader's subsequent open(), because fifofs doesn't like
it.