Dectecting dir changes
chuck
incoming FTP transfers. Under certain conditions, when the transfer is
complete I need to send an email notification, and do other stuff.
Win32 provides FindFirstChangeNotification(), but as best I can tell
this isn't supported on Solaris.
I am thinking of using the approach suggested here
http://tgolden.sc.sabren.com/python/win32_how_do_i/watch_directory_for_changes.html
which is:
import os, time
path_to_watch = "."
before = dict ([(f, None) for f in os.listdir (path_to_watch)])
while 1:
time.sleep (10)
after = dict ([(f, None) for f in os.listdir (path_to_watch)])
added = [f for f in after if not f in before]
removed = [f for f in before if not f in after]
if added: print "Added: ", ", ".join (added)
if removed: print "Removed: ", ", ".join (removed)
before = after
My concern with this is that a change may be detected before the ftp
daemon process is done writing the file to disk. I don't want to take
any action until the file is written and closed. I know that I could
pole a new file looping to check to see if it's file size is changing
but the timing of such a loop is subject to I/O buffering and is
otherwise not elegant.
Googling shows other solutions using fcntl
(http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/217829) but it
appears that this only works on Linux.
While I'm at it I'm going to throw in a grump about the Python
documentation of fcntl. The doco indicates to read the source for
fcntl.py to lookup the constants representing the different types of
events/signals that are avaiable. However fcntl on some platforms
seems to be implemented as a binary leaving no way to look up the
contants for the platform.
Suggestions?
Fredrik Lundh
there is no reliable way to do this, unless you can modify the sending
application to use a temporary filename during transfer (or find an FTP
server that does this automatically), or you're willing to write your own
FTP responder.
checking the mtime for new files might be good enough for some use
cases (poll the directory at regular intervals; if a file is newer than X
seconds, assume it's still being updated).
</F>
Sybren Stuvel
> The doco indicates to read the source for fcntl.py to lookup the
> constants representing the different types of events/signals that
> are avaiable. However fcntl on some platforms seems to be
> implemented as a binary leaving no way to look up the contants for
> the platform.
'pydoc fcntl' works fine on my system, even though the module is a
binary .so file.
Sybren
--
The problem with the world is stupidity. Not saying there should be a
capital punishment for stupidity, but why don't we just take the
safety labels off of everything and let the problem solve itself?
Frank Zappa
chuck
I think you may have missed my point. I don't have fcntl.py on my
Solaris box so how do I know what signals that I can used to monitor a
directory for modification. In other words will the following work?
fcntl.fcntl(self.fd, fcntl.F_NOTIFY,
fcntl.DN_DELETE|fcntl.DN_CREATE|fcntl.DN_MULTISHOT)
Without fcntl source, which the python documentation suggests that I
look at, I don't know if any of these constants apply to the Solaris
platform.
Fredrik Lundh
$ man fcntl
</F>
chuck
Steve Holden
> I need to write a daemon for Solaris that monitors a directory for
> incoming FTP transfers. Under certain conditions, when the transfer is
> complete I need to send an email notification, and do other stuff.
> Win32 provides FindFirstChangeNotification(), but as best I can tell
> this isn't supported on Solaris.
>
>
> Suggestions?
>
Write an FTP server in Python, then it will know exactly when each file
transfer is complete, and it can do the mailing itself!
regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/
Jean-Paul Calderone
>chuck wrote:
>> I need to write a daemon for Solaris that monitors a directory for
>> incoming FTP transfers. Under certain conditions, when the transfer is
>> complete I need to send an email notification, and do other stuff.
>> Win32 provides FindFirstChangeNotification(), but as best I can tell
>> this isn't supported on Solaris.
>>
>[...]
>>
>> Suggestions?
>>
>Write an FTP server in Python, then it will know exactly when each file
>transfer is complete, and it can do the mailing itself!
Or use an existing Python FTP server! ;)
Untested:
from zope.interface import implements
from twisted.protocols import ftp
class UploadNotifyingAvatar(ftp.FTPShell):
def uploadCompleted(self, path):
"""
Override me!
"""
# But, for example:
from twisted.mail import smtp
smtp.sendmail(
'127.0.0.1',
'ftp_server@localhost',
['ftp_admin@localhost'],
'Hey! Someone uploaded a file: %r' % (path,))
def openForWriting(self, path):
writer = ftp.FTPShell.openForWriting(path)
return CloseNotifyingWriter(
writer,
lambda: self.uploadCompleted(path))
class CloseNotifyingWriter(object):
implements(ftp.IWriteFile)
def __init__(self, wrappedWriter, onUploadCompletion):
self.wrappedWriter = wrappedWriter
self.onUploadCompletion = onUploadCompletion
def receive(self):
receiver = self.wrappedWriter.receive()
receiver.addCallback(
CloseNotifyingReceiver,
self.onUploadCompletion)
return receiver
class CloseNotifyingReceiver(object):
def __init__(self, wrappedReceiver, onUploadCompletion):
self.wrappedReceiver = wrappedReceiver
self.onUploadCompletion = onUploadCompletion
def registerProducer(self, producer, streaming):
return self.wrappedReceiver.registerProducer(producer, streaming)
def unregisterProducer(self):
# Upload's done!
result = self.wrappedReceiver.unregisterProducer()
self.onUploadCompletion()
return result
def write(self, bytes):
return self.wrappedReceiver.write(bytes)
Hook it up to a Realm and a Portal and override uploadCompleted to taste and you've got an FTP server that should do you (granted, it's a little long - some of this code could be be factored to take advantage of parameterizable factories to remove a bunch of the boilerplate).
Jean-Paul
chuck
Twisted, thinking about picking it up.
I assume that this little snippet will handle multiple/concurrent
incoming transfers via threading/sub-process, is scalable, secure, etc?
I could even run it on a non-standard port making it a bit more
(ob)secure.
Zem
Linux...and I think I saw Solaris support somewhere. Under Linux, the
FAM daemon doesn't use inefficient polling of the filesystem instead it
monitors file changes through the kernel. Under Solaris, it'd probably
fall back to polling.
If you can use it, then you can use the nice python-fam module. Very
simple to work with and generally works very well. I wrote a python
app that monitored a directory for file creation and deletion and sent
emails based on certain events.
http://python-fam.sourceforge.net/
Good luck.
Jean-Paul Calderone
>Hmmm, that is an interesting idea. I've noticed the new book on
>Twisted, thinking about picking it up.
>
>I assume that this little snippet will handle multiple/concurrent
>incoming transfers via threading/sub-process, is scalable, secure, etc?
Correct, except for the threading/sub-process part. Events from sockets are handled in a single thread in a single process.
>
>I could even run it on a non-standard port making it a bit more
>(ob)secure.
Indeed, although I mainly suggested it because I thought you were tied to FTP. If you actually have security concerns, you might want to use SFTP instead (which Twisted also supports, but to which the code I sent is only partly applicable).
Jean-Paul
chuck
different port. Limit comes more from the client side capabilities.
Did some reading about twisted and I now understand that things in
general are single threaded.
I started working my way through the twisted finger tutorial. While it
appears that it may be useful for someone writing a new protocol it
doesn't seem to useful for someone just trying to hook into an existing
one.
While I do appreciate the suggestions but I have to say that if the
twisted folks spent half the time writing documentation as they do code
- twisted would probably get used a lot more Python folks. Didn't get
much encouragement/assistance from the twisted irc channel either.
Perhaps the fella I chatted with hadn't had his coffee yet ;)
Fredrik Lundh
> While I do appreciate the suggestions but I have to say that if the
> twisted folks spent half the time writing documentation as they do code
> - twisted would probably get used a lot more Python folks. Didn't get
> much encouragement/assistance from the twisted irc channel either.
> Perhaps the fella I chatted with hadn't had his coffee yet ;)
or maybe it's a medication issue. your experience isn't exactly unique:
http://twistedmatrix.com/pipermail/twisted-python/2005-May/010380.html
"In this e-mail thread, someone wrote that people who say Twisted
has no docs ought to be stabbed in the face. Obviously, this was
intended as humor, but I believe that this social custom, along with
as other related social customs in Twistedland, deter people from
some other cultures from participating."
</F>
Cameron Laird
chuck <cmed...@gmail.com> wrote:
.
.
.
>While I do appreciate the suggestions but I have to say that if the
>twisted folks spent half the time writing documentation as they do code
>- twisted would probably get used a lot more Python folks. Didn't get
.
.
Please be aware there's a dead-trees book (!) devoted entirely to
Twisted <URL: http://www.oreilly.com/catalog/twistedadn/index.html >.
chuck
Amazon and at the bookstore I've noted that it does not touch on FTP
server stuff or how to grok the framework in general.