FileSystemWatcher, Underlying Windows APIS do not work?
InvestecLondonDevelopers
I have created a service which monitors a particular directory, and places
any new files on a queue. The system which places the files in the directory
is legacy, and cannot be changed. It places the files in the directory
always with the same name. Therefore, I need to process this file ASAP or it
disappears forever!
Now the FileSystemWatcher appeared to be exactly what I wanted. But for some
reason, when I place a file in the folder from another userid, it ignores
the new file. Only when I place the file in the folder on my machine, and
with the logged on user does it work. The location of the production folder
to be watched is on a share in another domain, with a userid and password on
the folder. I can connect to the share in the program, I can monitor the
folder from the service, I can place a file in the share from the machine
monitoring the folder, and it works. Get someone else to place a file in
there, and the service stops working, no files will be processed no matter
who puts it in there.
I went on to groups.google.com, and it seems that there are LOTS of
complaints about files not being picked up by the FSW. Seems that this is
not an object you should rely on.
Byt heres the thing that worries me. I had to delete about 30 files off my
HDD from various places. I performed a search on the c:\ for the file mask,
and the 30 files were shown. Now normally if I delete them one by one, they
disappear out of the search list. But if I select all files in the search
results list, and hit delete, about half of them disappear from the list.
The other files HAVE been deleted, its just that the search window did not
get the notification. So this appears to be a fundamental flaw with the OS,
not the FSW object. This also describes exactly the results of a lot of
complaints I found in google groups searches. performing a lot of operations
on a watched folder(s) will cause some events to not fire.
Also, I have no many many occasions deleted a folder from Windows Explorer,
but the folder does not disappear from Windows Explorer. It is like it dod
not get the event either. With so many occurrences of this happening in the
Windows Operating system (Explorer and Search), I do not hold out much hope
of the applications I build onto this technology working.
I would like to know if this is a known Bug within Microsoft, and what is
likely to be done about it. Should people be relying on this
FileSystemWatcher or not? At this stage I would be saying not. Please prove
me wrong...
BTW, I am using the .NET framework 1.0.3705, Visual Studio .Net 7.0.9466,
and Windows 2000 Build 2195 SP3 on all PCs, related ot this post.
Thank you
David
Keywords: FileSystemWatcher, Bug, Message Queue, Visual Studio .Net,
Framework.
Bharat Patel [MSFT]
There are some issues with FSW as mentioned on the following URL:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vbcon/html/vbtb
sTroubleshootingUNCPathNamesNotAcceptedOnNT4Machines.asp
However, since you are using it in a service, it may add to other issues. Are
you running this service under a specific Domain User account or is it running
under Local System account? If it is a system account, you need to change it to
a Domain account which has sufficient permissions to monitor that folder.
Also, it will be a good idea to test it out in a windows application and see if
it works OK. Another reason is that the legacy application might be keeping a
lock on that file which will prevent FSW from caching the events. Have you
enabled raising events from your code? I guess, you did as it worked fine when
you added some files.
Hope this helps!
Bharat Patel
Microsoft, Visual Basic .NET
This posting is provided "AS IS" with no warranties, and confers no rights.
Please reply to newsgroups only. Thanks.
InvestecLondonDevelopers
OK, I do not use NT4, Win 95, Win 98, I do not get multiple file
notifications, and I do not get a file and a directory notification at once.
So without repeating my original question, is there an anwer to my question?
Why is it that I keep getting answers to my questions that have nothing to
do with my questions? Are these questions put through some pattern matching
to generate answers?
The real question here is -- is there a problem with WINDOWS as far as file
notifications. Considering that Explorer itself misses half the
notifications, how am I sopposed to trust the FileSystemWatcer, considering
it seems to be EASILY fooled into not receiving an event. I am working in a
banking field, and when I write a program, it is expected to work, otherwise
there is a lot at stake. I do not get a good feeling about my applications
when building on top of flawed systems...
Thank you
David
"Bharat Patel [MSFT]" <bha...@online.microsoft.com> wrote in message
news:w1cSa6q3CHA.1664@cpmsftngxa06...
Bharat Patel [MSFT]
I am sorry I don't know the details at operating system level. I will find
someone who knows the win32 API's who can answer your question.
Bharat Patel [MSFT]
I have sent this information to the appropriate team and they are researching
this issue.
Frank Kim
I have duplicated this issue.
I am currently researching this to determine
if this is a bug or by design.
As soon as I get more info, I will post it.
Please let me know if you have any other
comments.
thanks
Frank Kim
Microsoft Support
InvestecLondonDevelopers
I look forward to the final answer.
David
"Frank Kim" <franki...@microsoft.com> wrote in message
news:7VHHLB14CHA.1456@cpmsftngxa06...
Frank Kim
After further testing I was unable to reproduce the problem again.
I am able to successfully copy files between different security
contexts and filesystemwatcher detects all the changes in the
directory.
I test with 3 accounts.
* Localsystem
* Administrator
* User.
I have user start the test app watching directory Z. I have files
copied/deleted
by the other users to directory Z and the test application appears to be
getting notified by all the changes done by the other users.
Directory A is a local directory.
All users are running on Machine A while the share is located on Machine B.
I changed Directory Z to a file share. I copied files to the file share
from localsystem
and the administrator account and the application was still getting
notified.
I'll need more information on the exact scenario for reproducing this.
InvestecLondonDevelopers
I will get back to you with more details. I wonder why you could reproduce
it one time, and not the next? It still comes down to the reliability issue.
Also, while there, try this. Drag about 50 files in to the watched folder at
the same time. Varying size files. some large.
How many files does the application get notifications for?
And also, on the other question I asked, why is it that the search window
and explorer do not get notifications? This is when I search for a file
mask, find about 50 or more files, Select all, delete them. Now half of
those files dissappear from the search window. They were deleted, but the
search window did not get the notification. Also, almost every time I delete
a directory and all files adn directories under it, the parent directory
stays in explorer. I get an error saying the directory does not exist when I
try to select it. Sometimes there even appears to be subdirectories under
it, if there were enough files to delete.
What I am trying to say is that I do not trust the underlying OS that FSW is
built on. It does not give me confidence. Can you explain why Search window,
and Explorer itself miss so many file events?
Thank you for your time
David
"Frank Kim" <franki...@microsoft.com> wrote in message
news:uwz43HP5CHA.1460@cpmsftngxa06...
InvestecLondonDevelopers
I have had a look at it again.
I think there are a couple of issues:
1) Too many files added at once fools FSW -- I created a share on a PC on
the next desk to me. Not a server, not a hiddden share ($), no security. I
monitored that share from my PC. I placed 200 2kb files in that share from
my PC. 66 files were processed, and the rest were ignored. I then got the
owner of that PC with the share to place the 200 files in there. 16 files
were processed. I then placed the 200 files in there again from my PC. All
files were processed slowly. I then got the owner of the PC with the share
to place the 200 files into that shared folder. It slowly processed 33
files, and then stopped. The thing here is that I never get exceptions, just
miss a lot of event fires.
2) FTP Share fools and Breaks FSW -- In production I am dealing with an FTP
folder on a different machine than the one the monitoring service is
running. The FTP folder has a single userid and password set up on it, no NT
Authentication. I must pass along the userid and password when mapping the
drive. This still does not work at all. What I do is I use the
WNetAddConnection2A Windows API call to map to that FTP folder, passing the
userid and password. I specify NO drive letter, as I have found that it is
impossible to remove that drive letter later when the drive letter was
created from a Windows Service. So now I have been authenticated, and can
refer directly to the \\server\share. I set up the FSW against that share
name. I run from the command line:
copy c:\file.log \\server\share\
The file is copied. The event is fired. Yay. I then get someone else on a
third PC to place a file in that directory. The watch event NEVER fires.
Now I run the copy command again, and the event DOES NOT fire! It is now
officially broken, and only a reboot of my PC with the FSW service will fix
it.
3) Even Explorer does not get it right -- The third thing that still worries
me is that The search window, and Explorer itself seem to miss file deletion
events all the time. Any clues as to why this is?
Thank you for your time, and I look foward to further answers.
David
"Frank Kim" <franki...@microsoft.com> wrote in message
news:uwz43HP5CHA.1460@cpmsftngxa06...
AWP
thought I'd bring it up if it is a bug. I have a FWS component setup on a
machine and it does not generate events for files that have a readonly bit
set. It sets off events in all other cases. I have copied the same files
into the same directory with only the read attribute being different and FWS
behaves differently.
/andrew
"InvestecLondonDevelopers" <londonde...@investecmail.com> wrote in
message news:OYjs$c85CH...@TK2MSFTNGP09.phx.gbl...
Frank Kim
>>1) Too many files added at once fools FSW -- I created a share on a PC on
>>the next desk to me. Not a server, not a hiddden share ($), no security. I
>>monitored that share from my PC. I placed 200 2kb files in that share from
>>my PC. 66 files were processed, and the rest were ignored. I then got the
>>owner of that PC with the share to place the 200 files in there. 16 files
>>were processed. I then placed the 200 files in there again from my PC. All
>>files were processed slowly. I then got the owner of the PC with the share
>>to place the 200 files into that shared folder. It slowly processed 33
>>files, and then stopped. The thing here is that I never get exceptions,
just
>>miss a lot of event fires.
What are you setting the InternalBufferSize property to? It is quite
possible
that if a lot of changes occur in a short amount of time that the buffer can
overflow and events can be missed.
You may want to increase the buffer size from the default size of 8K but
remember
this memory is coming from non-paged pool. It also helps to use the
NotifyFilter
to only receive the events that you really require.
>>2) FTP Share fools and Breaks FSW -- In production I am dealing with an
FTP
>>folder on a different machine than the one the monitoring service is
>>running. The FTP folder has a single userid and password set up on it, no
NT
>>Authentication. I must pass along the userid and password when mapping the
>>drive. This still does not work at all. What I do is I use the
>>WNetAddConnection2A Windows API call to map to that FTP folder, passing
the
>>userid and password. I specify NO drive letter, as I have found that it is
>>impossible to remove that drive letter later when the drive letter was
This is a known issue. Please read the following:
180362 INFO: Services and Redirected Drives
http://support.microsoft.com/?id=180362
>>created from a Windows Service. So now I have been authenticated, and can
>>refer directly to the \\server\share. I set up the FSW against that share
>>name. I run from the command line:
>>copy c:\file.log \\server\share\
>>The file is copied. The event is fired. Yay. I then get someone else on a
>>third PC to place a file in that directory. The watch event NEVER fires.
>>Now I run the copy command again, and the event DOES NOT fire! It is now
>>officially broken, and only a reboot of my PC with the FSW service will
fix
>>it.
I am unable to reproduce this problem as well. I created a interactive
localsystem
command prompt (similiar to a service) on Machine A. I established
credentials
to access a share on Machine B (since localsystem only has access to local
resources). I then fire my application in the localsystem prompt to watch
for files on
the share on Machine B.
On Machine A running in a difference security context (interactive user), I
copy a file
to the share. Localsystem command prompt gets notified. I then went to a
Machine
C and copied a file to the share, again localsystem prompt on Machine A
received
notification.
Machine A was a Windows 2000 Server, Machine B is Windows XP and Machine C
is Windows XP.
>>3) Even Explorer does not get it right -- The third thing that still
worries
>>me is that The search window, and Explorer itself seem to miss file
deletion
>>events all the time. Any clues as to why this is?
You'll need to provide a concrete repro for this. I am not aware of this.
Mike Barthold
you wrote in your answer:
> You may want to increase the buffer size from the default size of 8K but
> remember
> this memory is coming from non-paged pool.
WHAT DOES THIS MEAN??? What is the "non-paged" pool and why it is so
important????
I'm interested in this topic too, because I wrote a "Software Mirror"
service for my pc to reflect any changes on disk (or folder) A to disk (or
folder) B.
I never encountered the problems david had, not even if I copy TONS of files
(once I restored the entire harddrive, consisting about 200.000 files (!!!)
with a total amount of about 67 GIGABYTES and my software mirror did not
have a single miss --- everything was replicated PERFECT. ---- so i think
the problems of david must have another source than the FSW itself.
But I have ONE problem, I've never been able to solve:
My Software mirror runs on my fileserver at home and I always have some
folders (like the develop-folder) as "offline folder" available on my
notebook, so I can always have a look at my sources even in the office.
When I take my notebook back home at end of the week and synchronize it with
the fileserver, my software mirror does NOT replicate the changes to the
mirror-drive! I suppose this is because the folder (or the entire drive???)
that is being synchronized is (some sort of) "taken offline" in the
system...
I also can't access the develop share on the server from any other pc in the
network while the notebook is synchronizing...
any ideas on this topic???
TIA
mike
"Frank Kim" <franki...@microsoft.com> schrieb im Newsbeitrag
news:Il4zY7A6...@cpmsftngxa08.phx.gbl...
InvestecLondonDevelopers
I will tackle the Explorer not getting file deletion events in this post.
Its easy to reproduce.
Windows 2000. Create a folder on your local HDD. Create 200 files in that
folder of 2kb size. Name them all with the same extension. (*.log). Right
click on folder. Click search. Search for *.log. Select all. Delete. in my
case files 2 to 10 of 200 were removed from the search window. All others
remain. The files have been deleted, but the Search window does not know.
I tried again. This time hit Shift + del to not move to recycle bin. Same
result. Tried one more time with just del, and 2 to 11 were deleted.
Also, create the following Directory structure
New Folder
New Folder
New Folder (1)
New Folder (2)
New Folder
New Folder (3)
New Folder
New Folder
New Folder
New Folder (4)
Place the 200 files in each of these folders but not the root NewFolder. Now
shift + Del on the root permenantly delete the files and folders. All files
are deleted, but the full directory structure remains in Explorer. Try
clicking on a folder, and you get a message that it does not exist.You must
exit Windows Explorer and start it again to get the refresh. Also, just
hitting delete only to move the files to the Recycle bin does update windoes
explorer.
This is 100% reproducable for me. It happens always. Also, please try this
on a PC with the windows component update, and Explorer 6 and Visual
Studio.Net, as I am not sure if this will happen on a clean base Win2000
install. I would be interested to know if it does though if you can check
it.
As I said, with this sort of behavior with the OS tools, I don't trust file
update notifications. Maybe it is that buffer size you mention, and this is
an issue with Win Explorer and Search window.
David
"Frank Kim" <franki...@microsoft.com> wrote in message
news:Il4zY7A6...@cpmsftngxa08.phx.gbl...
InvestecLondonDevelopers
In this post I will deal with the linking to a share from a windows service.
I need to use a service for this task, as it must start whenever the server
is running. Unfortunately the service MUST go on a different server than the
FTP share. This means I must access that FTP share via a UNC path. This will
not work though, as the FTP share has a userid and password on it. This is
not an NT authentication thing, just a plain simple userid and password I
must pass when accessing the NC path. he only way I can see to do this is to
mapt to the FTP share using WNetAddConnection2A, and pass the password and
userid. If I do not specify a drive letter, no drive letter is mapepd, but
the connection is allowed, as are all further UNC references to that FTP
share, because I have already supplied the userid and password.
Can you suggest a way around this if I must use a windows service, and I
must access a password protected FTP share on another server?
I do not think this is the problem thought, as a I am testing all of this
with a normal windows forms app, while sorting out these issues. I still do
not supply the drive letter, but I use WNetAddConnection2A to connect to the
FTP share.
Thank you
David
"Frank Kim" <franki...@microsoft.com> wrote in message
news:Il4zY7A6...@cpmsftngxa08.phx.gbl...
InvestecLondonDevelopers
In this post I will deal with the FSW not getting updates from the FTP share
when a third PC is used to place a file in the FTP share.
In my case all PCs are Win2k. But I am not sure about your setup. Is the
share you are monitoring an FTP share, or a normal windows shared folder. It
must be an FTP share, as a normal windows share seems to work fine. (Except
for missing some files if too many are placed in at once.)
Also, it must have a userid and password on it. In our case, the FTP share
is outside of our domain, so I can not to my knowledge set up "credentials".
If so, how would I do this. I just use WNetAddConnection2A, and supplied the
FTP share name, userid, and password, but no drive letter. This then has set
up the FSW server to be allowed to access the FTP share with the UNC name
without a Userid and password from then on.
So if it was not an FTP share, try it with an FTP share. If it did not have
a userid and password set, try it with that. If I can set up these
"credentials" to access an ftp share on a server outside our domain, and
through a firewall, please let me know how to do that.
By FTP share, I mean setting up a folder in the FTP service in IIS using
MMC, and pointing it to a folder. Unfortunately I cannot do this myself to
test if, as I have only access to workstations, and they will not allow me
to set up an FTP share with a password. Without a password is fine. I am not
sure why this is, and have not worked with setting up FTP shares before.
Thank you for your help so far.
David.
"Frank Kim" <franki...@microsoft.com> wrote in message
news:Il4zY7A6...@cpmsftngxa08.phx.gbl...
> Hi David,
>
> >>2) FTP Share fools and Breaks FSW -- In production I am dealing with an
> FTP
> >>folder on a different machine than the one the monitoring service is
> >>running. The FTP folder has a single userid and password set up on it,
no
> NT
> >>Authentication. I must pass along the userid and password when mapping
the
> >>drive. This still does not work at all. What I do is I use the
> >>WNetAddConnection2A Windows API call to map to that FTP folder, passing
> the
> >>userid and password. I specify NO drive letter, as I have found that it
is
> >>impossible to remove that drive letter later when the drive letter was
>
Frank Kim
I forgot to mention that explorer is not using FileSystemWatcher class
but both Explorer and the FileSystemWatcher class do use the underlying
Win32 API ReadDirectoryChangesW().
It is quite possible there is an issue with the way Explorer is calling
ReadDirectoryChangesW() which is different from how you are using the
FileSystemWatcher class.
I would suggest posting this issue in a end user Windows newsgroup or
using one of the following other support options listed below since this
does not directly pertain to your issue.
thanks for the feedback.
Frank Kim
Microsoft Developer Support
This posting is provided "AS IS" with no warranties, and confers no rights.
Please reply to newsgroups only. Thanks.
============================================================================
=====
Microsoft support home page: http://support.microsoft.com.
To view support options:
http://support.microsoft.com/default.aspx?scid=sz;en-us;top.
To submit an online request:
http://support.microsoft.com/default.aspx?scid=fh;en-us;incidentsubmit.
Frank Kim
My tests were done using a windows shared folder.
Can you please tell me what path you were using to establish
a connection with the FTP share? What was the remote name
that you used in WNetAddConnection2?
It sounds like you are using the webdav protocol to access
the file share. This is a stateless protocol where it doesn't
have concepts of notifications. The FileSystemWatcher class
will not work in this scenario.
We are working on changing this for Longhorn.
The only work around is to watch the directory that the ftp share
represents locally on the machine and send the change notifications
to the client.
thanks
Frank Kim
Microsoft Developer Support
InvestecLondonDevelopers
organisation. It is on a W2k server, and is a IIS hosted FTP folder. How can
I find out if it is a webdav protocol? I have complete control over the FTP
server, and the monitoring server (Within guidelines) so, I can change
things if need be.
I was using the following code to map the drive
NETRESOURCE [] nr = new NETRESOURCE [1];
nr[0].lpRemoteName = uncPath;
nr[0].lpLocalName = driveLetter;
nr[0].dwType = 1; //disk
nr[0].dwDisplayType = 0;
nr[0].dwScope = 0;
nr[0].dwUsage = 0;
nr[0].lpComment = "";
nr[0].lpProvider = "";
// Try to Connect
Err = WNetAddConnection2A (nr, password, user, Flags);
I pass in the following values (Not real values)
uncPath = "\\server\ftpshare"
driveletter = ""
user = "username"
password = "thepassword"
Then the FTPserver knows that this server (monitoring server) has access
rights to that UNC path. All further references to that UNC path do not
require a userid and password. I just use \\server\ftpshare for the monitor
path.
Now the strange thing here is that you say FSW does not work with an FTP
share. This is not totally true, as it does work if a file is placed in the
FTP share by the userid that the FSW is running under, and on the PC that
the FSW is running on. If anyone else places a file in there though, it
completely breaks the FSW. So it partially works, but is not very useful.
If there is some way around this, some settings I can use, please let me
know.
Thanks for your help
David
"Frank Kim" <franki...@microsoft.com> wrote in message
news:6UjLVpN6...@cpmsftngxa08.phx.gbl...
InvestecLondonDevelopers
Thanks for this. I figured they would use the same underlying API.
I will post as you suggested. I would say that this is an explorer bug or
issue. I will see if there is a newsgroup for Explorer itself, or bugs in
the OS.
Is there a standard way to report bugs in MS products?
BTW, could you reproduce the behavior I described?
David
"Frank Kim" <franki...@microsoft.com> wrote in message
news:cuxNgSN6CHA.160@cpmsftngxa06...
Frank Kim
>>Then the FTPserver knows that this server (monitoring server) has access
>>rights to that UNC path. All further references to that UNC path do not
>>require a userid and password. I just use \\server\ftpshare for the
monitor
>>path.
>>Now the strange thing here is that you say FSW does not work with an FTP
>>share. This is not totally true, as it does work if a file is placed in
the
>>FTP share by the userid that the FSW is running under, and on the PC that
>>the FSW is running on. If anyone else places a file in there though, it
>>completely breaks the FSW. So it partially works, but is not very useful.
Can you test with a SMB share and see if you get the same results with
FSW and using different IDs? I have no problems with SMBs and different
user IDs.
The way the redirector works, UNC paths are used by both SMB and
WebDav. The redirector determines whether the UNC path is SMB or
WebDav.
It is quite possible that the redirector gets confused such that FSW does
work and then stops working.
InvestecLondonDevelopers
OK, I have given false info. I do not control that share, and am doing this
project for someone else. I was told that that share was an FTP share. But
now I have talked to the administrator for that share, and got the correct
info. I hope, as I still cannot see the server itself.
Apparently the share is just a normal windows share. But I think I might
have a lead on the problem. As I said earlier, there is a firewall between
the FSW and the share it is monitoring.
- Server A = Server with share on it that I want to monitor
- Server B = Server with FSW on it that I am using to monitor the share
- As I said in an earlier post, there is a firewall between these two shares
- I have just learned that this firewall blocks all trafic from the Server A
into the domain where Server B is.
- I assume though that data itself can travel from Server A into the domain
with Server B, as I can read files off the share from Server B.
Now I wonder if this means that when I copy a file onto the share from
Server B, I have initiated the transaction, and therefore there is a free
flow of info from Server A to Server B, and backwards. Therefore, Server B
receives all the normal calls from Server A saying the file copy completed
successfully, etc, and it also gets the signal to the FSW saying that a new
file was placed in the monitoring directory as part of that.
Now if someone else places a file in the share on Server A, I would assume
that it is Server A that tries to communicate this to the FSW on Server B.
This is most probably being stopped by the firewall.
I will be trying this out soon, and will confirm results.
David
"Frank Kim" <franki...@microsoft.com> wrote in message
news:09LygLd6...@cpmsftngxa08.phx.gbl...
InvestecLondonDevelopers
I have given up on the possibility of this working. We have decided to go
with a service that checks for new files every minute. Increased traffic,
but at least it works.
Thanks for your help
David
"InvestecLondonDevelopers" <londonde...@investecmail.com> wrote in
message news:uHmBU#h6CHA...@TK2MSFTNGP11.phx.gbl...
Frank Kim
For future reference.
Here are some kb articles on firewalls and SMB and domain/trusts.
The key item is know which ports to open up.
179442 How to Configure a Firewall for Domains and Trusts
http://support.microsoft.com/?id=179442
298804 The Internet Connection Firewall Can Prevent Browsing and File
Sharing
http://support.microsoft.com/?id=298804