WMServer.exe 'locked' local video files when issuing file delete

[Kokkers]

Dear microsoft.public.windowsmedia.server readers,

I am experiencing an inconvenient file locking problem using
WMS9.01.01.3862 on Windows Server 2003 x64 Enterprise (various
builds).

We are serving a substantial amount of on-demand video to our viewers
through the Windows Media platform.
Using local RAM-disks we are able to push 3.5Gbit+ of traffic
delivering 2000+ concurrent on-demand streams out of a single beefy
server dedicated to serving on-demand Windows media video files.
We try to keep our most popular video files on these fast servers to
offload our back-end Windows Media Servers.

We have implemented our own (higher level) caching mechanism and chose
not to use a local caching plug-in for WMS. At the time the cache plug-
in was not publicly available and Windows Server 2008 was not yet
ready to be deployed in a production environment.

In our environment WMV files are automatically being pushed to these
cache servers when in high demand.
These cache servers have a limited 64GB local RAM-disk capacity for
storing WMV files. This is all working quite well and there's
rejoicement all around.

Except for one situation, when a new file needs to get pushed onto a
cache server an older, less popular, file needs to leave. If I am not
mistaking, by design, a Windows Media Server does not lock local WMV
files, even when it is serving the file to a viewer. Usually I am able
to delete WMV files that are being streamed to viewers.

Once in a while the operating system does not allow a windows media
file to be removed though.

This is what happens:

A request to delete a particular file is sent to the system.
Message 'Cannot delete xxxxx.wmv: Access is denied message' is
returned.
The particular file remains available on the file system but it cannot
be accessed anymore by any means.
When viewing the properties of the file in Windows Explorer the
'security tab' is not available.
I have seen this before on Windows and this usually indicates the file
already has been issued a delete command but has not yet been freed
because of another process accessing it.

When using the Sysinternals 'Process explorer' utility I can see the
particular file has an active handle by process 'WMServer.exe'. Locked
files usually are released after some time but I have seen this
lasting for several hours and days as well. Whenever Windows Media
Services is stopped the 'locked' files disappear instantly. I have
seen this behavior when using both regular local storage and software
RAM disk devices.

Stopping and starting Windows Media Services on a 24/7 production
server on regular intervals obviously is not an option.

I have tried running the 'Windows Media Services' service using the
'Network service' account and only giving read access to 'Network
service' on the file system to no avail.
Since I am out of ideas on how to solve this inconvenience I am
turning to the experts here.

Now for my questions:
- Has anybody else seen this behavior?
- Why does this not happen to all files?
- Is this behavior by design?
- Would there be ways to prevent / circumvent this, hard unlinking the
file without crashing WMServer.exe for example?
- Since we pay a fair amount of money for our Enterprise licenses I
was wondering what the best support procedure would be at Microsoft.

Silently I am hoping someone with low-level knowledge of Windows and/
or WMS is able to provide some background information on what I am
seeing. Where are the WMS developers hiding ;-)

Regards,

Kokkers

[Neil Smith [MVP Digital Media]]

That's a great bug report - you should be a professionsl ;-)

On this page, there's a description of how to optimise the server when
content it located on a remote (network connected) network share :
http://support.microsoft.com/default.aspx?scid=kb%3ben-us%3b812633

It has lots of useful inside info, but in particular it notes that
this value when present in the server namespace file :

<node name="Only File Read Share" opcode="create" type="int32"
value="0x1" />

has the exact result that you've described - "opens files on the
remote share in a mode ... other users or applications cannot modify
this file"

It's probably a long shot, but if you've added this non-default
optimisation then try to revisit the settings to see if that's present

Obviously, intermittent bugs are a PITA to track down.

Something else to add : I've seen hard-or-impossible-to-delete files
before in this scenario : One of our servers had several possible
users as owner of a file.

For some reason - we suspected linux to windows fileshare
interoperability problems - inspecting the extended ownership
properties of the file Object, showed a username something like
S-I-S-<machineGUID>/Unknown, instead of the more usual
<machineName>/NetworkService account.

Once the file permissions were reassigned by resetting / taking
ownership on the top level / parent directory and cascading those to
contents, the ownership update seemed to remove this "unknown" user ID
on the files and allow for file handle to be manipulated (delete,
update, rename etc)

HTH
Cheers - Neil

On Wed, 15 Oct 2008 03:36:49 -0700 (PDT), Kokkers <kok...@gmail.com>
wrote:

------------------------------------------------
Digital Media MVP : 2004-2008
http://mvp.support.microsoft.com/mvpfaqs

[Kokkers]

Thanks for a thorough answer Neil.

I had seen the article regarding caching and shared mode when using
remote shares as a content source, problem might be that we source the
content from local disk.

The article has some background information though, I quote:

"When streaming from a file on the local disk, the OS file system
cache manager will buffer blocks of the file in memory. This helps
improve performance by reducing disk I/O operations. However, when
accessing content on a remote share, this OS file system caching
process may not always occur. This behavior occurs because WMS Server
allows files to be opened in a sharing mode. This sharing mode then
allows other applications to read, modify, and delete files while in
use by WMS."

Default when using local disk: Files are being cached in (file) system
cache.
Default when using remote share: Files are not being cached in (file)
system cache because of being opened in 'shared' mode.

The default mode is 'sharing mode', right? Which is the mode we
require because I want other processes to be able to delete the file
whenever they need.

This makes sense, however the article is unclear whether the "Only
File Read Share" parameter in the server namespace XML is applied to
local files on local disk as well. I assume this setting only affects
the way the server opens files on remote shares and not the ones
residing on local storage.

Regards,

Kokkers

On 18 okt, 22:40, "Neil Smith [MVP Digital Media]" <n...@nospam.com>
wrote:

> On Wed, 15 Oct 2008 03:36:49 -0700 (PDT), Kokkers <kokk...@gmail.com>

> Digital Media MVP : 2004-2008http://mvp.support.microsoft.com/mvpfaqs- Tekst uit oorspronkelijk bericht niet weergeven -
>
> - Tekst uit oorspronkelijk bericht weergeven -