Hi everyone,
We are having some issues around Cassette's locking of cache files. Our configuration is a bit unusual; we're using a deployment pattern similar to the
blue/green deployment pattern for easy testing and rollback in case of any problems. We have two sites configured in IIS, one called "Blue" and the other called "Green", only one of which is started at any given time. Each of these has a "test" virtual directory underneath it that points to the opposite environment. That is:
site_blue = Points to 'blue' code
/test = Points to 'green' code
site_green = Points to 'green' code
/test = Points to 'blue' code
The idea is that we deploy to the opposite environment (if blue is currently active, we deploy to green), do testing on /test, and if everything is successful the two sites are swapped (the one that's currently stopped is started, and vice versa). These are set up as separate sites to support core changes like .NET version upgrades, although some smaller sites have a single site in IIS and the physical paths are swapped instead.
The initial issue we were having was this one:
System.IO.IOException: The process cannot access the file 'manifest.xml' because it is being used by another process. My theory was that because Cassette uses
per-assembly isolated storage and both
site_blue and
site_green/test point to the same codebase, both worker processes recycling at the same time would cause both sites to try refreshing the cache at the same time. I tried working around this by switching to
per AppDomain isolated storage, which seemed to work fine in development and QA testing but failed in production because the file paths on Windows Server 2003 were too long (
C:\Documents and Settings\All Users\Application Data\IsolatedStorage\dk3geooi.iwo\21tsoexn.ldv\Url.qfz4dmpnwa4ezvdpdsc2yyk5r0iaf4ep\Url.vublg4ertjwnv0oy3gecgjtjkjqlufcr\Files\....).
I have since switched to using a cache directory containing a hash of the site's
IIS application ID, ensuring that it's unique per site (the configuration code is here:
https://gist.github.com/Daniel15/6066589). However, I am still seeing similar crashes around file locking. We have sites configured as web gardens which I don't know much about, but from what I've heard every app pool has multiple worker processes. A coworker's theory is that all the worker processes are trying to refresh the cache at once which is causing this issue.
Our production environment is mostly Windows Server 2003 (IIS 6) with some Windows Server 2008 (IIS 7.0) and 2008 R2 (IIS 7.5).
Does anyone have any ideas or possible workarounds for this issue? I haven't been able to replicate it in development but I think that'd be because of the amount of load we get to the production environment is hard to replicate in dev.
Here's the stack trace:
System.Exception: Bundle collection rebuild failed. See inner exception for details. ---> System.IO.IOException: The process cannot access the file 'c:\temp\Cassette\05D5BB2DCCB968287963A7D4DCE9B79EC3C0F469\manifest.xml' because it is being used by another process.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
at Cassette.IO.FileSystemFile.Open(FileMode mode, FileAccess access, FileShare fileShare)
at Cassette.Caching.BundleCollectionCacheWriter.WriteBundleContentFile(Bundle bundle)
at Cassette.Caching.BundleCollectionCacheWriter.WriteBundleContentFiles(IEnumerable`1 bundles)
at Cassette.Caching.BundleCollectionCache.Write(Manifest manifest)
at Cassette.CacheAwareBundleCollectionInitializer.WriteToCache()
at Cassette.CacheAwareBundleCollectionInitializer.Initialize(BundleCollection bundleCollection)
at Cassette.RuntimeBundleCollectionInitializer.Initialize(BundleCollection bundles)
at Cassette.ExceptionCatchingBundleCollectionInitializer.Initialize(BundleCollection bundleCollection)
--- End of inner exception stack trace ---
at Cassette.BundleCollection.GetReadLock()
at Cassette.ReferenceBuilder.Reference(String path, String location)
at Cassette.Views.BundlesHelper.Reference(String assetPathOrBundlePathOrUrl, String pageLocation)
at PageUp.People.Controls.PUPage.SetRenderModeInMaster()
at PageUp.People.Controls.PUPage.OnPreRender(EventArgs e)
at System.Web.UI.Control.PreRenderRecursiveInternal()
at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
Thanks! (and sorry for the long post)