Bundle rebuild file locking issue - Web garden issues?

[Daniel Lo Nigro]

Hi everyone,

We are having some issues around Cassette's locking of cache files. Our configuration is a bit unusual; we're using a deployment pattern similar to the blue/green deployment pattern for easy testing and rollback in case of any problems. We have two sites configured in IIS, one called "Blue" and the other called "Green", only one of which is started at any given time. Each of these has a "test" virtual directory underneath it that points to the opposite environment. That is:

site_blue = Points to 'blue' code

  /test = Points to 'green' code

site_green = Points to 'green' code

  /test = Points to 'blue' code

The idea is that we deploy to the opposite environment (if blue is currently active, we deploy to green), do testing on /test, and if everything is successful the two sites are swapped (the one that's currently stopped is started, and vice versa). These are set up as separate sites to support core changes like .NET version upgrades, although some smaller sites have a single site in IIS and the physical paths are swapped instead.

The initial issue we were having was this one: System.IO.IOException: The process cannot access the file 'manifest.xml' because it is being used by another process. My theory was that because Cassette uses per-assembly isolated storage and both site_blue and site_green/test point to the same codebase, both worker processes recycling at the same time would cause both sites to try refreshing the cache at the same time. I tried working around this by switching to per AppDomain isolated storage, which seemed to work fine in development and QA testing but failed in production because the file paths on Windows Server 2003 were too long (C:\Documents and Settings\All Users\Application Data\IsolatedStorage\dk3geooi.iwo\21tsoexn.ldv\Url.qfz4dmpnwa4ezvdpdsc2yyk5r0iaf4ep\Url.vublg4ertjwnv0oy3gecgjtjkjqlufcr\Files\....).

I have since switched to using a cache directory containing a hash of the site's IIS application ID, ensuring that it's unique per site (the configuration code is here: https://gist.github.com/Daniel15/6066589). However, I am still seeing similar crashes around file locking. We have sites configured as web gardens which I don't know much about, but from what I've heard every app pool has multiple worker processes. A coworker's theory is that all the worker processes are trying to refresh the cache at once which is causing this issue.

Our production environment is mostly Windows Server 2003 (IIS 6) with some Windows Server 2008 (IIS 7.0) and 2008 R2 (IIS 7.5).

Does anyone have any ideas or possible workarounds for this issue? I haven't been able to replicate it in development but I think that'd be because of the amount of load we get to the production environment is hard to replicate in dev.

Here's the stack trace:

System.Exception: Bundle collection rebuild failed. See inner exception for details. ---> System.IO.IOException: The process cannot access the file 'c:\temp\Cassette\05D5BB2DCCB968287963A7D4DCE9B79EC3C0F469\manifest.xml' because it is being used by another process.

at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)

at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath)

at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)

at Cassette.IO.FileSystemFile.Open(FileMode mode, FileAccess access, FileShare fileShare)

at Cassette.Caching.BundleCollectionCacheWriter.WriteBundleContentFile(Bundle bundle)

at Cassette.Caching.BundleCollectionCacheWriter.WriteBundleContentFiles(IEnumerable`1 bundles)

at Cassette.Caching.BundleCollectionCache.Write(Manifest manifest)

at Cassette.CacheAwareBundleCollectionInitializer.WriteToCache()

at Cassette.CacheAwareBundleCollectionInitializer.Initialize(BundleCollection bundleCollection)

at Cassette.RuntimeBundleCollectionInitializer.Initialize(BundleCollection bundles)

at Cassette.ExceptionCatchingBundleCollectionInitializer.Initialize(BundleCollection bundleCollection)

--- End of inner exception stack trace ---

at Cassette.BundleCollection.GetReadLock()

at Cassette.ReferenceBuilder.Reference(String path, String location)

at Cassette.Views.BundlesHelper.Reference(String assetPathOrBundlePathOrUrl, String pageLocation)

at PageUp.People.Controls.PUPage.SetRenderModeInMaster()

at PageUp.People.Controls.PUPage.OnPreRender(EventArgs e)

at System.Web.UI.Control.PreRenderRecursiveInternal()

at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

Thanks! (and sorry for the long post)

[João Bragança]

Build the manifest with the msbuild task, rather than 'lazy loading' it.

Check out the PageUp People Blog  |  Privacy Policy  | Careers on facebook

PS. PageUp People was recently positioned as a visionary company in Gartner's 2013 Talent Management Magic Quadrant. Access the Full Research Report.

--
You received this message because you are subscribed to the Google Groups "Cassette" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cassette+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Sent from my regular computer

http://twitter.com/thefringeninja

http://www.thefringeninja.com/

[Daniel Lo Nigro]

Thanks for the suggestion, I'll try that out and see if it solves the issue.

Daniel Lo Nigro - Web Developer Level 10, 91 William Street, Melbourne 3000 Tel: +61 3 8677 3777 x819

--
You received this message because you are subscribed to a topic in the Google Groups "Cassette" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cassette/IFLHpNid_lI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cassette+u...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jon Wynveen]

Daniel, did the msbuild task solve your problem? We are seeing the same error message in some of our sites and are hoping that the msbuild will solve the problem, so it would be great to know that over the long term it did or did not solve it.

[Daniel Lo Nigro]

Yeah I eventually got it to work using the MSBuild task. I don't work at the same employer any more so unfortunately I don't have access to the notes I wrote any more.