Spark Memory Issue with Many Custom Master Layouts

[James G]

We have an application that uses Spark as the view engine, but we haven’t figured out a way to precompile the views due to the way we have things setup. Due to this (at least we think it is due to this), we have what appears to be a memory leak (although, I’m not sure that is an accurate description).

Our app is an MVC3 web app. Now, each of our customers can fully theme the way our app looks for them (for example, when you hit www.example.com/app/customer1 it can and usually will be “themed” completely differently than www.example.com/app/customer2 ). The way we do that is we let our customers upload an html page that looks just like their website – but without the content. We then use that html page as a Spark master layout for that customer and stick a named content section in the page where the content of our app will be rendered.

Here is how we do this:

1. Our themes are stored in a folder separate from our app (not included in the VS project). So, in our App_Start, we call SparkSettings.AddViewFolder and pass in this folder. Note, the contents of this folder would look like this:
   
   
   drive:\\path\themefolder
   \customer1\master.html
   \customer2\master.html
   \customer3\master.html
   …. (we have several hundred customers)
   
   
2. We have an attribute that we place on our controllers which checks to see if a “theme” exists for this customer – if so, it changes the MasterName on the filterContext.Result to the sub-folder and name of the master layout for that customer (for example, customerId/master – this name is then relative to the themefolder path within which all of our themes are stored, which is set in Step 1 above).
   
   
3. Finally, we have a custom descriptor builder that adds this custom master layout name to the list of locations to search.

With those 3 things in place, we are able to replace the default master layout with a custom designed master layout for each customer and this has worked really great and provided tons of flexibility to our customer base. However, it appears that doing this creates a memory problem on our server. Essentially, what we see on the server is that the Memory (Private Working Set) for our app pool just continues to grow until it runs out of memory. So, for the moment we've tuned our app pool to recycle when it hits a certain memory limit and after some trial and error we've gotten it to a reasonably stable setting, but obviously that is not optimal.

From what I’m able to tell based upon some memory profiling, what we’re seeing is similar to what Steve Stevenson reported in this issue: https://groups.google.com/d/topic/spark-dev/HYexZD_pu3s/discussion . In that post, Louis says it wouldn't be a problem for people who pre-compile which I can only infer means that it WILL affect people who are not pre-compiling, like us. And given our situation above, we haven’t been able to figure out how to pre-compile due to the dynamic nature of our master layouts. 

So, a couple of questions:

1. Is what we’re doing and the way we’re doing it the best way to do it? Or is there some other way that might be better?
   
2. If what we’re doing is ok, is there a way we can somehow pre-compile not only our application views but also our customers custom master layouts?

Thanks for any help, guidance or pointers!

James

[Louis DeJardin]

The other thread wasn't as much about a leak as it was about how much memory was still referenced by a compiled view. Basically there's a reference to the parse tree kept, which showed up as a lot of small objects.

 

But that shouldn't show up as steady growth if the views are compiled once and used over and over. Is it possible they're being recompiled each request?

[RobertTheGrey]

Hi James,

Hopefully you're making some progress on this. I'm sure there are a number of ways to skin this particular cat, but it's probably not wise for me to give advice on whether your approach is the "right way" or not since I just don't know enough about the variables. But something that occurred to me while reading what you wrote - you said that users upload their own master page in the form of an html template and that you guys insert a content section in order to allow rendering. Is that an automated or manual step? If it's manual, then I'd probably start there in trying to find a way to include the customer template as part of an overall precompilation routine that puts it live.

Without knowing more about your solution architecture (and on this public list is probably not the best place to discuss it), I can't really give you better advice on how to get precompilation working, suffice it to say, we've got some really complex projects out there, even as far as embedding Spark in SharePoint as a CMS across websites in massive global organisations, and those are all precompiled, so I'm pretty sure that it's possible for it to work on your project.

Good luck!

Rob

[James G]

A follow up.

After working with Rob on this offline (thanks again Rob), we resolved this. We cleaned up the way we were loading our themes (although in practice, it's not a whole lot different that what was described in the OP) and we found a way to precompile all of our themes in a windows service. So, now we precompile our themes into DLLs that we store on disk and then load those DLLs into the AppDomain when our web app starts up. This of course makes our site faster and indeed eliminates the memory issue described, more on that below.

In building our Windows Service to pre-compile the themes, we got some interesting results. So, a little background. Initially this service created and configured a SparkViewFactory by manually setting up all the ViewFolders and adding the appropriate assemblies. Then it would loop through each of the themes (we have about 250 in our test environment), build all the descriptors using the appropriate master templates from the theme and finally call sparkViewFactory.Engine.BatchCompilation to build the dll. Each call to BatchCompilation would grab a chunk of memory and not release it. So, after running 250 themes through our service, the service was holding on to about 8GB of RAM. Once it stopped compiling new themes, the memory consumption would of course stop growing - but it would not be released until we killed the service. 

So, as Louie noted, it's not a leak, but it does hold onto that memory for the life of the AppDomain. In our case (which is clearly not the normal case), our web app would compile (on the fly) probably 100-200 themes (just a guess) in memory before the AppPool would hit a memory limit and recycle (or until IIS would start having memory problems and start throwing OutOfMemory exceptions and we'd have to manually recycle the AppPool). At that point, all those compiled themes get unloaded and the process would start over again. So, in a memory constrained environment like IIS, it feels like a memory leak even though it is not.

This repository demonstrates most of this:

https://github.com/jglinsek/SparkDynamicTheme

It has 2 projects. The first is a web application that demonstrates how we are doing the themeing. (This web app also has the precompile happening in the app startup so it's a bit slow when it starts up.) The second is a pre-compilation service - which is just a console app that runs through and precompiles each theme. NOTE: this precompile service is built to demonstrate that memory is not released when precompiling. If you wanted to use this in production, the workaround is to precompile each theme in a temporary AppDomain which avoids consuming potentially large amounts of memory AND locking the precompiled DLLs.

- James

[Robert Greyling]

Many thanks JAmes & Ben for working with me to find the issue and providing a solution to reliably reproduce this. I'll let you know once I've had a chance to dig into the BatchCompilation stuff and release the memory being held onto.

Cheers,

Rob

--
You received this message because you are subscribed to the Google Groups "Spark View Engine Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-dev+...@googlegroups.com.
To post to this group, send email to spar...@googlegroups.com.
Visit this group at http://groups.google.com/group/spark-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.