Proposal for the lightweight asset pipeline and initial thoughts about SPA development

[Jeremy D. Miller]

I’d love feedback of course, but I’m writing this as much as anything to compose my plans so I can get the code to start flying tomorrow morning.

Just for some background, I’m trying to hustle and get FubuMVC 2.0 (and SM3) ready for big project we’re doing at Extend Health this year.  I’m working under the assumption that we’re building along SPA lines (w/ Angular.js as our framework, Require.js for dependency mgmt, and Mimosa.js for all asset transformations) so hence, time to get moving on the asset pipeline replacement pronto.

After quite a bit of conversation w/ Matt Smith, Bob Pace, and Corey at Extend, I think this is the game plan:

Proposals for the Lightweight Asset Pipeline

All static content will be served directly by IIS if you have to be on ASP.Net or the new OWIN StaticMiddleware introduced in 1.3.  Either way, the assets will NOT be served by a behavior chain.  All asset transformations will be assumed to be done at build or deployment time.  We will no longer do anything whatsoever to order script or stylesheet dependencies.  No on the fly transformations, no magic combinations, no nothing.  I give up, go use Require.js or some other sort of AMD approach or use Cassette/SquishIt/etc.

I vote that we keep all of the existing page extension methods from the old asset pipeline (https://github.com/DarthFubuMVC/FubuMVC.Core.Assets/blob/master/src/FubuMVC.Core.Assets/AssetFilePageExtensions.cs), but the behavior is going to be slightly different:

1. When you requite an asset, fubumvc is going to try to locate a file matching that name and resolve the Url for wherever that file is in the file system.  So if you ask for “foo.js” and FubuMVC finds that file under “content/scripts/foo.js”, the Url will be “~/content/scripts/foo.js”.  Obviously, there needs to be a little bit of thought to naming collisions, specifying the deeper path, and so on.
2. There will be no dependency resolution.  The asset pipeline is going to write out what you tell it to and nothing else.

About the Require.js thing….

Some of you love it, some of you hate it, and others are going “huh?.”  The new lightweight assets will not mandate the usage of Require.js, but I think we should build just a tiny bit of Require.js awareness into the new assets (see below about cache busting).  

What about the old asset.config DSL?

KEEP:  The functionality for aliases like “jquery is jquery.1.7.1.min.js” so you can go (in Spark): <Script src=“jquery”/> 

KEEP:  ordered set [name] is… —> but it’s going to be a literal ordering of exactly those things. 

THROW AWAY: Everything else

ADD: Some syntax to specify CDN locations for certain assets, see below for more

CDN Support

Not sure about the exact syntax yet, but…

[alias] is [url] fallback to [file]

# and the next line might be someway to define fallbacks like what Hanselman is showing here:  http://www.hanselman.com/blog/CDNsFailButYourScriptsDontHaveToFallbackFromCDNToLocalJQuery.aspx

In the views, it would still be 'this.Asset(“jquery”)’ and the new asset pipeline would write out the script tag pointed at the CDN w/ the fallback tag as well

Caching in Production

• Our thinking at EH is to publish all the assets to a single folder at deployment time that embeds the application version.  So instead of just “/public” for the assets, it’ll be “/public/[app version]”.  That way you can both aggressively cache assets on the browser and still be able to cleanly update.  This workflow is directly supported by the Require.js/Mimosa.js combination.  We would add helpers to FubuRake for all of the new workflow.

• In production mode by default, the static middleware will use aggressive browser caching —> cache-control=“private, max-age=[one year]” and expires=“one year”.  IIS behaves similarly, but I’ll let y’all fight w/ that configuration on your own.
• We can make the exact caching header behavior be configurable easily enough
• *If* we could mandate using an OWIN enabled web server like Helios on IIS, we could instead do some easy URL rewriting tricks and eliminate the need to publish the files to the versioned path, but I don’t want anything to do w/ that in the IIS/ASP.Net combo.

The Development Experience

In development, I’m wanting to strongly encourage the usage of the “fubu run” development server — saying that I *think* that there is no longer any substantial technical hurdle keeping you from using the Katana based server.

If you use “fubu run”, the default fubu mode is “Development”, so we can use that mode to behave differently in development:

1. In “Development” mode, do not write any caching headers (cache-control or expires)
2. Have Require.js use its own query string cache busting trick and make the asset pipeline append a query string w/ an ETag of the file’s timestamp and contents.  We can’t use query strings in production for cache busting because some caching proxies don’t play nice w/ query strings, but I think it should be fine in the live Katana hosted server.
3. Mimosa.js has a “watched folder” function that does all the coffeescript/SASS/you name it conversions anytime the source files change.  Just leave that on as you development.
4. “fubu run” already has an auto-refresh mode that refreshes the browser when anything changes (asset files, spark views, recompile, config file changes).  The drawback is that I use WebDriver to pull that off and it breaks on occasional browser updates.  Maybe we can go to a client side only approach for the auto-refresh instead (maybe embed Fleck in fubu run?  Use the server sent events?????)

Crazy Ideas for Single Page Application development:

For now, I’m not giving up on the idea of using html conventions/Spark or Razor pages to generate HTML templates because it has so much goodness for automated testing, localization, validation, and reducing repetitive code.  That being said, we need to create the fastest development feedback cycle we can work more efficiently.

1. We have a new command in fubu.exe (call it “fubu templates”) that can exercise a FubuMVC applications to pre-build html templates from Spark or Razor views that match some yet to be determined naming conventions.  Maybe “all .spark or .cshtml files under a /Templates directory in the app or any bottle?”
1. As an fyi, I’ve got a new InMemoryFubuMvcServer in the 2.0 codebase that allows you to run a single request through a behavior chain without any need for a web server that should enable the html generation pretty easily
2. Add a “—watched” mode to the new “fubu templates” command that re-generates the HTML templates anytime a view file changes or re-generate them all when the application binaries change.
3. Rebuild the old Serenity Jasmine Runner (if you haven’t used it, think “AutoTest for Jasmine”) into fubu.exe so that you could seamlessly update your jasmine tests in a browser anytime either the fubumvc views, fubumvc application, or client assets change
4. Get more ReSTful —> the pesky problem of smuggling server side data like Url’s down to client side markup has always been a minor PITA, especially w/ our philosophy of never hard coding Url’s in the app.  If we get serious about Hypermedia API’s on the server side, I think we can drastically remove that longstanding problem.

[Jeremy D. Miller]

And a couple other things I forgot:

* Still going to allow you to use client assets from Bottles

* Lean on Bower and/or NPM to resolve assets as possible.  Probably going to be some FubuRake integration to make that easier.

* I’m planning to build the new lightweight asset pipeline directly into FubuMVC.Core 2.0, but if it’s deemed valuable, make a separate assembly later if we want to use the lighter weight model w/ FubuMVC 1.*

* The hope is that you won’t have to change existing apps much at all except for the script and stylesheet ordering

--
You received this message because you are subscribed to the Google Groups "FubuMVC Development Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fubumvc-deve...@googlegroups.com.
To post to this group, send email to fubumv...@googlegroups.com.
Visit this group at http://groups.google.com/group/fubumvc-devel.
For more options, visit https://groups.google.com/d/optout.

[Kevin Miller]

Looking at the FubuMVC issues work being done I see a lot of Issues that are likely addressing the pain we were feeling in using Spark as a view engine. That should be a huge improvement. Glad to see it. 

The SPA capabilities sound like they would help with build time templates. One of the needs we had because parts of the system were dynamic was to have runtime template building. We had rolled our own but have since moved away from Spark due to the pain I mentioned. 

Details on our client side template science project. We are finding a lot of goodness in client side templates and lack of friction and while I do miss HtmlConventions for some things template helpers and localization plugins have made up the difference there. The side advantage is there is less Magic in the views so it seems more approachable for developers looking to customize our app. 

I’ve given in on the whole don’t hard code URLs thing. We so seldom change a url once it is baked and the cure, hoops we were jumping through to avoid hard coding them, was worse than the disease. In the end we typically have modules that are coupled to backend endpoints which are usually in Bottles so anything we can do to make it easy to serve JavaScript modules needed for Bottle endpoints is helpful.  Currently we simply resort to ~/_content/{asset path}/module.js for bottle modules. 

RequireJS presents some problems with it’s config as bottles might want to throw paths or shims and such into it. Right now we avoid that by AMDifying everything and having the root url be ~/_content/scripts/. I’d be interested in hearing how you manage Bottles + RequireJS config.  

I really need to start looking into fubu run as a development workflow. How do I debug code against Katana is it an Attach to Process like workflow?

- KevM 

[Corey Kaylor]

I can't remember if Jeremy baked the F5 experience into fubu new or not, but it's essentially going to start a cmd line tool (in your project properties), the debugger will get attached if debugging, or CTRL + F5 starts without debugging so nothing new to learn there. We were already doing something similar for IISExpress because we didn't want to continuously change the profile specific applicationHost.config, so we pass it a unique config using the same technique.

The current proposal is that mimosa integration will be bottles aware and will allow you to combine assets from anywhere in your app + anything development time under the .links file.

I've kind of wondered the same thing you mentioned about client templates minus HtmlConventions, but we're not ready to throw in the towel just yet.

[Corey Kaylor]

Sounds like a good list. Curious, why private for "aggressive" caching in production mode?

[Jeremy D. Miller]

Because that’s what we do now?  I’d be a liar if I told you that I remembered why I did that (in the Firefly days w/ Josh I believe, but I coded it).  I need to research that one a bit.

[Jeremy Miller]

Ok, I think I change my mind a little bit about the asset page extensions. *Today*, you have:

Today

ImageUrl()

Image/ImageFor() -- they do effectively the same thing w/ different mechanics.  Duplication FTL!

Script(string[] names) -- only marks an asset as required, but doesn't write anything out

Asset(string[] names)-- only marks an asset as required, but doesn't write anything out

OptionalScript(string[] names)

OptionalAsset(string[] names)

Css(string names)-- only marks an asset as required, but doesn't write anything out

WriteCssTags() -- writes out all stylesheets that are queued up

WriteScriptTags() -- writes out all scripts that are queued up

WriteAssetTags() -- writes out all stylesheets & scripts that are queued up

ImmediatelyWriteCssTags() 

ImmediatelyWriteScriptTags()

ImmediatelyWriteAssetTags()

In the world of tomorrow:

ImageUrl()

Image()/ImageFor() ImageFor delegates to Image()

Script(string[] names) -- writes script tags immediately

Css(string[] names) -- writes <link> tags immediately 

OptionalScript(string[] names) -- writes the <script> tags if the scripts exist

OptionalAsset(string[] names)-- writes the <link> tags if the scripts exist

for all the other older extensions, I vote that we just have them delegate to the ones above.

The current asset pipeline was designed for a model where html conventions and html helpers could register assets that would be dumped into the page in the footer when the WriteScriptTags() method was called.  It demo'd really well and sounded cool, but I'm a little indifferent now to going down that path again.  I *think* that I vote for a much simpler model where we make users be explicitly responsible for declaring client side assets.

What do you think?

[Jeremy D. Miller]

The F5 experience is still outstanding.  Ryan has made it work w/ manual changes to a csproj file, but I haven’t yet brought that into the fubu.exe templating yet.  We can either make it a templating choice or maybe add something new to fubu.exe so you can quickly change a csproj file from F5 w/ Katana or w/ IISExpress or w/IIS (boo).  I’m thinking that it might require some FubuCsProjFile changes too though.

[Matthew Smith]

I'm in the middle of refactoring FubuCsProjFile to support F# libraries. I'll keep an eye out for an extension point to support that.

[Jeremy Miller]

Thanks for the feedback Kevin.  Couple thoughts on what you wrote:

* I've already got a working "in memory" host that's happily running Spark & Razor views in integration tests.  Exporting HTML at build time is not going to be a technical issue.

* In regards to the goal of not hard coding Url's in client side code, the magic word is "hypermedia".  Do that and at most you might just be embedding a single Url.  

* The other option that Bob has kicked around is having FubuMVC generate a single (or few) JavaScript class with all the Url's ala Play.  Essentially a generated proxy to the functions on the server side that'll also be easy to stub/mock in Jasmine testing.

* For the client side templating, my thought all along has been to invest very heavily in more Spark bindings.  Now that Spark bindings can use optional attributes, it's effectively going to be HTML with some custom tags to invoke the conventions.  

- Jeremy

[Matt S.]

"Now that Spark bindings can use optional attributes...". I missed that feature. Does that mean I don't have to create 10 bindings with all the variations of attributes I want them to support for a single element name?