spark

[Josh Taylor]

I would like to start using spark. Can we get some documentation?

[Geoff Schmidt]

Cool :)

Spark is a live page update engine -- it's the machinery that you put underneath a templating system like Handlebars to make it reactive. Most people will probably not use Spark directly; they'll use a templating system built on top of it. It is a rewrite of the old liveui package and will replace the old Meteor.ui.render / Meteor.ui.chunk / Meteor.ui.listChunk API with a new, more powerful and general API. Spark is designed to be totally separable from the rest of Meteor so that you could use it instead of/alongside jquery in any app you please. Spark and all its dependencies come to ~8k minified and gzipped (at least for modern browsers that have querySelectorAll -- old browsers will also need something like Sizzle.)

Spark is on the 'spark' branch in the package 'spark'. Also on that branch are the packages 'liverange', 'universal-events', and 'domutils' which were factored out of liveui and are spark dependencies. On the 'spark' branch, Meteor is now running fully on top of Spark and all tests pass.

We're still changing the Spark API every few commits and don't have any docs together. The reason the API is changing is that we're validating it by implementing other things on top of it and seeing what is missing. This situation should improve rapidly over the coming week or two. In the mean time, you can look at how the 'templating' package uses Spark (deftemplate.js) to get a rough idea of usage. It's basically Meteor.ui.chunk's options exploded out into a set of annotation functions. A quick guide:

- Spark.render: take a function that returns a string of HTML, and return a DocumentFragment. A little like setting innerhtml, but creates a Spark context. Inside the Spark context you can call the following functions to annotate the HTML with special behaviors that will be attached when the HTML is parsed into DOM nodes.

- attachEvents: set event handlers on a region.

- setDataContext: attach some data to a region, which you can retrieve later with getDataContext.

- isolate: make a region update automatically whenever the data backing it changes (using our standard Meteor.deps mechanism to automatically wire up callbacks.)

- list: repeat a block of HTML for each element in a database cursor, and keep it updated live (by creating, moving, and removing DOM nodes) as the contents of that database cursor changes.

- labelBranch, createLandmark: "labels" are arbitrary markers on regions of the DOM. "landmarks" are areas of the DOM that are preserved across rerendering. When you create a landmark, you can get create/render/destroy callbacks to track when your templates are created, come onscreen, are rerendered, and are taken offscreen and destroyed; you can mark certain elements (eg, <input> element) by selector to be preserved across rerenderings so that they are === identical before and after rendering (their attributes and children are patched up, but the node itself is not disturbed -- useful for animations and many other things); or you can mark the entire contents of a landmark as never to be disturbed or updated no matter what (say, a Google Maps embed.) A landmark is matched after rerendering by the fact that it has the same "label path", that is, it's enclosed by the same set of labels. Templating systems should automatically create labels for you where necessary.

-- Geoff and David

On Tue, Aug 7, 2012 at 1:00 AM, Josh Taylor <josh...@gmail.com> wrote:

I would like to start using spark. Can we get some documentation?

--
You received this message because you are subscribed to the Google Groups "meteor-talk" group.
To view this discussion on the web visit https://groups.google.com/d/msg/meteor-talk/-/7ZqHzJEdhZ4J.
To post to this group, send email to meteo...@googlegroups.com.
To unsubscribe from this group, send email to meteor-talk...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/meteor-talk?hl=en.

[Iain Shigeoka]

Thanks for the rundown Geoff and David. Will Spark provide a mechanism for integrating/driving libraries like D3 and jQuery Mobile?

Right now it's difficult (in the case of D3) because

1) D3 needs to hook onto, and then control a DOM node for the lifetime of a "page/view".
2) D3 assumes either data in the page or data via ajax - tying them into the reactive data and hooking data updates in minimongo to the internal third party data model is tricky - it feels like it should be much easier.
3) Being efficient with large datasets in the browser is currently difficult. Meteor wants to store a copy in the client (in minimongo), and the third party library must have its own copy. Is there a way to say, "Meteor, just feed be the collection deltas" and then feed those into the library without storing redundant copies in minimongo?

As a specific use-case, how do we tightly integrate and exploit Meteor to create the example D3/Crossfilter flight analysis page:

http://square.github.com/crossfilter/

It seems that with Meteor we should be able to recreate that example with improvements to the user experience because Meteor could stream the large data set to D3/Crossfilter and the UI should progressively build up as data streams in. But in practice this is extremely difficult in Meteor.

The same goes for support for jQuery Mobile - since the third party library assumes it's controlling the DOM and browser lifecycle, it's a real struggle to hook into Meteor in useful ways (or vice versa - to have jQuery mobile hook into Meteor pages in useful ways).

-iain

[Geoff Schmidt]

Yes, the constant region support in Spark is explicitly intended to solve #1. Spark will hold that region constant and not touch it, and use a tree patching algorithm to update the nodes around it in-place (!) You also get "created" and "destroyed" callbacks to set up the region when it comes onscreen, and tear it down if necessary once it's offscreen, as well as a way to store per-template-instance data such as your D3 handle. This has been a top request (it comes up in cases as simple as embedding a Twitter follow button as on meteor.com), we identified it as a tech risk, and that's one of the reasons we prioritized Spark.

For #3, yes, there are (undocumented) APIs for being a DDP data consumer. In fact, Mongo (in the mongo-livedata package) uses these to receive the deltas off the wire and apply them to the appropriate minimongo collection. These interfaces will probably change, but they're intended to be public eventually. Your experiments might help it mature faster :) I see two options:

* The easiest way would be to use the 'registerStore' method on LivedataConnection. You pass the collection name that you want to receive updates for, and four callbacks -- beginUpdate, update, endUpdate, reset. See packages/mongo-livedata/collection.js for an example use of registerStore. This way doesn't give you latency compensation; if you want that you're on your own (collection.js provides an example.)

* The harder way is to create a Meteor.Collection as usual, but pass a third argument to specify a different database driver (the default on the client is Meteor._LocalCollectionDriver, which is minimongo.) In this case, you'll essentially need to duplicate minimongo's interface. The upside is that this gets you latency compensation (in other words: you'll be able to write to the collection locally, not just wait for diffs to come down from the server. You have a fully functional Meteor.Collection.) The downside is that you must implement the methods snapshot, restore, pauseObservers, and resumeObservers, meaning that you will have to figure out how to make D3's datastore snapshottable, which may or may not be easy. There is a little bit of documentation on those methods on minimongo.js's implementation. You should have a much easier time than minimongo, since you presumably don't have to compute exact diffs on snapshot restore as minimongo does (minimongo/diff.js, quite a pain but worth it since it ends up producing minimal sequences of DOM operations when updating {{#each}} blocks.)

I'm not familiar with D3's API so I can't speak to #2, but would be happy to hear suggestions (though MDG probably won't be able to act on them anytime soon -- we have to stay focused on the accounts project and on spark.) The Crossfilter example looks like a great starting point for a D3 exploration, thank you. Maybe I will have time at a hackathon sometime to dig into it :)

Likewise, I haven't had a chance to educate myself about jQuery Mobile's requirements but will add it to my todo list. On the other hand, if you are able to suggest specific API changes that would help jQuery Mobile, there might be things we could get into the current cycle of Spark work. If all goes well we'll have an experimental Spark-based build with {{#constant}} available in about a week. We'll announce this on meteor-core, just like the auth branch.

Thanks,

Geoff

[Iain Shigeoka]

Hi Geoff,

Thanks for the in-depth response. I was thinking that digging into DDP was the only way to create an efficient "bridge" between Meteor's data and a third party js library's. The amount of work seems a bit intimidating (and with the APIs still in flux) I've been trying to avoid it. :) I'm guessing there may be a more third-party-javascript friendly generic Meteor.Collection/custom database driver solution that would make integrations easier but I don't have any concrete ideas. I'll probably bite the bullet and give it a shot when I have the time. I am very curious to learn more about how DDP works so it will be a good exercise.

As for jQuery Mobile (JQM) - I think support for it (and similar libraries) will help Meteor quite a bit - especially once there is Meteor support for browser detection and UI variants (to support localization, mobile, etc). One of the tricks with JQM is it hooks the document ready event, and then processes the entire DOM to "mobilize" it by adding DOM elements, css classes, etc based on the "raw" DOM contents, and then it issues its own "mobile document ready" event. Once JQM processes a page it pretty much steps out of the way leaving a js scripting environment that feels - to the developer - like plain old jquery. The problem with Meteor is the content that JQM should process is not in the DOM when JQM normally would processes the DOM.  A quick work around would be to load the JQM library into the browser _after_ Meteor loads and finishes reactively rendering the UI. One API change that could help is a "load this JS file/files after everything is completely done rendering" (or a smart package option since JQM would ideally be a smart package). I suppose we can hook the "created" callback and use javascript to pull in the JQM library but that seems error prone and complicated for less savvy developers. A really "magic" solution would be for Meteor to suppress the real browser DOM ready event and then issue its own virtual one every time it finishes rendering the UI - that would let jQuery mobile and similar libraries work as-is even when routing and pages are supported in Meteor (e.g. a virtual route triggers the rendering of a virtual page which triggers the virtual dom-ready event and JQM processes the DOM properly for each virtual page displayed even though the browser didn't actually reload the DOM).

A secondary problem is that jQuery mobile relies on additional ajax page requests to load more content which doesn't fit in with the current Meteor stack - I think that part will sort itself out once Meteor's ability to create REST interfaces emerges so probably doesn't need any solution now.

Thanks

-iain

You received this message because you are subscribed to the Google Groups "meteor-core" group.

To post to this group, send email to meteo...@googlegroups.com.

To unsubscribe from this group, send email to meteor-core...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/meteor-core?hl=en.

[steeve]

Geoff and Iain;

Great stuff and helps me understand the difficulty I had when I built a JQM app served from Meteor.

While JQM basically has built in RESTful support for request server side resources (data, etc...) I think it would be much more powerful to have the to do it the Meteor (or I guess Spark) way.

I think there are two cases to consider with mobile.  First, mobile apps/sites that are served from Meteor.  Second, mobile apps that are written in HTML5/CSS3/Javascript like JQM and wrapped in things like PhoneGap or Trigger.io and then installed natively (with native support).  

I built a JQM app served from Meteor but as Iain points out there are issues with DOM render sequences and it gets really hairy and messy.

I think the second case is very important because it allows for apps to be built and deployed to App stores and installed native rather than served as a mobile web-app from the Meteor server.  

My app has both a web-app and will have a native installed mobile app so I will have to cross the bridge at some point and figure out how to get data back and forth.  I would much rather it be real-time rather than REST.  I was able to build a JQM Knockout app that used the VertX EventBus to move data in real time and their solution is a javascript library that sits on the client.

Just some food for thought.

Steeve

[Josh Taylor]

Thanks so much for the run down - really helpful.

I've been skimming over the demo and this stuff is exactly what I've been looking for.

Quick question. What is the difference between the `find`s in the following:

Template.timer.render = function (landmark) {

...

self.node = landmark.find(".elapsed");

Template.circles.render = function () {
  var self = this;
  self.node = self.find("svg");

Josh

[Josh Taylor]

also what does preserve do? as in :

Template.preserveDemo.preserve = [ '.spinner', '.spinforward' ];

[Geoff Schmidt]

Nothing, I think -- this was a last loose end in the API (what is in 'this' and what is in arguments, and whether the user-writable per-template state object, and the object you call find() on, are the same or different.) David and I plan to have some docs together by Monday. The idea is to do a round of discussion/design critique/testing on meteor-core/meteor-talk, and once it passes muster with it you guys, push it out to the whole community as Meteor 0.4.0.

To view this discussion on the web visit https://groups.google.com/d/msg/meteor-talk/-/Hb3kRnE7r0sJ.

[Geoff Schmidt]

also what does preserve do? as in :

Template.preserveDemo.preserve = [ '.spinner', '.spinforward' ];

Preserve ensures that find(".spinner") remains the same ("===" equal) when the template is redrawn. It causes Spark to *patch around* an element, rather than *replace* it, so that it remains the same physical DOM node. Moreover, the node is guaranteed to be left in the DOM (it will not be removed and reinserted.)

This means that, unlike setting innerHTML...

* CSS animations will run without interruption

* cursor position and selection in <input> elements will be preserved

* focus is preserved

* if the user is composing an international character, the input manager won't get closed

* if the user is holding the mouse down on a button, it won't pop out unexpectedly

* event handlers installed by non-Meteor means (eg, jQuery) will be preserved

* pointers to the node in JavaScript data structures will continue to be valid

The rule is: by putting a selector in the preserve list, you guarantee that *only one* node matches that selector. Meteor finds the node that matched the selector before the update, and the node that matched after the update, and adds the pair (oldNode, newNode) to the list of invariants that will be preserved by the DOM patcher.

Other than the guarantee that the node will be reused and will not be taken out of the DOM, preserved elements behave just like any other element. For example, their attributes continue to update reactively (the patcher updates the attributes in place.)

Preserve applies only to a single node, not to its children. If you are trying to preserve a range of nodes and all of their children, you want {{#constant}}...{{/constant}}. Note that {{#constant}} has some limitations in the current release. For example, Meteor event handlers in {{#constant}} may not fire reliably. It's really just intended to give you a dead-simple way to embed a non-Meteor widget inside a Meteor template. If the limitations are a problem, they are fixable.

Preserve will fail if you create an impossible situation, for example:

* preserving two siblings, but changing their order (there's no way to handle that without taking one of them out of the DOM and reinserting it..)

* changing the type of a preserved element, eg DIV to SPAN (the DOM spec doesn't allow the type of an element to be changed once it's been created)

* changing the depth (number of parents) of a preserved element, or the types of the parents (obviously, this would require reparenting)

In these cases the patcher will do its best, but in the worst case it will simply ignore the preservation rule for that one update. Subsequent updates will continue trying to preserve the node. If you don't create an impossible situation (per the rules above), preserve will always work.

You can also bulk-preserve nodes by passing functions to preserve. The function should return a unique label for each node that matches a selector. Nodes will be preserved if they have the same key (the keys only need to be unique within each selector.) This code preserves all input elements with 'id' attributes, using the id as the unique key:

Template.foo.preserve = {

  'input[id]': function (node) { return node.id; }

};

Pre-Spark, Meteor would automatically preserve some nodes for you, based on a heuristic. (Quoting the docs: "To activate this feature, give each such element a unique id, or give it a unique name attribute inside the nearest enclosing element with an id.") Now the heuristic is gone and you have to ask for preservation if you want it. This is a breaking change. I believe the following preserve declaration will give you back the old heuristic:

Template.foo.preserve = {

  '*[id], #[name]': function(n) {

    var label = null;

    if (n.nodeType === 1) {

      if (n.id) {

        label = '#'+n.id;

      } else if (n.getAttribute("name")) {

        label = n.getAttribute("name");

        // Radio button special case:  radio buttons

        // in a group all have the same name.  Their value

        // determines their identity.

        // Checkboxes with the same name and different

        // values are also sometimes used in apps, so

        // we treat them similarly.

        if (n.nodeName === 'INPUT' &&

            (n.type === 'radio' || n.type === 'checkbox') &&

            n.value)

          label = label + ':' + n.value;

      }

    }

    return label;

  }

};

Very happy to hear any feedback on this or any other part of Spark. I hope it isn't too confusing. Preserve is very powerful -- it's the key to writing your HTML in a declarative style, where you don't have to worry about updating the DOM manually but also don't have to worry about exactly what is getting redrawn when.

The hope is that preserve will be a low-level API that novice users will not normally need to use. For example, a forms package should autopreserve the <input> elements it creates. An animation package should autopreserve the animated nodes. It would also be possible to make the template compiler (eg, Handlebars) autogenerate preserve maps, but that's a little too heuristic/magical for my taste.

Geoff

To view this discussion on the web visit https://groups.google.com/d/msg/meteor-talk/-/W5Hg88AROKcJ.

[Geoff Schmidt]

+ core

[David Greenspan]

That part of the demo was actually broken (not adapted for the latest API changes).  Fixed. :)

-- David

[Josh Taylor]

I noticed the following comment in the demo: 

// XXX actually want to create a ReactiveVar on the template!

// (but how will it be preserved across migration?)

// (maybe template.get, template.set?? rather than form??)

I would like to give implementing that a go. Do you think we could use the dom path of the template at on_migrate time to store the ReactiveVar and then on reload Templates can restore any ReactiveVars with the same dom path?

[Josh Taylor]

I implemented the ReactiveVar on the template and submitted a pull request. Could use some feedback on how to make the migration more robust.

Thanks.

[Blake Miner]

Can Spark be downloaded and used independently from Meteor?  If so, is there a "Spark bundle" out there somewhere (i.e. a compressed archive containing only the Spark smart package)?  If not, what's the best way of "stripping out" Spark from Meteor?

I love Meteor, but for a few projects, I'd just like to use the functionality that Spark provides.

Thanks!

[Blake Miner]

Geoff,

Could you please explain, in more detail, when it is appropriate for a templating solution to call labelBranch and createLandmark?

http://stackoverflow.com/questions/12896403/in-spark-whats-the-difference-between-a-landmark-and-a-branch-label

Thanks,

Blake

On Tuesday, August 7, 2012 9:57:28 PM UTC-4, Geoff Schmidt wrote:

[David Greenspan]

Hi Blake,

In Handlebars, every template and every {{#constant}}...{{/constant}} region is wrapped with a landmark.  When a region of the DOM is reactively updated, the old and new landmarks are matched for the purpose of DOM preservation, lifecycle callbacks (created/destroyed), and landmark-local state.

Landmarks are identified by the "path" formed by the branch labels that contain them.  (Landmarks themselves don't contribute to the path.)  It's illegal to have two landmarks with the same path.  It's also important that you compute the HTML that will go inside the labeled region or the landmark from the body of the function argument to labelBranch or createLabel.  This is because Spark is actually already matching the DOM when you call these functions so that it knows whether to call "created" (because this is a new landmark) or not (because it is a re-rendering of an old one) before running the function the calculate the HTML.

Handlebars calls labelBranch in several places to distinguish any template (landmark) invocations that should be distinct.  Basically, the labels should determine the "call stack" that led to the template invocation.  Each template invocation in the template source ({{> someTemplate}}) gets a label, and loops are also instrumented, labeling each iteration of the loop based on the _id of the object in question or some other heuristics if there isn't one.  (Note that there are two code paths for loops in Meteor templating, one for observable cursors from the database and one for everything else, like normal arrays.)

In this snippet from a template that includes the "foo" template several times, each invocation of "foo" (including each iteration of the loop) is done within a different branch label:

{{> foo}}

{{> foo}}

{{#each collection}}

  {{> foo}}

{{/each}}

{{> foo}}

What if a helper invokes template functions directly?

var myHelper = function () {

  return Template.foo() + " " + Template.bar();

}

In this case, the day is saved because the template package also wraps each template in a label like "Template.foo" or "Template.bar" (labels are cheap).  However, there will be a duplicate landmark problem if a helper calls the same template multiple times with different arguments; in this case, it is up to the helper to drop additional branch labels to disambiguate.

The implementation of Template.foo that is generated by the templating package (in deftemplate.js) uses the following nested annotations, from outer to inner: label, data, landmark, events, isolate.  This order is pretty constrained.  For example, the landmark enclosing the events annotation is used to find the data.  The isolate annotation is intentionally on the inside.

Finally, an important note/warning:  The current factoring of the handlebars and templating packages may not be great for adding additional templating packages!  This is a new frontier for us.  We're very interested in having more templating languages, though, and happy that you're working on this.

-- David

To view this discussion on the web visit https://groups.google.com/d/msg/meteor-talk/-/Uk4BLQQMfPYJ.

[Blake Miner]

David,

Thank you very much for your prompt response.  I look forward to working with you to get Blade integrated with Meteor + Spark.  I'm definitely excited to get this rolling, especially with the recent release of Meteor 0.5!  I still have a few questions for you...

1.) Just need to clarify the purpose of labelBranch.

2.) Need to chat about Blade file includes and how they differ from Handlebars.  Learn about dynamic file includes in Blade.

3.) Need to address the problem that "blocks" might create.  Learn about Blocks in Blade.

So... sorry in advance for such a long post, but here it goes...  :)

In Handlebars, every template and every {{#constant}}...{{/constant}} region is wrapped with a landmark.  When a region of the DOM is reactively updated, the old and new landmarks are matched for the purpose of DOM preservation, lifecycle callbacks (created/destroyed), and landmark-local state.

So, just so I understand.  Landmarks are created whenever a specific part of a template needs to have lifecycle callbacks, DOM preservation, or landmark-local data.  Inserting a landmark injects a special tag into the HTML, used to identify the landmark's location in the DOM, but this tag is completely unique each time createLandmark is invoked.  Spark.render() strips these "special tags" from the HTML while building the actual DOM structure and then tracks the actual DOM nodes that make-up a given Landmark.

Branch labels, on the other hand, are nothing more than a tool to identify Landmarks between re-renderings.  Other than identifying Landmarks between re-renderings, branch labels are completely useless, correct?  Why not pass a unique "branch label" as an argument to the createLandmark function and then get rid of the labelBranch function altogether?  Why separate the functionality of the two?

 

It's also important that you compute the HTML that will go inside the labeled region or the landmark from the body of the function argument to labelBranch or createLabel.  This is because Spark is actually already matching the DOM when you call these functions so that it knows whether to call "created" (because this is a new landmark) or not (because it is a re-rendering of an old one) before running the function the calculate the HTML.

This means that once the HTML is computed and Spark Annotations are added to the HTML, one should not change the previously computed HTML?  For example, "blocks" in the Blade templating language, are locations in the template where HTML can be injected later.  That is, you can define a named block (call it "nav") where your site's navigation goes.  Then, later, perhaps in an included file, you can append/prepend to or completely replace the "nav" block.

Anyway, if a block is defined inside of a Landmark, isolate, or branch label, then Blade would need to convert this whole area to HTML.  Unfortunately, this whole process would probably need to delete/clean-up any blocks defined in this area.  Again, this is because code elsewhere (even in a completely different file) could potentially alter the contents of the block.  Thoughts on this?

File includes, for example, may define blocks.  And, later, from another file entirely, one can append to that block and change the resulting HTML of the former template.  Right now, all file includes should probably be wrapped in a Spark.labelBranch call, which needs a function to return HTML.  How can we account for blocks, though?  What if each block definition was wrapped in an "isolate" and the resulting HTML was generated by a reactive variable?  Then, "append block" statements would modify the reactive variable and require Spark to re-render the whole block?  Thoughts?

In addition, one can use pure JavaScript code within a template and create looping structures.  Even if a branch label is unique to the filename and line in the template's source code, several instances of the same template can be included when an "include" statement is placed inside of a while loop, for loop, etc.  I thought about keeping a line-specific counter that would increment for each "include" call on the same line, which would make the label something like `filename + line_number + line_specific_counter`.  Somehow I feel like that's not going to solve the problem.  Thoughts?

 

Please let me know if you have any questions for me.  I'm available on Google Chat quite often (although probably not tomorrow or this weekend), and you have my email.

Thanks again for your help!

Blake

[Kaoru Aoi]

Is the unreliable events in constant going to get fixed? Why is it a problem anyway, I wouldn't thought that event delegation would let it work.