DOM reaction experiments

[Mike Bannister]

There's been a couple times when I've felt unsure about which parts of the DOM would be blown away when a certain reactive element's data changes and Avital showed me an example where he was having similar concerns. I really wanted to get a better grasp of this. As suspected it's pretty simple but, for me, knowing for sure and verifying with real experiments is the only road to peace.

You can see some experiments here (the code for this is in the repo)

    http://dom-reaction-experiments.meteor.com

and the smart package repo is here

    https://github.com/possibilities/meteor-dom-reactions

The package exposes a helper called `showReactions`. Include it anywhere in a template passing a unique label as the only argument:

    {{showReactions "an very cool element"}}

This will insert a block in the DOM indicating when it was last rendered.

I have some wild ideas for a much less obtrusive UI but won't implement it until I need it or others are interested.

-Mike

[Tom Coleman]

Hey, great. That looks like it'll really helpful for debugging!

Meteor devs: can you guys let us in on what the plan is regarding when DOM elements will be _replaced_ and when they will be _updated_? It seems the most recent release has some work in it to persisting form DOM elements[1]. Is the plan to do the same for all DOM elements e.g. when the class changes? Or is it too hard to do it right (are there edge cases I'm not thinking of?)?

Cheers,

Tom

[1] For instance, check out the unfinished/controls example in meteor-core with this stylesheet: https://gist.github.com/2958636; it should animate, showing how the first technique from here: http://bindle.me/blog/index.php/658/animations-in-meteor-state-of-the-game COULD work generally.

--
You received this message because you are subscribed to the Google Groups "meteor-talk" group.
To view this discussion on the web visit https://groups.google.com/d/msg/meteor-talk/-/U0XVLf1RmIsJ.
To post to this group, send email to meteo...@googlegroups.com.
To unsubscribe from this group, send email to meteor-talk...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/meteor-talk?hl=en.

[Nick Martin]

On Fri, Jun 22, 2012 at 7:29 PM, Tom Coleman <t...@thesnail.org> wrote:

Hey, great. That looks like it'll really helpful for debugging!

Agreed! This looks awesome, Mike.

 

Meteor devs: can you guys let us in on what the plan is regarding when DOM elements will be _replaced_ and when they will be _updated_? It seems the most recent release has some work in it to persisting form DOM elements[1]. Is the plan to do the same for all DOM elements e.g. when the class changes? Or is it too hard to do it right (are there edge cases I'm not thinking of?)?

Good question! That's actually what David is working on right now. I don't know the details and don't want to speak for David, but suffice it to say this is still an area of some research. We're trying to nail this down, as it's the basis for solid patterns for forms. It's also related to making it easy to embed other javascript libraries that modify the DOM like d3 or google maps.

Cheers,

-- Nick

[Mike Bannister]

Thanks for the feedback Tom & Nick...

You can see some steps towards the "less obtrusive UI" I mentioned. Less wild than I initially imagined but that's a good thing (;

http://dom-reaction-improved-ui.meteor.com/

Mouseover the messages in the console on the right and the affected area of the DOM will be highlighted.

Thinking out loud here (in code) but I think you'll get the idea. Better approach?

-Mike

[David Greenspan]

Hey Mike,

I love the idea of a pop-up debugger like that!  That's great.  I'm a big fan of making things visible, and I've been thinking for a while that developers will eventually need to be able to debug reactive updates.  I hadn't thought of the "log with mouseover" format.

This is indeed an active area of work, but I can tell you exactly how we update the DOM currently.  I've been meaning to write it up.

The biggest thing to realize is that even if a template is re-rendered, that doesn't mean all its nodes are replaced.  We use the new HTML as a starting point and essentially generate a patch to apply to the DOM, using the IDs of the nodes to inform the diff.  (The way to test for sure whether a given element was updated or replaced is to assign the node to a variable before the update and compare it after.)  For example, if you take the skeleton app and add an ID to the H1, you'll find the H1's attributes are updated in place.  Internally we construct the new DOM in a fragment and copy over the new nodes and differing attributes.

Tom:  Your square animation will work if you add an ID attribute to the square!

Mike:  What you are observing is the dependency logic for when a template is re-rendered, which is interesting in its own right.  As you may know, certain calls like Session.get register a data dependency.  When the data changes, it sends an invalidation signal to whatever the "current" context was at the time of the call, basically the current template, which is re-run from scratch, including all its sub-templates.  The available techniques to reduce redraw at this level are coarse; basically, move the data access as far down the tree as you can, and wrap it in its own template if you don't want that part of your display to redraw its siblings.  (If you want to get fancy, you can create a context boundary without starting a new template by calling Meteor.ui.chunk from a helper.)  But with the advent of DOM patching, this kind of care is much less necessary, and there is little reason to create additional templates for this purpose.

Here are some questions and answers.

## When is it important to preserve DOM elements instead of replacing them?

Text input:  At first glance, it seems like you could replace a text field if you save and restore 1) the focus and 2) the selection or location of the insertion point.  However, when you take into account international text input and assistive input methods, there is more state than this, and the state is not accessible to JavaScript.

For example, to type French diacritical marks on my Mac, I can type option-{E,I,U,`} and then type a vowel.  This feature has been around for 20 years and is called "dead keys."  After the initial key combo, the OS waits indefinitely to see what letter comes next.  (Originally there was no visual feedback, but now you see a little accent with an underline or a colored box while it's waiting.  Even more recently, there's an additional UI carried over the iPhone -- you can hold down a vowel to see a little menu of variants; try it!)  For Asian language users (e.g. Japanese) multi-key input is a way of life; words are composed into a little pop-up box or an inline autocorrect-like interface.  Then there's dictation and other alternative ways of text input.  All OSes ship with this stuff, it's just not usually turned on for normal English-speaking users.  Browsers are starting to have some events to tell JavaScript when this stuff is happening (compositionstart, compositionend, oninput), but there's way to access the state.

Controls in general:  When you think about it, every control has some state.  If the user is holding down a button, or navigating a pop-up menu, replacing the element will abort that interaction and be disruptive.  Even moving or removing and re-adding the same element is no good.

Animation:  What Tom said. :)  I hadn't thought through the animation case before, but basically it has to be the same element, new property -- not a new element -- to get the CSS transition.

iframes:  There's no way to replace an iframe, or even move it, without completely reloading it.

Widgets, in-DOM state:  You may have code (third-party or your own DOM hackery) that manipulates an element and expects it to stick around with the same identity and expando properties.

## Why does Meteor diff DOM nodes, even with structured templates and dependency tracking?

In theory, we could construct a system with perfectly narrow reactive sources and sinks, where a data change maps to a precise DOM mutation.  Consider an #if statement in a template whose condition depends on some calculation on the database.  Ideally, we would connect a wire from the true/false outcome of the test -- not, say, the contents of an entire database document -- to the location in the DOM where the #if occurs -- not the entire template.  When the condition toggles, and only then, we would go in and do a minimal replacement in the DOM.  This technique is favored by frameworks like DerbyJS and Asana.

The problem is that requiring the dependencies to be 100% exact is too constraining, and it places a burden on the developer and the framework.  One of Meteor's principles is that a template is just a function; it doesn't matter if, internally, it uses stock Handlebars, custom helpers, snippets of HTML, some other template language, or pure JS.  We can't necessarily reach inside and find the control structures and where they map to in the DOM (if anywhere).  To do that, we'd need full reflection capabilities on the template, the code, or both.  This is why Asana started by creating a language and DerbyJS is hard-coded for Handlebars.

Instead, we use template boundaries as the places where we know both 1) how to recalculate the HTML and 2) where to put it in the DOM when we do.  We allow both data invalidations and DOM updates to be overly broad, and the DOM diff/patch system provides an escape valve by catching the over-draw.

That said, we aren't afraid to draw inspiration from the "ideal" model when it's practical.  For example, Session.equals allows you to depend on a true/false outcome, and we hack Handlebars' #each to call Meteor.ui.listChunk and implement a fully reactive list.

## How does diff/patch work?

When a template is re-rendered, the new DOM tree for that template is held in a fragment.  The old DOM tree is in the document.  We don't have any more information about the deeper relationship between the old nodes and the new nodes.  At first it seems like we could use some rough heuristics to interpret what has changed.  Surely if both renderings consist of a single IMG tag, or a DIV followed by a TABLE, then it's the attributes and contents of these tags that have changed, not their presence at all?  But consider a template that contains three input fields before the update, and three input fields after.  Are they the same input fields, or completely different ones?  Perhaps the middle one disappeared and a new field appeared at the end.  Payment processing forms often have fields that come and go, for example if the user selects "pay with PayPal" or chooses a foreign country that doesn't have zip codes.  If we are too eager in our guesses, the results could be very confusing for the user and developer.  The user's focus or entered text could jump from one field to another, for example.

We call this problem of finding the nodes that correspond the "matching problem."  You could pose it generally as:  A user-specified function is executed and creates a bunch of objects (in this case, nodes).  Then we execute the function a second time, and it creates a different set of objects.  Which ones are which?  That is, which of the objects created the first time are still present, so we don't have to tear them down?

The way we go about it is to look for nodes in the new DOM that seem to match existing nodes in the old DOM, based on their ID attribute.  We also look at the "name" attribute, which is traditionally set on INPUT elements (and only INPUT elements, according to the latest HTML spec), and treat it as a locally unique rather than globally unique label.  (Developers often use the "class" attribute for non-unique names, but it makes a poor signal for matching.  So what if an old and new button both have the class "highlighted btn_group my_button", does this mean they're the same button or not?)

A match indicates that a node remains present across the update, and so we try to keep the node in place.  If there was a <div id="foo"> and a <div id="bar"> in the old DOM, somewhere in the template, and there is in the new DOM, too, we try to leave these two nodes in place and work around them, keeping their parents but replacing their siblings and children.  I say "try" because this is only possible if the nodes in question haven't in fact moved, been re-ordered, gotten new parents, etc.

The rule for getting an element preserved is:  Give it a unique ID, or a unique name within an enclosing unique ID (or name).  Don't have duplicate IDs (which sometimes happens by accident since browsers don't complain).  Also, the elements you want to preserve can't change order.  If you have elements with IDs changing order, they will throw off element preservation -- some of the elements within the immediately enclosing element with an ID will be replaced instead of updated.

Avi encountered this "no changing order" constraint in an app recently, and it bears some explanation.  Browsers can't really "move" nodes, even internally; they just remove and re-insert them into the DOM.  Pretty much all the problems with element replacement (listed above) are still problems if we move the element in any way -- input state, iframes, animations.  Only the last kind of state, state attached to the DOM element from JavaScript, is preserved if we keep the same node and move it.  If you are trying re-order iframes, or animate re-ordering, you'll have to do some custom stuff with the element positioning no matter what.

You actually can re-order elements while preserving the DOM nodes, though, using lists.

## How do lists (#each) work?

We hack Handlebars #each to behave in a special way when passed a database cursor.

Each time the body of the #each is run, it's run in a new dependency context, so in a sense it's as if you defined a separate template and invoked it multiple times.  When the set of documents changes, the list is never completely re-rendered, only on a per-document basis.  Internally, the database cursor is observed for added, removed, moved, and changed callbacks, and each callback results in a fine-grained mutation to the DOM.  "Added" renders the body of the #each for the new document and inserts the rendered fragment as a list item.  "Removed" removes the rendering of the document from the list.  "Changed" causes the list item to be re-rendered in place.  "Moved" pulls out the DOM nodes of the list item and re-inserts them, without re-rendering it.

## What's next?

Forms; storing data in templates and preserving it across updates; preserving embeds like Google Maps and D3; callbacks when a template is rendered; access to the template hierarchy at runtime.

The biggest issue is that right now, when a template is re-rendered, we lose all the state associated with the sub-templates; at best, we hold onto some of the DOM nodes.  The basic reason for this is another version of the matching problem; if template "foo" calls template "bar" in a bunch of places with different arguments, possibly from inside #ifs and loops, how do we know which invocation is which?  Does the developer have to annotate them?  Meteor doesn't get to look at the template source code -- it's template language agnostic -- only to watch what it does when run.

Once we have "template invocation matching," we can allow you to stash data locally in a template invocation, instead of in the DOM or in Session with a unique name.  We can also have a helper for "make this region immune from reactive updates," to better support embeds, and if an update happens that would normally replace it, the patcher will instead patch around it.

I'll probably put some of this text up on the wiki.

-- David

--

You received this message because you are subscribed to the Google Groups "meteor-talk" group.

To view this discussion on the web visit https://groups.google.com/d/msg/meteor-talk/-/rnaC2bw40foJ.

[Tom Coleman]

Wow, David, great stuff. Extremely helpful, thanks!

I've updated my animation post to reflect what you've said about using ids. 

Regarding animations, it's already a limitation of CSS transitions that you can't animate an element when it appears / disappears / changes position (which a combination of the two as you've pointed out). So meteor is in the clear here :). On the other hand I guess we really need those 'callbacks when a template is rendered' so we can animate those things with jQ[1]. 

-------

I think I understand what you are saying with all this and I understand the challenge of doing this stuff whilst viewing templates as simply functions that return html snippets. I have a question/suggestion however:

It seems that although you match elements on id and avoid replacing them as a result, you don't the same with the content. For instance, if I have this very simple test case:

  <div id="foo" class="{{open}}">

    <span>Bar</span>

  </div>

Although the <div> sticks around, the <span> is redrawn when {{open}} changes (unless the <span> has an id too). 

I would propose that if when re-rendering a template, if the html content of an element is _identical_ to what it was when if rendered last, then it should be left as is. In this way you could 'protect' non-meteor generated content by wrapping it in an element with an id. It also means that you can do things like add a class to a top level element on the page[2] without having to worry about everything re-drawing/attaching..

Some devs might rely on the content redrawing to 'clear out' stuff that was added e.g. by jQ, but I'd argue that if you want meteor to clean up after you, you should use meteor to make the mess.

I guess this suggestion probably creates more problems than it solves, but I'd love to hear your thoughts.

Cheers,

Tom

[1] I think there's a use case for both a) after it's attached to the DOM, and b) after the browser has rendered.

[2] ideally the <body> although that's not possible right now.

[David Greenspan]

Hi Tom,

You hit the nail on the head in regards to a couple key problem cases -- attributes on container elements, and protecting non-Meteor content.

Consider some scenarios:

1)  Google Map in a div.  In our template we have:

<div id="the_map"></div>

We call the Google Maps API which inserts a bunch of non-Meteor content inside the div, e.g.:

<div id="the_map">

  <div>

    <div>...</div>

    <div>...</div>

  </div>

  <div>...</div>

</div>

When Meteor now goes to update the template, it calculates the new DOM as the empty div <div id="the_map"></div>, while the old DOM is the map!  So it deletes the map.  If it somehow knew that the div was empty when last rendered, it might be able to leave it alone, which I think is what you were suggesting.  Right now, it doesn't remember what the HTML was on last render, it only knows what the DOM looks like now, so it can't tell Meteor from non-Meteor content.

I'm not sure if the "same content" criterion will generalize -- let's look at the reactive attribute case.

2) Attribute on container div with reactive data access.

At the top level of the body, we have this template:

<template name="main">

  <div id="main" class="{{mainclass}}">

    ... entire page contents ...

  </div>

</template>

And this helper:

Template.main.mainclass = function() {

  return Session.get("foo") ? "foo" : "";

};

(Note: I agree that being able to set attributes on <body> would be nice!)

Right now, the helper will register a dependency from the whole template on the "foo" session variable.  When "foo" changes, the entire page is re-rendered into a fragment, and at this point there isn't much to do to improve on the current patching. It's likely that the div contents aren't *exactly* the same as before, and even if they are, we haven't saved a lot of work because we've had to recalculate everything and compare it.

My leading proposal for this case is to wrap all helpers in dependency contexts, even inside attributes.  So when "foo" changes, only {{mainclass}} would be recalculated, not the whole template, and only the attribute would be patched.

However, consider this:

3) Attribute on container div, data already fetched

We have the same template, but instead of mainclass being a helper, it's a property of the data context.  For example, perhaps we invoke the main template as follows:

{{> main opts}}

Where opts is a helper like:

function() {

  return {mainclass: Session.get("foo") ? "foo" : ""};

}

The significant difference is that now there is no reactive data access that's isolated to the class="{{mainclass}}" attribute, as there was before.  Once we've called Session.get("foo"), Meteor can no longer track who is using the result.  It has to invalidate and recalculate all subsequent templates in the hierarchy because they might somehow depend on the outcome, as the value of "foo" flows in unknown ways through the program.

The answer to this case might be "don't do that" -- Mike's debugger would reveal that the whole page depends on the Session variable "foo", and it would be the developer's responsibility to break this dependency as an optimization.

I appreciate your interest in understanding the details of this stuff!

Thanks,

David

[Mike Bannister]

Wow, David, thanks for getting that off your chest! ...and Tom, you're way ahead of me on this so your questions are fantastic, thanks. I'm anxious to see if I can use this knowledge to improve the usefulness of the tool. Not sure when I'll be able to think more about this but my end goal of better understanding DOM reactivity has definitely been advanced! -Mike

To unsubscribe from this group, send email to meteor-talk+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/meteor-talk?hl=en.

--
You received this message because you are subscribed to the Google Groups "meteor-talk" group.
To post to this group, send email to meteo...@googlegroups.com.

To unsubscribe from this group, send email to meteor-talk+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/meteor-talk?hl=en.

--
You received this message because you are subscribed to the Google Groups "meteor-talk" group.
To post to this group, send email to meteo...@googlegroups.com.

To unsubscribe from this group, send email to meteor-talk+unsubscribe@googlegroups.com.

[Avital Oliver]

This is an awesome write-up! Just note that Asana's framework is called "Luna" (http://www.quora.com/Web-Application-Frameworks/What-is-Asanas-Luna-technology-framework)

[Tom Coleman]

Hi David,

Yeah, wrapping helpers in their own dependency context sounds like a really good step. Although I suppose it will introduce a new challenge due to the fact that helpers don't always output discrete html tags, as templates do-- for example this could pose a bit of a challenge:

<div id="x" class="{{class_helper_1}} {{class_helper_2}}">

What do you set #x's class to when re-rendering class_helper_1 only? I guess you'd need either a) store the result of class_helper_2 or b) re-run class_helper_2 even though it hasn't invalidated[1]. I guess there are probably other similar challenges, for instance when helpers are generating attributes as well. 

Regarding the data context, I think yes, if the data changes, everything that renders based on that data has to re-render. Given a lot of the time the data context for a template is an object pulled from mongo, and that the reactivity of minimongo cursors is done at the object level rather than the attribute level anyway, there's probably not a big loss there.

OTOH, I have run into situations where I've wanted to do something like

{{> main opts}}

and prepare opts.mainclass in the outer template rather than defining Template.main.mainclass (ie. this how to arguments for templates). But wouldn't doing something like this solve the problem:

Template.outer.opts = {

  mainclass: function() { return Session.get('foo') ? 'foo' : ''; }

}

That way the Session.get call won't get made until the {{mainclass}} call is actually made and there's been a chance to set up a new 'helper' dependency context?

Appreciate you letting me in on your thinking regarding all this..

Cheers,

Tom

[1] It all feels a lot like functional programming doesn't it? The two solutions are equivalent if you can assume that calling class_helper_2 is purely functional with no side-effects;

[David Greenspan]

Hi Tom,

Spot on on all points!  Replies inline.

On Mon, Jun 25, 2012 at 6:03 PM, Tom Coleman <t...@thesnail.org> wrote:

Hi David,

Yeah, wrapping helpers in their own dependency context sounds like a really good step. Although I suppose it will introduce a new challenge due to the fact that helpers don't always output discrete html tags, as templates do-- for example this could pose a bit of a challenge:

<div id="x" class="{{class_helper_1}} {{class_helper_2}}">

What do you set #x's class to when re-rendering class_helper_1 only? I guess you'd need either a) store the result of class_helper_2 or b) re-run class_helper_2 even though it hasn't invalidated[1]. I guess there are probably other similar challenges, for instance when helpers are generating attributes as well. 

This is the challenge precisely.  One way to address this case is to say that the helpers don't get their own dependency contexts here, but the whole open tag does, or more specifically the string between "<div" and ">" that contains the attributes.  We already have a routine for copying over attributes in the node preservation code.

Currently block helpers are broken inside attributes (as in <div class="{{#if hasfoo}}foo{{/if}}">).  You end up with what are supposed to be HTML comments (<!-- STARTRANGE -->) inside the attribute, because the template parser doesn't know when a helper is inside the angle brackets of a tag.  But you'll eventually be able to use #if, #each, etc. inside an attribute too.

 

Regarding the data context, I think yes, if the data changes, everything that renders based on that data has to re-render. Given a lot of the time the data context for a template is an object pulled from mongo, and that the reactivity of minimongo cursors is done at the object level rather than the attribute level anyway, there's probably not a big loss there.

OTOH, I have run into situations where I've wanted to do something like

{{> main opts}}

and prepare opts.mainclass in the outer template rather than defining Template.main.mainclass (ie. this how to arguments for templates). But wouldn't doing something like this solve the problem:

Template.outer.opts = {

  mainclass: function() { return Session.get('foo') ? 'foo' : ''; }

}

 

Yes, that would work great. :)

That way the Session.get call won't get made until the {{mainclass}} call is actually made and there's been a chance to set up a new 'helper' dependency context?

Appreciate you letting me in on your thinking regarding all this..

Cheers,

Tom

[1] It all feels a lot like functional programming doesn't it? The two solutions are equivalent if you can assume that calling class_helper_2 is purely functional with no side-effects;

It does indeed. :)  Basically the developer plumbs the data pipelines, and we worry about the execution, which is often how functional programming feels.  The concept of a "pure function" for our purposes is just a function which you don't know exactly when and how often it will be called, so side effects aren't that useful or natural (except for debugging).  Still, we try to avoid surprises, so if you happen to be watching what is called when, the result is about what you'd expect.

-- David

[Gezim Hoxha]

Hi David, everyone,

I'm wondering if you could explain what's happening with my situation.

My template looks like this:

<template name="task_list">

<form id="tasks" class="form-vertical">

{{#each tasks}}

{{> task}}

{{/each}}

</form>

</template>

<template name="task">

<label class="checkbox {{completed_class}}" data-priority="{{priority}}" id="label-{{_id}}">

<input type="checkbox" id="{{_id}}" {{{completed}}} /> {{text}} {{#if assignee}}{{#if show_assignee}}Assigned to: {{assignee.name}}{{/if}}{{/if}}<i class="icon-list"></i>

</label>

</template>

Template.task_list.tasks returns a collection of Tasks.

I'm trying to allow sorting of this list using drag and drop (jQuery sortable). Basically, in the event that's fired when the user drops the item, I update the collection to save this new order.

What happens is that when I drag down, nothing unusual happens, it shows the item in the new position. However, when they move an item up, it duplicates the item for some reason.

I found that when there is no duplication, the old DOM element is gone and is not the same as the new one. However, when duplication happens, the old element is there and Meteor adds another one like it.

Could someone shed some light on this?

Thanks,

-Gezim

To unsubscribe from this group, send email to meteor-talk+unsubscribe@googlegroups.com.

[David Greenspan]

Hi Gezim,

It probably depends heavily on the implementation of jQuery sortable.  There isn't a drop-in solution for sortable reactive lists in Meteor right now.  Sorry I can't be more helpful.

-- David

To view this discussion on the web visit https://groups.google.com/d/msg/meteor-talk/-/fUiCswV-v7gJ.

To post to this group, send email to meteo...@googlegroups.com.

To unsubscribe from this group, send email to meteor-talk...@googlegroups.com.