Server-side VDOM for Lift

[Joe Barnes]

Hello Lifters!

A good friend of mine is the founder of pakyow and a recent conversation gave me an idea to try server-side VDOM calculations for Lift.  The idea is that any updates to your page's DOM state (such as an ajax call or an event that would trigger a comet update) perhaps could be calculated by re-rendering the template server-side, and sending a VDOM-style diff to the client which would the patch your existing DOM for you.

As an example, I have begun an implementation of the Ubiquitous Chat App using this approach.  (Currently there are no comet async updates, so don't try that).  I implemented it using a Lift ajax form.  Rather than having to write code that adds an li element to my ul list of chat messages, I simply return UpdateDOM here:

def submit = {
  var msg = ""

  def onAjax():JsCmd = {
    append(msg)

    new UpdateDOM & SetValById("chat-in", "")
  }

  "name=in" #> (SHtml.text(msg, msg = _, "id" -> "chat-in") ++ SHtml.hidden(onAjax))
}

As you can see, I even combine the UpdateDOM object with a SetValById to clear the text field.

My Lift branch that implements this is extremely primitive. I've only implemented enough code to do a naive VDOM diff calculation that will find my added li elements (or any added element for that matter).  It performs the diff over the entire body tag. and finds my new elements, and the javascript will add the new li  and its child text node.

This idea poses a few challenges.  Firstly is VDOM diffing and patching.  The good news here is this is a very popular approach today, so there are several implementations available for our study and mimicking.  The other challenge is the VDOM diffs generally need to be applied in order, so any server communication issues will need to be handled so that the order isn't broken.  We also have to think about how client-side DOM changes will interact (I would argue that in general you wouldn't want to do such a thing, but perhaps we can envision a way to make this work well).

I feel that in principle this is a really good idea for Lift.  Even though you all know I've enjoyed integrating Angular 1 with Lift via lift-ng, I think that client-side rendering is a bit overused these days.  I love the idea of being able to stick with Scala and Lift's templating to handle not only my initial page loads, but page updates.  My discussion with Bryan Powell of the pakyow framework has me convinced that server-side rendering also has the advantage of building a better user experience.  Initially I objected to this because of the round-trip to the server delay, but he makes a great point that it is better than the client pretending the change has been made (what if server communication fails?).  With this approach, the server remains the source of truth and the user is never tricked into thinking something happened which has not actually happened yet.  (He talks at length about user trust over user experience)  It seems this could not only be a powerful developer experience, we'll be driving towards building better webapps than the current heavy-client trend.

I would love to hear everyone's thoughts and reactions to these ideas!

Joe

[Antonio Salazar Cardozo]

This sounds awesome to me! It's worth noting that the comet mechanism already

ensures proper ordering… so as long as we feed through that for any particular

implementation of VDOM we should get guaranteed ordering out-of-the-box. So,

this might be something worth building into or alongside round-trip streaming

promises somehow. idMemoize may also be a good starting point. So many options!

Super exciting :)

Thanks,

Antonio

[Joe Barnes]

Cool, good to know that! What I've done to get it off the ground uses Ajax and a RequestVar. Maybe it makes more sense to maintain this state with a per-page comet so all updates are synchronized and ordered. Although, I'd still like the state to be in a RequestVar so that it'll be able to take advantage of the fail-over clustering work I've been doing. 

Hmm... That also means I need a way to slap a comet on every page. I'll have to see how round trips and such accomplish that today. 

Joe

Sent from my iPhone

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to a topic in the Google Groups "Lift" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/liftweb/2B2fzK1touc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to liftweb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matt Farmer]

I’m also +1 on this idea.

Much of the time I’m writing client side code it’s something that idMemoize wouldn’t handle efficiently

because it resets the entire block. I essentially end up using comets to push DOM events to do delta

based changes on the client. Which is exactly what this code would do without me having to write

any JavaScript.

I think I have a few coworkers who have dabbled in some of this kind of stuff before - so I’ll post it in

our slack thread to see if they’re interested in the conversation.

—
Matt Farmer | Blog | Twitter
GPG: CD57 2E26 F60C 0A61 E6D8  FC72 4493 8917 D667 4D07

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---

You received this message because you are subscribed to the Google Groups "Lift" group.
To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.

[Matt Farmer]

> That also means I need a way to slap a comet on every page

I think S.appendJs can do this. At the very least idMemoize does on its own somehow. :)

—
Matt Farmer | Blog | Twitter
GPG: CD57 2E26 F60C 0A61 E6D8  FC72 4493 8917 D667 4D07

You received this message because you are subscribed to the Google Groups "Lift" group.

To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.

[Joe Barnes]

I'll take a looks. It's really more of how to figure out the developer intends to utilize this mechanism without it needing to be explicit. 

Joe

Sent from my iPhone

[Joe Barnes]

Awesome, Matt! Hopefully they'll have time and interest to contribute!

Joe

Sent from my iPhone

You received this message because you are subscribed to a topic in the Google Groups "Lift" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/liftweb/2B2fzK1touc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to liftweb+u...@googlegroups.com.

[Joe Barnes]

Check it out... I'm using HtmlUnit to test the javascript that updates the DOM with the VDom transforms: https://github.com/lift/framework/blob/joe-vdom/web/webkit/src/test/scala/net/liftweb/http/UpdateDOMSpec.scala

Once I get the three basic transforms working (add, remove, reorder), I'm going to look into writing some scalacheck tests for this.  I'm curious if I could take it even further and use the approach for performance testing too, since I certainly want to make sure this Lift feature is highly performant.  One thing that's fun about performance in our context, is we're pretty unique in that we'll be diffing on the server, serializing the transforms on the wire, and applying them on the client.  My guess is that minimizing what we need to serialize and what the client needs to do to mutate the DOM will yield the best results, suggesting that we may be able to have a more sophisticated diffing algorithm than is currently available in the wild today.

On that note... virtual-dom algorithm considerations are complicated. It's going to be a lot of fun. :)

Joe

[Joe Barnes]

I wanted to update the community on the rather slow progress of my favorite idea in the world right now. :)

Britany and I now have our branch of Lift at a point that we pass 100% of our generative property tests which inject a single mutation into our sample DOM. When we crank up to multiple (even if only 2), we fail after a good handful of attempts. She has some ideas on how to change our algorithm to make it cleaner (friendly reminder: VDOM diff is really hard lol) that might help. I need to review the failures closely and make sure we aren't generating unreasonable mutations that aren't worth expecting/supporting.

About two months ago we had a successful demo of Lift with this feature. It was my favorite demo of Lift yet because it was a split demo with a designer. He spent the first half demoing how to use a design tool he had been playing with. We switched laptops, I downloaded his HTML, and live coded and cussed our way into a working chat app with no JS written by us. Was beautiful.

Next month Britany will be presenting at DevSpace which is local to us in north Alabama. Then in November, she got accepted to present at Scala By the Bay, but unfortunately her schedule is just too busy for her to make the trip. Fortunately, I've been able to reach out to Alexey and offer to fill in for her unless they wanted to replace the talk. He accepted my offer, so the show will go on. I'm not sure if they'll update the schedule to my mug on there, so I may need to spray my hair with silver glitter to look like Britany. Either way, I'll have 20 minutes to soapbox about how client-side rendering is ruining the web and then present our alternative which solves world hunger in the process.

We're busy and slow moving, but we're gonna have something great in store for Lift 3.1!!

Joe

[Antonio Salazar Cardozo]

Awesome! Super excited for this :)

Antonio

[Peter Petersson]

Thanks for the update Joe really exiting stuff.

Peter

[Henrik Härkönen]

Hi!

I have totally missed this VDOM thing. I actually read your post earlier, but I didn't understand it at first, but now when I read it again, I have to say this is super cool!

When you say VDOM diff is hard, I guess you mostly mean logically? How about computationally, like load-wise?

-Henrik

[Joe Barnes]

Henrik,

I mean logically. I'm not concerning myself with the load at this time. Once we get the logic right, then if we feel so compelled when can write some performance tests. But for the most part, I don't worry too much about it. Conceptually we have some nice advantages over mutation based VDOM implementations such as React because with our immutable objects, a referentially-equal comparison guarantees we needn't examine that subtree. Not that computation is cheap, but at least this is a per-session cost so it is scalable. I would be concerned if it impacted the ability to scale, but I see no reason that would be the case.

Let me know if you have more questions!

Thanks,

Joe

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---

You received this message because you are subscribed to a topic in the Google Groups "Lift" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/liftweb/2B2fzK1touc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to liftweb+unsubscribe@googlegroups.com.

[Henrik Härkönen]

Hi Joe!

This VDOM for Lift sounds better and better every time I learn something new about it. :) Awesome!

-Henrik

To unsubscribe from this group and all its topics, send an email to liftweb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to a topic in the Google Groups "Lift" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/liftweb/2B2fzK1touc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to liftweb+u...@googlegroups.com.

[Alberto Colombo]

Hello,

how did the presentation go? Was it videoed, by any chance, or are slides available?

Anyway, I'm following this thread closely. You mention having it available in 3.1, but 3.0 only came out a few days ago. Is there some kind of roadmap? 

Server-side React-style programming in Scala, this is so awesome!

Alb

[Joe Barnes]

Hey Alb,

Thanks for the interest! Yeah, the talk was recorded. I'll certainly reply back on this thread when it's ready. It'll be on youtube. 

If we get something working that we're confident is solid enough for inclusion in Lift 3.1, we'll merge it into master and it'll be available in whatever next milestone build comes out. It's certainly in a crude alpha proof-of-concept state at the moment.

I'd love to finish it out if it looks like something the community is interested in, so please keep reminding me that it's a good idea. My main concern at this time is that I'm not currently working for an employer who utilizes Lift. If I finish this, I need to find a way to utilize it so I eat my own dog food. I found and fixed some nasty bugs in lift-ng because I was using it every day.

So as it stands, it's something I'll work on when I get a chance to, which at the moment is low on my priority list. Part of my motivation for my talk at SBTB was to poll the community if it is in fact something folks would use, or am I just working really hard on an idea no one will use and I still have to maintain.

Joe

 

To unsubscribe from this group and all its topics, send an email to liftweb+unsubscribe@googlegroups.com.

[Antonio Salazar Cardozo]

It may be worth opening a PR and doing a custom snapshot build... Not sure

what that would look like off the top of my head.

Another option is developing it as something that can be added on to Lift---if

it hooks deep into internals, perhaps looking at how we can modularize those

so you can hook into them in a module would a path we could pursue in 3.1.

Then once it's ready and/or has someone who's using it day-to-day, it can be

pulled into mainline.

Just throwing some ideas out there, as I'm also curious to see where this little

yellow brick road goes :)

Thanks,

Antonio

[Joe Barnes]

Thanks for those ideas, Antonio. I'll take a look back over the code next chance I get. I'm certainly not shy when it comes to building a Lift module. :)

But yeah, off the top of my head I don't recall how deeply into Lift I had to reach to make it possible. The cool thing is most of the internals is just reusing stuff we had all along.

Joe