Lift Delegation

[Matt Farmer]

Hey all,

I wanted to start a conversation on something I've been turning over in my head for awhile called Lift Delegation. It's essentially a reverse-proxy type setup to allow you to split up large lift applications into multiple smaller ones but still present it to the user as if its one cohesive application.

I did the full write-up on my blog because it is long to try and explain and I find that trying to compose things that are that long really obnoxious on the list. Likewise, I'm hoping that it'll be more accessible toward other people in the community who follow me on Twitter and might find the idea interesting. Reading long things in email form is also something I find quite annoying...

Anyway, the full write up is here: http://farmdawgnation.com/blog/2015/3/21/lift-delegation-proposal

Since this is a Lift issue, I'm going to link to this thread as soon as I've posted it so anyone who has comments about it should remark here. That should help keep the conversation about the idea itself on the ML, which I think is what we all want. :)

Would love to hear your thoughts on the idea after you've had a chance to read through it.

Thanks,

Matt

[Diego Medina]

Hi Matt,

One of the points that you outline on your blog is about having the same look and feel across all delegate apps, which nowadays is not happening. I would say that this is already possible now, and when two or more "parts" of the system don't look the same, it is more of a developer's lack of care, than a technical issue.

As an example where I'm actually involved, I have been working with a client for about a year an a half on this one Lift app, it essentially is 11-12 separate modules, in your proposal, they could be converted into 11 or 12 delegates. We originally had about 5 modules, and each of them were a separate web app, running on their own port, and nginx on the front would send requests to the right app based on the subdomain.

This worked, and allowed us to deploy a single module update to production without messing with the other apps, but in practice, many changes to one module affect other modules, one of the more common cases is when you change something on the database. And we were able to keep the same look and feel on all modules just fine, by keeping the surrounding templates, and css files in a main location.

A negative point on this setup was that by having 4 or 5 jars running on the server, we had to dedicate a lot of ram to the web servers, and many times it was wasted ram, one module may need more than the other.

Development was pretty painful, where you had to run two or more instances of sbt if your changes were targeting two modules

Eventually we moved to a single jar, so all modules are on the same webapp, same port, development is a lot simpler now, and we still have the same look and feel across all modules. And now we just have one jar to dedicate as much ram as we need to, so the ops part is simpler.

===========
Now, to specific questions about your proposal:

Let's say user A goes to http://mattsawesomeapp.com

The browser hits the master web app, does the master make an http request to one of the delegates and returns whatever html the delegate gave?
Does the master app need any knowledge of how the delegates respond to certain requests?

Because if it does, then I cannot just deploy one of the delegates with new code, I need to also deploy a new master, and if one delegate needs code from another delegate, then I need to keep track of the graph of dependencies.

> The result should have already been processed through the rendering system of the masterSo which lift app talks to the database, the master or the delegates?

> The DelegateSessionVar will probably exhibit the same lifetime as a RequestVar so that it doesn't constantly have to go fetch the value

Typo ordo you mean DelegateRequestVar ?

However, I did have a somewhat similar idea of a reverse proxy with knowledge of the backend servers, but applied to REST APIs to we can have secure rest apis with random parameter names/URI, so your rest apis can come and go as they please and it is just this one proxy that keeps the map of random strings => rest api end points, but I don't want to hijack your email with it :)

Let's start with these points and I'll ask more as we go along.

Thanks

Diego
Sent from my cell

--
--
Lift, the simply functional web framework: http://liftweb.net
Code: http://github.com/lift
Discussion: http://groups.google.com/group/liftweb
Stuck? Help us help you: https://www.assembla.com/wiki/show/liftweb/Posting_example_code

---
You received this message because you are subscribed to the Google Groups "Lift" group.
To unsubscribe from this group and stop receiving emails from it, send an email to liftweb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matt Farmer]

Regarding the presentation of templates, I don’t entirely agree. Sure, you can rig it such that things are consistent by practice. But I do think that this approach might encourage and enable sharing of that common content without duplication or having to store it in a common library that you then have to version bump in every module every time it changes. Or, worse, result in developers sticking such content in a common folder that all of the components are just expected to have access to – which breaks (or at least cripples) the actual modularity I’m seeking.

In my ideal world, I want to be able to have delegates move around within an EC2 VPC and as be able to come up and down at will. As long as their Elastic IP follows them, it would “just work” because the entire contract for common resources is through an API and changes to those common resources wouldn’t inherently require a new release of the delegate.

Now, to the problems that you mentioned with regard to development and operations: These are all things that are true of the microservice model in general. As with microservices, you must have well defined contracts or making any changes sucks (I would argue sharing database schemas directly would be a no-no for example). As with microservices, you have to figure out how to arrange and allocate resources to more parts on a server. As with microservices, local development can become more problematic. In spite of those issues, this is a model that engineers sometimes find beneficial.

I’m merely seeking to give people a “Lifty” way to do that in a large web application when they decide they need it. Something that provides a secure, uniform way for state/resource sharing between a delegate and a master. It’s certainly not something for the faint of heart. I’ll probably never touch it on my side projects. But I do think this provides a big enough benefit to the people that need it (of which my employer is one) that it’s worth having in the Framework.

Phew! Now on to your questions!

The browser hits the master web app, does the master make an http request to one of the delegates and returns whatever html the delegate gave?

Yes.

Does the master app need any knowledge of how the delegates respond to certain requests?

Because if it does, then I cannot just deploy one of the delegates with new code, I need to also deploy a new master, and if one delegate needs code from another delegate, then I need to keep track of the graph of dependencies.

No, the master is a dumb proxy in this arrangement. It doesn’t care how the delegate responds except in the case the delegate isn’t there, in which case the master would return a 503. And the user code can already customize how that would appear to the end-user through an existing LiftRule.

So which lift app talks to the database, the master or the delegates? 

They both can. As with microservices, the engineering team has to define contracts that make sense for what they’re building but cross-polination should be kept to a minimum. So, the delegates might be required to have some concept of a User, right? And the engineering team has to figure out how to handle the fact that the specification for a User could change and how that might affect things. But if you’re building something and find that you’re sharing more than one or two model objects directly across modules then I’d argue you either need to abandon using this architecture or rethink your modeling.

Typo ordo you mean DelegateRequestVar ?

No, it would be DelegateSessionVar. A DelegateSessionVar should only exist on the delegate application, but should retain its value for a Request. At the beginning of each new request it’s initializer should fetch the current value from the master node.

—
Matt Farmer | Blog | Twitter

[Diego Medina]

On Sun, Mar 22, 2015 at 12:03 AM, Matt Farmer <ma...@frmr.me> wrote:

Regarding the presentation of templates, I don’t entirely agree. Sure, you can rig it such that things are consistent by practice. But I do think that this approach might encourage and enable sharing of that common content without duplication or having to store it in a common library that you then have to version bump in every module every time it changes. Or, worse, result in developers sticking such content in a common folder that all of the components are just expected to have access to – which breaks (or at least cripples) the actual modularity I’m seeking.

Are you talking about separate (git|etc) repositories for master, delegate1, delegateN ?

 

In my ideal world, I want to be able to have delegates move around within an EC2 VPC and as be able to come up and down at will. As long as their Elastic IP follows them, it would “just work” because the entire contract for common resources is through an API and changes to those common resources wouldn’t inherently require a new release of the delegate.

Is this along the lines of what Joe was talking about on another thread? where even the server side binded functions will still work? If so, are you saying the function name => function map stays on the master node?

 

Now, to the problems that you mentioned with regard to development and operations: These are all things that are true of the microservice model in general. As with microservices, you must have well defined contracts or making any changes sucks (I would argue sharing database schemas directly would be a no-no for example). As with microservices, you have to figure out how to arrange and allocate resources to more parts on a server. As with microservices, local development can become more problematic. In spite of those issues, this is a model that engineers sometimes find beneficial.

Not to go too far off topic but this is actually something I want to find out, if all those who use the microservice idea just live with the development pain or if there is a "nice" way to develop them, but again, may be off topic here.

 

I’m merely seeking to give people a “Lifty” way to do that in a large web application when they decide they need it. Something that provides a secure, uniform way for state/resource sharing between a delegate and a master. It’s certainly not something for the faint of heart. I’ll probably never touch it on my side projects. But I do think this provides a big enough benefit to the people that need it (of which my employer is one) that it’s worth having in the Framework.

Phew! Now on to your questions!

The browser hits the master web app, does the master make an http request to one of the delegates and returns whatever html the delegate gave?

Yes.

Does the master app need any knowledge of how the delegates respond to certain requests?

Because if it does, then I cannot just deploy one of the delegates with new code, I need to also deploy a new master, and if one delegate needs code from another delegate, then I need to keep track of the graph of dependencies.

No, the master is a dumb proxy in this arrangement. It doesn’t care how the delegate responds except in the case the delegate isn’t there, in which case the master would return a 503. And the user code can already customize how that would appear to the end-user through an existing LiftRule.

Ok, so if in this case the master is a dumb proxy, in which case is the master "smart", I think I'm really missing the point of what you are trying to say. 

So which lift app talks to the database, the master or the delegates? 

They both can. As with microservices, the engineering team has to define contracts that make sense for what they’re building but cross-polination should be kept to a minimum. So, the delegates might be required to have some concept of a User, right? And the engineering team has to figure out how to handle the fact that the specification for a User could change and how that might affect things. But if you’re building something and find that you’re sharing more than one or two model objects directly across modules then I’d argue you either need to abandon using this architecture or rethink your modeling.

Typo ordo you mean DelegateRequestVar ?

No, it would be DelegateSessionVar. A DelegateSessionVar should only exist on the delegate application, but should retain its value for a Request. At the beginning of each new request it’s initializer should fetch the current value from the master node.

Ah ok, and then the delegate has to somehow tell the master of the delegate changed the value of the DelegateSessionVar ?

Also, DelegateSessionVar are individual per delegate, right? or can I expect to set a DelegateSessionVar on server! and read it on server2 ?

 

Thanks.

--

Diego Medina
Lift/Scala consultant
di...@fmpwizard.com
http://fmpwizard.telegr.am

[Matt Farmer]

Are you talking about separate (git|etc) repositories for master, delegate1, delegateN ?

For the specific scenario I’m thinking about? Yes. But at the very least separate sbt projects for each.

Is this along the lines of what Joe was talking about on another thread? where even the server side binded functions will still work? If so, are you saying the function name => function map stays on the master node?

No, as stated below, the master is “dumb.” It has no knowledge of function information from any other modules. Likewise, delegates have no knowledge of function numbers from a master. In the event a node goes down and comes back up session state would be still be lost as it is today.

Not to go too far off topic but this is actually something I want to find out, if all those who use the microservice idea just live with the development pain or if there is a "nice" way to develop them, but again, may be off topic here.

I haven’t dug enough into that world to know yet, to be honest.

Ok, so if in this case the master is a dumb proxy, in which case is the master "smart", I think I'm really missing the point of what you are trying to say. 

The point is that we’re providing a way to share session state with the delegates. The master is a dumb proxy, but its sitemap will drive which delegate gets accessed for what URL patterns. It could also, foreseeably, handle access control into the module itself using its LocParams. I haven’t played too much with that yet, tbh.

A “dumb proxy” isn’t investigating the content it’s relaying. A “smart proxy” would. The master doesn’t care what the delegate sends back. It doesn’t investigate it. It doesn’t cache it. It just forwards the response. This architecture is certainly smart in other ways. Not that I’m biased. ;)

Ah ok, and then the delegate has to somehow tell the master of the delegate changed the value of the DelegateSessionVar ?

Also, DelegateSessionVar are individual per delegate, right? or can I expect to set a DelegateSessionVar on server! and read it on server2 ?

Ah, sorry, this maybe wasn’t clear.

DelegateSessionVars are read only. The only way they obtain a value is by reading it from the master. That’s an important detail. So this is for storing things like “the current user’s ID” right? The master would handle authentication, store the value in a MasterSessionVar, and then all DelegateSessionVars in the delegate apps that have the same name would have the same value.

Whatever you store in one of these guys would have to be JSON serializable so it can be sent over the wire.

—
Matt Farmer | Blog | Twitter

[Diego Medina]

On Sun, Mar 22, 2015 at 12:38 AM, Matt Farmer <ma...@frmr.me> wrote:

Are you talking about separate (git|etc) repositories for master, delegate1, delegateN ?

For the specific scenario I’m thinking about? Yes. But at the very least separate sbt projects for each.

In this case I think the example project I mentioned isn't along the lines of what you are talking about here. I know you gave an example on your post, but just so I have a better idea. If Google were to adopt this architecture, would it be along the lines of, Google Search is one delegate and Gmail is another ? or more like, Displaying emails and sending emails in Gmail is one delegate, and the contact part of Gmail is another delegate?

What I mean is, Gmail and Search are clearly separated, displaying emails, managing the contact that appear on gmail are closer related, but I can see how you can have an API in between the two, so getting the list of contact to autocomplete can query a REST api from the contacts delegate, instead of doing a search on their database.

 

Is this along the lines of what Joe was talking about on another thread? where even the server side binded functions will still work? If so, are you saying the function name => function map stays on the master node?

No, as stated below, the master is “dumb.” It has no knowledge of function information from any other modules. Likewise, delegates have no knowledge of function numbers from a master. In the event a node goes down and comes back up session state would be still be lost as it is today.

ok
 

Not to go too far off topic but this is actually something I want to find out, if all those who use the microservice idea just live with the development pain or if there is a "nice" way to develop them, but again, may be off topic here.

I haven’t dug enough into that world to know yet, to be honest.

Ok, so if in this case the master is a dumb proxy, in which case is the master "smart", I think I'm really missing the point of what you are trying to say. 

The point is that we’re providing a way to share session state with the delegates. The master is a dumb proxy, but its sitemap will drive which delegate gets accessed for what URL patterns. It could also, foreseeably, handle access control into the module itself using its LocParams. I haven’t played too much with that yet, tbh.

Can't nginx do this, where url patterns sends traffic to certain server? (Minus the LocParam)

 

A “dumb proxy” isn’t investigating the content it’s relaying. A “smart proxy” would. The master doesn’t care what the delegate sends back. It doesn’t investigate it. It doesn’t cache it. It just forwards the response. This architecture is certainly smart in other ways. Not that I’m biased. ;)

I didn't mean to call the architecture dumb, sorry if it came out like that. I'm trying to see *what* the master does to help.

Ah ok, and then the delegate has to somehow tell the master of the delegate changed the value of the DelegateSessionVar ?

Also, DelegateSessionVar are individual per delegate, right? or can I expect to set a DelegateSessionVar on server! and read it on server2 ?

Ah, sorry, this maybe wasn’t clear.

DelegateSessionVars are read only. The only way they obtain a value is by reading it from the master. That’s an important detail. So this is for storing things like “the current user’s ID” right? The master would handle authentication, store the value in a MasterSessionVar, and then all DelegateSessionVars in the delegate apps that have the same name would have the same value.

Whatever you store in one of these guys would have to be JSON serializable so it can be sent over the wire.

Ah, I missed the read-only part

OK, let's see if I'm getting closer to what you are proposing.

* The master node would act like reverse proxy in the case where based on some url pattern (SiteMap or something else), it will know which delegate to query, to satisfy the http request from the user's browser.

* The master also acts like a datastore, where you can store things like the current userID, maybe locale, and it would also keep some html templates

Is this more or less accurate?

[Matt Farmer]

If Google were to adopt this architecture, would it be along the lines of, Google Search is one delegate and Gmail is another ? or more like, Displaying emails and sending emails in Gmail is one delegate, and the contact part of Gmail is another delegate?

Email and contacts could be closely related enough in my mind to allow there to be a master/delegate situation. Though in the case of GMail the email is more important so perhaps the entire email portion is the master and the contact management is a delegate of that master.

The rub with that example is that there is a higher level of data model cross-pollination between the two, so it’s not perfect, but it would work.

Can't nginx do this, where url patterns sends traffic to certain server? (Minus the LocParam)

Sure, but nginx can’t:

• Use application state and configuration to determine the IPs of the delegates (note that all the LiftRules additions are ()=>Box[String])
• Provide the application developer nice hooks for error recovery in the event a delegate disappears. It would be forseeable, for example, on a future iteration of this to enable SiteMap to be aware that a particular delegate isn’t reachable right now and hide the menu item. Or present a page to the user with the applications main menu (bound according to the css selectors and the current user’s session as it normally would be) but with a message saying that service isn’t available now.
• Enable state and resource sharing by correlating the master’s session with the delegate’s.

Ah, I missed the read-only part

OK, let's see if I'm getting closer to what you are proposing. 

* The master node would act like reverse proxy in the case where based on some url pattern (SiteMap or something else), it will know which delegate to query, to satisfy the http request from the user's browser.

* The master also acts like a datastore, where you can store things like the current userID, maybe locale, and it would also keep some html templates

Is this more or less accurate?

Sure.

—
Matt Farmer | Blog | Twitter

[Andreas Voeth]

Hello,

before i start writing down my thoughts let me say that i never worked
on an lift or even web application which should have been an
"application system". So maybe some thoughts i have are not really the
way one should go. Nevertheless you asked for thoughts:

On 21.03.2015 21:13, Matt Farmer wrote:
> Would love to hear your thoughts on the idea after you've had a chance
> to read through it.

As i understood it, the main idea is about delegating whole pages to
other lift apps. They are identified by the URL and mapped in the
Sitemap, right?

But in this case the delegate would then need to know the main template
(which should be part of the master) or would need to ask the master for
template data and increase system internal communication more than
needed (as the master would send the default page template to the
delegate, which would fill it with content and send everything back).

Wouldn't it be more elegant to only delegate parts of the page which
rely on the specific delegate? My idea would be to delegate only
snippets calls:

- master parses template
- there is a snippet not found in the master, so it will be delegated
(here is some kind of mapping needed)
- delegation could be done asynchronous, the delegate only needs the
html node the snippet is inside.
- the master could then replace the snippet call with the answer from
the delegate

Of course this would mean more work done by the master, as it would be
needed to parse the answers again to remap function calls (as the random
id generated by the delegated snippet needs to be mapped to this
delegate later).

Some of this work could be done by the delegate: The master could give
each delegate for each snippet a random prefix and the delegate could
tell the master the used function names => parsing the answer again
wouldn't be necessary as the combined function names would be unique.

I would be interested on your thoughts on this alternative approach to
delegating. There are for sure drawbacks in this solution, as the master
would need to be much more than a simple reverse proxy. But maybe this
would be ok and it would be possible to render pages where different
parts are generated from different delegates.

Thanks,
Andreas Voeth

[Matt Farmer]

As i understood it, the main idea is about delegating whole pages to other lift apps. They are identified by the URL and mapped in the Sitemap, right?

Delegating entire groups of pages, really, but that’s the gist. So at work we have entire sub-applications that have no URL and almost no data overlap between other sub-applications within the larger application. That’s really the scenario I’m thinking of.

Wouldn't it be more elegant to only delegate parts of the page which rely on the specific delegate? My idea would be to delegate only snippets calls:

- master parses template
- there is a snippet not found in the master, so it will be delegated (here is some kind of mapping needed)
- delegation could be done asynchronous, the delegate only needs the html node the snippet is inside.
- the master could then replace the snippet call with the answer from the delegate

This wouldn’t work because it would mean the html structure for delegate pages would be stored in the master application. That breaks the ability to deploy delegates independent of the master.

Yes, under my architecture the delegate will have to know a few things about what is going on in the master application. It’ll have to have a few names so its templates render correctly and it can access the right session information, but I very specifically wanted all things specific to a single delegate to remain within that delegate. So, HTML structure specific to the delegates pages, delegate-specific CSS files, snippet implementations, comets, actors, database models, etc.

The purpose here isn’t really spreading around computational complexity to different nodes, which is something your solution would be good for. Though my proposal certainly can do that, I’m more concerned with building an architecture where different modules are silo’d from each other and that breaking those silos requires very intentional effort. What I’m finding is that once an application reaches a certain level of complexity, this becomes beneficial.

So, this means that if Bob introduces a bug into Module B v2 (a delegate) that causes the JVM to start throwing OutOfMemoryErrors after a certain point, the master and all other delegates in the application remain running. That module just becomes unreachable until engineering and operations can intervene. In addition to that, engineering and operations can release an old JAR (let’s say Module B v1) while they’re trying to debug the issue because the contract between delegates and masters is strong enough to withstand minor version alterations. They can then release the fixed Module B whenever they want without affecting the availability of the application as a whole or any of the other delegates.

Thanks for the feedback!

—
Matt Farmer | Blog | Twitter