How to write a plugin or other solution to combine multiple upstream responses

[Aaron Trevena]

Hi All,

It looks like we need a nice way of combining responses from multiple
upstream sources into a single downstream response - a simple
concatonation of header, body, footer - maybe some very simple ESI or
transforms.

Would this be very do-able using Perlbal - ESI seems overkill and less
scalable than combining responses from upstream app-servers into a
single page.

I thought it might be possible in a similar way to the mogilefs
plugins available

Any thoughts or suggestions?

A

--
Aaron J Trevena, BSc Hons
http://www.aarontrevena.co.uk
LAMP System Integration, Development and Consulting

[Ask Bjørn Hansen]

On Aug 24, 2012, at 7:29, Aaron Trevena <aaron....@gmail.com> wrote:

> It looks like we need a nice way of combining responses from multiple
> upstream sources into a single downstream response - a simple
> concatonation of header, body, footer - maybe some very simple ESI or
> transforms.

I'm not sure if this is easy to do with a plugin ("fetch three requsts, return them to the user in this order").

If not, then this seems like either you need to just use ESI (use Varnish) or you need something more custom (make a node.js or Go server).


Ask

[Aaron Trevena]

On 25 August 2012 01:30, Ask Bjørn Hansen <a...@develooper.com> wrote:
>
> On Aug 24, 2012, at 7:29, Aaron Trevena <aaron....@gmail.com> wrote:
>
>> It looks like we need a nice way of combining responses from multiple
>> upstream sources into a single downstream response - a simple
>> concatonation of header, body, footer - maybe some very simple ESI or
>> transforms.
>
> I'm not sure if this is easy to do with a plugin ("fetch three requsts, return them to the user in this order").

It's more about assembling a page than getting them in order,
essentially it would be modifying one response by fetching 2 more and
adding them, either in parallel or consecutively.

> If not, then this seems like either you need to just use ESI (use Varnish) or you need something more custom (make a node.js or Go server).

Yes, looking at ESI with varnish is an option, but not sure it's going
to be fast or flexible enough - and writing bespoke node.js code
sounds like a world of pain given we have precisely 0 backend
javascript knowledge and thus an enormous learning curve (I don't
consider it to be some sort of magic pixie dust that does stuff
faster, and I'm concious we could invest time in that route and end up
with something no faster or easier to maintain than if we'd hacked
together something in perlbal or mojolicious)

Oh well.

My preferred option is to embed some of the dynamic content using
javascript in the client, that enables caching large ammounts of less
frequently changing or non-user specific content, and has worked
before nicely.

Cheers,

[Antti Vähäkotamäki]

It's more about assembling a page than getting them in order,
essentially it would be modifying one response by fetching 2 more and
adding them, either in parallel or consecutively.

> If not, then this seems like either you need to just use ESI (use Varnish) or you need something more custom (make a node.js or Go server).

Yes, looking at ESI with varnish is an option, but not sure it's going
to be fast or flexible enough - and writing bespoke node.js code
sounds like a world of pain given we have precisely 0 backend
javascript knowledge and thus an enormous learning curve

 

I understand why Ask suggested you use one of the mentioned event driven technologies. However your use case looks like it could most probably be solved with any server side component and a sufficient number of machines running multiple threads. You can put a couple of Perlbal instances behind an IP load balancer, configure a pool of backend servers for each Perlbal and choose the technology for the backend servers freely according to your skill sets.

For example simply a prefork Apache running a simple mod_php/mod_perl script would probably do fine for fetching the parts sequentially and serving them together. This would most arguably require you to invest a bit more in the hardware than you would have to with an event driven technology but you would get additional redundancy and ease of scaling with the same cost. And if at some point you acquire some talent who can whip up and maintain a node.js (or any other event driven) server, you can speed things up and free resources with concurrent and asynchronous fetching - the node.js server will be a drop in replacement for the old Apaches.

If you ever happen to end up needing node.js, I have absolutely loved this module for concurrent fetches: https://github.com/caolan/async

My preferred option is to embed some of the dynamic content using
javascript in the client, that enables caching large ammounts of less
frequently changing or non-user specific content, and has worked
before nicely.

What is the reason why you would like to change this? To me this looks like a valid approach. 

But if you move to the server side joining, memcached does a marvelous job of being both really simple to use and really fast for caching stuff.

- Antti Vähäkotamäki

[Aaron Trevena]

I think this could be a quick/easy win :

http://mojolicio.us/perldoc/Mojolicious/Guides/Cookbook#Backend_web_services

The recipe is pretty close to what we're looking for.. definately
worth a try, and doesn't mean mixing in yet more unfamiliar (to both
developers and sysadmin) tech.

> If you ever happen to end up needing node.js, I have absolutely loved this
> module for concurrent fetches: https://github.com/caolan/async

Handy - noted.

>> My preferred option is to embed some of the dynamic content using
>> javascript in the client, that enables caching large ammounts of less
>> frequently changing or non-user specific content, and has worked
>> before nicely.
>
>
> What is the reason why you would like to change this? To me this looks like
> a valid approach.

I think we may need to retain more control server-side than that
allows - depends, may work well for part of the solution if not all of
it.

> But if you move to the server side joining, memcached does a marvelous job
> of being both really simple to use and really fast for caching stuff.

Yes, I definately see at least a couple of roles for memcached in the
big picture here,

regards,

[Leo Lapworth]

On 25 August 2012 01:30, Ask Bjørn Hansen <a...@develooper.com> wrote:

Or if you are running Perl:

https://metacpan.org/module/Plack::Middleware::ESI ( doesn't seem to be Async... but shouldn't be a hard patch )

Personally I'd use that for development, then Varnish for production

But I've not actually tried ESI yet as I've never found a case I needed it (most stuff I can serve out of memcached)

Good luck

Leo