Smart Batching

[James Bunch]

Hi All,

Has anyone used Akka to implement something akin to the 'Smart Batching' pattern that Martin Thompson describes in this blog post? 

http://mechanical-sympathy.blogspot.co.uk/2011/10/smart-batching.html

I've seen this topic: https://groups.google.com/forum/#!topic/akka-user/todrna9GRS8/discussion

Which looks to be pretty close, but not quite the same approach - if I'm understanding it correctly!

The use-case I'm interested in is to batch lots of small network calls before they are sent over a high latency link to an external system.

JB

[Akka Team]

Hi James,

You can achieve something similar with akka.io (http://doc.akka.io/docs/akka/2.3.2/scala/io.html). For example, while your actor waits from an Ack from a corresponding Tcp connection actor, it can batch up the all the incoming ByteStrings and send them off in one piece resulting in one write. I have an implementation of this pattern in the first version of our Tcp stream support but it was not yet measured in any form.

-Endre

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

--

Akka Team

Typesafe - The software stack for applications that scale

Blog: letitcrash.com
Twitter: @akkateam

[James Bunch]

Thanks, that looks useful.

I'm using spray.client at the moment to make the network call (to a web-service). I'm not sure how the akka.io stuff would fit in to this though?

[Akka Team]

Hi James,

Spray sits on top of akka.io. Unfortunately I don't think you can modify how it talks to the Tcp connection. Anyway, since akka-http will use reactive streams this batching behavior will be used by default. I am not sure how much it will buy though, spray is already pretty fast.

-Endre

On Wed, Apr 30, 2014 at 11:13 AM, James Bunch <james....@gmail.com> wrote:

Thanks, that looks useful.

I'm using spray.client at the moment to make the network call (to a web-service). I'm not sure how the akka.io stuff would fit in to this though?

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[James Bunch]

That's ok. I think I'll try the batching pattern (with max size and max latency constraints) described in https://groups.google.com/forum/#!topic/akka-user/todrna9GRS8/discussion

It seems to improve throughput substantially - in a quick test over a simulated high latency link.