Going to Strangeloop this week? Which talks are on your "must see" list?

[Nathan Tippy]

I am planning to see Martins talk on Aeron but how does one decide when we have so many cool talks going on at the same time?

If you are going, please share your "must see" suggestions.

[Ariel Weisberg]

I don't really have a must see list. Just the things that I am going to make sure I go to. I won't know what is must see until I done seen it.

Concurrency options in the JVM - Jessica Kerr

Shenandoah - Christine Flood

Apache Samza - Martin Klepperman

Finagle and Clojure - Sam Neubardt

Akka Cluster - Jonas Boner

Ludicrous Speed - Cliff Moon

Cap'n Proto and Rust - David Renshaw

And of course Aeron.

[Martin Thompson]

No pressure! :-)

I'm finishing writing my presentation right now before flying out on Wednesday.

So looking forward to it.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rajiv Kurian]

Wow, Aeron looks really interesting. Has it been released yet? I tried looking it up, but didn't find much.

[Martin Thompson]

Not just yet! But soon.

On 15 September 2014 17:32, Rajiv Kurian <geet...@gmail.com> wrote:

Wow, Aeron looks really interesting. Has it been released yet? I tried looking it up, but didn't find much.

[ben.c...@alumni.rutgers.edu]

Aeron indeed looks interesting. Especially enticing is the ambition of
offering native OS Shared Memory as an "as required" IPC transport.
This will be especially attractive to those long pining to deploy Java
solutions onto HPC supercomputing platforms (where scaling out compute
resources via NIC hops and/or OSI loopback transports is considered a
soul-crushing compromise).

[Rajiv Kurian]

Hopefully the talk is recorded!

On Monday, September 15, 2014 9:34:44 AM UTC-7, Martin Thompson wrote:

Not just yet! But soon.

On 15 September 2014 17:32, Rajiv Kurian <geet...@gmail.com> wrote:

Wow, Aeron looks really interesting. Has it been released yet? I tried looking it up, but didn't find much.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

[Nathan Tippy]

As for tomorrow (day 1) I really like the "Distributed Systems" track but the Shenandoah talk also looks cool.

[Rajiv Kurian]

Looks like a bunch of strangeloop talks have been released on their YouTube channel, but I can't find the Aeron one yet. Does anybody know if it's going to be released?

[Martin Thompson]

Aeron is in performance testing and tuning, then once complete our client is preparing their own press release. Expect it in about a months time.

Martin...

On 24 September 2014 08:38, Rajiv Kurian <geet...@gmail.com> wrote:

Looks like a bunch of strangeloop talks have been released on their YouTube channel, but I can't find the Aeron one yet. Does anybody know if it's going to be released?

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[Rajiv Kurian]

Thanks Martin! Do you know if the we should expect your strangeloop talk recording to be also released around then?

[Martin Thompson]

Yes. I'll let Alex know and I'm sure he will release it promptly.

On 24 September 2014 09:04, Rajiv Kurian <geet...@gmail.com> wrote:

Thanks Martin! Do you know if the we should expect your strangeloop talk recording to be also released around then?

[Alex Miller]

It's ready to go, I just need to make it public when Martin says the word.

On Wednesday, September 24, 2014 3:05:40 AM UTC-5, Martin Thompson wrote:

Yes. I'll let Alex know and I'm sure he will release it promptly.

On 24 September 2014 09:04, Rajiv Kurian <geet...@gmail.com> wrote:

Thanks Martin! Do you know if the we should expect your strangeloop talk recording to be also released around then?

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

[Rajiv Kurian]

Looking forward to both the software and the talk!

[Rajiv Kurian]

Hey Martin. Any updates on the talk or the project?

[Martin Thompson]

We are waiting for permission for our clients press department. The project is now ready to open.

On 26 October 2014 02:13, Rajiv Kurian <geet...@gmail.com> wrote:

Hey Martin. Any updates on the talk or the project?

[ymo]

+1 for waiting on that !

For folks like me that did/could not go to strangeloop the wait is a killer )))

On Sunday, October 26, 2014 4:25:42 AM UTC-4, Martin Thompson wrote:

We are waiting for permission for our clients press department. The project is now ready to open.

On 26 October 2014 02:13, Rajiv Kurian <geet...@gmail.com> wrote:

Hey Martin. Any updates on the talk or the project?

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

[Martin Thompson]

I'll be doing an updated version of that talk, including more lessons learned, at GOTO Berlin next week :-)

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[ymo]

I soooo wish .. again )))

On Tuesday, October 28, 2014 1:57:37 PM UTC-4, Martin Thompson wrote:

I'll be doing an updated version of that talk, including more lessons learned, at GOTO Berlin next week :-)

On 28 October 2014 17:53, ymo <ymol...@gmail.com> wrote:

+1 for waiting on that !

For folks like me that did/could not go to strangeloop the wait is a killer )))

On Sunday, October 26, 2014 4:25:42 AM UTC-4, Martin Thompson wrote:

We are waiting for permission for our clients press department. The project is now ready to open.

On 26 October 2014 02:13, Rajiv Kurian <geet...@gmail.com> wrote:

Hey Martin. Any updates on the talk or the project?

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Todd Montgomery]

I will be talking at QCon SF next week about the Java 8 things that Martin, Richard, and I have learned wrt Aeron.

If you are in the Bay Area and want to talk about Aeron, just hit me up at QCon.

-- Todd

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[Todd Montgomery]

I will be talking at QCon SF next week about the Java 8 things that Martin, Richard, and I have learned wrt Aeron.

If you are in the Bay Area and want to talk about Aeron, just hit me up at QCon.

-- Todd

On Wed, Oct 29, 2014 at 7:53 AM, ymo <ymol...@gmail.com> wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[ymo]

sorry man ... i cant attend anything for a (good) while. Maybe one day i will get to meet you folks ! But for now all i can do is looking forward to it !

Regards.

[Rajiv Kurian]

Will the GOTO Berlin talk be released on their channel by any chance?

On Tuesday, October 28, 2014 10:57:37 AM UTC-7, Martin Thompson wrote:

I'll be doing an updated version of that talk, including more lessons learned, at GOTO Berlin next week :-)

On 28 October 2014 17:53, ymo <ymol...@gmail.com> wrote:

+1 for waiting on that !

For folks like me that did/could not go to strangeloop the wait is a killer )))

On Sunday, October 26, 2014 4:25:42 AM UTC-4, Martin Thompson wrote:

We are waiting for permission for our clients press department. The project is now ready to open.

On 26 October 2014 02:13, Rajiv Kurian <geet...@gmail.com> wrote:

Hey Martin. Any updates on the talk or the project?

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Martin Thompson]

Looks like many of the the talks are not recorded here in Berlin. Mine was not recorded.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[Todd Montgomery]

My talk yesterday at QCon SF (which contains some info on the work) was recorded, but won't be made public for a couple months on the InfoQ site.

-- Todd

[Rajiv Kurian]

Martin, your talk looks interesting from some of the slides on Twitter. Will wait a couple of months for yours Todd.

[Rajiv Kurian]

Martin, will your GOTO Berlin slides be released?

[Martin Thompson]

Yes they plan to release the slides. I don't know when yet.

--

[Martin Thompson]

They have been released.

http://gotocon.com/dl/goto-berlin-2014/slides/MartinThompson_AeronTheNextGenerationInOpenSourceHighPerformanceMessaging.pdf

[Alex Miller]

Hey Martin, 

Is it ok to release your Strange Loop video then?

Alex

--
You received this message because you are subscribed to a topic in the Google Groups "mechanical-sympathy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mechanical-sympathy/fr7moLcsuiI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mechanical-symp...@googlegroups.com.

[Martin Thompson]

Yes :-)

--

You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[Rajiv Kurian]

Awesome!!

[Alex Miller]

Here you go!

https://www.youtube.com/watch?v=tM4YskS94b0

[Norman Maurer]

Awesome! Watching it now :)

-- 
Norman Maurer

[Georges Gomes]

I think this illustrate very well genius designs: simple and obvious... when explained.

Well done guys!

On Tue Nov 11 2014 at 8:24:38 PM Norman Maurer <norman...@googlemail.com> wrote:

Awesome! Watching it now :)

-- 
Norman Maurer

On 11 Nov 2014 at 20:21:00, Alex Miller (al...@puredanger.com) wrote:

Here you go!

https://www.youtube.com/watch?v=tM4YskS94b0

On Tue, Nov 11, 2014 at 1:13 PM, Martin Thompson <mjp...@gmail.com> wrote:

Yes :-)

On 11 November 2014 18:57, Alex Miller <al...@puredanger.com> wrote:

Hey Martin, 

Is it ok to release your Strange Loop video then?

Alex

On Tue, Nov 11, 2014 at 12:51 PM, Martin Thompson <mjp...@gmail.com> wrote:

They have been released.

http://gotocon.com/dl/goto-berlin-2014/slides/MartinThompson_AeronTheNextGenerationInOpenSourceHighPerformanceMessaging.pdf

On Saturday, 8 November 2014 02:46:25 UTC, Rajiv Kurian wrote:

Martin, will your GOTO Berlin slides be released?

On Thursday, November 6, 2014 9:10:31 AM UTC-8, Rajiv Kurian wrote:

Martin, your talk looks interesting from some of the slides on Twitter. Will wait a couple of months for yours Todd.

--
You received this message because you are subscribed to a topic in the Google Groups "mechanical-sympathy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mechanical-sympathy/fr7moLcsuiI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "mechanical-sympathy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mechanical-sympathy/fr7moLcsuiI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

[Martin Thompson]

Thanks Georges! It can take some time and many iterations to get to simple. We are still cleaning up a few things in the design.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[Daniel Compton]

Hi Martin

Thanks for your work on this, I really enjoyed your talk. From a high level, a lot of Aeron seems quite similar to Kafka e.g. persistent sequential logs, and clients holding their own offset in the log. Some of the differences I can see are around choice of data structures, transport, latency, and mechanical sympathy. 

Are you able to comment on Kafka and how it compares to Aeron?

--

Daniel

[Martin Thompson]

On 12 November 2014 10:32, Daniel Compton <de...@danielcompton.net> wrote:

Thanks for your work on this, I really enjoyed your talk. From a high level, a lot of Aeron seems quite similar to Kafka e.g. persistent sequential logs, and clients holding their own offset in the log. Some of the differences I can see are around choice of data structures, transport, latency, and mechanical sympathy. 

Thanks! You have summed up most of the differences :-) 

Many things seem similar from a high level. The devil is in the detail. 

With Aeron the goal is to keep things pure and focused. The benchmarking we have done so far suggests a step forward in throughput and latency. What is quite unique is that you do not have to choose between throughput and latency. With other high-end messaging transports this is a distinct choice. The algorithms employed by Aeron give maximum throughput while minimising latency up until saturation.

Are you able to comment on Kafka and how it compares to Aeron?

For now I'd like to see others independently compare the performance of Aeron to the alternatives out there. We still have plenty of work to do on our roadmap to keep us busy.

If you have a quick look into the implementations you'll see Kafka is brokered and Aeron is peer-to-peer. Aeron is a pure OSI layer 4 implementation, Kafka is doing much more such as providing clustering and archived messages. Kafka is using TCP for the underlying transmission media; Aeron uses UDP and will have SHM IPC, and Infiniband soon.

Look at the below for latency and throughput then try similar with Aeron.

    http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines

As we move to a world of micro services then we need very low and predictable latency from our communications otherwise the coherence component of USL will come to rain fire and brimstone on our designs.

Martin...

[Dan Eloff]

That was a fascinating talk! I was floored when Martin described the algorithm for the wait-free log, I wrote a similar one a couple weeks ago using the same strategy of atomic add and then writing the header last, with a release-store of the frame length in the publisher and acquire-load in the consumer. I guess there is only one obvious way to do it!

I fell right into the trap of using a big file for the log though. It makes perfect sense to rotate through smaller buffers to avoid the paging and page cache overheads, that taught me something.

You guys have implemented a low-latency, high-throughput replicated log much better than I could have. I was working on a better performing Raft consensus implementation. I will work on building that on top of Aeron instead.

[Adam Warski]

Hello,

great to finally see some details of Aeron!

As far as I understand, it provides a persistent log of messages, however I was under the impression that it also mentioned replication? I couldn’t find anything in the samples which would indicate some inter-broker communication, so maybe it’s not there, or I just can’t find it? :)

Also, another project that is similar in some respects is ZeroMQ - although it doesn’t offer persistence or replication, such services could be built on top of it. Do you have any thoughts on how ZeroMQ and Aeron compare?

Thanks,

Adam

-- 
Adam Warski

http://twitter.com/#!/adamwarski
http://www.softwaremill.com
http://www.warski.org

[Martin Thompson]

On 14 November 2014 08:45, Adam Warski <ad...@warski.org> wrote:

As far as I understand, it provides a persistent log of messages, however I was under the impression that it also mentioned replication? I couldn’t find anything in the samples which would indicate some inter-broker communication, so maybe it’s not there, or I just can’t find it? :)

Aeron essentially replicates (in the normal use of the word) the log buffers from one machine to another. The buffers are persistent in a functional sense in that the stored records are not mutated, not persisted to disk. The fun we have with language in this subject :-)

However in the new year I'm doing some work for another client that will go open source which provides replication in a clustered machine context that will also provide a persisted log to disk as an archive. This is likely to be built on top of Aeron.

Aeron is peer-to-peer and does not use brokers. This is one of the reasons it has such low-latency.

 

Also, another project that is similar in some respects is ZeroMQ - although it doesn’t offer persistence or replication, such services could be built on top of it. Do you have any thoughts on how ZeroMQ and Aeron compare?

I'll keep saying it until people get what it actually means. :-) Tongue firmly in cheek!

Aeron provides an OSI layer 4 message transport. The 5 services of this layer are well defined. To compare another message transport you need to compare to these 5 services. Do you they provide more or less. Archiving and cluster replication are not part of OSI layer 4. For example with flow control, many other messaging products rely on TCP. Aeron provides its own flow control with pluggable hooks from providing alternative algorithms. ZeroMQ could sit on top of Aeron, just as it does with PGM.

Many messaging products are a Swiss Army knife; Aeron is a scalpel.

 

[Adam Warski]

On 14 Nov 2014, at 10:22, Martin Thompson <mjp...@gmail.com> wrote:

On 14 November 2014 08:45, Adam Warski <ad...@warski.org> wrote:

As far as I understand, it provides a persistent log of messages, however I was under the impression that it also mentioned replication? I couldn’t find anything in the samples which would indicate some inter-broker communication, so maybe it’s not there, or I just can’t find it? :)

Aeron essentially replicates (in the normal use of the word) the log buffers from one machine to another. The buffers are persistent in a functional sense in that the stored records are not mutated, not persisted to disk. The fun we have with language in this subject :-)

However in the new year I'm doing some work for another client that will go open source which provides replication in a clustered machine context that will also provide a persisted log to disk as an archive. This is likely to be built on top of Aeron.

Aeron is peer-to-peer and does not use brokers. This is one of the reasons it has such low-latency.

Ok, thanks, I think I’m starting to understand, though I’m quite fixed on what “traditional” messaging systems provide, so it’s probably going to take some time ;) So, for example, a higher-level messaging service could use Aeron as the transport, to communicate between nodes. And using that transport more advanced services, like on-disk persistence or cluster-replication can be built. 

Also, another project that is similar in some respects is ZeroMQ - although it doesn’t offer persistence or replication, such services could be built on top of it. Do you have any thoughts on how ZeroMQ and Aeron compare?

I'll keep saying it until people get what it actually means. :-) Tongue firmly in cheek!

Aeron provides an OSI layer 4 message transport. The 5 services of this layer are well defined. To compare another message transport you need to compare to these 5 services. Do you they provide more or less. Archiving and cluster replication are not part of OSI layer 4. For example with flow control, many other messaging products rely on TCP. Aeron provides its own flow control with pluggable hooks from providing alternative algorithms. ZeroMQ could sit on top of Aeron, just as it does with PGM.

So in fact comparing ZeroMQ and Aeron or Kafka and Aeron don’t make much sense, as they are built for different roles. It at all, then probably more with ZeroMQ, as it provides “better sockets” - p2p communication as well - however on top of existing transport protocols, instead of implementing its own one.

Thanks again,

Adam

[Richard Warburton]

Hi,

Ok, thanks, I think I’m starting to understand, though I’m quite fixed on what “traditional” messaging systems provide, so it’s probably going to take some time ;) So, for example, a higher-level messaging service could use Aeron as the transport, to communicate between nodes. And using that transport more advanced services, like on-disk persistence or cluster-replication can be built.

So in fact comparing ZeroMQ and Aeron or Kafka and Aeron don’t make much sense, as they are built for different roles. It at all, then probably more with ZeroMQ, as it provides “better sockets” - p2p communication as well - however on top of existing transport protocols, instead of implementing its own one.

Indeed. The most common example of a level 4 protocol is TCP. You can think of Aeron as sitting at the same level in your stack as TCP.

Now in practice many applications don't want to use the "stream of bytes" abstraction that TCP provides. Libraries like ZeroMQ wrap TCP and then provide a messaging abstraction on top of TCP. Now this ends up confusing people because they view messaging as a layer above TCP, when in actual fact it could be provided by a level 4 protocol, if that protocol chose a messaging abstraction.

This isn't to say that there's anything wrong with ZeroMQ - its a good library that's used by lots of people for a good reason. Having said that, if you've got the option to use a protocol that provides the right abstraction out of the box, rather than having to add an "adapter" layer of abstraction you'll probably find that you'll get a better performing and simpler system as a result.

I personally hope that a future version of ZeroMQ comes with support for the Aeron protocol since the abstractions that its API provides are incredibly sympathetic to the design of the Aeron system.

I believe the comparison between kafka and Aeron has really come more out of an overloading of the terms "replicate", "log" and "message". Terminology is both the cause of and solution to communication problems in our industry ;)

regards,

  Richard Warburton

  http://insightfullogic.com

  @RichardWarburto

[Todd Montgomery]

I agree with Martin and Richard. Aeron is a slightly different beast than Kafka at this point. However, there is some overlap.

Aeron being an ISO layer 4 protocol provides a number of things that messaging systems can't and also doesn't provide several things that some

messaging systems do.... if that makes any sense. Let me explain slightly more wrt all typical messaging systems (not just Kafka and 0MQ).

One way to think more about where Aeron fits is TCP, but with the option of reliable multicast delivery. However, that is a little limited in that Aeron

also, by design, has a number of possible uses that go well beyond what TCP can do. Here are a few things to consider:

- individual identification, in time, of a "persisted" stream of data with full record boundaries. This is the {channel, sessionId, channelId, offset, length} tuples

that is at the heart of log buffer strategy. This allows a non-volatile storage of the stream with arbitrary playback to fall out... which is quite interesting. Which

leads to doing reconnect and persistence of data streams in a truly mechanical sympathetic way. I can't stress enough what this opens up. In fact, it

also opens up true location transparency within the transport protocol. i.e. a local subscriber can read directly from the publishers log buffer while

the driver transparently sends data to off-box subscribers. These are the things we've identified so far, but I think this can also apply to unique ways

to handle proactive/reactive forward error correction, carouseling, arbitrary replay, etc.

- Aeron doesn't have topics. It has individual non-contended streams. Most messaging systems provide a topic space. Which is a blessing and a curse.

By keeping the stream space for Aeron deliberately bounded implementation-wise (but unbounded design-wise), Aeron allows topic spaces to be

built on top (which allows 0MQ and a host of other systems to leverage it) without locking the implementation into wasting resources for those use

cases where it is not needed.

- the reliable unicast design is a familiar, firewall traversable design that mimics TCP. But the reliable multicast design allows for pub/sub semantics with

infinitely configurable flow control strategies. This opens up all kinds of use case possibilities beyond just normal messaging, streaming, etc.

-- Todd

--

[Jan van Oort]

I've been thinking, for over half a year now, to extend the http protocol ( PUT, GET, POST, TRACE, DELETE, the whole shebang ) with two extra methods: INDEX and SEARCH, just for the needs of Stratus, my indexing logic. 

The great advantage of using Aeron viz. building on top of it would be exactly what Todd points out here :-)  Extending http with INDEX and SEARCH methods would be quite easier than with other libraries ( and here, along with Martin Thompson, I'll say: **libraries**, don't get me started on <<frameworks>> ). 

Fortuna audaces adiuvat - hos solos ? 

[ymo]

Maybe this should be copy pasted as is on the github project page as a section titled "How does it comapre to X" ? I am sure this is soon to turn into a FAQ !

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.