Azul for HFT

[Janny]

Just curios are there any examples of Azul usage in HFT? I have never heard that the biggest banks use it. Also what is the future of investment with value types coming in the future JDK versions?

[Martin Thompson]

Most of the biggest banks in capital markets and quite a few trading companies use Azul Zing. Many are just not that open about their technology stack.

On 18 February 2015 at 01:43, Janny <winnie...@gmail.com> wrote:

Just curios are there any examples of Azul usage in HFT? I have never heard that the biggest banks use it. Also what is the future of investment with value types coming in the future JDK versions?

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Janny]

Hi Martin, 

Do you have facts that most of the biggest banks in Capital Markets use Azul Zing? Especially I'm sceptical about the word most. I can believe in a few (as a guys from this world), but also I can't find any azul/zing job among 9920 investment banking jobs worldwide (http://www.efinancialcareers.com/search?keywords=Zing%2Cazul). 

I can give more serious facts also.

Thanks.

On Wednesday, February 18, 2015 at 4:39:05 PM UTC+8, Martin Thompson wrote:

Most of the biggest banks in capital markets and quite a few trading companies use Azul Zing. Many are just not that open about their technology stack.

On 18 February 2015 at 01:43, Janny <winnie...@gmail.com> wrote:

Just curios are there any examples of Azul usage in HFT? I have never heard that the biggest banks use it. Also what is the future of investment with value types coming in the future JDK versions?

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

[Martin Thompson]

I cannot give more details because as I said before financial organisations are often, unnecessarily, secretive about such things. Like all good science you should not take what I say for any more than the evidence that is provided. The only thing you can read into it is that I do not work for Azul and it benefits me in no way offering this. I know Gil because we often end up helping the same clients.

Regarding the jobs search. In my experience you develop plain old Java for Zing and it requires almost no special tuning for use in low-latency. It does much better with idiomatic OO than other JVMs. This is not that case with Hotspot which requires very specialised skills to tune and develop for in a low-latency environment. Also consider that the biggest banks are also very big. This means that they will not use the same technology in every department. In my experience it is not uncommon for one department to not know what other departments are doing.

On 18 February 2015 at 10:52, Janny <winnie...@gmail.com> wrote:

Hi Martin, 

Do you have facts that most of the biggest banks in Capital Markets use Azul Zing? Especially I'm sceptical about the word most. I can believe in a few, but also I can't find any azul/zin job among 9920 investment banking jobs worldwide (http://www.efinancialcareers.com/search?keywords=Zing%2Cazul). 

I can throw more serious facts also.

Thanks.

On Wednesday, February 18, 2015 at 4:39:05 PM UTC+8, Martin Thompson wrote:

Most of the biggest banks in capital markets and quite a few trading companies use Azul Zing. Many are just not that open about their technology stack.

On 18 February 2015 at 01:43, Janny <winnie...@gmail.com> wrote:

Just curios are there any examples of Azul usage in HFT? I have never heard that the biggest banks use it. Also what is the future of investment with value types coming in the future JDK versions?

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jean-Philippe BEMPEL]

On Wednesday, February 18, 2015 at 12:10:04 PM UTC+1, Martin Thompson wrote:

 Also consider that the biggest banks are also very big. This means that they will not use the same technology in every department. In my experience it is not uncommon for one department to not know what other departments are doing.

+1000 on this statement, This is without doubt the reason of this discrepancy :) 

[Jan van Oort]

Coincides 100% with my experience with large insurers and banks. One department may have been using X with great success, with another department struggling to choose between Y and Z. 

Fortuna audaces adiuvat - hos solos ? 

--

[Janny]

"+1000" here and there makes me feel better about my job security. ;-)

[Janny]

This is all good in theory, but in practice if a good number of banks uses Zing, why I have never heard about it?

Someone from my friends who work in other banks or hedge funds would definitely mention it. I went through many interviews with big banks for equities and FX trading in the past, but have never heard a world about Zing. Here where I live all these hedge funds and big banks don't afraid to mention about fpga, C++, STL or just core java, kdb+ for HFT roles - but they don't mention Zing in the job descriptions. I know what kind of projects we have in our company and what kind of technologies we use. Nobody will be using Zing, for example, for fixed income trading, usually for Equities and FX (the usage is quite narrow, so not so many departments in the banks who potentially may use it). I'm very very sceptical...

On Wednesday, February 18, 2015 at 9:47:55 PM UTC+8, Jan van Oort wrote:

Coincides 100% with my experience with large insurers and banks. One department may have been using X with great success, with another department struggling to choose between Y and Z. 

Fortuna audaces adiuvat - hos solos ? 

On 18 February 2015 at 14:20, Jean-Philippe BEMPEL <jpbe...@gmail.com> wrote:

On Wednesday, February 18, 2015 at 12:10:04 PM UTC+1, Martin Thompson wrote:

 Also consider that the biggest banks are also very big. This means that they will not use the same technology in every department. In my experience it is not uncommon for one department to not know what other departments are doing.

+1000 on this statement, This is without doubt the reason of this discrepancy :) 

--

You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

[Vitaly Davidovich]

FWIW, I worked in a big bank for a while and also never heard of any group using Azul.  I believe it was evaluated at some point, but was not picked (don't know the reasons).  That's not to say I don't believe it's used in banks, but I've also not heard of it much from other folks I know who work in big financial corps.

I do know that the groups using java for low latency are particularly careful about the way they write code.  In a nutshell, it consists of avoiding allocations in hot paths and sizing eden such that you make it through the day without any GC.

sent from my phone

On Feb 18, 2015 9:08 AM, "Janny" <winnie...@gmail.com> wrote:

This is all good in theory, but in practice if a good number of banks uses Zing, why I have never heard about it?

Someone from my friends who work in other banks or hedge funds would definitely mention it. I went through many interviews with big banks for equities and FX trading in the past, but have never heard a world about Zing. Here where I live all these hedge funds and big banks don't afraid to mention about fpga, C++, STL or just core java, kdb+ for HFT roles - but they don't mention Zing in the job descriptions. I know what kind of projects we have in our company and what kind of technologies we use. Nobody will be using Zing, for example, for fixed income trading, usually for Equities and FX (the usage is quite narrow, so not so many departments in the bank may be using it). I'm very very sceptical...

On Wednesday, February 18, 2015 at 9:47:55 PM UTC+8, Jan van Oort wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Janny]

I may probably post the final very bold statement until Gil decide to answer. Big Banks do not care about the latency/throughput  so much as Azul may think they do (they care, but they may all achieve they need with JDK or they have big salary budget and they use C++), and hedge fund usually choose C/C++, because usually scalping is the only source of income - straight from the horse mouth

On Wednesday, February 18, 2015 at 10:22:18 PM UTC+8, Vitaly Davidovich wrote:

FWIW, I worked in a big bank for a while and also never heard of any group using Azul.  I believe it was evaluated at some point, but was not picked (don't know the reasons).  That's not to say I don't believe it's used in banks, but I've also not heard of it much from other folks I know who work in big financial corps.

I do know that the groups using java for low latency are particularly careful about the way they write code.  In a nutshell, it consists of avoiding allocations in hot paths and sizing eden such that you make it through the day without any GC.

sent from my phone

On Feb 18, 2015 9:08 AM, "Janny" <winnie...@gmail.com> wrote:

This is all good in theory, but in practice if a good number of banks uses Zing, why I have never heard about it?

Someone from my friends who work in other banks or hedge funds would definitely mention it. I went through many interviews with big banks for equities and FX trading in the past, but have never heard a world about Zing. Here where I live all these hedge funds and big banks don't afraid to mention about fpga, C++, STL or just core java, kdb+ for HFT roles - but they don't mention Zing in the job descriptions. I know what kind of projects we have in our company and what kind of technologies we use. Nobody will be using Zing, for example, for fixed income trading, usually for Equities and FX (the usage is quite narrow, so not so many departments in the bank may be using it). I'm very very sceptical...

On Wednesday, February 18, 2015 at 9:47:55 PM UTC+8, Jan van Oort wrote:

Coincides 100% with my experience with large insurers and banks. One department may have been using X with great success, with another department struggling to choose between Y and Z. 

Fortuna audaces adiuvat - hos solos ? 

On 18 February 2015 at 14:20, Jean-Philippe BEMPEL <jpbe...@gmail.com> wrote:

On Wednesday, February 18, 2015 at 12:10:04 PM UTC+1, Martin Thompson wrote:

 Also consider that the biggest banks are also very big. This means that they will not use the same technology in every department. In my experience it is not uncommon for one department to not know what other departments are doing.

+1000 on this statement, This is without doubt the reason of this discrepancy :) 

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Vitaly Davidovich]

I wouldn't be that bold :).  "Big banks" is too broad; it's definitely the case that groups within these banks are oft not aware of what others are doing (just like most large organizations), I agree with that observation.  It's also the case that only certain groups would require that type of latency SLA crucial to their business, so certain divisions simply don't need this at all.

There are definitely low latency players using java, but keep in mind that it also doesn't have to be black and white: you can mix java with native code where it makes sense.

sent from my phone

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[Jean-Philippe BEMPEL]

I think financial industry is pretty large even vast. From my views I do not agree with your statement, but I doubt I have an exhaustive view of what it is used in the whole sector.

In my company we have big banks as customers, we are doing low latency, low & high throughput and we are using Zing (but not only!).

Saying all Big Banks are knowing about Zing or using it, I am not in a position to affirm it, but I have experiences and examples.

[Martin Thompson]

You seem to be posting statements and disagreeing with others who are simply offering their viewpoint. We all have a limited sample set and all we can do is share what we know.

What are you looking to discover with the question? Do you have a need at work for which you need to know some technical answer?

[Janny]

it is okay to disagree with facts. tried to understand who may use Zing and hear about experience vs optimized JDK, also I asked a question about future as java will be soon moving to off-heap structures - just have interest to hear what people in low latency field may think about it and where else Zing may be better.

[Vitaly Davidovich]

What do you mean by java moving to offheap structures? Value types? If so, that in no way will displace Azul.

sent from my phone

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[Martin Thompson]

Well it looks like some peoples sample set have observed Zing being used in banks and trading firms. For others they have not. You then jumping to the conclusion that Gil is wrong about latency needs is not good analysis of the facts given the responses. Would you call that good analysis?

Regarding your other question about value types. Value types are very much an on-heap construct and not off-heap as you seem to think. Due to the ability to stack allocate them we should see benefits from locality of reference and less GC pressure. However we would not have guarantees for the locality of reference. I'd expect more benefits to throughput rather than predictable latency without applying a deep understanding of how the JVM will implement them.

[Vitaly Davidovich]

While it does remain to be seen how they're implemented, unless it's completely botched (which I doubt), I do expect it to improve throughput *and* latency in *some* cases (assuming code is updated to use value types where applicable).  It should also facilitate writing cleaner code without sacrificing performance.  That's certainly been the case with .NET's structs (for a slight flavor of that: http://samsaffron.com/archive/2011/10/28/in-managed-code-we-trust-our-recent-battles-with-the-net-garbage-collector).

Ultimately, if done right, they should come with similar performance gains to using primitives over custom value classes today.

--

You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[Martin Thompson]

Yes they should be a good thing for the language when they arrive. We could kill off the horrid Pair<V1, V2> style classes to deal with returning more than one thing as an example :-)

On a single thread I would expect a mean reduction in latency when you see an increase in throughput. I guess this is the "some" cases as you carefully highlight. Depending on coding pattern and stack limits they may not always be stack allocated. They also may not be co-located in memory with an aggregate class of which they are a part. That is why I cautioned about "predictable latency" not just low latency.  I'm more interested in the distribution rather than the mean.

I'm fascinated to see how they can be implemented.

[Vitaly Davidovich]

On a single thread, taking GC pressure out of the equation (which of course is a big reason why they could reduce "global" app-level latency), they should act just like if you were using a primitive (assuming the underlying storage is of same size).  There is this whole thing about "we're allowed to heap allocate if the value type is over some size threshold", but I'm assuming that threshold is going to be sufficiently large such that no reasonable/sane value type will hit that limit (this is par of what I meant by "not botching" the implementation).  They will come with copying costs given they're intended to be immutable, whose performance hit will depend on the shape of the value type, but given the alternative (allocating in the heap), I'll take that any day of the week.

They can also reduce overall heap footprint if many instances of value classes are switched over to value types (i.e. lose header bytes on each instance).

In terms of layout control, yeah, there's no guarantee of anything; that's somewhat similar to primitives today, although the layout algorithm used is known and can be played with to varying degree of success.  However, again given the alternative of chasing a pointer, I'll take that any day of the week as well :).

For the (I think) expected common case of a value type with just a few primitives underneath, the scalarizing of that would yield pretty good benefits since data will be placed into registers (assuming no register pressure, but at worst, spilled/reloaded to/from stack) directly.

The two dampers on them are (1) always immutable (there are some cases where that's not ideal) and (2) no way to pass them by-ref (to avoid copying cost for large-ish ones.  Both scenarios are somewhat rare, so shouldn't be too big of a deal.

--

[Jimmy Jia]

Much love to Gil and everyone else, but the truth regarding Zing and HFT is that it depends a lot on what your definition of "low latency" is.

If your issues are around GC pauses caused by object allocation in FIX handling, it's pretty likely that Zing will help you out quite a bit. That's a different level of "low latency" than, say, worrying about shaving off the last µs or two on your critical path. Zing v HotSpot just isn't going to matter there.

It really is bizarre how outside of this community, most discussions of how to tune Java GC pausing don't have a footnote that just says "or go use Zing", though.

On Wed, Feb 18, 2015, 10:48 Martin Thompson <mjp...@gmail.com> wrote:

Yes they should be a good thing for the language when they arrive. We could kill off the horrid Pair<V1, V2> style classes to deal with returning more than one thing as an example :-)

On a single thread I would expect a mean reduction in latency when you see an increase in throughput. I guess this is the "some" cases as you carefully highlight. Depending on coding pattern and stack limits they may not always be stack allocated. They also may not be co-located in memory with an aggregate class of which they are a part. That is why I cautioned about "predictable latency" not just low latency.  I'm more interested in the distribution rather than the mean.

I'm fascinated to see how they can be implemented.

On 18 February 2015 at 15:29, Vitaly Davidovich <vit...@gmail.com> wrote:

While it does remain to be seen how they're implemented, unless it's completely botched (which I doubt), I do expect it to improve throughput *and* latency in *some* cases (assuming code is updated to use value types where applicable).  It should also facilitate writing cleaner code without sacrificing performance.  That's certainly been the case with .NET's structs (for a slight flavor of that: http://samsaffron.com/archive/2011/10/28/in-managed-code-we-trust-our-recent-battles-with-the-net-garbage-collector).

Ultimately, if done right, they should come with similar performance gains to using primitives over custom value classes today.

On Wed, Feb 18, 2015 at 10:09 AM, Martin Thompson <mjp...@gmail.com> wrote:

Regarding your other question about value types. Value types are very much an on-heap construct and not off-heap as you seem to think. Due to the ability to stack allocate them we should see benefits from locality of reference and less GC pressure. However we would not have guarantees for the locality of reference. I'd expect more benefits to throughput rather than predictable latency without applying a deep understanding of how the JVM will implement them.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

[Jimmy Jia]

The other thing is, for very low latency trading applications, if you are using Java and have been for a while (i.e. before you found out about Zing), you've probably already written all of your code to not generate any garbage at all (or gone out of business). Zing's ReadyNow! looks pretty amazing, but if GC pauses are causing issues for you, you're not really doing HFT.

If you were starting from scratch? Or have a non-HFT application in Java where you want to get rid of pauses? Then Zing looks pretty damn good. Although given the current respective states of C++11/14 v Java, it's true that you might just want to go with C++ unless you like Java a lot better, if you're targeting very low latency for HFT - but you'll still be slower than people who aren't only using software.

Zing is a pretty broadly applicable product rather than a niche one, but there are plenty of vagaries about the shape of the space that mean there are quite large niches where Zing isn't that applicable or helpful, and unfortunately for Zing, a lot of stuff lives in those niches.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsubscribe...@googlegroups.com.

[Michael Barker]

You could always look at the list of customers on their website:

http://www.azulsystems.com/content/case-studies

There's section for finance, with a number of well known banks.

On 18 February 2015 at 14:43, Janny <winnie...@gmail.com> wrote:

Just curios are there any examples of Azul usage in HFT? I have never heard that the biggest banks use it. Also what is the future of investment with value types coming in the future JDK versions?

--

You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[Jan van Oort]

<quote>There's section for finance, with a number of well known banks</quote>

Including LMAX, Peter Lawrey's outfit. Which in and by itself is already quite the reference, I daresay. 

Fortuna audaces adiuvat - hos solos ? 

[Martijn Verburg]

Hi all,

For reference I run jClarity (We provide Censum, the GC analysis tool for Hotspot & some other performance tooling).

We are often called in to assist clients with their Hotspot GC problems around latency (in particular for financial services), if we're unable to tune Hotspot to meet the customers latency requirements then we don't hesitate for a second to recommend Zing. Sometimes that requirement is obviously low/stringent enough that it simply elicits the "Go buy Zing" response from us. There are certain latency requirements that Hotspot collectors are never going to meet consistently (G1 is slowly starting to get push down, but I suspect most people on this list wouldn't consider it a 'low-latency' collector). *Maybe* further G1 improvements and/or Shenandoah will push Hotspot a little further into that space, but there is a lot of work to be done there yet and we don't pretend to customers otherwise.

I'd say that I've personally seen Zing in a couple of the Tier 1's, several of the Tier 2/3's + a bunch of other players in the market. From what I've observed these aren't typically "replace the whole 100,000 JVM plant" type installations, more like strategic ones in areas where latency matters a great deal and the system is written in Java. Again, as Martin points out, I'm just one sample data source here, only Azul's CRM likely knows the truth ;-).

Cheers,
Martijn

[Michael Barker]

Including LMAX, Peter Lawrey's outfit. Which in and by itself is already quite the reference, I daresay. 

It is true that we (LMAX) use Zing, but worth noting that Peter Lawery does not work for us.

[Jan van Oort]

Wasn't he the author of the Disruptor pattern and implementation with you guys ? I always thought so. 

Fortuna audaces adiuvat - hos solos ? 

On 18 February 2015 at 22:32, Michael Barker <mik...@gmail.com> wrote:

Including LMAX, Peter Lawrey's outfit. Which in and by itself is already quite the reference, I daresay. 

It is true that we (LMAX) use Zing, but worth noting that Peter Lawery does not work for us.

--

[Jimmy Jia]

BTW, the counter-point to "LMAX use Zing" is probably "NASDAQ don't use Zing".

What I'm really getting at is - if you want to optimize latency in a Java application where GC pauses are a concern, you probably want to use Zing. If you think through and understand why NASDAQ don't use Zing, then you'll understand why their reasons don't really apply to the scenario I just pointed out, which should probably point you toward using Zing.

[Martin Thompson]

Check the original commit log for the Disruptor :-)

https://code.google.com/p/disruptor/source/browse/#svn%2Ftrunk%2Fcode%2Fsrc

[Jan van Oort]

Point taken, Martin. IOU a beer :-)

Fortuna audaces adiuvat - hos solos ? 

[Martin Thompson]

I was also the co-founder and CTO of LMAX but that is in the past these days.

[Michael Barker]

Peter has done some really interesting work in a similar space.  Checkout https://github.com/OpenHFT.

Mike.

[Martin Thompson]

There is no one true technology for success right now. Last time I looked the top exchanges by volume and their implementation language in the US were in no particular order:

NYSE/NASDAQ - C

ICE - Java

Direct Edge - C#

CME - Java

Bats - C

All keep re-evaluating and updating. It is likely most will be native language implementation with FPGA support in the future. Languages like Java could possibly compete in this space but we need true concurrent collectors and control of memory layout. Getting this sort of support for managed runtimes would be great benefit to the world in general for high performance computing.

[Jan van Oort]

@Mike Yep, he has. I am using OpenHFT ChronicleMap right now to build a LRUCache<byte[], Object >. 

99-percentile latency is down to 6 µs on a Windows 7 / Core-i7 / SSD laptop with a Hotspot JVM :-)

Fortuna audaces adiuvat - hos solos ?