[erlang-questions] trading systems
Joel Reymont
This basically involves loading price and timestamp pairs
and iterating over them calculating various things like
moving averages, correlation, etc.
Off the top of my head, Erlang does not seem to be the most
efficient platform for this because prices would need to be
stuffed into ETS or copied around otherwise.
The most efficient way to manage price data is to simply keep
appending quotes to the end of an mmap-ed file. I would love
to represent the mmap-ed memory as a binary and I think it may
even be possible. It would require some tricky work at the
driver level, though.
What do you think?
Thanks, Joel
---
fastest mac firefox!
http://wagerlabs.com
________________________________________________________________
erlang-questions mailing list. See http://www.erlang.org/faq.html
erlang-questions (at) erlang.org
Robert Raschke
> Has anyone used Erlang to backtest trading systems?
>
> This basically involves loading price and timestamp pairs
> and iterating over them calculating various things like
> moving averages, correlation, etc.
>
> Off the top of my head, Erlang does not seem to be the most
> efficient platform for this because prices would need to be
> stuffed into ETS or copied around otherwise.
>
> The most efficient way to manage price data is to simply keep
> appending quotes to the end of an mmap-ed file. I would love
> to represent the mmap-ed memory as a binary and I think it may
> even be possible. It would require some tricky work at the
> driver level, though.
>
> What do you think?
>
> Thanks, Joel
>
>
You'll want a column based data store, K comes to mind (http://www.kx.com/).
If you want something a little bit more open, look at J (
http://www.jsoftware.com/) and it's JDB effort.
Staying with the one letter theme (must be an in joke of stats heads,
methinks), there's also S and R. Good look searching for those ;-)
Robby
Robert Raschke
Sorry, meant to say "have a look at ... for inspiration".
Robby
Joel Reymont
On Oct 1, 2009, at 11:58 AM, Robert Raschke wrote:
> You'll want a column based data store, K comes to mind (http://www.kx.com/
> ).
I have this one and actually know how to use it.
> If you want something a little bit more open, look at J (http://www.jsoftware.com/
> ) and it's JDB effort.
A nice free replacement.
> Staying with the one letter theme (must be an in joke of stats
> heads, methinks), there's also S and R. Good look searching for
> those ;-)
Searching for QuantMod is easier. That's a module that provides
trading tools for R, including charting.
Still, I'd like to know how this can be done with Erlang, as optimally
as possible. And no, I don't think I want to use byte arrays, gb_trees
or the array module ;).
Robert Raschke
>
> On Oct 1, 2009, at 11:58 AM, Robert Raschke wrote:
>
> You'll want a column based data store, K comes to mind (
>> http://www.kx.com/).
>>
>
> I have this one and actually know how to use it.
>
> If you want something a little bit more open, look at J (
>> http://www.jsoftware.com/) and it's JDB effort.
>>
>
> A nice free replacement.
>
> Staying with the one letter theme (must be an in joke of stats heads,
>> methinks), there's also S and R. Good look searching for those ;-)
>>
>
> Searching for QuantMod is easier. That's a module that provides trading
> tools for R, including charting.
>
> Still, I'd like to know how this can be done with Erlang, as optimally as
> possible. And no, I don't think I want to use byte arrays, gb_trees or the
> array module ;).
>
>
>
Shooting from the hip, how about files containing Erlang binaries for fixed
index ranges. So, per column have a series of files col0, col1000000,
col2000000, ... If you're on 64bit, then mmapping whole columns without
segmentation might be an option (that's how J does it). Each column contains
fixed size data values. And then have an access layer with potential
caching. Speedy things needed would be the loading of a file to memory and
sequential access through the data. If live update is important, you'll want
zero cost append.
And over the Erlang binaries containing your fixed sized data you can
implement map and fold operations. Easy. <tongue firmly in cheek>
Robby
Kenneth Lundin
As usual you are jumping right into low level optimizations without
explaining why that
would be necessary:).
It would be nice if you could explain what the system needs to do and after that
why you think it is a problem to store prices in an ETS-table.
I am sure there are other options as well.
At least for me it is not obvious what performance that is needed for
an application like this.
And pure performance might not be the only reason to use Erlang. It can also be
that it is so very simple to implement and maintain the solution and
that is scales very well over
multiple cores without extra work for the programmer.
Why are you talking about mmapped files? Please explain for a novice
in trading systems.
/Kenneth
Joel Reymont
On Oct 1, 2009, at 4:50 PM, Kenneth Lundin wrote:
> It would be nice if you could explain what the system needs to do
> and after that
> why you think it is a problem to store prices in an ETS-table.
> [...]
> Why are you talking about mmapped files? Please explain for a novice
> in trading systems.
It's high-frequency trading systems I'm talking about. Where's Serge
Aleynikov [1] when you need him?
I'd like to process an incoming price quote, make a decision and
submit a trade order with minimal latency. I know that this is
normally done using C++ (or OCaml [1]) but I'm wondering how this can
be achieved using Erlang.
Minimal latency is usually achieved by hitting the CPU cache as
frequently as possible and minimizing memory copying overhead. ETS
copies memory and I suspect it's use of the cache is rather poor by
virtue of being a "hash table" and pointing all over the place in
memory.
A simple array of floats or doubles groups data together in memory and
is very cache friendly. I figure that an Erlang binary is the closest
I can get to an array in Erlang. mmap-ing a file of floats or doubles
and wrapping it in a binary should avoid unnecessary copying of data
from disk to memory.
I think operating on binaries representing mmap-ed files of price
quotes is as close as you can get to low latency in Erlang. I know
that premature optimization is the root of all evil but I'd rather
have my teeth pulled than optimize Erlang. This is speaking from
experience. I figure a much better way is to choose a concept that
guarantees fast code from the start.
Thanks, Joel
Andrew Thompson
There's always the undocumented hipe bit and bytearrays (which are
mutable) if you're willing to sacrifice everything on the altar of
performance.
http://erlang.org/pipermail/erlang-questions/2009-April/043460.html
Andrew
Paul Fisher
http://www.acm.org/ubiquity/volume_10/v10i3_hyde.html
Joel Reymont
On Oct 1, 2009, at 5:40 PM, Andrew Thompson wrote:
> There's always the undocumented hipe bit and bytearrays (which are
> mutable) if you're willing to sacrifice everything on the altar of
> performance.
You cannot store doubles in a bytearray from what I know.
Also, you need to populate the byte array which involves loading data
from disk into buffers, into erlang, etc. mmap for the win!
---
fastest mac firefox!
http://wagerlabs.com
________________________________________________________________
Andrew Thompson
On Thu, Oct 01, 2009 at 05:04:12PM +0100, Joel Reymont wrote:
There's always the undocumented hipe bit and bytearrays (which are
mutable) if you're willing to sacrifice everything on the altar of
performance.
http://erlang.org/pipermail/erlang-questions/2009-April/043460.html
Andrew
Artur Matos
I tried to implement a backtesting/execution system in Erlang a few years
ago, but eventually gave up and moved to Java. I was not trading intraday at
that time (just using daily data) so performance was never much of an issue.
My biggest problem with Erlang is that it was lacking essential libraries,
and I guess this is still pretty much a problem today. Of the top of my
head:
- Converting dates between different timezones - I needed this because I
trade in several exchanges around the world (NYSE, London, ...). You can
kinda do this in Erlang by setting the TZ environment variable before
calling date functions, but it is hacky and I could never put it to work.
- Fixed point arithmetic, for things like money. You don't need that for
share prices and such, but you will need it at least in a few places if you
care for accuracy, like for computing PnLs and such. This is not so much of
an issue when you are backtesting, but at least in my case I want stuff to
be as accurate as possible when doing real trading.
- Mathematical and scientific libraries. For things like statistical
arbitrage you will eventually need to compute more things besides
correlations, e.g. to do linear regressions and such. Erlang has no
libraries for this.
- FIX connectivity, or interfaces for standard brokers, like Interactive
Brokers.
- Graphical user interfaces. Implementing them in pure Erlang doesn't make
much sense because it is just too limited (although now you have bindings
for wx, so it might be better now).
- Charts. Even if you are not doing technical analysis, you still need to
generate simple charts like equity curves and such when backtesting.
Most of these things can be implemented in Erlang and some of them are not
even that hard, but it slows you down considerably. And of course you can
always write parts of your application in other languages and interface them
with Erlang, but that is quite cumbersome as well.
I eventually went down that way - I implemented parts of my application in
Java - but then I got to a point where it doesn't make much sense to use
Erlang anymore. If you are using Java for the GUI, and Java for talking with
the broker, and Java for generating reports, what are you using Erlang for?
I could easily be much more productive in a language like Java that has
libraries for everything, even if Java is such a pain to use. Of course, I
would much prefer to use Erlang, but at least for this project the lack of
libraries was really a problem.
Things might be better now there is the new FFI interface and such, but for
now I would just avoid Erlang for trading. Of course you might have a
different set of requirements - this might not apply to you if you just want
to implement a simple trading bot, and have another system in place for
managing orders, executions, etc...
Artur.
On Oct 1, 11:45 am, Joel Reymont <joe...@gmail.com> wrote:
> Has anyone used Erlang to backtest trading systems?
>
> This basically involves loading price and timestamp pairs
> and iterating over them calculating various things like
> moving averages, correlation, etc.
>
> Off the top of my head, Erlang does not seem to be the most
> efficient platform for this because prices would need to be
> stuffed into ETS or copied around otherwise.
>
> The most efficient way to manage price data is to simply keep
> appending quotes to the end of an mmap-ed file. I would love
> to represent the mmap-ed memory as a binary and I think it may
> even be possible. It would require some tricky work at the
> driver level, though.
>
> What do you think?
>
> Thanks, Joel
>
> ---
> fastest mac firefox!http://wagerlabs.com
Joel Reymont
On Oct 1, 2009, at 6:06 PM, Artur Matos wrote:
> Things might be better now there is the new FFI interface and such,
Is there a new FFI interface? If you are talking about loadable BIFs
then I would very much like to make use of it.
> Of course you might have a different set of requirements - this might
> not apply to you if you just want to implement a simple trading bot,
> and have another system in place for managing orders, executions,
> etc...
The platform I'm thinking about is rather simple. Consider
EasyLanguage (EL)
for example [1]. This is a Pascal-like language, the Basic of trading
languages judging by popularity. It's the language used by
TradeStation [2].
It's a neat and very capable language but using it requires you to
babysit
your desktop. You cannot put EL on the server for unattended but
monitored
trading. The language itself does not have heavy-duty statistics or
anything
of the sort. It does have a bunch of technical indicators like moving
averages.
In fact, most if not all the indicators are implemented in EL rather
than built in.
I have already written compilers from EL to C# in Haskell, OCaml and
Lisp [3].
I plan to write yet another compiler, this time in and to Erlang. I
then plan
to sell this "black box" execution system to brokerage houses and
enable them
to offer a value-added service to their clients.
Server-side (co-located?) execution does not need any graphics or
charting,
although a web interface would come in extremely handy.
[1] http://lambda-the-ultimate.org/node/2201
[2] http://www.tradestation.com
[3] http://groups.google.com/group/topdog/browse_thread/thread/d5fd8feac8e68331
---
fastest mac firefox!
http://wagerlabs.com
________________________________________________________________
Paul Mineiro
>
> On Oct 1, 2009, at 6:06 PM, Artur Matos wrote:
>
> > Things might be better now there is the new FFI interface and such,
>
> Is there a new FFI interface? If you are talking about loadable BIFs
> then I would very much like to make use of it.
While I'm also excited about the new FFI interface, I got really good
results (i.e., low latency, high throughput) doing something analogous
(interfacing with localmemcache, an mmap based db) from a linked-in
driver using port_control and PORT_CONTROL_FLAG_BINARY.
So I think you already have what you need to pull this off.
-- p
Kenneth Lundin
Please explain more.
I have also heard about the latency requirements for trading systems
but they are still very diffuse
to me. I got the impression that latency is a big selling point for
trading systems but in reality there might be
other factors that are more limiting and makes the hunting for latency
under a certain level unnecessary.
Please explain more.
A backtest system is not run in real time and there should be no
absolute requirements on latency?
Correct me if I am wrong.
On Thu, Oct 1, 2009 at 6:04 PM, Joel Reymont <joe...@gmail.com> wrote:
> Kenneth,
>
> On Oct 1, 2009, at 4:50 PM, Kenneth Lundin wrote:
>
>> It would be nice if you could explain what the system needs to do and
>> after that
>> why you think it is a problem to store prices in an ETS-table.
>> [...]
>> Why are you talking about mmapped files? Please explain for a novice
>> in trading systems.
>
> It's high-frequency trading systems I'm talking about. Where's Serge
> Aleynikov [1] when you need him?
>
> I'd like to process an incoming price quote, make a decision and submit a
> trade order with minimal latency. I know that this is normally done using
> C++ (or OCaml [1]) but I'm wondering how this can be achieved using Erlang.
>
You have a server supporting some protocol (probably FIX) , the price
quote arrives this way?
The decision is taken at the server?
The trade order is handled in the server or sent to another system?
From where , to where is the latency measured?
From your reasoning it seems that the encoding/decoding +
communication latency is not part of
the important latency? In that case why not?
Anyway there must be som latency value that is just good enough and
going below that is just unnecessary?
/Kenneth
Jayson Vantuyl
anything it will give you something to use as a baseline for
comparison and profiling will tell you how much you can eventually
save with a more complex approach.
Also, play with HIPE. It currently is a bit unstable for general use,
but judicious HIPE compilation of key modules might be very helpful.
Out of curiosity, what features of Erlang make you want to try this?
It would seem that it's made more for fault-tolerance / scalability
and less for the absolute lowest-latency.
Joel Reymont
On Oct 1, 2009, at 7:43 PM, Kenneth Lundin wrote:
> Please explain more.
Feel free to ask pointed questions.
> I got the impression that latency is a big selling point for
> trading systems
Correct. If you can react to the market faster than the next guy
then you win. There's even a new kind of high-frequency trading now
called latency arbitrage [1].
> but in reality there might be
> other factors that are more limiting and makes the hunting for latency
> under a certain level unnecessary.
There may be other factors but people pull tricks like disabling RTTI
in their C++ code, not using exceptions and moving as much as they can
to the compilation stage with C++ templates. This is in addition to
using
cache-friendly and lock-free data structures.
No need to put myself at a disadvantage from the start by picking an
inefficient approach to price data management and processing.
> A backtest system is not run in real time and there should be no
> absolute requirements on latency?
It's pointless to build one system for backtesting and another one
for execution. I'd rather have a single system handle everything.
> You have a server supporting some protocol (probably FIX) , the price
> quote arrives this way?
In my case it's gonna be via a C++ API, over the network.
> The decision is taken at the server?
Right.
> The trade order is handled in the server or sent to another system?
The order is sent to the broker via a C++ API, over the network.
> From where , to where is the latency measured?
The latency I'm talking about is measured from the moment the price
quote is received by Erlang to the moment Erlang tells the C++ API
to send the trade to the broker.
There's also the network latency but it should be minimal for a
server co-located at the exchange data center or close to it.
> From your reasoning it seems that the encoding/decoding +
> communication latency is not part of
> the important latency? In that case why not?
Encoding and decoding is part of the latency but network latency is not.
>
> Anyway there must be som latency value that is just good enough and
> going below that is just unnecessary?
No such thing :D [1]. It's a war out there over who can go down faster.
People are embedding FIX adapters and trading strategies into FPGAs
for microsecond latency [2]. Incidentally, the HPC Platform software
is written in Lisp. It's mostly compilation and VHDL generation, though.
Thanks, Joel
[1] http://www.google.com/search?client=safari&rls=en&q=latency+arbitrage&ie=UTF-8&oe=UTF-8
[2] http://www.hpcplatform.com/
---
fastest mac firefox!
http://wagerlabs.com
________________________________________________________________
Jayson Vantuyl
Basically, you are writing code that eventually figures into executing
trades (sometimes automatically). In the automatic systems, you are
literally racing other systems. This particular race is so
competitive (and lucrative) that companies build datacenters closer to
the trading house just so that they have less latency. It is not
unheard of to even tweak the communications timing on their uplinks,
use fibre just to shave off a few ms, or even embed assembly code.
In extreme cases, people have systems that watch for large buys /
sells, predict the follow-on effect it will have on the market, and
try to wedge in their own order to capitalize on the inevitable (but
lagging) price shifts. In any other industry, this level of
optimization would be well into the "diminishing returns" area.
However, financial markets operate at a big enough scale this sort of
thing is still quite profitable.
Jayson Vantuyl
> inefficient approach to price data management and processing.
Premature optimization *IS* the root of all evil. Specifically,
because you make decisions on optimization in the absence of
information. Processor cache is a very unpredictable beast in the
face of multiple cores. Trying to predict how it behaves when other
code is in the pipeline is unfortunately optimistic. Multicore
systems inherently increase latency and decrease cache performance.
Erlang is optimized for these. Any information you have regarding
cache behavior is highly unlikely to be directly applicable. Test and
profile, early and often. It's the only way to go.
It is entirely possible (due to the design goals of Erlang) that you
might not even be able to wring the speed you need out of it. Get
something running, profile it, and fail fast if necessary. Putting
2000 lines of C before running a profiler is not a timesaver--even in
the world of cutthroat performance.
--
Jayson Vantuyl
kag...@souja.net
Michael Truog
cost of the data to actually do the type of trading you want to do. If
you are trying to do low latency arbitrage type trading you might want
to relocate to New York for obvious reasons. The data costs of a
real-time feed with reliable data might be a bit bothersome and would
dig into any profit. I think that after you figured out the constraints
on the system due to network and data the design would take shape. The
design could be as simple as using Tradestation libraries, but any
slippage on order fills would be something you would discover after you
finished creating the system (and that sort of discovery is negative).
- Michael
Joel Reymont
On Oct 1, 2009, at 7:49 PM, Jayson Vantuyl wrote:
> I would really recommend starting out with a naive implementation.
> If anything it will give you something to use as a baseline for
> comparison and profiling will tell you how much you can eventually
> save with a more complex approach.
True. I think I'll still start by experimenting with mmap-ed binaries,
or at least a binaries-based approach. These can always be mmap-ed
later.
> Also, play with HIPE. It currently is a bit unstable for general
> use, but judicious HIPE compilation of key modules might be very
> helpful.
Yes, I may try porting HiPE to x86-64 when I have time to spare.
> Out of curiosity, what features of Erlang make you want to try
> this? It would seem that it's made more for fault-tolerance /
> scalability and less for the absolute lowest-latency.
Let's consider a few alternatives from the point of view of someone
who is trying to built a platform for sale, me. The platform is to
allow many users to host and run their trading strategies unattended,
somewhere on a server co-located near the exchange. The platform has
to have a web-based user and admin interface. I want to be able to
compile strategies to machine code for fast execution. I also want
some kind of typing to catch my errors and give me an additional layer
of assurance.
I can write the whole thing in Lisp. Lisp has a built-in compiler and
good web servers. I can take an HTTP POST-ed strategy and compile it,
making sure it will run fast. I have a free license to a commercial
Lisp but it requires royalties. I don't think I want to go there and I
don't want to spend money on a non-royalty commercial Lisp either.
Neither Lisp will give me fault tolerance, scalability or typing.
OCaml. I like the static typing, speed and ease of use (for me). I had
the most pleasant experience of all writing and debugging my compiler
in OCaml. OCaml does not have a built-in compiler, though. I will need
to schedule jobs to compile strategies. The OCaml web server (Ocsigen)
is not exactly proven and OCaml does not give me scalability of fault
tolerance.
C++. I like the static typing and Boost Spirit v2.1 seems like the
ticket for writing the compiler. Performance can truly be squeezed
here. I could build my engine into a web server (nginx?) and use LLVM
or NanoJIT to compile strategies. I will surely spend a hell of a long
time developing compared to other solutions. C++ does not give me
scalability but I could use ZeroMQ or something similar. Fault
tolerance will need to be ensured by making each trading strategy into
a separate process.
Erlang. There's a layer of typing with dialyzer. The web servers are
good, there's scalability and fault tolerance. Building a compiler
should be straightforward with leex and yecc. I can take HTTP POST-ed
strategies and compile them. There's always HiPE for that ultimate
performance boost. Performance may not be top-notch in the end but
development should be fast and debugging straightforward.
I can safely discard Lisp. I would like to use OCaml but I won't since
I'll have to built a distribution infrastructure, bind to ZeroMQ for
scalability and NanoJIT or LLVM for compilation to machine code. I
don't really see the point.
I see migrating to C++ as need arises, chunk by chunk. Erlang may end
up as the message bus or I may swap it out entirely. This won't happen
for a while or may never happen at all. Erlang just seems like the
lesser of all evils or, perhaps, the golden middle.
Joel Reymont
On Oct 1, 2009, at 8:07 PM, Jayson Vantuyl wrote:
> Multicore systems inherently increase latency and decrease cache
> performance. Erlang is optimized for these. Any information you
> have regarding cache behavior is highly unlikely to be directly
> applicable. Test and profile, early and often. It's the only way
> to go.
Erlang is often slow, even on multicore. Erlang is damn hard to
optimize. I have spent literally months trying to understand the
behavior and optimize OpenPoker and recently another system I build
for a client. The latter system scales to hundreds of thousands of
users on Amazon EC2 and makes use of up to 200 servers in simulation.
Both projects need high performance. I still bear the scars and I'm
still recovering.
I feel that Erlang is a good start but I also feel that I must start
the right way. After a good deal of time spent with Erlang, I'm very
wary of "use the naive approach first" suggestions. It's all good in
theory but does everyone making these suggestions have my amount of
practice ;-).
---
fastest mac firefox!
http://wagerlabs.com
________________________________________________________________
Jayson Vantuyl
> binaries, or at least a binaries-based approach. These can always be
> mmap-ed later.
modifies them, only makes new ones. Given that the compiler optimizes
for this (or should), I would be careful about modifying binaries
after they make it to Erlang.
> Yes, I may try porting HiPE to x86-64 when I have time to spare.
If you're very speed conscious, I would recommend running in 32-bit
mode. With everything I've done, it seems faster to page a GB of data
in/out of memory than it does to deal with the larger pointers--
especially in something pointer-heavy (like a dynamic language).
> Let's consider a few alternatives from the point of view of someone
> who is trying to built a platform for sale, me. The platform is to
> allow many users to host and run their trading strategies
> unattended, somewhere on a server co-located near the exchange. The
> platform has to have a web-based user and admin interface. I want to
> be able to compile strategies to machine code for fast execution. I
> also want some kind of typing to catch my errors and give me an
> additional layer of assurance.
In my experience, typing really doesn't catch errors anymore. That
said, dialyzer is wonderful.
> I can write the whole thing in Lisp. Lisp has a built-in compiler
> and good web servers. I can take an HTTP POST-ed strategy and
> compile it, making sure it will run fast. I have a free license to a
> commercial Lisp but it requires royalties. I don't think I want to
> go there and I don't want to spend money on a non-royalty commercial
> Lisp either. Neither Lisp will give me fault tolerance, scalability
> or typing.
Agreed.
> OCaml. I like the static typing, speed and ease of use (for me). I
> had the most pleasant experience of all writing and debugging my
> compiler in OCaml. OCaml does not have a built-in compiler, though.
> I will need to schedule jobs to compile strategies. The OCaml web
> server (Ocsigen) is not exactly proven and OCaml does not give me
> scalability of fault tolerance.
Okay. Haskell might be another option. If you're going typed, it's
the best, but it's probably suboptimal due to lack of libraries.
> C++. I like the static typing and Boost Spirit v2.1 seems like the
> ticket for writing the compiler. Performance can truly be squeezed
> here. I could build my engine into a web server (nginx?) and use
> LLVM or NanoJIT to compile strategies. I will surely spend a hell of
> a long time developing compared to other solutions. C++ does not
> give me scalability but I could use ZeroMQ or something similar.
> Fault tolerance will need to be ensured by making each trading
> strategy into a separate process.
Okay. Also, avoid templates and try compiling for size. In cache-
sensitive environments, I've found that -Os gets me better performance
that -O99 just because of the smaller code size.
>
> Erlang. There's a layer of typing with dialyzer. The web servers are
> good, there's scalability and fault tolerance. Building a compiler
> should be straightforward with leex and yecc. I can take HTTP POST-
> ed strategies and compile them. There's always HiPE for that
> ultimate performance boost. Performance may not be top-notch in the
> end but development should be fast and debugging straightforward.
In particular, debugging of concurrent code can be very nice. http://souja.net/2009/04/making-sense-of-erlangs-event-tracer.html
> I see migrating to C++ as need arises, chunk by chunk. Erlang may
> end up as the message bus or I may swap it out entirely. This won't
> happen for a while or may never happen at all. Erlang just seems
> like the lesser of all evils or, perhaps, the golden middle.
That makes sense. It sounds like the selling points are
debuggableness, fault-tolerance, scalability, and concurrency. That
all makes sense. Good luck on the latency front, though, as each of
those traits increases latency.
--
Jayson Vantuyl
kag...@souja.net
Joel Reymont
On Oct 1, 2009, at 8:13 PM, Michael Truog wrote:
> One thing you might want to consider before building the system is the
> cost of the data to actually do the type of trading you want to do.
> If you are trying to do low latency arbitrage type trading you might
> want
> to relocate to New York for obvious reasons.
There's no need to relocate to New York. I'm not going to run the
strategies
on my desktop and a server co-located in Chicago does not care that
I'm in Tenerife.
Jayson Vantuyl
> optimize. I have spent literally months trying to understand the
> behavior and optimize OpenPoker and recently another system I build
> for a client. The latter system scales to hundreds of thousands of
> users on Amazon EC2 and makes use of up to 200 servers in simulation.
in any language. That said, Erlang certainly takes a specific mindset
to optimize. Are these built with Distributed Erlang? I find that
it's rife with latency / weirdness. Working on an alternative to that
is one of my pet projects, although I strongly doubt that I can make
it as transparent as Distributed Erlang (a distribution plugin is
HARD, but links and monitors require deep integration with the VM).
> Both projects need high performance. I still bear the scars and I'm
> still recovering.
Please distinguish between low-latency, high-throughput, and both.
"High performance" and "fast" are hard terms to work with.
> I feel that Erlang is a good start but I also feel that I must start
> the right way. After a good deal of time spent with Erlang, I'm very
> wary of "use the naive approach first" suggestions. It's all good in
> theory but does everyone making these suggestions have my amount of
> practice ;-).
In my experience, optimizing first has never been better. I always
start out thinking so, but it's just not.
--
Jayson Vantuyl
kag...@souja.net
Raoul Duke
>> right way. After a good deal of time spent with Erlang, I'm very wary of
>> "use the naive approach first" suggestions. It's all good in theory but does
>> everyone making these suggestions have my amount of practice ;-).
>
> In my experience, optimizing first has never been better. I always start
> out thinking so, but it's just not.
the wrong thing: premature optimization.
the right thing: spending time on designing an appropriate architecture.
?
sincerely.
Joel Reymont
On Oct 1, 2009, at 8:43 PM, Jayson Vantuyl wrote:
> Erlang assumes that binaries don't change. Note that it never
> modifies them, only makes new ones. Given that the compiler
> optimizes for this (or should), I would be careful about modifying
> binaries after they make it to Erlang.
An Erlang binary in the driver is an iovec structure or something like
that. I assume I'll just create a new binary (structure) once I have
new data. I'll have to experiment with this.
> If you're very speed conscious, I would recommend running in 32-bit
> mode. With everything I've done, it seems faster to page a GB of
> data in/out of memory than it does to deal with the larger pointers--
> especially in something pointer-heavy (like a dynamic language).
I'm assuming it's an Erlang-specific suggestion, right? http://www.kx.com
flies with 64 bits, Mozilla's TraceMonkey Javascript engine is up to
20% faster with 64 bits on some benchmarks. There are more registers
available for the compiler to use, etc. In short, I disagree since
your suggestion is quite general and is not backed up by known use
cases.
> Okay. Haskell might be another option. If you're going typed, it's
> the best, but it's probably suboptimal due to lack of libraries.
I assume you have not used Haskell for heavy-duty projects but heard
of it and maybe tinkered with it. I used Haskell for a networking
project I extensively wrote about (google for Erlang vs Haskell). I
switched to Erlang in the end and felt much better. Haskell is lazy
which makes performance unpredictable.
> Okay. Also, avoid templates
Why? I interviewed with a hedge fund. They use templates extensively
and it makes sense. Partial template specialization is just the ticket
to optimize generic code at compile time.
> and try compiling for size. In cache-sensitive environments, I've
> found that -Os gets me better performance that -O99 just because of
> the smaller code size.
I agree with this.
> In particular, debugging of concurrent code can be very nice. http://souja.net/2009/04/making-sense-of-erlangs-event-tracer.html
Yes, thanks. I've been using tracing for years now.
> That makes sense. It sounds like the selling points are
> debuggableness, fault-tolerance, scalability, and concurrency.
And faster time to market.
> That all makes sense. Good luck on the latency front, though, as
> each of those traits increases latency.
Thanks. I'm not aiming to be the top dog but I would like to have a
very strong showing.
Joel Reymont
On Oct 1, 2009, at 8:50 PM, Jayson Vantuyl wrote:
> Concurrent + latency-sensitive + distributed == damn hard to
> optimize in any language.
Not necessarily hard. It's perfectly fine to be concurrent at the
process level, for example.
The difficulty with Erlang is tracking the state of all the message
queues at any given moment and just plain figuring out what's going on
among all the different processes. I have resorted to various tricks
apart from tracing, e.g. putting stuff into the process dictionary at
various stages of process life for ease of fetching via process_info.
> That said, Erlang certainly takes a specific mindset to optimize.
> Are these built with Distributed Erlang?
Distributed Erlang indeed. Very distributed. Check the recent netsplit
thread for some of the weirdness. Go back to the "Ultimate Erlang
Challenge" for more.
---
fastest mac firefox!
http://wagerlabs.com
________________________________________________________________
Joel Reymont
On Oct 1, 2009, at 8:54 PM, Raoul Duke wrote:
> the wrong thing: premature optimization.
>
> the right thing: spending time on designing an appropriate
> architecture.
I admit that I'm biased. My (contract) day job nowadays
is optimizing Firefox performance. I spend days staring
at profiles, I have become expert with DTrace and I'm trying
to think of how I can carry all that knowledge back to Erlang.
---
fastest mac firefox!
http://wagerlabs.com
________________________________________________________________
Jayson Vantuyl
> flies with 64 bits, Mozilla's TraceMonkey Javascript engine is up
> to 20% faster with 64 bits on some benchmarks. There are more
> registers available for the compiler to use, etc. In short, I
> disagree since your suggestion is quite general and is not backed up
> by known use cases.
in Python and Ruby. Unless you're generating machine code, I've never
seen 64-bit to be a big boost. While HIPE might make that a non-
issue, I've generally observed that VMs don't benefit from 64-bit in
that way.
> I assume you have not used Haskell for heavy-duty projects but heard
> of it and maybe tinkered with it. I used Haskell for a networking
> project I extensively wrote about (google for Erlang vs Haskell). I
> switched to Erlang in the end and felt much better. Haskell is lazy
> which makes performance unpredictable.
I've used it and loathed that lack of libraries. I was only pointing
out that it's type system is cleaner. Just trying to round out the
spectrum of options.
> Why? I interviewed with a hedge fund. They use templates extensively
> and it makes sense. Partial template specialization is just the
> ticket to optimize generic code at compile time.
Templates can drastically inflate code size and cause an extra
indirection at function call time. I've always found that, for
optimization, templates are middling, doing your own indirection in
straight C is king--especially when you give branching hints. Of
course, that might be premature optimization at this point. ;P
> Yes, thanks. I've been using tracing for years now.
I'll assume from your OpenPoker statement that you were doing tracing
with Erlang. I've found the documentation a bit lacking, so I like to
throw that example around to make sure that people can actually figure
out how to use it. Let's just say that dbg:tpl is not exactly an
obvious interface.
> And faster time to market.
Right on.
--
Jayson Vantuyl
kag...@souja.net
Jayson Vantuyl
> queues at any given moment and just plain figuring out what's going
> on among all the different processes. I have resorted to various
> tricks apart from tracing, e.g. putting stuff into the process
> dictionary at various stages of process life for ease of fetching
> via process_info.
mailboxes are the number one headache for me.
You can always get at the message queue with
erlang:process_info(self(),messages). I have a check in most of my
servers that, when I'm debugging, will dump the message queue to disk
if it gets over a certain size. Helps me track down places that need
to be synchronized to prevent run-ahead.
>> That said, Erlang certainly takes a specific mindset to optimize.
>> Are these built with Distributed Erlang?
>
> Distributed Erlang indeed. Very distributed. Check the recent
> netsplit thread for some of the weirdness. Go back to the "Ultimate
> Erlang Challenge" for more.
Yeah. Since distribution modules are C, I doubt it's going to get
fixed any time soon though. :(
--
Jayson Vantuyl
kag...@souja.net
Joel Reymont
On Oct 1, 2009, at 9:36 PM, Jayson Vantuyl wrote:
> Are you using vanilla tracing, sequential tracing, or both? Yes,
> the mailboxes are the number one headache for me.
I haven't tried sequential tracing, although I did consider it at one
point. See the message I just posted to the list for my tracing helpers.
> You can always get at the message queue with erlang:process_info(self
> (),messages). I have a check in most of my servers that, when I'm
> debugging, will dump the message queue to disk if it gets over a
> certain size. Helps me track down places that need to be
> synchronized to prevent run-ahead.
You can also crash the system and then inspect the dump for the
contents of the message queues, stacks, etc. It's not that difficult.
---
fastest mac firefox!
http://wagerlabs.com
________________________________________________________________
Alpar Juttner
> > Why? I interviewed with a hedge fund. They use templates extensively
> > and it makes sense. Partial template specialization is just the
> > ticket to optimize generic code at compile time.
> Templates can drastically inflate code size
It doesn't true today, due to inlining and optimization. Consider for
example 'std::vector<int> a;'. If you hane look at the header file you
will see that a single statement like 'a[5]=12;' results in a tremendous
amount of (nested) function calls. But the optimizer will reduce it to
same machine code as the corresponding C expression would result in.
> and cause an extra
> indirection at function call time.
I cannot even guess what do you mean by "extra indirection".
Regards,
Alpar
Michael Truog
>>> Why? I interviewed with a hedge fund. They use templates extensively
>>> and it makes sense. Partial template specialization is just the
>>> ticket to optimize generic code at compile time.
>>>
>> Templates can drastically inflate code size
>>
>
> It doesn't true today, due to inlining and optimization. Consider for
> example 'std::vector<int> a;'. If you hane look at the header file you
> will see that a single statement like 'a[5]=12;' results in a tremendous
> amount of (nested) function calls. But the optimizer will reduce it to
> same machine code as the corresponding C expression would result in.
>
better than g++ 2.95, and that we are reaching the new C++ changes with
tr1/C++0x or what will soon be C++1x. I think the main problems I have
seen with templates inflating code size have related to static
data/methods within the template class, so it is an issue with proper
design of the system (not the language itself).
Viktor Sovietov
Do you really think that software-based trading systems would survive
in the near future? Honestly, specifically accelerated hardware
solutions (most probably FPGA-based) should win the battle. This
discussion reminded me what Marc Battyani (you surely know him as the
Lisp expert) is doing right now in http://www.hpcplatform.com/ - it's
the compiler for FPGA accelerators, mostly targeted to serve financial
market's needs. Now it can look as a far perspective, but... it's so
typical for IT techs - most of them looked unobvious or unproductive
just a few years ago.
Sincerely,
--Victor
Joel Reymont
On Oct 2, 2009, at 5:48 PM, Viktor Sovietov wrote:
> Do you really think that software-based trading systems would survive
> in the near future?
Sure. Software can always be baked into hardware.
> Honestly, specifically accelerated hardware
> solutions (most probably FPGA-based) should win the battle. This
> discussion reminded me what Marc Battyani
I referred to Marc's solution in the beginning of this thread ;-).
---
fastest mac firefox!
http://wagerlabs.com
________________________________________________________________
Viktor Sovietov
> > Honestly, specifically accelerated hardware
> > solutions (most probably FPGA-based) should win the battle. This
> > discussion reminded me what Marc Battyani
>
> I referred to Marc's solution in the beginning of this thread ;-).
Ah, sorry, sometimes I'm so inattentive :)
But what I was going to say... Probably, low latency trading is too
specific problem area for Erlang. Erlang can find a niche as the
language for writing monitoring and management software (and there are
no others can beat Erlang on this field) but not for implementing main
functionality of such systems. It's like the situation with HPC.
Almost all production software for parallel supercomputers is MPI-
based, but Erlang is inapplicable in this area because it's great in
MPI but not in calculations, and even great MPI model of Erlang is too
expensive for HPC purposes.
Sincerely,
--Victor
andrew mmc
Have you got any idea of what level of performance you need? How close are
you to the exchange, for example? There isn't much point in slaving over
unit milliseconds when you are 400ms from the exchange, for example.
Personally, unless you are doing this with one of the very big banks, I
think you are fighting a losing battle. The big players (and the only
people actually making money doing this) colocate their trading servers with
the exchange servers - in the same room, with dedicated network connections.
Unless you can compete with that level of funding and access, there are
better alternatives to make money trading.
Serge Aleynikov
> Kenneth,
>
> On Oct 1, 2009, at 4:50 PM, Kenneth Lundin wrote:
>
>> It would be nice if you could explain what the system needs to do and
>> after that
>> why you think it is a problem to store prices in an ETS-table.
>> [...]
>> Why are you talking about mmapped files? Please explain for a novice
>> in trading systems.
>
> It's high-frequency trading systems I'm talking about. Where's Serge
> Aleynikov [1] when you need him?
Still here...
> I'd like to process an incoming price quote, make a decision and submit
> a trade order with minimal latency. I know that this is normally done
> using C++ (or OCaml [1]) but I'm wondering how this can be achieved
> using Erlang.
Don't want to be too repetitive to answers already posted here, but what
is your definition of "minimal latency"? If you are talking about
microseconds, having much logic coded in Erlang will not help you. At
the end you will end up with basic configuration/controlling
functionality in Erlang navigating a low-level component coded in some
low-level language (unless of course you are willing to invest time in
writing some domain-specific language for the task, which might be an
overkill given the scope of your work).
> A simple array of floats or doubles groups data together in memory and
> is very cache friendly. I figure that an Erlang binary is the closest I
> can get to an array in Erlang. mmap-ing a file of floats or doubles and
> wrapping it in a binary should avoid unnecessary copying of data from
> disk to memory.
Even if that component is fast, you'll still have a bytecode-related
overhead on the Erlang side reading/writing data to this structure
through copying. It is ok (and very convenient) for most applications,
but for what you are talking about that overhead may be a show stopper.
> I think operating on binaries representing mmap-ed files of price quotes
> is as close as you can get to low latency in Erlang. I know that
> premature optimization is the root of all evil but I'd rather have my
> teeth pulled than optimize Erlang. This is speaking from experience. I
> figure a much better way is to choose a concept that guarantees fast
> code from the start.
If you don't want to spend too much time coding C++, since you have
already done good work of abstracting mnesia backend, why don't you
create an Erlang driver for this database: www.fastdb.org. It's based
on memory mapped files, implemented in C++, is relatively fast compared
to other in-memory databases (about 250k lookups/s, 50k inserts/s), has
time-series capabilities, simple C-API. I implemented FastDB interface
in several languages, and generally it's good for systems that deal with
latencies in sub-milliseconds range. It may suffice for you unless you
need to stay in the low microseconds realm.
My 2c.
Serge