[erlang-questions] Lets build a stock exchange!
Joel Reymont
I want you to start an Amazon EC2 instance and join the "Hardcore
Erlang" cluster.
I want to build the biggest Erlang cluster in the world and push
Erlang to its limits.
I want to build a stock exchange and show you how to do it.
And of course I want to write a book about it.
The focus of the book is not changing, it's still fault-tolerance,
scalability, distribution, etc. What's changing is the software the
book is built around. Reading about how to build a stock exchange sure
as hell beats reading about a poker server.
What do you think?
Thanks, Joel
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://www.erlang.org/mailman/listinfo/erlang-questions
YC
Yes - agreed that a stock exchange is a lot more interesting than a poker server (but poker isn't bad either).
Is the goal to have a truly functional stock exchange that interfaces with other stock exchanges? Or one that has the capability but not the hook up?
yc
Joel Reymont
On Nov 11, 2007, at 2:24 AM, YC wrote:
> Is the goal to have a truly functional stock exchange that
> interfaces with
> other stock exchanges? Or one that has the capability but not the
> hook up?
I'm writing a book first and foremost so I'll probably leave the hook
up for later. It will depend on the amount of material in the book and
my time constraints.
Francis Norton
Francis.
Christian S
exchanges? Order matching procedures, what information do tax offices
want, typical features as in stop-loss rules...
Oh, and yes. A stock exchange is cooler than poker.
Torben Hoffmann
Anyone know a book on technical requirements that brokers put on stock
exchanges? Order matching procedures, what information do tax offices
want, typical features as in stop-loss rules...
Oh, and yes. A stock exchange is cooler than poker.
It may very well be the case that a stock exchange is cooler than poker, but I think that the poker server has some interesting aspects that could inspire others. E.g., the dynamic allocation of a server to run a new table as well as the handling of the messages between the players is similar in nature to some of the challenges there is in telecom with regards to group calls and where to "host" the call.
Furthermore, I would rather see a well-written book on hard-core Erlang hit its window of opportunity than see the whole thing turn into "let's make the coolest application that will never run in real life just in order to spice up a programming book"-kind of projects.
A well written book with a clear example of usage including discussions about where the general ideas can be applied is what I would bet my money on. If a potential buyer of the hard-core Erlang book cannot abstract from the specific case study used to teach the principles then it is time to utter "Joe, We have a problem!" ;-)
Cheers,
Torben
P.S. My first reaction to the stock exchange idea was "Wow!! Cool!!!", but then my analytical side issued a kill to that process for the reason listed above.
Joe Armstrong
I have no idea whether it would succeed or fail - if it succeeds then (gulp)
If it fails the interesting thing is not that it failed but WHY it failed.
If things fail and we know why we can do research and fix things. It
is the existence of things
that don't work that stimulates software research and development.
Interesting problems are very difficult to think up - asking the right
question is
often the really difficult bit - once asked lots of brains can try and
find the solution.
Joels suggesting poses a difficult question - great - let's try and answer it.
If Joel can in any way structure the problem then there are loads of
us that can try and
write the individual bits.
Breaking the system is great - a program that breaks the compiler is
far better than 100 that work.
A program that breaks mnesia is far better than 1000 that work.
If this breaks Erlang great - we can fix it and make an even better system.
Breaking stuff is easy - the interesting point is *where* does the system break!
/Joe Armstrong
jm
Was curious so when looking for info on the Australian Stock Exchange
(ASX). Try
-> Participant (top menu)
-> Library (top menu)
-> Manuals (in list of item)
Some other items listed at this level require a login.
I can't find the rules and regulations governing trading, but there is
discussion of it in a general sense floating around somewhere on the net
which would be accurate enough for the job at hand as the idea is to
show case erlang and not to provide a realistic simulation of a real
stock exchange. For this to be a viable demostration project, you could
ignore most of the trading rules and concentrate on
* the interface to the traders (I think that can be found in the above docs)
* the phases of the market (closed, pre-open, open, pre-close, etc)
* order matching
* order tracking
* reporting
* outstanding orders and market depth
* etc
As far as the actual trading rules goes you could write your own and
show how to make it "easy" to modify the system to change the systems
trading rules without taking the system down (hot code updates) or
needing a domain specific rule language which would be the port of first
call in many another language.
This looks to be a much better project than a poker site as this could
easily be modified into any type of clearing house site. This type of
project is more likely to be of interest to main stream business as they
can see the direct applicability to their situation where as a poker
site may not have the same impact as intended.
Jeff.
Michael Regen
Just my humble two cents:
On Nov 11, 2007 2:58 AM, Joel Reymont <joe...@gmail.com> wrote:
> I want to build the biggest Erlang cluster in the world and push
> Erlang to its limits.
>
> I want to build a stock exchange and show you how to do it.
This means either a lot of money and/or people are needed. So, how do
we motivate people to spend money or time?
I guess it will be easier to make people spend time than money.
Choosing the right project will affect very much how many people will
join and whether they need to spend money or time.
Building a stock exchange is an incredibly interesting aim from the
technical point of view and could certainly push Erlang to its limits
- if they exist :).
But if you want to build the biggest Erlang cluster than you need
something which is actually useable in real life for at least some
time. That would also be a much bigger benefit for the people to join
compared to just build a showcase example.
And here I'm not sure how big financial and legal hurdles will be to
actually operate a stock exchange. Does anybody have some insight in
this topic?
Just a half-baked idea and my empty brain was not able to figure out a
practical application so far. Anyway, maybe just think about it for a
short while and forget it soon if it leads to nothing: What about some
kind of socially desirable project. E.g. something one or better more
NGOs can use. Something that motivates people to join because they can
feel good if they contribute? Feel better than if they just
contributed code to the public.
Sometimes consulting companies compete hard for poor paid contracts
with NGOs just because it will be great PR for them. So there might
also be a bigger chance to push Erlang.
But I have to agree, this idea is not worth much without an
application. And of course, building something which is usable also
means responsibility...
So I hope this mail only drives people away from the stock exchange
idea if someone finds a good application soon.
Cheers,
Michael
YC
IMO the relevance depends on the depth and intimacy the system is explained and covered, and whether author places emphasis on knowledge transfer. I never find much use of the abundant trivial bank examples, and even the shoucast example popularized by Practical Lisp etc.
On the other hand, books such as Norvig's PAIP covers examples that are not in common use, but because the author focuses so much more on teaching the subject matter and how to use Lisp to solve them, I found myself walking learning something without ever using the example.
Without getting to that level of detail, then best examples are relevant ones - i.e. examples that can be immediately of use by simply copy and paste. To do that would mean narrowing the target audience, but that might actually be a good thing.
If going with a relevant angle - then a lot of people today are working on web-based applications and database related applications, so if the example solves at least some of the web and database problems than it won't hurt ;)
My 2 cents,
yc
Anthony Kong
**********************************************************************
***** IMPORTANT INFORMATION *****
This document should be read only by those persons to whom it is
addressed and its content is not intended for use by any other
persons. If you have received this message in error, please notify
us immediately. Please also destroy and delete the message from
your computer. Any unauthorised form of reproduction of this message
is strictly prohibited.
St George Bank Limited AFSL 240997, Advance Asset Management Limited
AFSL 240902, St George Life Limited AFSL 240900, ASGARD Capital Management Limited
AFSL 240695 and Securitor Financial Group Limited AFSL 240687 is not liable for
the proper and complete transmission of the information contained in
this communication, nor for any delay in its receipt.
**********************************************************************
Damien Morton
incorporation of streaming-database and continuous-query capabilities
into the underlying message bus.
Joel Reymont
On Nov 11, 2007, at 9:10 PM, Torben Hoffmann wrote:
> It may very well be the case that a stock exchange is cooler than
> poker, but
> I think that the poker server has some interesting aspects that could
> inspire others.
I can always describe the architecture of the poker server in a series
of blog articles.
> Furthermore, I would rather see a well-written book on hard-core
> Erlang hit
> its window of opportunity than see the whole thing turn into "let's
> make the
> coolest application that will never run in real life just in order
> to spice
> up a programming book"-kind of projects.
I firmly plan to hit my window of opportunity, that's my priority #1.
I may have confused a lot of people by forgetting to mention that I
don't plan to build a business. I do plan to write a book, one with
examples based around software. I'm passionate about all things
trading and I'm not passionate about poker, so I chose to build and
describe stock exchange software instead.
> A well written book with a clear example of usage including
> discussions
> about where the general ideas can be applied is what I would bet my
> money
> on.
It will be done!
Thanks, Joel
_______________________________________________
Joel Reymont
On Nov 11, 2007, at 11:14 PM, jm wrote:
> As far as the actual trading rules goes you could write your own and
> show how to make it "easy" to modify the system to change the systems
> trading rules without taking the system down (hot code updates)
Great point and excellent ASX pointers as well!
> This looks to be a much better project than a poker site as this could
> easily be modified into any type of clearing house site.
I doubt the world needs another stock exchange. Everybody knows what a
stock exchange is, though, and the concept could be extended to other
types of "prediction markets".
Thanks, Joel
_______________________________________________
Joel Reymont
On Nov 11, 2007, at 9:45 PM, Joe Armstrong wrote:
> I have no idea whether it would succeed or fail - if it succeeds
> then (gulp)
>
> If it fails the interesting thing is not that it failed but WHY it
> failed.
I plan to either succeed or fail spectacularly. Either way there will
be a great lesson to learn.
> If Joel can in any way structure the problem then there are loads of
> us that can try and
> write the individual bits.
I still haven't figured out how to incorporate community feedback. I
did create a Google group where the book and ideas can be discussed,
though. See http://groups.google.com/group/hardcore_erlang.
I assume that not everyone interested in the book or the stock
exchange software will be a subscriber to this list.
I doubt I will have time to run a full-scale open source project in
addition to writing the book and managing my regular consulting
assignments. I'll sure I will run into software problems along the
way, though. I will put those problems into context and ask for help
then.
Once the book is written, I should have plenty of time to further
develop the concept and associated software.
Thanks, Joel
_______________________________________________
Joel Reymont
On Nov 11, 2007, at 11:33 PM, Michael Regen wrote:
> This means either a lot of money and/or people are needed.
I'm looking to write a book and enough software to demonstrate
the concepts in the book. I will likely extend the software afterwards
but I definitely do not plan to go into the exchange business...
at least not at the moment and not a stock exchange.
There are other kinds of markets that could be built and run and then
there's venture capital!
> But if you want to build the biggest Erlang cluster than you need
> something which is actually useable in real life for at least some
> time. That would also be a much bigger benefit for the people to join
> compared to just build a showcase example.
I thought about this for quite a while (at least a few minutes).
I imagined a cluster of 1k EC2 instances and one of 100k instances.
You could probably run a few Erlang nodes per instance as well
which will positively make this a huge cluster.
I tried to figure out what could entice people to boot up their
Erlang EC2 instance and add it to a cluster and I couldn't come up
with anything specific, apart from sheer interest and scientific value.
You could crawl the internet Google-style or try to build some sort
of a virtual world. Trading is my passion so I finally settled on
a massive stock exchange.
It can be a stock exchange or a prediction market, the essence
of the book will stay the same regardless.
Thanks, Joel
_______________________________________________
Joel Reymont
On Nov 12, 2007, at 12:10 AM, YC wrote:
> Agreed that cool factor shouldn't be it for a book (since Joel
> confirmed
> that book is the first priority), but there probably is an element of
> marketing there :)
There's a large element of marketing here.
> On the other hand, books such as Norvig's PAIP covers examples that
> are not
> in common use, but because the author focuses so much more on
> teaching the
> subject matter and how to use Lisp to solve them, I found myself
> walking
> learning something without ever using the example.
I very much like your PAIP suggestion. I'll have to keep it handy
while I'm writing.
Overall, I received so much excellent feedback and ideas that it's
gonna take me
some time just to organize them. Between the book, the blog, this
mailing list
and the book list, I'll have to swear off Reddit, Wall St Journal and
my RSS feeds!
In fact, I have already unsubscribed from the Haskell and OCaml lists
just to keep
a tight focus :-).
> Without getting to that level of detail, then best examples are
> relevant
> ones - i.e. examples that can be immediately of use by simply copy and
> paste.
That would be more of a "cookbook" approach and I'm not looking to
write a cookbook.
> If going with a relevant angle - then a lot of people today are
> working on
> web-based applications and database related applications, so if the
> example
> solves at least some of the web and database problems than it won't
> hurt ;)
Web apps in Erlang is a whole separate book, one that I don't plan to
write.
Thanks, Joel
_______________________________________________
Joel Reymont
On Nov 11, 2007, at 11:23 PM, Anthony Kong wrote:
> But I think people here may want to model NYSE. After all more people
> are familiar with NYSE's operations, and the final implementation will
> be more attractive to general public out there.
Anyone familiar with the NYSE operations? Any documents out there?
> I guess a question back to Joel: how
> do you manage the scope? It is really no easy task.
I think I'll go bottom-up here. I will need chapters with in-depth
coverage
of Mnesia, debugging and testing, profiling and optimization, etc.
What I plan
to do is write those chapters first to show progress and to gauge how
many pages
I have written.
I will then need to go back and cherry-pick stock exchange modules
that would be
a good fit for each chapter. Market data (stock quotes) distribution
would fit
right in profiling and optimization, for example.
The book needs to be about 200 pages at a minimum so I may have to
include
just the choice portions of the stock exchange software and describe
the rest
in my blog or somewhere else.
The software project does not need to end once the book is published
and I will not
be able to cover everything in the book. I will be tough to strike a
balance
but I will often ask for your feedback!
Thanks, Joel
_______________________________________________
Joel Reymont
On Nov 11, 2007, at 11:29 PM, Damien Morton wrote:
> A key aspect of any next generation trading system will be the
> incorporation of streaming-database and continuous-query capabilities
> into the underlying message bus.
Speaking of the message bus, it already exists and is called AMQP which
stands for Advanced Message Queuing Protocol. Fortunately for me,
there's RabbitMQ which implements AMQP and is written in Erlang.
Brian Granger
as a model application of erlang:
1) The latency demands can be very important. Building a system that
scales while maintaining latency constraints is a real challenge.
Showing how erlang makes that possible would be very interesting.
2) Reliability. In any system dealing with finacial transactions,
reliability must be there in all respects. Erlang is perfect for this
type of thing and showing how it can be done would be fantastic.
Brian
Doug Edmunds
check out how they work. Maybe someone can smooze the
programmers for some general stock market design help:
http://www.howthemarketworks.com/
http://simulator.investopedia.com/
Doug Edmunds
check out how they work. Maybe someone can smooze the
programmers for some general stock market design help:
http://www.howthemarketworks.com/
http://simulator.investopedia.com/
Doug Edmunds
check out how they work. Maybe someone can smooze the
programmers for some general stock market design help:
http://www.howthemarketworks.com/
http://simulator.investopedia.com/
Michael Regen
> Anyone familiar with the NYSE operations? Any documents out there?
Not familiar with it. But there is a quite good description for how
their connectivity to the outside looks like:
http://www.nysetransacttools.com/
The following gives you some overview how things fit together:
http://www.nyse.com/press/1177426714683.html
I guess quite a lot of useful search terms can be extracted there.
They even published their data dictionary:
http://www.nysetransacttools.com/resources/
Cheers,
Michael
Dmitrii 'Mamut' Dimandt
> I will then need to go back and cherry-pick stock exchange modules
> that would be
> a good fit for each chapter. Market data (stock quotes) distribution
> would fit
> right in profiling and optimization, for example.
>
>
This reminds me of Deitel&Deitel's book on C++. They build an elevator
throughout the book adding newer and more advanced features with each
chapter. Don't know if this could be applicable to yor book though...
Torben Hoffmann
Torben,I can always describe the architecture of the poker server in a series
On Nov 11, 2007, at 9:10 PM, Torben Hoffmann wrote:
> It may very well be the case that a stock exchange is cooler than
> poker, but
> I think that the poker server has some interesting aspects that could
> inspire others.
of blog articles.I firmly plan to hit my window of opportunity, that's my priority #1.
> Furthermore, I would rather see a well-written book on hard-core
> Erlang hit
> its window of opportunity than see the whole thing turn into "let's
> make the
> coolest application that will never run in real life just in order
> to spice
> up a programming book"-kind of projects.
I may have confused a lot of people by forgetting to mention that I
don't plan to build a business. I do plan to write a book, one with
examples based around software. I'm passionate about all things
trading and I'm not passionate about poker, so I chose to build and
describe stock exchange software instead.
If you take a look a the mails in this thread you see a lot of enthusiastic people - which is good - but my concern for the book is that most of them are enthusiastic about the prospects of building a stock exchange. This is very good indeed, but what Erlang needs the most is to keep momentum and I feel that could be jeopardised if too many things are thrown up into the air at the same time.
Living inside a big company (Motorola) it is more important for me to get ammunition that shows the momentum than the particular case study in the book.
It will be done!
> A well written book with a clear example of usage including
> discussions
> about where the general ideas can be applied is what I would bet my
> money
> on.
Super!
Cheers,
Torben
P.S. The views expressed above are 100% my own and is not associated with my employer.
Aaron Feng
Aaron
http://wagerlabs.com/archives/129.html
I want you to start an Amazon EC2 instance and join the "Hardcore
Erlang" cluster.
I want to build the biggest Erlang cluster in the world and push
Erlang to its limits.
I want to build a stock exchange and show you how to do it.
And of course I want to write a book about it.
The focus of the book is not changing, it's still fault-tolerance,
scalability, distribution, etc. What's changing is the software the
book is built around. Reading about how to build a stock exchange sure
as hell beats reading about a poker server.
What do you think?
Thanks, Joel
--
http://wagerlabs.com
Valentin Micic
chance to do something similar (i.e. build a stock exchange system) in their
careers? To me, this sounds like teaching someone to build affordable
housing by explaining how to build a pyramid. If the main goal of writing a
book is to popularize a specific subject, the stock exchange system, IMHO,
would not cut. If you want a popular support, write a book about popular
subject. For example, I think that erlang's implementation of jabber
protocol (XMPP, etc), did a great deal to erlang's popularity...But then
again, how many books do we need?
V.
David Mercer
e-commerce engine, or a hospital information system, or something else that
may have very similar reliability, scalability, security, and audit
requirements.
For that matter, if I plan to build a poker server, I could probably apply
much of what is learned from building a stock exchange. The opposite is not
necessarily true. I'd much rather see an application that hits more the
sweet spot of Erlang (massive soft-real-time systems) than a toy application
conjured up to illustrate the same.
Cheers,
David
Christian S
I imagine the stock exchange to be an example of something that
needs throughput and high service levels. Lessons learned will be
applicable for other projects. A system related to the financial sector
can also help to popularize Erlang more there.
2007/11/12, Valentin Micic <vale...@pixie.co.za>:
Patrick Logan
> protocol (XMPP, etc), did a great deal to erlang's popularity...But then
> again, how many books do we need?
Given that the existing XMPP books are so old as to be labeled
"Jabber" maybe a book could serve multiple purposes:
* Building a large system with Erlang.
* Providing newer information about XMPP, e.g. Pub/Sub.
* Demonstrating the use of XMPP in an integration rather than IM scenario.
Joel Reymont
XMPP can just go and read the ejabberd source code or the spec. Any
book about the same is automatically destined to gather dust on the
shelves.
When I buy a book I want to read about something exciting, something
that challenges my skills and imagination. I may not get to build a
pyramid or a stock exchange but neither do I want to read about
affordable housing!
Joel
On Nov 12, 2007, at 7:00 PM, Patrick Logan wrote:
>> * Building a large system with Erlang.
> * Providing newer information about XMPP, e.g. Pub/Sub.
> * Demonstrating the use of XMPP in an integration rather than IM
> scenario.
David Mitchell
There's better coders than me around, but I work for a broker now, was
on the trading floor in Australia in October 1987 and thus experienced
first hand what happens when trading systems fail, and have traded my
own futures account since 1999 and thus have a significant interest on
several levels.
Bring it on
Dave M
G Bulmer
...
> I want you to start an Amazon EC2 instance and join the "Hardcore
> Erlang" cluster.
>
> I want to build the biggest Erlang cluster in the world and push
> Erlang to its limits.
> I want to build a stock exchange and show you how to do it.
Why not build something that we could use?
> And of course I want to write a book about it.
As long as it's interesting, the more the merrier.
> The focus of the book is not changing, it's still fault-tolerance,
> scalability, distribution, etc.
I'd like to read about that.
> What's changing is the software the
> book is built around. Reading about how to build a stock exchange sure
> as hell beats reading about a poker server.
I agree, poker is a bit dull.
> What do you think?
Sorry to be out of step, but I'd prefer something that *I* could
imagine using.
How about a replacement for eBay auctions (or is Erlang what the e
mans now?-)? eBay has fault-tolerance, scale, distribution challenges.
I could sell my old programming books, which have become useless to
me, now that I have started using Erlang ;-)
Craig's list (http://sfbay.craigslist.org/) has shown that a free
classified-ads system can become popular in over 50 countries.
Go the next step, and provide classified ads and auctions.
To make it more fun, you could provide several bidding systems, not
just highest bidder, like Dutch Auction, etc.
There has been some pretty sneaky work on designing auctions (a well
known one was the scheme used by the UK government when selling the
3G mobile phone spectrum), so there is lots of scope for extending
and enhancing the business-process side if anyone wants to do that.
Anyone who sells anything at auction could use it. Probably payment
would be pushed off to paypal, or any other payment mechanism.
A good solution would support 'live auctions', and keep the bids in
sequence.
It'd have some of the challenges of a stock market, but you'd avoid
having to provide liquidity to make it useable (I realise your focus
is a book, but wouldn't you like millions of people to use the
software?).
Just a thought,
G Bulmer
jm
>> What do you think?
>
> Sorry to be out of step, but I'd prefer something that *I* could
> imagine using.
>
> How about a replacement for eBay auctions (or is Erlang what the e
> mans now?-)? eBay has fault-tolerance, scale, distribution challenges.
> I could sell my old programming books, which have become useless to
> me, now that I have started using Erlang ;-)
>
> Craig's list (http://sfbay.craigslist.org/) has shown that a free
> classified-ads system can become popular in over 50 countries.
> Go the next step, and provide classified ads and auctions.
>
> To make it more fun, you could provide several bidding systems, not
> just highest bidder, like Dutch Auction, etc.
> There has been some pretty sneaky work on designing auctions (a well
> known one was the scheme used by the UK government when selling the
> 3G mobile phone spectrum), so there is lots of scope for extending
> and enhancing the business-process side if anyone wants to do that.
>
> Anyone who sells anything at auction could use it. Probably payment
> would be pushed off to paypal, or any other payment mechanism.
>
> A good solution would support 'live auctions', and keep the bids in
> sequence.
Stock market are well suited to the buying and selling any classes of
items which are identitical, eg, equity in a company, futures contracts,
commodities (beef, gold, etc) and are more common than may you at first
think. The auction model is more suited to individual and unique items,
eg. antiques, spectrum blocks, second hand goods, etc. Both are open
makets though. In the middle is a whole range of markets including the
one you bought your new fridge in. There's certainly a lot around on the
theory of designing a good auction just type "auction theory" into your
favorite search engine or look at wikipedia's page on the subject
http://en.wikipedia.org/wiki/Auction_theory The most common market at a
guess would be those in the middle either B2B or B2C trying to find a
plumber, a new desk, or a manufacturer for your new widget. So if your
shooting for the potential size of audience that's where you'd aim, but
I think that is also the weakest in that it is ill defined.
I suppose comes down to finding a topic which highlights Erlangs good
side, inspires the potential reader to buy the book, motivates the
reader to try things, and proves valuable as a reference far into the
future. As such I'd rather any proposed book not be too heavy on the web
stuff and much more on the infrastructure side of things.
That being said I don't think either the auction or stock market ideas
are bad and it would be nice to see something less teleco/web industry
centric discussed in depth.
Jeff.
Brian Granger
> mans now?-)? eBay has fault-tolerance, scale, distribution challenges.
> I could sell my old programming books, which have become useless to
> me, now that I have started using Erlang ;-)
>
> Craig's list (http://sfbay.craigslist.org/) has shown that a free
> classified-ads system can become popular in over 50 countries.
> Go the next step, and provide classified ads and auctions.
>
> To make it more fun, you could provide several bidding systems, not
> just highest bidder, like Dutch Auction, etc.
> There has been some pretty sneaky work on designing auctions (a well
> known one was the scheme used by the UK government when selling the
> 3G mobile phone spectrum), so there is lots of scope for extending
> and enhancing the business-process side if anyone wants to do that.
>
> Anyone who sells anything at auction could use it. Probably payment
> would be pushed off to paypal, or any other payment mechanism.
>
> A good solution would support 'live auctions', and keep the bids in
> sequence.
>
> It'd have some of the challenges of a stock market, but you'd avoid
> having to provide liquidity to make it useable (I realise your focus
> is a book, but wouldn't you like millions of people to use the
> software?).
These problem domains lack one thing that makes a stock exchange such
an interesting problem - the need for low latency! Building a large
scale, distributed, fault-tolerant system is one thing - doing it
while maintaining low latency is a whole different ball game. Showing
that Erlang is well matched to these problems is a huge thing.
Valentin Micic
V.
Well just wishing to do it is not going to teach you any of these things
you've mentioned.
G Bulmer
>> mans now?-)? eBay has fault-tolerance, scale, distribution
>> challenges.
>> A good solution would support 'live auctions', and keep the bids in
>> sequence.
>>
>> It'd have some of the challenges of a stock market, but you'd avoid
>> having to provide liquidity to make it useable (I realise your focus
>> is a book, but wouldn't you like millions of people to use the
>> software?).
>
> These problem domains lack one thing that makes a stock exchange such
> an interesting problem - the need for low latency! Building a large
> scale, distributed, fault-tolerant system is one thing - doing it
> while maintaining low latency is a whole different ball game. Showing
> that Erlang is well matched to these problems is a huge thing.
I think a 'live auction' may need this too.
Further, I am not sure how low the latency will be; I can't ping
amazon, but when I ping google I get round-trip averages over 50 mSec
Revisiting the original post:
> I want you to start an Amazon EC2 instance and join the "Hardcore
> Erlang" cluster.
>
> I want to build the biggest Erlang cluster in the world and push
> Erlang to its limits.
My problem is I don't see why *I* would run a stock exchange
application, so I don't see why *I* would run Amazon EC2 instances.
On the other hand, I do see why I and others might want to run EC2
instances for auctions.
A plausible explanation for why I'd run the stock exchange
application would help me a lot.
Adjusting resources by starting and stopping EC2 instances also seems
to be a reasonable fit to auctions. Having a 'hub' which contains the
product catalogue would seem to be an enabler for the whole system,
and separate EC2 instances would be used for auctions (with some 'low-
rent' instances for low-activity auctions).
I am not 'wedded' to any particular application.
I *would* like to see Erlang running well on EC2, because that seems
to be great complementary technology.
I would also like to see something which would actually get run at
scale, and exploit the EC2 model, so that Erlang+EC2 gets proven.
Joel Reymont
On Nov 15, 2007, at 12:55 PM, G Bulmer wrote:
>>> (I realise your focus
>>> is a book, but wouldn't you like millions of people to use the
>>> software?).
No, I wouldn't. The software I will create for the book is just
educational software. I will use this software to demonstrate Erlang
features and to try to build a huge EC2 cluster.
I will likely develop the software for millions of people to use after
I write the book. It will be a commercial effort based on the concepts
described in the book. I don't think it's possible or viable to
develop production software and write a book about this software at
the same time. There's not enough time or pages in the book to
accomplish such a feat.
>> Further, I am not sure how low the latency will be; I can't ping
>> amazon, but when I ping google I get round-trip averages over 50 mSec
I think latency here is the delay introduced by the software itself,
not the delay introduced by the internet.
> My problem is I don't see why *I* would run a stock exchange
> application, so I don't see why *I* would run Amazon EC2 instances.
You don't need to run a stock exchange application. You may want to
add your instance to the cluster to take part in the Erlang
scalability test.
> On the other hand, I do see why I and others might want to run EC2
> instances for auctions.
> A plausible explanation for why I'd run the stock exchange
> application would help me a lot.
Writing a book is a marketing venture with an element of teaching. It
helps to be passionate about the subject one is writing about. I'm not
passionate about auctions but I am passionate about all things
trading. I don't want to sell books or have a need to. I do, on the
other hand, want to show that Erlang is a good fit for high-
availability financial apps. This is my focus.
> Adjusting resources by starting and stopping EC2 instances also
> seems to be a reasonable fit to auctions. Having a 'hub' which
> contains the product catalogue would seem to be an enabler for the
> whole system, and separate EC2 instances would be used for auctions
> (with some 'low-rent' instances for low-activity auctions).
I'm putting together all the feedback that I received so far and
trying to organize it into a table of contents. The book is turning
out to be quite thick. May I suggest that you take up a parallel
effort and describe how to build auctions on EC2 in a series of blog
posts? The software you develop will be commercially viable and useful
to you. It will serve as additional proof of viability of Erlang on EC2.
Thanks, Joel
_______________________________________________
G Bulmer
Oh, okay. I am sorry to seem a bit slow.
Summary:
- You want something interesting enough to induce a bunch of us to
run EC2 instances in order to test a humongous Erlang+EC2 cluster.
- It has to be interesting to you so that you get it done.
- It does *not* need to be production quality.
> On Nov 15, 2007, at 12:55 PM, G Bulmer wrote:
>>>> (I realise your focus
>>>> is a book, but wouldn't you like millions of people to use the
>>>> software?).
>
> No, I wouldn't. The software I will create for the book is just
> educational software. I will use this software to demonstrate
> Erlang features and to try to build a huge EC2 cluster.
...
> I will likely develop the software for millions of people to use
> after I write the book. .... There's not enough time or pages in
> the book to accomplish such a feat.
Okay, I understand, and can agree that designing and building a
usable stock exchange, at the same time as writing a book, may be too
much work in the time window available.
>>> Further, I am not sure how low the latency will be; I can't ping
>>> amazon, but when I ping google I get round-trip averages over 50
>>> mSec
>
> I think latency here is the delay introduced by the software
> itself, not the delay introduced by the internet.
Well, Amdahl's law says the largest component of performance
dominates optimisations ...
So, is your plan to run the whole thing within EC2?
>
>> My problem is I don't see why *I* would run a stock exchange
>> application, so I don't see why *I* would run Amazon EC2 instances.
>
> You don't need to run a stock exchange application. You may want to
> add your instance to the cluster to take part in the Erlang
> scalability test.
Ah, ha! I think I've got it!
>> ...
> .... May I suggest that you take up a parallel effort and describe
> how to build auctions on EC2 in a series of blog posts? The
> software you develop will be commercially viable and useful to you.
> It will serve as additional proof of viability of Erlang on EC2.
I am not passionate about auctions, it just seemed more likely that
people would use it, and hence it'd get a large number of instances
being 'thrashed' by live traffic. I now understand that is a 'non-
goal'. You only need something big for a 'cluster-fest'.
Maybe I will wait for the extremely-large-scale Erlang+EC2 cluster
issues to be solved.
That I am interested in!
Good luck
G Bulmer
Brian Granger
> My problem is I don't see why *I* would run a stock exchange
> application, so I don't see why *I* would run Amazon EC2 instances.
> On the other hand, I do see why I and others might want to run EC2
> instances for auctions.
> A plausible explanation for why I'd run the stock exchange
> application would help me a lot.
I don't imagine that many people would want to run a stock exchange or
auction software themselves. But, I think that is not the point. The
point is that _many_ of us (potential readers of the book) might
want/need to build systems that have similar properties (distributed,
fault tolerant, low-latency, etc.) to these systems. At that level, I
also don't care what the book uses as an example - as long as the
example demonstrates the relevant ideas. To me low-latency is really
important.
G Bulmer
Okay, so we are pretty much on the same page.
>> My problem is I don't see why *I* would run a stock exchange
>> application, so I don't see why *I* would run Amazon EC2 instances.
>> ...
>
> I don't imagine that many people would want to run a stock exchange or
> auction software themselves.
Oh, I may be interested if I wanted to run a 'real-time' auction for
my own stuff. Paying for an EC2 instance for a few hours to avoid
eBay fees sounds interesting if it's easy. eBay and others
demonstrate that there is lots of content, and liquidity in auctions,
but who cares.
I will stop flogging the horse now, it is deceased, and going mouldy.
> But, I think that is not the point. The
> point is that _many_ of us (potential readers of the book) might
> want/need to build systems that have similar properties (distributed,
> fault tolerant, low-latency, etc.) to these systems.
Agreed.
> At that level, I also don't care what the book uses as an example -
> as long as the
> example demonstrates the relevant ideas. To me low-latency is really
> important.
I am interested in low-latency too.
In my simple machine-to-machine experiments, I observe approx. 1% of
message round-trips to be more than an order of magnitude slower than
the other 99%.
G Bulmer
PS - This is the sort of problem that I would attack using DTrace, if
I could see the function call or message send/receive within each
Erlang node.