Results for happs-tutorial on an 8GB server
tphyahoo
testing happs-tutorial.
The result was that we got up to about 20,000 users before using up
ram. So, this reinforces my growing belief that to scale up a happs
app to a large userbase, some sacrifices have to be made with regards
to keeping all state in memory.
I pasted a chat snip below if anyone's curious.
******
MightyByte: Don't add any more users.
It looks like my RAM is pretty much full now.
me: Ok, we're up to 2,000,000 jobs.
MightyByte: 6488M used by happs-tutorial.
me: I guess we should stop.
And think. :)
MightyByte: Yeah
Can I kill the server or do you want to do anything else?
me: Is the server using swap?
MightyByte: Not sure
me: top, then F, then choose field n, then return.
(assuming unix)
sorry, choose field p
It'll sort by the apps that are using the greatest amoutn of swap
MightyByte: No, happs-tutorial isn't on there.
me: Hm, I bet we could get up to 20,000 users, 4 million jobs.
Still, one thing's clear.
You can't just assume your megapopular app will scale out of the box
with 16G of ram.
With happs.
MightyByte: Yeah
That's too bad.
me: I think the next thing to try is a better (or at least
different) data model.
I'll try having jobs be a separate field in AppState, rather than a
subfield of users.
MightyByte: I heard an interesting podcast the other day.
It was a guy from ebay talking about their architecture.
me: 2) See what lemmih says about sharding. I'm skeptical, but maybe
that's just because I don't understand it.
MightyByte: He was saying that you have to give up ACID in a large-
scale system.
http://www.infoq.com/news/2008/03/ebaybase
me: 3) Stop keeping everything in macid, and start storing stuff on
disk.
But try to keep the goodness of serialized happs data structure, macid
style.
MightyByte: Yeah. There's got to be a balance.
me: You mind if I post this chat to the google group?
MightyByte: No
me: Ok.
Well, looking at it optimistically, 10000 users is still a lot of
users.
MightyByte: Yeah
But that's not much in the way of extra data.
me: What do you mean?
MightyByte: Well, is that just the user data?
me: There's 200 jobs attached to each user.
so it's really 2 million rows of data.
You can see this if you click on http://mightybyte.net:8080/tutorial/jobs
You might also see your ram/cpu spike up as happs attempts to handle
this expensive query.
Not sure if that answers your question.
MightyByte: Ok
Sent at 6:56 PM on Wednesday
MightyByte: Ok, it doesn't take too long to build that page.
Sent at 6:57 PM on Wednesday
me: One final question: when happs isn't doing anything but just
waiting for queries, are you still seeing it's using 6G of ram?
Sent at 6:58 PM on Wednesday
me: By the way thanks for pointing me to the ebay article. It's
extremely relevant to happs, I think.,
In that success of HAppS as a project may depend on addressing some of
these issues.
<<>>
Sent at 7:02 PM on Wednesday
me: By the way, I just attempted to view a jobs pagination link
rather far down on the list, and it's taking a long time to respond.
And in fact the whole app seems to have ground to a half.
halt.
Ah! I did get a page back. Took about a minute though.
Sent at 7:05 PM on Wednesday
MightyByte: Hang on, let me look.
Now my swap is filled up.
It still uses the memory even when it's waiting for queries.
me: Did you see my group post on limiting your swap via lulimit -v ?
MightyByte: happs is now using 7.2 gigs
me: I think this is worth doing, for sanity.
MightyByte: Yeah
Yeah, good idea.
I'm going to kill the server
me: Better to have the app die than the console stop responding.
MightyByte: Now it's creating a checkpoint.
Sent at 7:08 PM on Wednesday
me: It would be good to know how long the checkpoint takes, and what
the size of file is.
MightyByte: Wow, this is hammering my box now.
me: ls -lth _local/happs-tutorial-state | head
MightyByte: htop is showing a lot of usage over all 8 cores
But it's weird because no process is showing up as using a lot of CPU
Sent at 7:11 PM on Wednesday
MightyByte: The checkpoint still isn't done.
me: Yeah, it might take a while, especially if you're using swap.
Might want to just kill -9 it, and try another time after you've used
ulimit to prevent going deeply into swap.
So, memory is HAppS achilles heel? PerHAppS?
MightyByte: Seems to be
I'm a little skeptical of the time it takes to create checkpoints.
Thomas: Why?
MightyByte: Well, that just doesn't seem like wise behavior for a
production system.
Thomas: Or, to be concrete, how big is the event file, how big the
checkpoint file, and how much time did it take?
Oh, skeptical pragmatically. I agree.
MightyByte: Well, the whole idea of these massive checkpoints seems a
little dubious to me.
Yeah
Thomas: You don't have to checkpoint.
But if you don't checkpoint, startup is very slow.
MightyByte: Oh, well that's good.
Ok, so you're screwed either way.
Thomas: Not necessarily, you just shift the pain into starting up,
yeah.
tphyahoo
very relevant to the problems I'm having.
MightyByte: I heard an interesting podcast the other day.
It was a guy from ebay talking about their architecture.
scale system.
http://www.infoq.com/news/2008/03/ebaybase
Lemmih
>
> Mightybyte was kind enough to offer me his 8GB server for stress
> testing happs-tutorial.
>
> The result was that we got up to about 20,000 users before using up
> ram. So, this reinforces my growing belief that to scale up a happs
> app to a large userbase, some sacrifices have to be made with regards
> to keeping all state in memory.
I'm not quite sure what you're trying to show. We are well aware that
the number of items you can keep in memory is limited by the available
memory. We are also well aware that gc costs increase as memory usage
goes up.
This was all known when HAppS development started. We realised that
limited memory capacity wasn't the only problem. You also have to deal
with reliability. Since few would consider running their production
system without backups or failovers, we decided to resolve all of
these issues in a unified manner. That is, by running your application
across multiple machines. This way you get no single point of failure
and all the memory you can afford.
As a side note, you're filling up the memory with String's. They
require at least 12 bytes per character. Using a different
representation would be prudent.
As an additional side note, you're free to store data on disk, S3 or
anywhere else. This does not interfere with the HAppS model.
--
Cheers,
Lemmih
tphyahoo
the "real world." The "simple stupid" approach with HAppS seems to be
to put all state in macid, so I tried that first. I think I've showed
that this works for prototyping, but if you want to scale you have to
think hard about what is really state versus what is just data.
> we decided to resolve all of
> these issues in a unified manner. That is, by running your application
> across multiple machines. This way you get no single point of failure
> and all the memory you can afford.
the naive approach of "everything in macid" that I used with the
tutorial job board won't scale... correct? My understanding is that
sharding will help you scale in terms of dealing with more user
queries to the application, but not actually keeping more of the data
in ram, where its easier to access than on a hard disk. Please correct
me if I'm wrong.
> As an additional side note, you're free to store data on disk, S3 or
> anywhere else. This does not interfere with the HAppS model.
liked the conceptual simplicity of using the macid machinery for
everything. But I've learned my lesson. Probably in the next two
releases I will be moving a lot of the application to the hard drive,
and only using state for what is REALLY state. In the case of the job
board, actually, I don't think there is any real transactional state.
Users actions can happen in pretty much any order they want.
My understanding is that with the current released version of HAppS,
there is no support for running state across across multiple machines.
I also understand there is some support for this in the head version
of HAppS, and if you'll point me towards documentation or a demo of
how this is used, I'll try and have a look. Any intuition on when
there will be a release to hackage?
> As a side note, you're filling up the memory with String's. They
> require at least 12 bytes per character. Using a different
> representation would be prudent.
you recommend? Bytestrings?
Thanks for your help.
Thomas
On Oct 15, 9:36 pm, Lemmih <lem...@gmail.com> wrote:
Lemmih
>
>> I'm not quite sure what you're trying to show.
>
> I guess what I'm mainly trying to show is a working demo of happs in
> the "real world." The "simple stupid" approach with HAppS seems to be
> to put all state in macid, so I tried that first. I think I've showed
> that this works for prototyping, but if you want to scale you have to
> think hard about what is really state versus what is just data.
As Justin said, don't worry about that until it becomes a problem.
Moving stuff from memory to disk/s3 is quite easy.
>> we decided to resolve all of
>> these issues in a unified manner. That is, by running your application
>> across multiple machines. This way you get no single point of failure
>> and all the memory you can afford.
>
> I'm not sure if I understand what you're saying. Even with sharding,
> the naive approach of "everything in macid" that I used with the
> tutorial job board won't scale... correct? My understanding is that
> sharding will help you scale in terms of dealing with more user
> queries to the application, but not actually keeping more of the data
> in ram, where its easier to access than on a hard disk. Please correct
> me if I'm wrong.
Partitioning will allow you to keep subset of the data in memory. This
is different from replication which only scales w.r.t. queries.
>> As an additional side note, you're free to store data on disk, S3 or
>> anywhere else. This does not interfere with the HAppS model.
>
> Agree. I don't know why, but I've resisted this idea. I think I just
> liked the conceptual simplicity of using the macid machinery for
> everything. But I've learned my lesson. Probably in the next two
> releases I will be moving a lot of the application to the hard drive,
> and only using state for what is REALLY state. In the case of the job
> board, actually, I don't think there is any real transactional state.
> Users actions can happen in pretty much any order they want.
>
> My understanding is that with the current released version of HAppS,
> there is no support for running state across across multiple machines.
> I also understand there is some support for this in the head version
> of HAppS, and if you'll point me towards documentation or a demo of
> how this is used, I'll try and have a look. Any intuition on when
> there will be a release to hackage?
The released version already supports replication, I believe.
Partitioning, on the other hand, is still being developed privately.
When it will be released depends on funding.
>> As a side note, you're filling up the memory with String's. They
>> require at least 12 bytes per character. Using a different
>> representation would be prudent.
>
> Wow, 12 bytes. I'll definitely switch to something else. What would
> you recommend? Bytestrings?
Yeah, they're not perfect but they'll do.
--
Cheers,
Lemmih
Justin T. Sampson
> Partitioning will allow you to keep subset of the data in memory. This
> is different from replication which only scales w.r.t. queries.
Yeah. I actually like to think of "replication" as five distinct techniques:
1. Journaling: Replicate transactions to persistent storage. Doesn't
allow you to handle any more data, and actually slows things down
relative to not doing it, but is the simplest way to provide disaster
recovery.
2. Remoting: Move the state to another server and access it remotely.
Gives some flexibility in CPU and RAM usage.
3. Mirroring: Replicate the entire state locally on a client machine,
but still send transactions to be executed on the server. Provides
improved query speed on the client without impacting the server.
Client can also continue to run read-only if connection is lost, and
server doesn't care if connection is lost.
4. Partitioning: Split your state into multiple pieces, each of which
is transactional but which don't coordinate with each other. Allows
spreading data across multiple machines, for greater data sizes. Also,
can improve snapshot and startup times even on a single machine due to
isolation of data.
5. Clustering: Replicate the same state on multiple machines, and
coordinate them transactionally (a.k.a. "multimaster"). Provides
improved query latency but worse transaction latency (though not
necessarily reduced throughput). Most importantly, provides redundancy
such that as long as a majority of the nodes are still connected, they
can continue servicing requests.
The cool thing is they can be mixed and matched as desired -- for
example, partition the data, then deploy each partition as a cluster
with journaling on each node, and have clients access some partitions
remotely while mirroring others. And partitioning and remoting don't
even really require support from the framework itself -- just start up
multiple instances of your app and provide your own web services or
whatever.
Journaling, mirroring, and clustering need to be built-in since they
affect how transactions are handled. I can't speak to their state in
HAppS, but Prevayler has of course always had journaling and has a
reasonably functional mirroring implementation. Several people have
experimented with clustering but we haven't included an implementation
in Prevayler itself yet.
Cheers,
Justin
tphyahoo
> require at least 12 bytes per character. Using a different
> representation would be prudent.
instead of String had no affect on performance whatsoever. I am
getting the exact same sized checkpoint file on quit, and the same ram
memory usage when running.
Perhaps ByteStrings are already used behind the scenes somehow as part
of the macid machinery, which would be good -- except it means no
performance gains are possible by switching the representation.
Thomas.
stepcut
uses Data.Binary to serialize the Strings.
Data.Binary serializes a string by writing out the length of the
String as a 64 bit integer, followed by each character of the string
as UTF-8. So, in the case where all the characters in the string are
ASCII, the number of bytes required to store a string will be 8 +
length of the String.
> encode ("haskell")
Chunk "\NUL\NUL\NUL\NUL\NUL\NUL\NUL\ahaskell" Empty
> Data.ByteString.Lazy.length (encode ("haskell"))
15
> length "haskell"
7
>
However, I would expect the in-memory representation to be (length of
string * 12 bytes)...
j.
tphyahoo
done is run htop while the old code and the new code was running, each
with 10000 some-odd inserts. I also noticed that both versions of the
code give me an out of memory error at the same level of stress
testing.
I was surprised too. Maybe I'm doing something wrong with the testing.
thomas.