Help me prove macid can scale! (Macid stress tests results for happs-tutorial toy job board disappointing so far.)

[tphyahoo]

HAppS is a new, relatively unproven technology.

So I am asking myself this question. Will HAppS allow me to scale the
toy job board I created for happs-tutorial into a high-volume, high
concurrency, large-user count type job board? You know, the kind that
might make money?

I have done some preliminary testing to answer this question, and so
far the results have been disappointing.

I am hoping that I am doing something wrong, and that there is a way
of using macid effectively for more than just toy applications. If
there are solution for the problems I'm experiencing I will definitely
be integrating this knowledge into the tutorial, so stay tuned.

I am seeking feedback from HAppS experts and educated users on the
following questions:

* Is building a heavy duty website like monster.com in HAppS a
realistic goal -- say, in the next twelve months?
* Are there certain types of web apps that are unlikely to work
well with the HAppS web architecture?
* Are there changes I can make to my toy app's architecture -- be
it data structures, buying new hardware, whatever -- that will enable
me to get good performance against the stress test described below and
in the demo?
* Are there other HAppS stress tests in the public domain, and
what are the results so far?

A complete description of the stress test I did (basically, insert 200
users with 200 jobs apiece) along with a runnable demo, is in the
tutorial preview at

http://happstutorial.com:5002/tutorial/macid-stress-test.

(The actual stress test is disabled in the online demo because I don't
want to zap my server.)

But the summary is, it's slow, slow to start, and creates a huge
events file -- 356M. For around 40000 records. That's like, 8
megabytes per job, which is a few lines of text. OK, happs isn't a
database -- but still.

This is all is checked into darcs and tagged:

darcs get --tag='stress test' http://code.haskell.org/happs-tutorial

Thanks in advance for helping make happs usable in the real world!

thomas.

[Lemmih]

'AppStateSetBased.updateUser' is extremely expensive. Running it will
serialize both the old user and the new user (including all jobs).
This means that you're serializing almost 200 times too much
information. Changing this to something more sensible reduces the
runtime from 8m to 20s on my box.
The event file is quite a lot smaller at 8.9MB. This comes down to
230bytes per database insert (ControllerStressTest.insertu makes 202
inserts).

--
Cheers,
Lemmih

[Lemmih]

On Sun, Oct 5, 2008 at 9:22 PM, tphyahoo <thomash...@googlemail.com> wrote:
>

> HAppS is a new, relatively unproven technology.
>
> So I am asking myself this question. Will HAppS allow me to scale the
> toy job board I created for happs-tutorial into a high-volume, high
> concurrency, large-user count type job board? You know, the kind that
> might make money?
>
> I have done some preliminary testing to answer this question, and so
> far the results have been disappointing.
>
> I am hoping that I am doing something wrong, and that there is a way
> of using macid effectively for more than just toy applications. If
> there are solution for the problems I'm experiencing I will definitely
> be integrating this knowledge into the tutorial, so stay tuned.
>
> I am seeking feedback from HAppS experts and educated users on the
> following questions:
>
> * Is building a heavy duty website like monster.com in HAppS a
> realistic goal -- say, in the next twelve months?

HAppS is still quite experimental so I wouldn't put money on it unless
you're willing to fix bugs when needed.

> * Are there certain types of web apps that are unlikely to work
> well with the HAppS web architecture?

No released version of HAppS supports sharding as of this time. This
limits the amount of data you can manipulate.

> * Are there changes I can make to my toy app's architecture -- be
> it data structures, buying new hardware, whatever -- that will enable
> me to get good performance against the stress test described below and
> in the demo?

The changes mentioned in my previous mail results in throughput of
~2000 transactions per second (using run-of-the-mill desktop
hardware). More optimization and better hardware could probably
improve that number a bit.

> * Are there other HAppS stress tests in the public domain, and
> what are the results so far?

I'm not aware of any.

--
Cheers,
Lemmih

[stepcut]

Hello,

I modified your main function to do a checkpoint on startup:

runserver p = withProgName "happs-tutorial" $ do
control <- startSystemState (Proxy :: Proxy AppState) -- start the
HAppS state system
createCheckpoint control
putStrLn $ "happs tutorial starting on port " ++ (show p) ++ "\n" ++
"shut down with ctrl-c"

simpleHTTP (Conf {port=p}) controller -- start serving web pages

I then did a 20 user stress test. It created a 54MB event file. I
stopped the server and restarted. It took a while to restart, but when
it was done it generated a 264KB checkpoint file. After that,
restarting the server was almost instantaneous.

Looking at the contents of the event file, I see the tuple (3,32)
appear many times. So, it seems that you have some sort of space and
time leak going on, where every time you update the record you
serialize everything that has already been serialized?

I think that it is definitely the case that some best practices
guidelines need to be figured out and documented. I've documented some
of the stuff I know on my blog (http://nhlab.blogspot.com/), but
obviously there is still a lot of information to be uncovered.

j.
ps. putting the checkpoint at startup is not where I would put it in
practice -- but it was useful for testing purposes. Here is what my
main functions tend to look like:

main :: IO ()
main =
control <- startSystemState entryPoint
eConf <- liftM parseConfig getArgs
let conf = case eConf of
Left e -> error (unlines e)
Right c -> c { validator = Just wdgHTMLValidator }
tid <- forkIO $ simpleHTTP conf (impl store)
putStrLn "running..."
waitForTermination
killThread tid
createCheckpoint control
shutdownSystem control

This checkpoints the system on every shutdown, which means faster
startup time, and it is possibly safer if the next restart is a newer
version of the server which migrates data.

Additionally, by forking off simpleHTTP, you have the option of
running simpleHTTP on more than one port -- which would have been one
way to redirect from port 5001 to port 80 while only running one
process.

[tphyahoo]

>The changes mentioned in my previous mail

Care to share them?

Also, did you use any HAppS machinery -- (eg IxSet, which I just
discovered) -- or just continue on using vanilla Map and Set from the
standard libs?

I also just noticed, with regards to ixset, that the haddock...

http://hackage.haskell.org/packages/archive/HAppS-IxSet/0.9.2.1/doc/html/HAppS-Data-IxSet.html

is a lot more userful if you read the source code as well, a very long
documentary comment at the beginning doesn't show in the haddock.

I've decided that for an interim goal, I'd like to show my toy app
with 100,000 users created, and maybe 2000 transactions per second as
you had. Better would be a million users but I'll worry about that
after I am satisfied that 10^5 is doable.

Thomas.

On Oct 6, 1:20 am, Lemmih <lem...@gmail.com> wrote:

[tphyahoo]

I'm reposting an archived chat from #happs as I think it answers, or
at least outlines the problem space, of many questions I had about the
happs approach to doing "big" apps. In this case, there's a discussion
of what would be involved in doing something like google reader with
happs, and also a banking app, where acid is more important. An
important insight is that HAppS acid guarantees are limited by RAM,
however, if your app needs a bigger datastore than will fit in ram you
can still do this by storing BLOBs on disk (or even using a mysql
database or whatever as backend storage, I guess). You just need to
bear in mind that whatever is stored on disk isn't acid, ie no
transactional guarantees.

Thomas.


*******************

http://tuukka.iki.fi/tmp/happs-2008-01-16.html

15:39:17 <mightybyte> Anyone awake here?
15:40:52 <pulczynsk1> yup
15:40:53 <pulczynsk1> :)
15:42:33 <mightybyte> Aha
15:42:52 <mightybyte> I've been looking into happs for a web app.
15:43:12 <mightybyte> Is it useable for an app that needs more state
than can be stored in memory?
15:57:03 <mightybyte> The reason I ask is because I've seen the in-
memory state aspect emphasized.
16:07:46 <pulczynsk1> mightybyte: whole point is abount keepeing data
in memory (with acid guarantees), so short answer is : no. :)
16:09:38 <mightybyte> Hmmm, so it seems that happs wouldn't be an
appropriate choice for large-scale apps.
16:11:03 <mightybyte> Since it's advertised as a having the ability
scale, this is a little disappointing.
16:11:57 <pulczynsk1> mightybyte: What do you call "large-scale"
apps ?
16:12:27 <mightybyte> I'm mainly referring to apps that have large
storage requirements (and thus can't reside entirely in memory).
16:13:45 <mightybyte> i.e. it sounds like Google couldn't use happs
for web apps of their scale and storage requirements
16:14:04 <pulczynsk1> mightybyte: you should check this thread
http://groups.google.com/group/HAppS/browse_thread/thread/d2eea2f5170bbb0c
(about data sharding etc)
16:14:18 <mightybyte> Ok
16:14:22 <pulczynsk1> mightybyte: And that's true, google couldnt use
happs as its only persistence layer
16:14:48 <pulczynsk1> on previous version of mainpage there were some
numbers which showed that you could use happs as transaction-center of
ebay
16:15:11 <mightybyte> Hmmm, how would that be accomplished?
16:15:23 <mightybyte> ...the persistence aspect that is.
16:16:23 <alexj> you store blobs on disk.
16:16:28 <pulczynsk1> mightybyte: you dont need to have all data in
memory, just those which are needed for this kind of transaction. If
you keep them in memory you have non blocking acces to them , so you
are limited only by cpus (and main bus)
16:16:36 <alexj> the question is how much transactional state you
have.
16:17:04 <mightybyte> Ok, well let's choose a concrete example to talk
about. Take something like Google reader.
16:17:27 <mightybyte> I've heard that the size of their database of
links is huge.
16:18:23 <mightybyte> Could it be run with happs accessing a database
backend when necessary?
16:18:31 <alexj> mightybyte: yes.
16:18:39 <alexj> mightbyte: you can do IO to whatever you want.
16:18:39 <pulczynsk1> mightybyte: if I understand it properly you can
just shred users over happs instance
16:18:55 <pulczynsk1> s/instance/instances/
16:18:57 <alexj> right now sharding would be a manual process.
16:19:03 <alexj> we hope to automate it in the futur.
16:19:11 <mightybyte> sharding what?
16:19:23 <pulczynsk1> alexj: i have heard of plan of using ec2 btw, is
that true?
16:20:01 <alexj> yes.
16:20:19 <mightybyte> Does sharding refer to splitting users over
happs instances?
16:20:21 <pulczynsk1> mightybyte: users does not have common data in
google reader, so you can just put limited numer of users on one
server. to communicate with other users (eg reccomend them a link) you
can just use api of happs instance
16:20:29 <pulczynsk1> mightybyte: yes.
16:21:39 <mightybyte> So would a user be permanently restricted to a
specific physical server, or would it be load balanced appropriately
when the user logs in?
16:21:58 <pulczynsk1> mightybyte: it would be restricted to one
server, true
16:22:08 <mightybyte> That seems like it might not scale
16:22:23 <pulczynsk1> mightybyte: scale-up is something different than
scaling-out
16:22:32 <mightybyte> It's certainly not a stretch to suppose that
groups of users would not be evenly distributed.
16:23:08 <pulczynsk1> mightybyte: you are comparing it do common
database solution, which also you have to scale at some point, and the
key to scalabiltiy is keeping data in ram (eg mysql cluster)
16:23:59 <mightybyte> Yeah, I guess my picture of how to properly use
state and backend database storage is not quite there yet.
16:24:05 <pulczynsk1> mightybyte: so using happs you just consloidate
web-fronted with database ,(gain: dont have to use external caches)
16:24:26 <mightybyte> Yeah
16:24:44 <pulczynsk1> mightybyte: im not really an expernt in
databases, but at some point you have to split/shred you data over
multiple instances
16:25:02 <pulczynsk1> if you want to do this using multi-master
replication you are limited by INSERT or UPDATE statements (when using
SQL)
16:25:14 <pulczynsk1> because every INSERT must be propageted across
nodes
16:25:19 <mightybyte> So if users are split among servers, when a user
logs out and then back in, does he get the same server as the first
login?
16:25:33 <pulczynsk1> that would be one of solutions
16:25:36 <alexj> mightybyte: you would typically multimaster each
shard.
16:25:48 <mightybyte> What does that mean?
16:26:32 <alexj> there is multimaster code in the repos but we have
not made it production yet, but the concept is that each shard would
be fully replicated so a user could hit any server that holds his/her
data.
16:26:43 <alexj> and any update to any of those servers would get
propagated to the others.
16:26:51 <mightybyte> Ok
16:27:06 <alexj> so you load balance accross multimastered instances.
16:27:30 <pulczynsk1> alexj: what about acid guarantess in this case?
16:27:38 <alexj> you get acid per shard.
16:27:49 <pulczynsk1> alexj: so what about conflicts?
16:27:53 <alexj> you don't get acid accross shards.
16:27:58 <alexj> conflicts?
16:28:03 <mightybyte> I guess I need to figure out in my mind what
data goes in happs state and what would go in the database
16:28:56 <pulczynsk1> user hits server A and says DELETE X (and A
deletes X) and then hits B which has X in its state because state-
change hassn't been propageted to it yet
16:28:59 <alexj> mightybyte; think of it as hierarchical storage
management.
16:29:27 <mightybyte> I'm thinking that in an application like Google
Reader, there would be little to no user state.
16:29:28 <pulczynsk1> alexj: i am talking about some kind of
distributed lock (or smth)
16:29:45 <alexj> pulczynsk1: you get ACID per exposed state method.
16:30:08 <mightybyte> If a user deleted a link from his list, then
that would be reflected in the database, not the happs state.
16:30:40 <alexj> pulczynsk1: yes you could in principle see a
propagation delay. that is true of any form of replication. it is just
unlikely that the user would hit the other server before replication
happened.
16:30:41 <pulczynsk1> mightybyte: that would mean you are using happs
just as a cache layer to db?
16:30:54 <mightybyte> I guess so.
16:31:29 <pulczynsk1> alexj: yeah, but in this scenario you dont gain
so-called "scaling-out" (transparently) (eg i caonnt use it in banking
application)
16:31:44 <pulczynsk1> alexj: on the other hand its enough for most
web2.0
16:31:45 <alexj> in any case, you couldn't get a conflict because the
operations have a total order so serverB may still have X but it you
can't do anything with it.
16:31:46 <mightybyte> If the happs state only contains data specific
to a given session instance, then users would be able to get different
happs servers and it would be possible to load balance properly.
16:32:27 <alexj> pulcynzsk1: you can'
16:32:32 <pulczynsk1> alexj: right. i didnt get it at first.
16:32:34 <alexj> you can't get ACID accross shards.
16:32:50 <alexj> but you can get ACID within shards because within a
shard all operations have a total order.
16:33:02 <mightybyte> It seems like this approach would be a good way
to handle large amounts of data persistence because then you can let
the database guys take care of their own scaling issues.
16:33:53 <mightybyte> alexj: Yeah, and in the approach I've just
described, I don't think you'd need ACID across shards.
16:33:59 <alexj> pulczynsk1: you probably don't want to distribute the
bank accounts accross multiple shards unless you do something clever
algorithmicly like locking some of the balance until the transaction
is confirmed at both ends (which is how the credit card systen works)
16:34:21 <alexj> mightybyte: I really don't want to use an external
DBMS ever again.
16:35:04 <alexj> if you did a banking app you just need to think about
whether you really need to shard the accounts table.
16:35:05 <mightybyte> alexj, I have the same feelings, but they're
pretty well-understood, and can be outsourced to people who don't
mind.
16:35:31 <alexj> mightbyte: my hope for happs is that few apps are
sufficiently large that you need to outsource.
16:35:44 <mightybyte> Hmmm
16:35:45 <alexj> haskell coding efficiency is just soo much higher
than for other languages.
16:35:54 <mightybyte> Oh, I agree
16:36:00 <mightybyte> That's why I'm here discussing this right now.
16:36:07 <pulczynsk1> :)
16:36:14 <mightybyte> ...otherwise I'd be off coding in Python on Java
16:36:25 <alexj> the coordination costs of adding programmers to the
team dwarves the management savings of outsourcing.
16:36:47 <alexj> you can't produce a baby in one month by impregnating
nine women. etc.
16:36:54 <mightybyte> But I still can't help thinking about situations
like Google's.
16:36:54 <pulczynsk1> lol
16:38:42 <mightybyte> How would you do something like reader without
outsourcing the persistence to a database?
16:38:51 <mightybyte> ...solve the scaling issues that is.
16:38:58 <alexj> so lets spec out what reader is?
16:39:12 <mightybyte> Ok, you have a huge "database" of links
16:39:22 <mightybyte> ...and feeds
16:40:12 <mightybyte> Links are associated with feeds, and multiple
users can be subscribed to a given feed.
16:40:52 <alexj> ok so why not shard the link database?
16:40:57 <mightybyte> I think you can ignore other details like tags
for feeds and/or links.
16:41:23 <mightybyte> Ok, I don't understand all the ramifications of
that.
16:41:59 <alexj> mightybyte: you need to read up on sharding issues in
general. you have to do it for mysql too.
16:42:26 <mightybyte> Yes, I don't claim to be an expert, but I'm
trying to understand it in the context of happs.
16:42:42 <alexj> the gist is that you need to divide the link database
accross multiple servers. when you need to find something, you query
all of them and collect results toegether.
16:42:56 <mightybyte> Yeah, I understand that.
16:43:27 <mightybyte> So that would imply that the state sharding is
somewhat separate from the web server interface.
16:43:54 <mightybyte> If that's the case, I hadn't realized it until
now.
16:44:04 <pulczynsk1> i would recoomend reading http://www.highscalability.com/
and real world examples : http://highscalability.com/links/weblink/24
. at some point all of them has to shard datas of some kinds. you have
to do exactly this in happs, but are freed from using externals tools
and thinks like locking, IO-bounding etc (you dont wait for resources)
16:44:17 <mightybyte> Or rather, the implications weren't clear until
now.
16:44:37 <pulczynsk1> so, happs by itself isnt cure for scaling out
(just for scaling up, which is sufficient in most cases)
16:45:33 <mightybyte> So is the state management and sharding
independant from the web server parts of happs?
16:45:50 <alexj> mightybyte: happs is divided into different repos for
a reason. happs-http is your web server layer. happs-state is another
layer. the nice thing is that you get a single executable to handle it
all.
16:45:56 <alexj> yes
16:46:10 <mightybyte> Ahhh, the light is coming on.
16:47:28 <pulczynsk1> bah, i must leave, but for sure i will check
logs tommorow (as i am evangelist of happs in my company ;)
16:48:19 <mightybyte> I had assumed that if I used the happs state, I
would be implicitly splitting data into groups of users. But it sounds
like that's wrong.
16:49:28 <alexj> mightybyte: you might also shard your users
16:49:36 <mightybyte> Certainly
16:49:51 <mightybyte> But I didn't realize that happs would allow both
dimensions of sharding.
16:49:59 <alexj> though you may find that you need to have a lot of
users to justify bothering.
16:50:26 <alexj> mightybyte: just to be clear happs is not production
quality for this sort of thing yet so you are playing with fire if you
are planning to scale like this today.
16:50:40 <mightybyte> Right, I understand.
16:50:51 <mightybyte> But that's the goal
16:51:07 <mightybyte> right?
16:51:17 <alexj> yes.
16:51:27 <alexj> the next step is to get multimaster nice.
16:51:29 <alexj> then sharding.
16:51:56 <mightybyte> Well this is really encouraging.
16:52:28 <mightybyte> It sounds like I can forget worrying about a
database schema and just design my data structures in haskell.
16:53:02 <alexj> thats the idea.
16:53:29 <mightybyte> Wow, we need to improve the documentation so
that this is more accessible.
16:53:43 <alexj> very much. could definitely use help.
16:54:21 <mightybyte> So what does happs have today?
16:54:54 <alexj> working code for a single non-sharded server. we are
stabilizing apis now.
16:55:06 <mightybyte> apis?
16:55:44 <alexj> how you declare your state so that you get ACID query/
update operations on it.
16:56:06 <mightybyte> Ok
16:56:12 <alexj> check out the AllIn example on the homepage.
16:56:25 <alexj> and you'll get a flavor for what is happening.
16:56:58 <mightybyte> Hmm, I'm not seeing that. Link?
17:00:13 <mightybyte> Oh, is it in the repository in HAppS-HTTP/
Examples/AllIn.hs?
17:00:32 <alexj> yes
17:00:47 <mightybyte> I haven't found a way to access that from the
website.
17:00:49 <alexj> its labeled as State+HTTP together on the homepage.
17:01:02 <alexj> yeah, I need to fix repo directory indexing.
17:01:02 <mightybyte> Oh, ok
17:01:38 <mightybyte> What is the "$(deriveNewData..." bit?
17:03:46 <alexj> until the net version of haskell, we need that
because happs uses generic programming interfaces to reduce the amount
of boilerplate you need to write.
17:04:12 <alexj> "deriving Data" does not yet provide enought stuff so
we need templatehaskell to provide the rest.
17:04:32 <mightybyte> What's the meaning of the $()?
17:05:08 <alexj> it is templatehaskell
17:05:11 <mightybyte> Is that from templatehaskell? I'm not familiar
with it.
17:05:13 <mightybyte> Ok
17:05:16 <mightybyte> I'll look that up.
17:05:29 <mightybyte> (still learning haskell too)
17:08:05 <mightybyte> Is templatehaskell pretty standard?
17:10:12 <mightybyte> alexj: I have to go. Thanks for all the help.

On Oct 6, 1:20 am, Lemmih <lem...@gmail.com> wrote:

[Lemmih]

On Thu, Oct 9, 2008 at 8:50 AM, tphyahoo <thomash...@googlemail.com> wrote:
>
>>The changes mentioned in my previous mail
>
> Care to share them?
>
> Also, did you use any HAppS machinery -- (eg IxSet, which I just
> discovered) -- or just continue on using vanilla Map and Set from the
> standard libs?

Using a Map is half the solution, I'd think. It makes you realise that
only a key is needed to identify a user (as opposed to the entire user
info).
The methods that have User's in their arguments is what makes the
program slow. We can get around this by adding specialised functions
like: 'addUserJob :: UserName -> Job -> Update YourState ()'.

The 'addUserJob' is all you need to make the stresstest run fast.
However, it would be a good idea to get rid of 'updateUser'
altogether.

> I've decided that for an interim goal, I'd like to show my toy app
> with 100,000 users created, and maybe 2000 transactions per second as
> you had. Better would be a million users but I'll worry about that
> after I am satisfied that 10^5 is doable.

Inserting 100,000 users will be slow in any database if you use 200
transactions per user. Try reducing the number of transactions needed.

--
Cheers,
Lemmih

[Justin T. Sampson]

Howdy,

Storing blobs on disk actually can be transactional. Just treat them as read-only, and make sure you write the file to disk before you reference it elsewhere in your state. I've done this a couple times with Prevayler.

By the way, I also gave some thought to ACID properties across multiple shards (which I was calling "partitions") a few years ago:

http://www.krasama.com/prevayler-acid.html

Cheers,
Justin

[tphyahoo]

I pretty much scrapped the original schema I had and used maps for
everything, and performance is certainly a lot better.

I'm seeing 3.8M for 200 user / 40000 job stress test, and startup time
is also reasonable after I integrated stepcut's suggestions for
checkpointing on shutdown.

All that's in darcs.

I'm now going to see how things are if I apply more stress... eg
100000 users.

Thomas.

On Oct 9, 2:08 pm, Lemmih <lem...@gmail.com> wrote:

[tphyahoo]

The slow bit is now when I do a shutdown, which now involves a
checkpoint.

For ~40000 macid transactions, I'm seeing 5 minute+ shutdown times.

This might be good enough for the real world, but I'm not sure.

I'm wondering if there's a way to get better shutdown times if you
checkpoint more often and not just at shutdown.

Can checkpointing be made into a cron-like thing where it's just done
every 5 minutes say?

thomas.

[Lemmih]

On Mon, Oct 13, 2008 at 1:22 PM, tphyahoo <thomash...@googlemail.com> wrote:
>
> The slow bit is now when I do a shutdown, which now involves a
> checkpoint.
>
> For ~40000 macid transactions, I'm seeing 5 minute+ shutdown times.

That's only ~130 transactions per second. Are you maxing out your CPU?

Btw, I tried building the new patches but Safe.hs is missing.

> This might be good enough for the real world, but I'm not sure.
>
> I'm wondering if there's a way to get better shutdown times if you
> checkpoint more often and not just at shutdown.
>
> Can checkpointing be made into a cron-like thing where it's just done
> every 5 minutes say?

Yes, but I wouldn't recommend it. Making a checkpoint writes the
entire state to disk.

--
Cheers,
Lemmih

[Aycan iRiCAN]

Lemmih wrote:
>> This might be good enough for the real world, but I'm not sure.
>>
>> I'm wondering if there's a way to get better shutdown times if you
>> checkpoint more often and not just at shutdown.
>>
>> Can checkpointing be made into a cron-like thing where it's just done
>> every 5 minutes say?
>>
>
> Yes, but I wouldn't recommend it. Making a checkpoint writes the
> entire state to disk.
>
>

Also what about event execution when a checkpoint is being executed? I
think nobody wants to block entire state.

--
aycan

[tphyahoo]

> Btw, I tried building the new patches but Safe.hs is missing.

sorry, safe is just

cabal install safe

from hackage. Neil Mitchell's safe versions of read, head, tail, etc.

I'm thinking that some of my annoyance might be that I was running
this from inside ghci, and ghci doesn't seem to do a complete shutdown
on ctrl-c.

So I've started running from the command prompt, and also added some
timing information to runserver.

(all darcs pushed)


On Oct 13, 4:01 pm, Lemmih <lem...@gmail.com> wrote:

[tphyahoo]

>> That's only ~130 transactions per second. Are you maxing out your CPU?

Okay, I made things a bit more precise. The web frontend to the
stresstest now gives me timing feedback:

"Stress Test Completed
Users 642 to 841 have been inserted.
Stress test time: 202 secs "

And the checkpoint + shutdown gives me timing feedback

".....
added useruser840
added useruser841
C-c C-ccreating checkpoint
time: Mon Oct 13 17:07:50 CEST 2008
shutting down system
time: Mon Oct 13 17:10:41 CEST 2008
shutting down system, time:
time: Mon Oct 13 17:10:41 CEST 2008
"

So both the webform stresstest and the shutdown take about 3 minutes.
40000 transactions / 180 seconds = 222 transactions per seconds, which
is in the same ballpark as you say.

I'm not sure if I'm maxing out my cpu, what can I do to tell?

Are you seeing dramatically faster times?

It is kind of a junky old laptop I'm working on.

thartman@thartman-laptop:~/happs-tutorial>uname -a
Linux thartman-laptop 2.6.24-19-generic #1 SMP Wed Aug 20 22:56:21 UTC
2008 i686 GNU/Linux
thartman@thartman-laptop:~/happs-tutorial>cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Mobile Intel(R) Celeron(R) CPU 2.00GHz
stepping : 9
cpu MHz : 1995.006
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe up pebs
bts sync_rdtsc cid xtpr
bogomips : 3994.41
clflush size : 64

On Oct 13, 4:01 pm, Lemmih <lem...@gmail.com> wrote:

[Lemmih]

Event execution continues while the checkpoint is being written to
disk. It's just a waste of resources to serialize the state every 5
minutes.

--
Cheers,
Lemmih

[Flavio Botelho]

Just run "top" in a terminal while running the test.
If it stays at 99% most of the time, then the CPU is the bottleneck.

[tphyahoo]

When I do that cpu isn't at 99%, but certainly using a lot of ram.

This is quite an annoyance, essentially my laptop slowed down and
became unuseable after I inserted a bit over1000 users. (200000
records.)

I would really like to be convinced that 100000 users with a happs app
is doable, because to me this is the baseline of an economically
viable web 2.0 type app. I estimate if you're doing an ad driven site
each user is worth maybe .50 cents a year, so with 100000 users you're
likely profitable although probably not wildly so. But much less than
that and you're really not making money. Those are the numbers I'm
using anyway.

What I finally did, after some trial and error, is put

ulimit -v 262144 #limit virtual memory to 256M

into my .bashrc. At least now my happs app will just quit with an out
of memory error rather than slowly grinding to a halt that can only be
stopped with kill -9 if I notice what's happening before the keyboard
stops responding. Typically what happens now is my app quits when I
try to view any of the data driven pages (consultants, consultants
wanted, or jobs) whenever I add over 1000 users.

I'm growing more skeptical of the "put everything into memory" by
default strategy. However, I still like the macid style of tying your
data model directly with haskell records. Maybe there's a middle way.

Plans:

1) Try the toy job board on a box with more ram. I'll try to get one
of the 15GB amazon ec boxes going -- trying amazon ec is something
I've been meaning to do anyway. And see how much more performance I
get if I radically improve my hardware.

2) maybe improve the data structures again. perhaps jobs should be its
own data record in macid state, with an fk like relationship to users,
rather like how this would most likely be implemented if we were using
a database

3) Perhaps some things shouldn't be in RAM at all but rather
serialized to disk as Justin Samson suggested earlier. The only thing
that absolutely needs to be in ram is the transactional state, which
in a job board there basically isn't any. Well, perhaps sessions? Not
sure, need to think about this.

All those disappointing words having been said, my conclusion is that
macid is more than ready for one-off type websites with an expected
user count of a few hundred, or probably even a few thousand with
better hardware. I'd have no hesitation whatsoever trying for that.

And I still hope to convince myself that scaling up to 100000 users is
doable.

Thomas.

On Oct 13, 7:47 pm, "Flavio Botelho" <fezsent...@gmail.com> wrote:
> Just run "top" in a terminal while running the test.
> If it stays at 99% most of the time, then the CPU is the bottleneck.
>

[tphyahoo]

With regards to future plans and better hardware: does anybody in
happs land have access to a server with a lot of ram -- say 8G that I
could experiment with for a few hours? Maybe that's a ridiculous
question and all the 8G servers in the world are already being put to
use for more important things -- but I figure there's no harm in
asking.

Thomas.

[Lemmih]

On Mon, Oct 13, 2008 at 5:18 PM, tphyahoo <thomash...@googlemail.com> wrote:
>
>>> That's only ~130 transactions per second. Are you maxing out your CPU?
>
> Okay, I made things a bit more precise. The web frontend to the
> stresstest now gives me timing feedback:
>
> "Stress Test Completed
> Users 642 to 841 have been inserted.
> Stress test time: 202 secs "
>
> And the checkpoint + shutdown gives me timing feedback
>
> ".....
> added useruser840
> added useruser841
> C-c C-ccreating checkpoint
> time: Mon Oct 13 17:07:50 CEST 2008
> shutting down system
> time: Mon Oct 13 17:10:41 CEST 2008
> shutting down system, time:
> time: Mon Oct 13 17:10:41 CEST 2008
> "
>
> So both the webform stresstest and the shutdown take about 3 minutes.
> 40000 transactions / 180 seconds = 222 transactions per seconds, which
> is in the same ballpark as you say.
>
> I'm not sure if I'm maxing out my cpu, what can I do to tell?
>
> Are you seeing dramatically faster times?

I am. By fixing a space leak and tuning the GC options, these are the
times I'm getting:

"Users 801 to 1000 have been inserted.
Stress test time: 8 secs"

"added user: user998
added user: user999
added user: user1000
creating checkpoint
time: Tue Oct 14 21:23:13 CEST 2008
shutting down system
time: Tue Oct 14 21:23:16 CEST 2008
shutting down system, time:
time: Tue Oct 14 21:23:16 CEST 2008"

My hardware is slightly newer:

david@desktop:~$ uname -a
Linux lemmih-desktop 2.6.24-16-generic #1 SMP Thu Apr 10 12:47:45 UTC
2008 x86_64 GNU/Linux

david@desktop:~$ cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 75
model name : AMD Athlon(tm) 64 X2 Dual Core Processor 4600+
stepping : 2
cpu MHz : 1000.000
cache size : 512 KB
[snip]

Make sure you don't get bitten by laziness and try +RTS -c -A5m -RTS.

--
Cheers,
Lemmih

[Lemmih]

On Tue, Oct 14, 2008 at 8:55 PM, tphyahoo <thomash...@googlemail.com> wrote:
>
> When I do that cpu isn't at 99%, but certainly using a lot of ram.
>
> This is quite an annoyance, essentially my laptop slowed down and
> became unuseable after I inserted a bit over1000 users. (200000
> records.)
>
> I would really like to be convinced that 100000 users with a happs app
> is doable, because to me this is the baseline of an economically
> viable web 2.0 type app. I estimate if you're doing an ad driven site
> each user is worth maybe .50 cents a year, so with 100000 users you're
> likely profitable although probably not wildly so. But much less than
> that and you're really not making money. Those are the numbers I'm
> using anyway.

How much memory usage are using seeing? After inserting 200k jobs,
HAppS on my box is using 245megs. We could probably get that number
even further down by using a more compact representation of strings.

We, at HAppS, are going for option 4: Throwing more machines at the
problem. Limited memory isn't the only problem of individual machines.
They are also limited in CPU capabilities and reliability. We're
trying to solve all of these problems by making HAppS a distributed
application.

--
Cheers,
Lemmih

[parnell]

Could you post what you did to fix the space leak?

Cheers,

Parnell

On Oct 14, 2:53 pm, Lemmih <lem...@gmail.com> wrote:

[Justin T. Sampson]

Hi Thomas,

I posted a story a while ago describing a development process for performance...

http://groups.google.com/group/HAppS/msg/c6aa9912ec38fb64

In short, one thing to try is taking your performance targets in steps of 10x improvements: Start out with a performance target that you're already happy with -- maybe that's only 100 users. Then bump up your stress test to 1,000 users and profile carefully to identify what exactly is taking the most memory at that point, and address it creatively until you're happy with the performance again. Then bump up to 10,000 users and profile again to find the new culprit. And so on... You're likely to find one big memory hog at each step, which will be easier to identify and fix than if you go straight to 100,000 and already have poor performance.

Cheers,

Justin

[Lemmih]

On Tue, Oct 14, 2008 at 11:34 PM, parnell
<parnel...@ronin-capital.com> wrote:
>
> Could you post what you did to fix the space leak?

It seems to me that the 'jobs' entry in 'UserInfos' is never forced in
the stresstest. This probably isn't a big issue, though.

--
Cheers,
Lemmih

[tphyahoo]

I did pretty much what Lemmih suggested, and it's in darcs.

Switched from set to map, and put more action specific :: Update
actions and... well, it was a lot of little changes, but that's the
gist of it.

[tphyahoo]

Sorry, I answered parnell's question thinking it was addressed to me,
but I think he was asking lemmih.

Parnell, why do you think there is a space link in the new version of
job board state? My thinking was that 200,000 records in ram is simply
all my hardware can take.

Is this naive?

[parnell]

I do not think there is a space leak. I have the updated version and
I am modifying it for a little experiment of my own. I was responding
to Lemmih's remark:

"I am. By fixing a space leak and tuning the GC options, these are the
times I'm getting: "

I just wanted to know what the space leak was since I have not seen it
but I also have not played with the code as much.

I hope to be able to post some results of my own soon. I will be
pumping the system with millions of records.

Cheers,

Parnell

[tphyahoo]

> How much memory usage are using seeing? After inserting 200k jobs,
> HAppS on my box is using 245megs. We could probably get that number
> even further down by using a more compact representation of strings.

I've done a bit more work (up on darcs).

Interestingly I'm seeing much better performance on my linode than on
my development laptop, though both machines have similar specs. On my
linode, I've managed to insert 25,000 users (*200 jobs/user), which is
at least the same order of magnitude as 100,000, my goal. I find I can
insert 1000 users at a time, and thusly work my way up to higher user
counts. However if I try to insert more than 1000 at a time the uml
hypervisor kills my happs process. (linode server has 360M of ram and
256M of swap.) Maybe it's some quirk of virtualized os? Or maybe just
because it's saving ram not on not running xwindows and so on?

Anyway, there are now 3 stress tests:
-- 1 inserts all users, all jobs with one transaction
-- 2 one transaction per user, all jobs get inserted at once
-- 3 one transaction per job

One transaction per job is too slow for inserting a lot of dummy data,
though it looks like lemmih got better results in his own testing.
Inserting all users is of course the fastest, however I couldn't do
this for more than 200 users. The best option in practice was option
2, which allowed me to insert 1000 users at a time, eg by

time wget http://www.happstutorial.com:5002/tutorial/stresstest/atomicinsertsalljobs/1000

What I'm seeing with 25000 users is, I can only view the consultants
page. If I try to look at consultants wanted (which checks if the jobs
list is null) or show all jobs (which does a map fold over users to
get all the jobs) the hypervisor kills happs for using too much
memory.

> We, at HAppS, are going for option 4: Throwing more machines at the
> problem. Limited memory isn't the only problem of individual machines.
> They are also limited in CPU capabilities and reliability. We're
> trying to solve all of these problems by making HAppS a distributed
> application.

How would this work in the job board case? Could I accomplish 100,000
with the same data model I have now, but using four machines? What
would happen when I attempt to view all jobs? That's 20 million --
whoa -- but I did make a paginator that allows you to scroll around in
that many records keeping the actual browser page display down to a
reasonable size. To display the data you don't need all the jobs, just
the first 200, and pagination info for the rest. So really it comes
down to doing a count of jobs records.

Is there an example somewhere in the head repos for sharding, on a
large data set like what I have for the stress test?

Thomas.

On Oct 14, 10:34 pm, Lemmih <lem...@gmail.com> wrote:

[Lemmih]

On Wed, Oct 15, 2008 at 1:52 PM, tphyahoo <thomash...@googlemail.com> wrote:
>
>> How much memory usage are using seeing? After inserting 200k jobs,
>> HAppS on my box is using 245megs. We could probably get that number
>> even further down by using a more compact representation of strings.
>
> I've done a bit more work (up on darcs).
>
> Interestingly I'm seeing much better performance on my linode than on
> my development laptop, though both machines have similar specs. On my
> linode, I've managed to insert 25,000 users (*200 jobs/user), which is
> at least the same order of magnitude as 100,000, my goal. I find I can
> insert 1000 users at a time, and thusly work my way up to higher user
> counts. However if I try to insert more than 1000 at a time the uml
> hypervisor kills my happs process. (linode server has 360M of ram and
> 256M of swap.) Maybe it's some quirk of virtualized os? Or maybe just
> because it's saving ram not on not running xwindows and so on?

RAM has become quite cheap. Adding a couple of gigs can be cheaper
than using developer time.

> Anyway, there are now 3 stress tests:
> -- 1 inserts all users, all jobs with one transaction
> -- 2 one transaction per user, all jobs get inserted at once
> -- 3 one transaction per job
>
> One transaction per job is too slow for inserting a lot of dummy data,
> though it looks like lemmih got better results in his own testing.
> Inserting all users is of course the fastest, however I couldn't do
> this for more than 200 users. The best option in practice was option
> 2, which allowed me to insert 1000 users at a time, eg by
>
> time wget http://www.happstutorial.com:5002/tutorial/stresstest/atomicinsertsalljobs/1000
>
> What I'm seeing with 25000 users is, I can only view the consultants
> page. If I try to look at consultants wanted (which checks if the jobs
> list is null) or show all jobs (which does a map fold over users to
> get all the jobs) the hypervisor kills happs for using too much
> memory.

Inserting one million jobs isn't too bad on my machine (which has 4Gb of ram):

david@desktop:happs-tutorial$ time wget
http://localhost:5001/tutorial/stresstest/atomicinsertsalljobs/100000
-o /dev/null
real 3m17.576s
user 0m0.004s
sys 0m0.004s

At this point, happs-tutorial is using 423megs of ram. It's only a
tiny bit more than firefox (:
I'll try to rewrite the tutorial using the BerkeleyDB binding I
uploaded yesterday[1].

>> We, at HAppS, are going for option 4: Throwing more machines at the
>> problem. Limited memory isn't the only problem of individual machines.
>> They are also limited in CPU capabilities and reliability. We're
>> trying to solve all of these problems by making HAppS a distributed
>> application.
>
> How would this work in the job board case? Could I accomplish 100,000
> with the same data model I have now, but using four machines? What
> would happen when I attempt to view all jobs? That's 20 million --
> whoa -- but I did make a paginator that allows you to scroll around in
> that many records keeping the actual browser page display down to a
> reasonable size. To display the data you don't need all the jobs, just
> the first 200, and pagination info for the rest. So really it comes
> down to doing a count of jobs records.

You wouldn't use exactly the data model you have now. It should look
more like this (excuse the formatting):

> data AppState = AppState{
> appUsers :: Map UserId User,
> appUsersByName :: Map UserName UserId,
> appJobs :: Map JobId Job,
> appJobsByName :: Map JobName JobId }

This makes it very easy to partition the users and the jobs over an
arbitrary number of nodes.

Listing the jobs can be done in many different ways. A list of job ids
could be shared on all nodes, or you could request a subset of the
jobs from each node and merge the responses, or you could do something
different depending on what you want.

> Is there an example somewhere in the head repos for sharding, on a
> large data set like what I have for the stress test?

Unfortunately there isn't. No sharding code has been released yet.

--
Cheers,
Lemmih

[1] http://hackage.haskell.org/cgi-bin/hackage-scripts/package/berkeleydb

[Lemmih]

Here are my timings using compact-map instead of berkeleydb:

david@desktop:happs-tutorial$ time wget
http://localhost:5001/tutorial/stresstest/atomicinsertsalljobs/100000
-o /dev/null

real 1m3.774s
user 0m0.000s
sys 0m0.000s

creating checkpoint
time: Tue Nov 11 15:34:38 CET 2008
shutting down system
time: Tue Nov 11 15:34:48 CET 2008

Memory usage at 269 megs. I'm using the compacting garbage collector.

I'd love to discuss the results with you in real time. Come by #happs
on irc.freenode.net some day.

--
Cheers,
Lemmih

[tphyahoo]

I'll definitely try compact-map.

Is this module intended as a drop-in replacement for Data.Map?

If so, are there any performance tradeoffs involved?

Thomas.

On 11 Lis, 16:05, Lemmih <lem...@gmail.com> wrote:
> On Sat, Nov 1, 2008 at 4:49 PM, Lemmih <lem...@gmail.com> wrote:

> > On Wed, Oct 15, 2008 at 1:52 PM, tphyahoo <thomashartm...@googlemail.com> wrote:
> >> Anyway, there are now 3 stress tests:
> >> -- 1 inserts all users, all jobs with one transaction
> >> -- 2 one transaction per user, all jobs get inserted at once
> >> -- 3 one transaction per job
>
> >> One transaction per job is too slow for inserting a lot of dummy data,
> >> though it looks like lemmih got better results in his own testing.
> >> Inserting all users is of course the fastest, however I couldn't do
> >> this for more than 200 users. The best option in practice was option
> >> 2, which allowed me to insert 1000 users at a time, eg by
>

> >> time wgethttp://www.happstutorial.com:5002/tutorial/stresstest/atomicinsertsal...

>
> >> What I'm seeing with 25000 users is, I can only view the consultants
> >> page. If I try to look at consultants wanted (which checks if the jobs
> >> list is null) or show all jobs (which does a map fold over users to
> >> get all the jobs) the hypervisor kills happs for using too much
> >> memory.
>
> > Inserting one million jobs isn't too bad on my machine (which has 4Gb of ram):
>
> > david@desktop:happs-tutorial$ time wget
> >http://localhost:5001/tutorial/stresstest/atomicinsertsalljobs/100000
> > -o /dev/null
> > real 3m17.576s
> > user 0m0.004s
> > sys 0m0.004s
>
> > At this point, happs-tutorial is using 423megs of ram. It's only a
> > tiny bit more than firefox (:
> > I'll try to rewrite the tutorial using the BerkeleyDB binding I
> > uploaded yesterday[1].
>
> Here are my timings using compact-map instead of berkeleydb:
>

> david@desktop:happs-tutorial$ time wgethttp://localhost:5001/tutorial/stresstest/atomicinsertsalljobs/100000

[Lemmih]

On Wed, Nov 12, 2008 at 2:24 PM, tphyahoo <thomash...@googlemail.com> wrote:
>
> I'll definitely try compact-map.
>
> Is this module intended as a drop-in replacement for Data.Map?

Yes, all functions are tested against Data.Map for compliance.

> If so, are there any performance tradeoffs involved?

There are. CompactMap's are quite a bit slower than ordinary maps. Try
to void newtyped keys. The newtypes unfortunately keep dictionaries
around which hurt performance.
Here are some benchmarks:
http://darcs.haskell.org/~lemmih/compact-map/benchmarks/

--
Cheers,
Lemmih