A Critical Look at the Happstack Paradigm

[mightybyte]

In the thread "Serialization code seems to require excessive amounts
of memory", there was a discussion about how serialization affects the
memory footprint of programs using happstack-state. I think we need
to take a broader look at this issue.

I run a website that is built completely on Happstack and Happstack-
State. I've been running live to the public for about 10 weeks. I
have just over 1000 registered users (though I'm sure they're not all
regulars), and I'm averaging 100-150 visits per day. My checkpoint
file weighs in at about 8 megs. Three weeks ago, I had to upgrade my
Linode hosting server from 540 to 720 megs of RAM to have enough
memory to run the app. Today, I again had to upgrade. So in three
weeks my memory requirements increased by about 180 megs. I don't
consider my site to have very large data requirements. There is no
multimedia--all my data is entered by my users via forms (no file
uploads).

When I just load the app and write a checkpoint file with +RTS -s, it
says that I had 200 MB maximum residency with 452 MB total memory in
use. When I actually run the app and navigate around the site to
pages that perform queries on the data, -s tells me that I had 332 MB
maximum residency with 923 MB total memory in use. (Hence the need to
upgrade to more than a 720 MB server.)

I can understand how the checkpoint file may use a much more compact
format than the in-memory data structures, thus explaining the
difference between the 8 meg checkpoint file and 200 megs of memory
usage. But the 50% increase when I do queries on the data combined
with the almost 3x space increase imposed by Haskell's garbage
collection is making me question the viability of using happstack-
state to serve dynamic websites. I'm looking at an increase of maybe
200 megs of RAM / month with only 150 visits per day!

Now, I tend to like radical innovation when it makes sense--and I
think Happstack's in-memory state does make sense. I went into this
project knowing that I would be using a new technology that might not
have all its scalability features implemented by the time I needed
them. I'm nowhere near needing to shard my state, but if my costs
keep increasing at the same rate, it will not be economically viable
for me to continue using Happstack-state as my data store.

I can think of a few possibile avenues of investigation:

Investigate the jump from 200MB to 332MB RAM usage when all I did was
do a few IxSet queries.
Investigate the 3x garbage collector overhead imposed by GHC. This
isn't our problem per-se, but maybe we can optimize things to reduce
this amount.

Perhaps we could also look into augmenting our system with some disk-
based storage while still keeping some of the other unique aspects of
Happstack's state?

Note that I am fully aware that I could use a RDBMS or any other
persistent data store instead of happstack-state. I just like the
elegance and simplicity of happstack-state and would like to keep as
many of these aspects as possible.

Anyone have comments or possible solutions to this issue?

[Lemmih]

IxSet could be adapted to use CompactMap. That would eliminate the GC
overhead and give you the same compact binary format as with
checkpoints.

--
Cheers,
Lemmih

[MightyByte]

Hmmm, that sounds promising. I'll take a look at it.

[Kamil Dworakowski]

Sorry to hear about these problems. I imagine it might be quite
expensive with each new upgrade. When I was last looking around for
hosting with large amount of ram ovh had the cheapest offer. I pay
them around 25 pounds a month for a *machine* with 2 GBs of ram. That
could solve your immediate problem with hosting bills.

I am myself working off time on a site powered by happstack, and am on
purpose trying to keep things simple. For instance, I have evaded
IxSet. It gets quite verbose for me at times, but at a point when I
will have to debug stuff, I will have things as simple as possible.
Patterns like this: http://conal.net/blog/posts/semantic-editor-combinators/
help too. I have no users yet though, so I should probably shut up.

[Matthew Elder]

I too was going to recommend switching to a physical host at some
point, it is much more cost effective than a vps.

I think you can get a 2gb server for around 99 bucks / month (USD)

specs for the server from 1and1.com:

* Opteron 1216
* 2 x 2.4GHz
* 2 GB RAM
* 2 x 250GB Software-RAID


You can use raid 1 for redundancy in the case of disk failure.

--
Need somewhere to put your code? http://patch-tag.com
Want to build a webapp? http://happstack.com

[Marc Weber]

Excerpts from mightybyte's message of Sat Aug 15 02:54:04 +0200 2009:

> Anyone have comments or possible solutions to this issue?

I only have some comments:

* I haven't had time to test a haskell system yet. I'm still busy
integrating hackage packgaes into nix. I will be done soon.

Anyway I was wondering for a long while: Should haskell be used for some
apps because it's easy to write multithreading apps? I mean if you write
it in C you don't even need multithreading (unless you have many
cores which can make a difference then) because C probably is 4 times
faster in typical cases.

And if you have about 1000 users and maybe 100 visitors a day you should
not need to buy a dedicated server for that task.

I mean compare this to a typicall PHP + MySQL application. It can serve
1000 users as well.

Anyway I keep watching this project with joy.

Marc Weber

[Thomas DuBuisson]

Marc Weber:

> Anyway I was wondering for a long while: Should haskell be used for some
> apps because it's easy to write multithreading apps? I mean if you write
> it in C you don't even need multithreading (unless you have many
> cores which can make a difference then) because C probably is 4 times
> faster in typical cases.

Often, it isn't the easier multithreading that makes using Haskell
attractive but the type and memory safety it affords you over C.
That said...

Yes! I think Haskell should be used in part due to it's easy
threading - which is one reason I built Control-Engine. You can
always perform FFI to C for those performance critical parts and still
use the Haskell RTS to manage the threading. In some cases it isn't
so much a language issue as an algorithm issue - algorithms that scale
are frequently half or a quarter the speed of their sequential
equivalents so this current age with two and four core processors
isn't so friendly to those solutions - perhaps once we have 64 cores
in our laptops (Tilera anyone?) such solutions will finally give real
world benefit.

TomMD

[Don Stewart]

mightybyte:

> Investigate the 3x garbage collector overhead imposed by GHC. This
> isn't our problem per-se, but maybe we can optimize things to reduce
> this amount.
>

That seems like a fruitful path. In general you should aim for GC to be
< 10% of your app.

You can also experiment with the compacting collector, and having larger
old generations.

-- Don

[Lemmih]

On Sat, Aug 15, 2009 at 11:45 PM, Don Stewart<do...@galois.com> wrote:
>
> mightybyte:
>> Investigate the 3x garbage collector overhead imposed by GHC.  This
>> isn't our problem per-se, but maybe we can optimize things to reduce
>> this amount.
>>
>
> That seems like a fruitful path. In general you should aim for GC to be
> < 10% of your app.

He was referring to the memory overhead.

--
Cheers,
Lemmih

[Don Stewart]

lemmih:

Fire up the compacting GC then.

-c Enable compaction for all major collections

-G<n> Number of generations (default: 2)
-T<n> Number of steps in younger generations (default: 2)
-c<n> Auto-enable compaction of the oldest generation when live data is
at least <n>% of the maximum heap size set with -M (default: 30%)
-w Use mark-region for the oldest generation (experimental)
-I<sec> Perform full GC after <sec> idle time (default: 0.3, 0 == off)

Would be interesting to see results.

[MightyByte]

Well, I started with the easiest of all the suggestions. Here are two
runs, where I serve one web page that does a pretty significant query,
then write a checkpoint.

---------

./Main +RTS -s

6,066,667,080 bytes allocated in the heap
1,546,225,288 bytes copied during GC
294,886,096 bytes maximum residency (9 sample(s))
6,932,632 bytes maximum slop
723 MB total memory in use (11 MB lost due to fragmentation)

Generation 0: 11497 collections, 0 parallel, 11.86s, 11.95s elapsed
Generation 1: 9 collections, 0 parallel, 2.37s, 2.80s elapsed

INIT time 0.00s ( 0.00s elapsed)
MUT time 3.89s ( 30.44s elapsed)
GC time 14.23s ( 14.75s elapsed)
EXIT time 0.00s ( 0.00s elapsed)
Total time 18.12s ( 45.19s elapsed)

%GC time 78.5% (32.6% elapsed)

Alloc rate 1,558,320,281 bytes per MUT second

Productivity 21.5% of total user, 8.6% of total elapsed

---------

./Main +RTS -c -s

6,066,664,808 bytes allocated in the heap
819,800,776 bytes copied during GC
294,075,576 bytes maximum residency (9 sample(s))
7,241,336 bytes maximum slop
531 MB total memory in use (9 MB lost due to fragmentation)

Generation 0: 11497 collections, 0 parallel, 14.94s, 15.38s elapsed
Generation 1: 9 collections, 0 parallel, 7.51s, 7.56s elapsed

INIT time 0.00s ( 0.00s elapsed)
MUT time 4.15s ( 19.54s elapsed)
GC time 22.45s ( 22.94s elapsed)
EXIT time 0.00s ( 0.00s elapsed)
Total time 26.60s ( 42.48s elapsed)

%GC time 84.4% (54.0% elapsed)

Alloc rate 1,460,765,271 bytes per MUT second

Productivity 15.6% of total user, 9.8% of total elapsed

---------

So just -c by itself seems to be a pretty significant improvement.

[MightyByte]

> On Sat, Aug 15, 2009 at 2:15 AM, Kamil
> Dworakowski<ka...@dworakowski.name> wrote:
>>
>> Sorry to hear about these problems. I imagine it might be quite
>> expensive with each new upgrade. When I was last looking around for
>> hosting with large amount of ram ovh had the cheapest offer. I pay
>> them around 25 pounds a month for a *machine* with 2 GBs of ram. That
>> could solve your immediate problem with hosting bills.

I didn't see anything quite that cheap from OVH. The cheapest I saw
was 35 pounds / month which currently equals about $57 / month. (More
elaboration in my response below.)

On Sat, Aug 15, 2009 at 10:39 AM, Matthew Elder<ma...@mattelder.org> wrote:
>
> I too was going to recommend switching to a physical host at some
> point, it is much more cost effective than a vps.
>
> I think you can get a 2gb server for around 99 bucks / month (USD)
>
> specs for the server from 1and1.com:
>
>    * Opteron 1216
>    * 2 x 2.4GHz
>    * 2 GB RAM
>    * 2 x 250GB Software-RAID
>
>
> You can use raid 1 for redundancy in the case of disk failure.

Right now I'm paying $50 / month (USD) with 900 MB RAM. So they are
not much cheaper in terms of dollars / MB, but it's a significantly
higher absolute cost (and probably not as easy to upgrade).

My main point in all this is that a LAMP site with my traffic could
easily run on < $10 / month shared hosting. So my costs are already
5x higher than they should be. In short, it seems like an all
in-memory solution may not be cost-effective for some types of small
start-up.

That said, some of the other responses had promising ideas, and I
already got an improvement with the compacting garbage collector. I'm
definitely not giving up hope. I just think we should investigate
this issue more closely.

[Lemmih]

It would help a lot if we could see the code.

--
Cheers,
Lemmih

[MightyByte]

On Fri, Aug 14, 2009 at 9:24 PM, Lemmih<lem...@gmail.com> wrote:
>

> IxSet could be adapted to use CompactMap. That would eliminate the GC
> overhead and give you the same compact binary format as with
> checkpoints.

I adapted created a Happstack.Data.Compact.IxSet and modified it to
use CompactMap. It looks like the only thing needed to be able to use
it is the splitLookup function for CompactMap. I looked at the
CompactMap code, but it looks like it will take me a significant
amount of studying before I can implement it.

[Matthew Elder]

I definitely think that the resource cost of bringing the state online
needs to be as close to linear as possible. Running lookups / queries
/ insertions is another problem alltogether.

--
Sent from my mobile device

[MightyByte]

I took Lemmih's suggestion and implemented Compact-IxSet based on the
CompactMap library. I haven't had time to test it yet to see what
kind of a difference it makes, but I put the code up on GitHub in case
anyone else is interested.

http://github.com/mightybyte/compact-ixset/tree/master

This is just a temporary home util we decide if this is worth keeping
and the best ultimate location for the code.

[Kamil Dworakowski]

On Aug 16, 4:07 pm, MightyByte <mightyb...@gmail.com> wrote:
> > On Sat, Aug 15, 2009 at 2:15 AM, Kamil
> > Dworakowski<ka...@dworakowski.name> wrote:
>
> >> Sorry to hear about these problems. I imagine it might be quite
> >> expensive with each new upgrade. When I was last looking around for
> >> hosting with large amount of ram ovh had the cheapest offer. I pay
> >> them around 25 pounds a month for a *machine* with 2 GBs of ram. That
> >> could solve your immediate problem with hosting bills.
>
> I didn't see anything quite that cheap from OVH.  The cheapest I saw
> was 35 pounds / month which currently equals about $57 / month.  (More
> elaboration in my response below.)

I am using http://www.ovh.co.uk/products/rps3.xml which is 20 pounds
per month.