Learning or not learning Scala/Lift?
Jonas
doesn't mean much until it has been adopted by a larger community (and
that is only possible with a good documentation).
I still haven't decided to use Lift because I want to be safe if it'll
support some features in which I'm interested. But the main problem
(the same of another people) is to have to learn another new
developing language and a different paradigm as it's the functional
language.
I believe that I'll risk myself to learn another language.
David Pollak
Jonas,
I want to be very honest. I don't think that lift is ready for external hardcore projects yet. "WHAT!??!?!" you ask. "Aren't you using lift in a project that you're doing?"
Yes. I believe that Scala is an amazing language and it's been growing and maturing at a nice pace and the LAMP team has been doing a great job of moving from an "Experiment" to an "Experience."
I think that lift has the right pieces to make it a world class web framework and it's been very stable and scalable for us at CircleShare.
However, I'm doing radical surgery on the APIs on a weekly basis. I'm cool with this as I have both lift and CircleShare source so that when I make a change to lift I can go in and change all the dependencies in CircleShare.
Also, I have limited availability to make API changes for other folks and cannot commit to doing so. Yes, there are a couple of non-CircleShare committers, but I don't think any of them can commit to making necessary API changes for other folks.
While I'd really, really love to see 100 public web sites based on lift by the end of the summer, anyone who builds something with lift right now is taking a huge risk that they're going to have to change up their code 20 times in the next 3 months because APIs change and they're going to lack certain features.
Thanks,
David
--
lift, the fast, powerful, easy web framework
http://liftweb.net
Derek Chen-Becker
I completely agree with what you've outlined here. While I'm looking at
playing around with lift, I'll probably wait until it's a little more
stable for a production app. In the meantime, I think that having the
flexibility to make radical API changes is necessary at this stage
(infancy) of lift. The current amorphous nature makes allows for a lot
of experimentation while we(you) find the right balance for the API. I'm
definitely looking forward to seeing what it crystalizes into...
Derek
Jonas
wrote:
> Yes. I believe that Scala is an amazing language and it's been growing and
> maturing at a nice pace and the LAMP team has been doing a great job of
> moving from an "Experiment" to an "Experience."
same problem that Java (because is using its JVM), the overhead and
the consume of memory. In fact, as you can see in that benchmark,
Scala is one of the languages that is consuming more memory.
David Pollak
Memory consumption is a strange thing. Yes, for a single benchmark, Scala consumes more memory than even Java (and it will always consume more memory because of the Scala libraries on top of the Java libraries.)
The realities of running a server is that Java and Scala allow you to support far more users per byte of server footprint than Ruby/Rails and any other "share nothing" platform.
I have been doing benchmarks of lift and Butterfly (our internal name for our CircleShare server.) On an Amazon EC2 instance, lift and Butterfly (running on Tomcat 6) are able to support 3,400 simultaneous logged in users making 220 request per second. That's a single core (actually a fraction of a single core) machine running both the app server and MySQL instance and they were able to service 1/3rd of Twitter's load (it takes Twitter 16 cores and a lot more RAM to service 600 requests per second.) The JVM size in the benchmark was 512MB and the actual memory usage was consistent around 128M. I ran the test (220 requests per second) for 22 hours.
So, yes, the JVM and Scala libraries have a non-trivial footprint. If you had to fire up a separate instance of the JVM to support a single request (like Rails), you'd need 16GB+ of RAM to support the same request load.
Thanks,
David
bartvdo
It's basicly a game to see which can conform to the benchmark the
best. Take any result with a huge amount of salt.
It looks at a specific problem domain, and webapplications aren't a
part of it.
so to come back to the problem of memory overhead. Yes JVM
applications have a large memory overhead. But compared to LAMP
systems, this overhead is negligeble. One JVM process might consume
256 MB in memory (usually the standard upping for a web application),
but it can handle (due to good multithreading) say 100 concurrent
requests (jetty has something like a max of 125 concurrent threads
configured). With most LAMP implementations you have one process per
concurrent user. These processes are shared nothing, and have to
create their own pool of classes and caches etc.
Even though these processes might use less memory per process (say 20
MB, although I've read about 200+ MB of rails process sizes ) together
they will use alot more than the 256MB the JVM uses.
There has been alot of discussion on this the last month in the
blogosphere, especially in replies to DHH's claim in his blog that
rails scales fine as it is.
I think that either a self made implementation in d,c++,c,ocaml as a
webapplication or maybe a webapplication in erlang _might_ improve the
memory usage of your webapplication.
So A) benchmarks are only usefull when doing the right one
B) the memory overhead of the JVM is not bad if you figure in the
excelent concurrency possibilities (especially with Scala).
On 16 jun, 14:09, Jonas <jonas....@googlemail.com> wrote:
> On 11 jun, 17:15, "David Pollak" <feeder.of.the.be...@gmail.com>
> wrote:> Yes. I believe that Scala is an amazing language and it's been growing and
> > maturing at a nice pace and the LAMP team has been doing a great job of
> > moving from an "Experiment" to an "Experience."
>
> It's true that Scala is an amazing language but it follows having the
> same problem that Java (because is using its JVM), the overhead and
> the consume of memory. In fact, as you can see in that benchmark,
> Scala is one of the languages that is consuming more memory.
>
> http://shootout.alioth.debian.org/sandbox/benchmark.php?test=all&lang...
Erik Engbrecht
In order to get a really good comparison you have to separate out the fixed memory consumption (JVM, standard libraries, etc) from memory consumption from the memory consumption associated with "solving the problem," be it serving web requests or analyzing data. In cases where you can amortize the impact over many, many sessions (like a web application) or over a large problem size (analyzing a large dataset that must be kept in memory) the fixed cost disappears into the noise.
For very long running processes the JVM is often also much better than native languages like C/C++, because the VM can take care of issues like heap fragmentation that can make a memory-leak free program that theoretically has a lot of memory available run out of memory because none of it is contiguous. In C/C++ you would have to use a custom allocator and essentially manage your own memory, and it will have to be a really, really good one in order to match the efficiency of JVM memory management.
-Erik
Isaac Gouy
On Jun 16, 6:19 am, bartvdo <bart...@gmail.com> wrote:
> I take it you read the home page of the shootout?
I have :-)
> It's basicly a game to see which can conform to the benchmark the
> best. Take any result with a huge amount of salt.
Saying - take any result with a pinch of salt - suggests there's
something wrong with the results themselves, but that isn't what you
say in the following comments.
Rather than " take any result with a huge amount of salt" you seem to
mean - don't generalize from those specific results to other different
situations (100m times would be a bad predictor of marathon finish
times and vica versa).
> >http://shootout.alioth.debian.org/sandbox/benchmark.php?test=all〈...
David Pollak
Perhaps this suggests another test for the shoot-out: servicing
simultaneous web requests for dynamically generated pages that do not
involve database requests.
What say you?
Thanks,
David
bartvdo
>
> I have :-)
I was asking Jonas :-)
>
> > It's basicly a game to see which can conform to the benchmark the
> > best. Take any result with a huge amount of salt.
>
> Saying - take any result with a pinch of salt - suggests there's
> something wrong with the results themselves, but that isn't what you
> say in the following comments.
>
> Rather than " take any result with a huge amount of salt" you seem to
> mean - don't generalize from those specific results to other different
> situations (100m times would be a bad predictor of marathon finish
> times and vica versa).
>
Yes that was my main point. And of course to the fact that the
shootout benchmarks aren't meant as a way to make discisions. Maybe
the salt mountain was a wrong picture to paint (I think faster than I
write, which usually results in wrongfull use of imagery :-).
My main problems with the shootout (which is probably the best
implemented benchmark across languages, and loads of fun and
informational) are that startup is penalized, while memory leaks
really aren't (I'm not saying that any implementations leak, but leaks
are hard to detect in programs that only run a short amount of time).
I've been toying with the idea to fork the benchmarks with a change
where the benchmarks would take atleast an hour per test (no matter
how impractical in implementation :-) . I think that would be a more
honest result as most important applications these days are longer
running applications.
Although saying this I do realize that most applications are just
wainting for input from a user.
All of the above is my very humble opinion ;-) and I might be very
wrong (which I can accept :)
As far as going back to the topic, I think Scala is a great starting
point for scalable web applications based on my experiences/research
with LAMP and java based applications.
Isaac Gouy
On Jun 16, 9:49 am, "David Pollak" <feeder.of.the.be...@gmail.com>
wrote:
> Isaac,
>
> Perhaps this suggests another test for the shoot-out: servicing
> simultaneous web requests for dynamically generated pages that do not
> involve database requests.
>
> What say you?
That would be another test, but not the same kind of test as the
others shown in the benchmarks game.
It's hard enough for people to figure out what's going on in those 50
line programs - once you layer on a web framework, forget about it.
Isaac Gouy
-snip-
> startup is penalized
I have more patience with that complaint when someone makes the effort
to find out whether or not it is significant for those specific
programs (I also have more sympathy for the Erlang guys - they are
starting up an application server!).
> I've been toying with the idea to fork the benchmarks with a change
> where the benchmarks would take at least an hour per test (no matter
> how impractical in implementation :-)
Please do! (Just don't call it the benchmarks game.)
> I think that would be a more honest result as most important applications these
> days are longer running applications.
Be careful about saying "honest" - most people take the slightest
suggestion that they are less honest very personally.