Clojure vs Scala - anecdote

[Sean Corfield]

I just wanted to share this experience from World Singles...

Back in November 2009, we started developing with Scala. We needed a
long-running process that published large volumes of changes from our
member database as XML packets published to a custom search engine.
The mapping from half a dozen tables in the database to a flat XML
schema was pretty complex and the company had tried a number of
solutions with mixed success in the past. I introduced Scala based on
the promises of performance, concurrency and type safety - and
conciseness (especially with XML being a native data type in Scala).

We've been running the Scala publishing daemons in production for most
of two years. Generally they work pretty well but, under stress, they
tend to hit Out of Memory exceptions and, after a lot of poking
around, we became fairly convinced it was due (at least in part) to
the default actor implementation in Scala. Scala is going to fold in
Akka soon and we had been considering migrating to Akka anyone...

But having introduced Clojure this year (after experimenting with it
since about May last year), we figured we'd have a short spike to
create a Clojure version of the Scala code to see how it worked out.

It took about 15 hours to recreate the publishing daemon in Clojure
and get it to pass all our tests. Today we ran a "soak test"
publishing nearly 300,000 profiles in one run. The Scala code would
fail with OoM exceptions if we hit it with 50,000 profiles in one run
(sometimes less). The Clojure code sailed thru and is still happily
running - so we'll be replacing the Scala code during our next
production build.

The other aspect that's interesting is that the Scala code totaled
about 1,000 lines (about 31k characters of code). The Clojure
replacement is just under 260 lines (around 11.5k characters of code).
Neither code base has much in the way of comments (*ahem* - I'm not
proud of that, just pointing out that there's no "noise" offsetting
the code comparison). That doesn't include unit tests either, it's
just the raw production code. The form of the Clojure code mostly
follows the form of the Scala code, most of the same functions - it
was very functional Scala - with some refactoring to helper functions
to make it more modular and more maintainable.

The net result is (obviously) that we'll be taking the Clojure
publishing daemon to production and we'll be dropping Scala
completely.

Kudos to Rich Hickey and the Clojure/core team for creating a great
general purpose language that can solve big problems - thank you!
--
Sean A Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/
World Singles, LLC. -- http://worldsingles.com/
Railo Technologies, Inc. -- http://www.getrailo.com/

"Perfection is the enemy of the good."
-- Gustave Flaubert, French realist novelist (1821-1880)

[Ambrose Bonnaire-Sergeant]

Thanks for sharing Sean, very interesting!

Ambrose

[Luc Prefontaine]

Hi,

We have been running Clojure daemons 24/7 in prod. since Jan. 2009.
We also considered Scala back in 2008.

We could not agree more with your conclusions :)

Luc P.

--
Luc P.

================
The rabid Muppet

[Marko Kocić]

While we are at this topic, how do you run Clojure deamons. Do you have some scripts to set it up how?

Is there a simple way to daemonize lein project?

Regards,

Marko

[Bronsa]

the lein-daemon plugin seems to do that

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en

[Dennis Haupt]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

so the scala actors add much more overhead than the clojure equivalent?

- --

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJOZ6cgAAoJENRtux+h35aGsxoQAKAkmQb/8cxVsHSw1bH6mZjW
Hoea0zi90eO2ds9Wk1wrFjtc0wfPPHdrp0FpZ3w1090BkwyKcRBj/iDM45sP4IY5
grc2I6vaRhfgIIVuaxgUt9HzTCbyjCOxk9xJHpCyY7sIfEIcFwNuzQWVHxgdqG/l
CY/9mDe1Wex3rt2QxCSsUX/+yB5uXaxmAoX5m0jyEAmZzw/46+cVzZ8xMi9Gw1o/
mjI/mvpwTmdGcPkh7DamIEU8QjYbNBosgPWpNktJzmhtUaFdXhEMxdyDhldzUcJZ
J8tZZkTWZoQqPfVdMPgfe1blDtV+nse8X2HDqed+Df42TU1YY+1VJ7e8jfr3vV62
cI+6SAqYTT91UC57GkmYKVOm01vNMpp98+fxaxBHUQi64tv/hIkWG4iHRgBCvncR
hdIKfmzVwcPGrOZu6QT0RrVQzeEbz83+3l4CZQ7KOdL8k5vjd5b1T/LsPrQM1rod
jDAn481tmpZKtSLe8+QbSakxfIFT9oTKUXbtDEEkN2CbJOkE4/EQwuCc/gnlo9Mr
YPlPfx96JLxBfVq6JZ92VSdrpnEBS65HjKhWF587XjGjTqzYbbCNJIekwRdqga8e
zkonzIj+IgnuZznV/fbKZ2yCEnO85TXoj0ZWUDnw0Ffvu2vUFvSF0ykR2BHxZBFD
a1yhe/wr8AGyvIff6Hj8
=N1zz
-----END PGP SIGNATURE-----

[Sean Corfield]

On Wed, Sep 7, 2011 at 10:17 AM, Dennis Haupt <d.ha...@googlemail.com> wrote:
> so the scala actors add much more overhead than the clojure equivalent?

The main problem is that the current implementation of actors in Scala
suffers from known memory leaks and performance problems - problems
that are completely addressed by Akka, which is why they're going to
incorporate it and replace the current implementation.

Our choices at World Singles were: migrate to Akka, wait for Scala
2.10 (which is when I think Akka will be folded in). Neither were
appealing solutions. Migrating to Clojure was less work and more
timely (for us).

[Aaron Bedra]

I have used jsvc in the past and found it to be a great tool. It allows you to configure which user the application runs as, and does proper detaching. It allows you to configure output streams and pid files to your liking. I have written some simple init scripts as well to make it very unix service like.  All in all, I like it a lot more than lein daemon. Lein is a development and packaging tool. I wouldn't want it to be a dependency of my production systems.

Cheers,

Aaron Bedra

--

Clojure/core

http://clojure.com

[Marko Kocić]

Thanks for the tip about jsvc. I'll give it a try. Do you have some script examples to share, since having Linux service is exactly what I need?

Thanks,
Marko

[Aaron Bedra]

I'll try and put together a few things including the code that
implements the interface to get hooked up to jsvc.

--

[Tal Liron]

On Wednesday, September 7, 2011 1:53:43 PM UTC-5, Marko Kocić wrote:

Thanks for the tip about jsvc. I'll give it a try. Do you have some script examples to share, since having Linux service is exactly what I need?

I strongly recommend Tanuki's wrapper over jsvc:

http://wrapper.tanukisoftware.com/

It's better in every possible way, and will make your deployment life so much easier! It's something I would very much like to see as baked into to the JVM.

If the GPL licensing is any concern (i.e., you need to distribute your product), YAJSW is somewhat Tanuki-compatible and also has an interesting architecture:

http://yajsw.sourceforge.net/

-Tal

[Alex Ott]

I've used Apache Commons Daemon in my projects

On windows I had following registration .bat to run it as service:

@echo off
cd /D "$INSTALL_PATH\sbin"

SET UHOME=$INSTALL_PATH
SET UMAINCLASS=myprog.main

SET ULOGDIR=%UHOME%\var\log
MKDIR %UHOME%\var\tmp\myprog\
MKDIR %ULOGDIR%

%UHOME%\sbin\%PROCESSOR_ARCHITECTURE%\myprog.exe //IS//Myprog
--DisplayName="Myprog"
--Install=%UHOME%\sbin\%PROCESSOR_ARCHITECTURE%\myprog.exe --Jvm=auto
--Startup=auto --StartPath=%UHOME%\sbin --StartMode=jvm
--StopMode=jvm --StartClass=%UMAINCLASS% --StartMethod=main
--StartParams=start --StopMode=jvm --StopClass=%UMAINCLASS%
--StopMethod=main --StopParams=stop --LogPath=%ULOGDIR%
--StdOutput=auto --StdError=auto
--Classpath=%UHOME%\lib\myprog-0.1-jar-with-dependencies.jar
--JvmOptions=-Djava.library.path=%UHOME%\lib;-Dwservice.configuration=..\etc\wservice.properties;-Dlog4j.defaultInitOverride=true;-Dlog4j.configuration=..\etc\log4j.properties

sc start Myprog

on Linux I just followed instructions from ACD page

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en

--
With best wishes,                    Alex Ott
http://alexott.net/
Tiwtter: alexott_en (English), alexott (Russian)
Skype: alex.ott

[Nathan Sorenson]

I adore Clojure as well, but could this success not be partially due
to the "reimplementing for the second time" phenomenon? i.e. if you re-
wrote the entire thing in Scala again, perhaps you would see similar
gains in brevity etc?

> Railo Technologies, Inc. --http://www.getrailo.com/

[Laurent PETIT]

Isn't it Brooks who said "you will throw it away at least 3 times", or something like this ? :)

2011/9/13 Nathan Sorenson <nd...@sfu.ca>

[Meikel Brandmeyer]

“Plan to throw one away.”

[Sean Corfield]

On Tue, Sep 13, 2011 at 9:48 AM, Nathan Sorenson <nd...@sfu.ca> wrote:
> I adore Clojure as well, but could this success not be partially due
> to the "reimplementing for the second time" phenomenon? i.e. if you re-
> wrote the entire thing in Scala again, perhaps you would see similar
> gains in brevity etc?

Well, the Scala world has moved on quite a bit since 2009 so I could
certainly make it somewhat more concise (I'd use the parallel
collections in 2.9 instead of actors and I hope there's a better SQL
abstraction by now so I could drop the ResultSet collection wrapper I
wrote). I doubt I could reduce it by a factor of three which is what
it would take to get close to the Clojure code.

I don't know who posted it on HN but I see it's also on DZone and so
it's generated a lot of noise out there and now I'm probably going to
do a more detailed comparison and analysis to post on my blog, to
answer some of the critical voices on HN... It was intended to be
purely anecdotal but that doesn't seem to satisfy anyone! :)

--
Sean A Corfield -- (904) 302-SEAN

An Architect's View -- http://corfield.org/
World Singles, LLC. -- http://worldsingles.com/

Railo Technologies, Inc. -- http://www.getrailo.com/

[Laurent PETIT]

Oh, it was just one, after all ? 

Please, don't tell this to my boss :-D

2011/9/13 Meikel Brandmeyer <m...@kotka.de>

“Plan to throw one away.”

--

[cig]

Impressive, wonder if they were running this on a single node or more
widespread?
In a wide spread environment I think Erlang would be the true winner,
though it does not natively have macros :-(
There is an implementation of Lisp for Erlang called LFE (lisp
flavored Erlang) which I looked at, which does have macros and a real
engine underneath.
But clojure is an awesome combination

> An Architect's View --http://corfield.org/
> World Singles, LLC. --http://worldsingles.com/

> Railo Technologies, Inc. --http://www.getrailo.com/

[Sean Corfield]

On Wed, Sep 14, 2011 at 11:15 PM, cig <clifford...@gmail.com> wrote:
> Impressive, wonder if they were running this on a single node or more
> widespread?

We run an instance of the process on multiple nodes, configured
slightly differently. We needed "some" parallelization to improve
throughput but didn't need a massive net of processes. And we needed
JVM interop so Erlang is out (and Erjang isn't yet mature enough - at
least, not last time I looked).

> But clojure is an awesome combination

Indeed.

--
Sean A Corfield -- (904) 302-SEAN

An Architect's View -- http://corfield.org/
World Singles, LLC. -- http://worldsingles.com/

Railo Technologies, Inc. -- http://www.getrailo.com/

[Edward Garson]

Native Erlang does have a macro facility, but it is not as powerful as
Lisp/Clojure's.

On Sep 15, 2:15 am, cig <clifford.goldb...@gmail.com> wrote:
[snip]

> In a wide spread environment I think Erlang would be the true winner,
> though it does not natively have macros :-(

[snip]

[Raoul Duke]

On Thu, Sep 15, 2011 at 9:24 AM, Edward Garson <ega...@gmail.com> wrote:
> Native Erlang does have a macro facility, but it is not as powerful as
> Lisp/Clojure's.

lfe, baby, though of course that is not "native" erlang.

[Tal Liron]

On Tuesday, September 13, 2011 1:44:09 PM UTC-5, Sean Corfield wrote:

It was intended to be purely anecdotal but that doesn't seem to satisfy anyone! :)

Homer: "You know, when I was a boy, I really wanted a catcher's mitt, but my dad wouldn't get it for me. So I held my breath until I passed out and banged my head on the coffee table. The doctor thought I might have brain damage."
Bart: "Dad, what's the point of this story?"
Homer: "I like stories."