A large Clojure deployment
123 views
Skip to first unread message
Stuart Sierra
Apr 7, 2009, 10:41:21 AM4/7/09
to clo...@googlegroups.com
Here: http://www.altlaw.org/
About 4000 lines of Clojure code, 2500 of Java, powering a web site
with well over a million pages, averaging around 10,000 visitors a
day.
Some of what I'm using:
Restlet
StringTemplate
Solr
Hadoop
Apache Java Commons
markdownj
cpdetector
JRuby
Most of the code is online, GPL'd, at http://github.com/lawcommons
-Stuart Sierra
Sean
Apr 7, 2009, 10:47:51 AM4/7/09
to Clojure
Okay wow... it'll take some time to fully appreciate this.
Can you comment on your hardware stack? How many servers are you
using? Is there an RDBMS in there somewhere?
How was deployment?
Looks awesome, thanks for sharing!
Can you comment on your hardware stack? How many servers are you
using? Is there an RDBMS in there somewhere?
How was deployment?
Looks awesome, thanks for sharing!
David Nolen
Apr 7, 2009, 11:40:00 AM4/7/09
to clo...@googlegroups.com
Congrats!
Perhaps soon there should be a Projects Using Clojure section on the main site?
Good way to get the word out that people are using Clojure in the "real world"
David
Michael Wood
Apr 7, 2009, 12:10:26 PM4/7/09
to clo...@googlegroups.com
On Tue, Apr 7, 2009 at 5:40 PM, David Nolen <dnolen...@gmail.com> wrote:
> Congrats!
> Perhaps soon there should be a Projects Using Clojure section on the main
> site?
> Good way to get the word out that people are using Clojure in the "real
> world"
> Congrats!
> Perhaps soon there should be a Projects Using Clojure section on the main
> site?
> Good way to get the word out that people are using Clojure in the "real
> world"
Sounds like a good idea to me. Maybe a "Powered by Clojure" logo
would also be in order?
--
Michael Wood <esio...@gmail.com>
Stuart Sierra
Apr 7, 2009, 12:24:38 PM4/7/09
to Clojure
Hi Sean,
It's deployed on an Ubuntu server on Amazon EC2. Just one server for
now, but designed to scale to more. No RDBMS!
I'm not using any Amazon services besides EC2 and S3, but I'm looking
at SQS and SimpleDB.
Deployment was pretty easy; my only problems have been non-Clojure-
related. For example, system properties set in Ant don't persist into
Log4j, so I have to use a shell script to launch the server.
Restlet's default logging doesn't match Apache's format, so making
AWStats work has been a hassle.
If you're interested in size:
There are ~870,000 documents on the site.
The Lucene/Solr index is ~6 GB.
The HTML content is ~4 GB, compressed.
99% of the pages are static HTML, pre-rendered in a Hadoop job.
-Stuart Sierra
It's deployed on an Ubuntu server on Amazon EC2. Just one server for
now, but designed to scale to more. No RDBMS!
I'm not using any Amazon services besides EC2 and S3, but I'm looking
at SQS and SimpleDB.
Deployment was pretty easy; my only problems have been non-Clojure-
related. For example, system properties set in Ant don't persist into
Log4j, so I have to use a shell script to launch the server.
Restlet's default logging doesn't match Apache's format, so making
AWStats work has been a hassle.
If you're interested in size:
There are ~870,000 documents on the site.
The Lucene/Solr index is ~6 GB.
The HTML content is ~4 GB, compressed.
99% of the pages are static HTML, pre-rendered in a Hadoop job.
-Stuart Sierra
Laurent PETIT
Apr 7, 2009, 12:35:42 PM4/7/09
to clo...@googlegroups.com
Waooh, congrats !
Surely an important step towards demonstrated to our employers the power and presence of clojure !
Could you elaborate on the rationale behind using java or clojure files ?
In particular, was it more due to a (some) limitation(s) in the current abilities to generate java from clojure ? Or maybe due to the fact that certain call to third party java APIs was easier to write with java code assist in a java editor ? Or was it for performance concerns ? ...
thanks,
--
Laurent
Surely an important step towards demonstrated to our employers the power and presence of clojure !
Could you elaborate on the rationale behind using java or clojure files ?
In particular, was it more due to a (some) limitation(s) in the current abilities to generate java from clojure ? Or maybe due to the fact that certain call to third party java APIs was easier to write with java code assist in a java editor ? Or was it for performance concerns ? ...
thanks,
--
Laurent
2009/4/7 Stuart Sierra <the.stua...@gmail.com>
Allen Rohner
Apr 7, 2009, 3:11:13 PM4/7/09
to Clojure
> It's deployed on an Ubuntu server on Amazon EC2. Just one server for
> now, but designed to scale to more. No RDBMS!
>
app, but it looks you beat me to it. :-)
Can you go into detail about what/how you are persisting since you
aren't using an RDBMS?
Allen
Stuart Sierra
Apr 7, 2009, 3:17:01 PM4/7/09
to Clojure
The java files were just stuff I'd already written in Java that I
didn't feel like rewriting.
-Stuart
On Apr 7, 12:35 pm, Laurent PETIT <laurent.pe...@gmail.com> wrote:
> Waooh, congrats !
>
> Surely an important step towards demonstrated to our employers the power and
> presence of clojure !
>
> Could you elaborate on the rationale behind using java or clojure files ?
> In particular, was it more due to a (some) limitation(s) in the current
> abilities to generate java from clojure ? Or maybe due to the fact that
> certain call to third party java APIs was easier to write with java code
> assist in a java editor ? Or was it for performance concerns ? ...
>
> thanks,
>
> --
> Laurent
>
> 2009/4/7 Stuart Sierra <the.stuart.sie...@gmail.com>
didn't feel like rewriting.
-Stuart
On Apr 7, 12:35 pm, Laurent PETIT <laurent.pe...@gmail.com> wrote:
> Waooh, congrats !
>
> Surely an important step towards demonstrated to our employers the power and
> presence of clojure !
>
> Could you elaborate on the rationale behind using java or clojure files ?
> In particular, was it more due to a (some) limitation(s) in the current
> abilities to generate java from clojure ? Or maybe due to the fact that
> certain call to third party java APIs was easier to write with java code
> assist in a java editor ? Or was it for performance concerns ? ...
>
> thanks,
>
> --
> Laurent
>
Stuart Sierra
Apr 7, 2009, 3:26:09 PM4/7/09
to Clojure
There isn't an RDBMS mostly because I have an irrational prejudice
against them. In this case, the content is mostly static. The Hadoop
jobs process collection of source documents -- a few dozen GB, a big
ol' mess of PDF, XML, JSON, even WordPerfect! -- and, many hours
later, output two things: 1) the Solr index, and 2) a few million HTML
files. Everything you see on the site comes from one of those two.
There is no run-time data to persist.
When I need a DB, I'll probably use Amazon SimpleDB.
By the way, I've been using Clojure for the Hadoop/back-end code for
over a year. The front-end used to be Ruby on Rails, I switched it to
Clojure a few months ago.
-Stuart
against them. In this case, the content is mostly static. The Hadoop
jobs process collection of source documents -- a few dozen GB, a big
ol' mess of PDF, XML, JSON, even WordPerfect! -- and, many hours
later, output two things: 1) the Solr index, and 2) a few million HTML
files. Everything you see on the site comes from one of those two.
There is no run-time data to persist.
When I need a DB, I'll probably use Amazon SimpleDB.
By the way, I've been using Clojure for the Hadoop/back-end code for
over a year. The front-end used to be Ruby on Rails, I switched it to
Clojure a few months ago.
-Stuart
Baishampayan Ghose
Apr 7, 2009, 3:38:51 PM4/7/09
to clo...@googlegroups.com
>> It's deployed on an Ubuntu server on Amazon EC2. Just one server for
>> now, but designed to scale to more. No RDBMS!
>>
>
> Awesome, Congratulations. I was hoping to be the 2nd "famous" Clojure
> app, but it looks you beat me to it. :-)
>> now, but designed to scale to more. No RDBMS!
>>
>
> Awesome, Congratulations. I was hoping to be the 2nd "famous" Clojure
> app, but it looks you beat me to it. :-)
Excuse my ignorance, but which one is the first famous Clojure app?
Regards,
BG
--
Baishampayan Ghose <b.g...@ocricket.com>
oCricket.com
http://oCricket.com
Abhishek Reddy
Apr 8, 2009, 2:06:52 PM4/8/09
to clo...@googlegroups.com
On 4/8/09, Baishampayan Ghose <b.g...@ocricket.com> wrote:
>
> Excuse my ignorance, but which one is the first famous Clojure app?
>
>
> Excuse my ignorance, but which one is the first famous Clojure app?
>
That could be a reference to Luc Prefontaine's veterinarian hospital
application. It got some press at InfoQ:
http://www.infoq.com/news/2009/01/clojure_production
--
Abhishek Reddy
http://abhishek.geek.nz
Luc Prefontaine
Apr 8, 2009, 4:41:38 PM4/8/09
to clo...@googlegroups.com
Yep, looks like we where the first of hopefully many production implementations...
It's quite funny, we are expanding the bus logic presently and without Clojure, I wonder how much effort
and Java code it would have required us.
We store rules in a database (hey Stuart, we really need one !:))), it remains simple to enter them from our Rails GUI.
We convert them to full sexpr before applying them to the messages on the bus.
Some are routing rules, others apply transformations to messages to adapt to endpoints limitations (field length limitations, field substitutions,...)
All of this is quite dynamic and Java was not a workable option. We were juggling with JRuby when we discovered
Clojure. JRuby was not quite complete at the time we looked at it and Clojure looked usable and expandable right now.
In retrospective, I find it a lot easier with Clojure than Ruby, immutability is a key point here.
We do not want any side effects on the original message. More fun, less debugging hours...
Welcome to the club Stuart :)))
I looked at your app and it's impressive. Ours is more nuts and bolts and not generally exposed to the public aside from the vets of course.
Yours has a larger audience and that's good for all of us.
Luc
It's quite funny, we are expanding the bus logic presently and without Clojure, I wonder how much effort
and Java code it would have required us.
We store rules in a database (hey Stuart, we really need one !:))), it remains simple to enter them from our Rails GUI.
We convert them to full sexpr before applying them to the messages on the bus.
Some are routing rules, others apply transformations to messages to adapt to endpoints limitations (field length limitations, field substitutions,...)
All of this is quite dynamic and Java was not a workable option. We were juggling with JRuby when we discovered
Clojure. JRuby was not quite complete at the time we looked at it and Clojure looked usable and expandable right now.
In retrospective, I find it a lot easier with Clojure than Ruby, immutability is a key point here.
We do not want any side effects on the original message. More fun, less debugging hours...
Welcome to the club Stuart :)))
I looked at your app and it's impressive. Ours is more nuts and bolts and not generally exposed to the public aside from the vets of course.
Yours has a larger audience and that's good for all of us.
Luc
On Thu, 2009-04-09 at 06:06 +1200, Abhishek Reddy wrote:
On 4/8/09, Baishampayan Ghose <b.g...@ocricket.com> wrote: > > Excuse my ignorance, but which one is the first famous Clojure app? > That could be a reference to Luc Prefontaine's veterinarian hospital application. It got some press at InfoQ: http://www.infoq.com/news/2009/01/clojure_production
|
Luc Préfontaine Armageddon was yesterday, today we have a real problem... |
dysinger
Apr 8, 2009, 5:49:58 PM4/8/09
to Clojure
We have several TB of indexes w/ literally billions of documents and
will be using clojure w/ hadoop soon. I appreciate the hadoop mapr
job examples in clojure. This is great stuff.
will be using clojure w/ hadoop soon. I appreciate the hadoop mapr
job examples in clojure. This is great stuff.
Reply all
Reply to author
Forward
0 new messages