Do web apps need Clojure?

[Marcus Blankenship]

Hi Folks,

We’re a Python / Django shop, and some folks are getting excited about using Clojure for building web apps.  Certainly there are numerous open-source options to assist us (Pedastal, Ring, Compojure, Caribou, etc), but I think it begs a larger question: as a rule, do web applications need the power that Clojure brings to the table?

Other folks on my team are telling me that solutions built with Python / Django (or even RubyOnRails) fully satisfy the needs of 99% of the web apps we have built, and that Clojure offers nothing new to this problem space.  

So, here’s the question: How are you are actually using Clojure, and why did you choose to use it, particularly in the “web application” space?  

Thanks,

Marcus

marcus blankenship

\\\ Partner, Problem Solver, Linear Thinker

\\\ 541.805.2736 \ @justzeros \ skype:marcuscreo

[Justin Smith]

Hi. I'm part of the Caribou team, which started as an in-house tool (and continues to serve that purpose).

A few advantages of clojure in the webapp space, off the top of my head:

Clojure provides execution efficiency that Ruby or Python cannot match. This translates to lowered hosting costs.

The lein tool provides very clean and isolated dependency management, that makes setting up dev environments and deployment very straightforward.

Targeting the JVM means that each deploy can be a single jar or war file that you upload to the server.

The ring middleware system is a very clean way of including functionality in an app.

Working with immutable data structures and threadsafe bindings as a pervasive default does a lot for stability, and rules out many of the round about ways one of heisenbugs end up in the system.

[Marcus Blankenship]

Thanks, Justin.  These are great points!  I especially like the simplicity of deployment.  Do you folks use Heroku, AWS, or some other hosting service?

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Bastien]

Hi Marcus,

Marcus Blankenship <mar...@creoagency.com> writes:

> Thanks, Justin. These are great points! I especially like the
> simplicity of deployment. Do you folks use Heroku, AWS, or some
> other hosting service?

I started with Heroku but then switched to using a VM with Immutant.
Immutant is great, you should have a look!

--
Bastien

[Justin Smith]

Usually elastic beanstalk or ec2 (beanstalk is simpler and trivial to make scale, but less flexible so sometimes we need to fall back on using an ec2 instance). In either case we typically a clojure war into a tomcat container. As long as you remember to keep your permgen large enough it is pretty simple.

[Marcus Blankenship]

Ah, thanks! I’ll check it out!

[James Reeves]

On 13 November 2013 22:38, Marcus Blankenship <mar...@creoagency.com> wrote:

We’re a Python / Django shop, and some folks are getting excited about using Clojure for building web apps.  Certainly there are numerous open-source options to assist us (Pedastal, Ring, Compojure, Caribou, etc), but I think it begs a larger question: as a rule, do web applications need the power that Clojure brings to the table?

Surely that depends on the web application? The internal architecture between web applications varies enormously, perhaps more than any other class of system.

You mention your team works with Django, so presumably your applications tend to be backed by a single SQL database, and are form-based to some degree. Django has some excellent tools for generating web applications based around that structure.

However, I've run into a number of situations where that structure becomes a hindrance. For instance, maintaining an immutable history of events in SQL is a pain in the ass, which is a requirement for some systems, particularly those that record clinical data. Another problem with clinical data is that you can't discard invalid data, even if it fails something as basic as a type check. This subverts a lot of assumptions web frameworks (and databases) typically make.

There's also a lot of interesting research going on in Clojure right now, some of which affects web development. For example, core.async can be used on the client and server side to great effect, and support for Ring request and response data structures is, I believe, planned for core.typed.

- James

[Marcus Blankenship]

Got it, thanks!  I’ve always wanted to use Beanstalk, but figured it would have some limitations…

[Adrian Mouat]

On Wednesday, 13 November 2013 23:23:52 UTC, Justin Smith wrote:

Hi. I'm part of the Caribou team, which started as an in-house tool (and continues to serve that purpose).

A few advantages of clojure in the webapp space, off the top of my head:

Another nice advantage is being able to use the same language on the backend and frontend, if you are willing to embrace Clojurescript.

[Marcus Blankenship]

Hi James,

Good points, and I figured someone would say “Well, what kind of web app are you creating?”  Just like you, we’ve built a huge variety of web apps, and they each have different requirements.  Some of what we have done would have benefited from it, and others probably would have been a wash.

Thanks for chiming in!

Marcus

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Brian Craft]

On Wednesday, November 13, 2013 3:23:52 PM UTC-8, Justin Smith wrote:

Clojure provides execution efficiency that Ruby or Python cannot match. This translates to lowered hosting costs.

It should be noted that the go-to solution for performance in python is to drop to native code for bottlenecks, which is pretty easy, and which still out-performs the jvm. Whether clojure runs faster than python in a particular application is not a given.

[Brian Craft]

I also work in a python/django shop, and have been experimenting with clojure for about nine months. Before yesterday I would have told you that clojure web tooling does not come remotely close to the power of django. With a large amount of effort in piecing different libraries together, you might cover 30 or 40% of what django provides out of the box. I will have to reevaluate that with the announcement of Caribou, which looks good at first glance.

As a language, clojure is much more enjoyable. Python may seem clumsy and ineffective after writing clojure for awhile.

On Wednesday, November 13, 2013 2:38:49 PM UTC-8, Marcus Blankenship wrote:

[Marcus Blankenship]

Brian, that’s really interesting.  I think we’re seeing something similar, and are going to look at Pedestal and Caribou as options for a project we’re working on.  Are their others we should consider?

Best,

Marcus

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Bastien]

Marcus Blankenship <mar...@creoagency.com> writes:

> Brian, that’s really interesting. I think we’re seeing something
> similar, and are going to look at Pedestal and Caribou as options for
> a project we’re working on. Are their others we should consider?

Perhaps you should consider starting from scratch, in parallel.

Maybe that's because I'm a beginner in both the language and in web
development, but so far I've found it's the best way to understand the
choices behind framaworks. Otherwise I would confuse "web development"
with those choices, and I feel the richness of the Clojure ecosystem
is precisely to open your mind about "web development".

2 cents,

--
Bastien

[Islon Scherer]

For me it's about 1 thing: Data.

A web application is about taking data from the user, transform it and store it on the database and take data from the database, transform it and show it to the user.
Clojure is the best language I used to work with data, it just gives you a composable set of tools and then get out of your way, and there's always macros for the more complex use cases.
We have a web application that serves edn data to our clojurescript frontend, our webdevelopers created a new site for mobile in backbone.js that used json, I had just to create a function (ring middleware) that transformed my edn data to json based on the accept header.

My 2¢

[Ryan Spangler]

Marcus,

To answer your original question, the main reason we use Clojure is that we were originally using Ruby, and once our sites got to a certain level of traffic they fell over badly.  Rewriting the system in Clojure meant literally two orders of magnitude performance increase, which at the level of clients we are dealing with now is not just a huge win, it is critical.

I've also used Python extensively (and we still have some projects in it), but there is a lot of complexity in just dealing with the environments and dependencies (virtualenv is a standard practice now?)  Once people have lein installed, they can download one of our Clojure projects and we know everyone is using the same version of everything without any hassle.  It just works.  

Also, the benefit of being able to spin off a new thread to do some background task is not to be overlooked.  Python and Ruby are still fundamentally single threaded and each has its own flavors of what are basically hacks to get around that limitation.

As our applications begin to demand more and more asynchronous handling Clojure really is starting to shine.  Once you have core.async at your disposal, all of a sudden it is hard to imagine how you ever lived without it.  Seriously.  What a brilliant abstraction and solution for what was previously a complex architecture task.

So can you still use Django and Rails to build webapps?  Of course, these are mature and tested platforms for doing standard things.  But Clojure is the future.  There is always a choice to be made between going with the sanctioned solutions of today or blazing the trails of tomorrow.  I remember when Django and Rails were the platforms of the future and people thought they were risky or strange and untested.  People are always making a case for the safe established choice.  But somehow progress gets made anyway.

(I am biased of course.  But I made that decision years ago and never looked back.  I can't imagine going back to python or ruby now, but I can't speak for everyone.)

On Wednesday, November 13, 2013 2:38:49 PM UTC-8, Marcus Blankenship wrote:

[Sean Johnson]

I'm a "Rails guy" who now uses Clojure for all new web development, so don't take what I'm about to say as disagreeing with Justin. 

As someone with a foot firmly in both worlds, a couple of Justin's points aren't really marks in Clojure's favor, at least compared to Ruby/Rails (I can't say as much about Python/Django). I also have a few other benefits to bring up that Justin didn't:

Clojure provides execution efficiency that Ruby or Python cannot match. This translates to lowered hosting costs.

Agreed. This is a huge point. And not just in lowered hosting costs, but also potentially a better user experience (lower latency).

 

The lein tool provides very clean and isolated dependency management, that makes setting up dev environments and deployment very straightforward.

The pair of Bundler/rake does this very effectively for  Ruby.

Targeting the JVM means that each deploy can be a single jar or war file that you upload to the server.

True. Clojure offers potentially much simpler deployment schemes, but if this is important to you, there's JRuby.

 

The ring middleware system is a very clean way of including functionality in an app.

Rack is every bit as clean in the Ruby world.

 

Working with immutable data structures and threadsafe bindings as a pervasive default does a lot for stability, and rules out many of the round about ways one of heisenbugs end up in the system.

True, but this comes up less frequently than you might expect with the typical web app. With the stateless nature of HTTP, and the MVC (of a sort) convention of Rail/Django, typical web app code in these frameworks ends up being more short lived, imperative, and object based in nature rather than long running, object-oriented, and stateful.

Here are several additional points to consider:

Conciseness. Terseness. Or to put a more positive spin on it, expressiveness. Idiomatic Clojure is more concise than idiomatic Ruby or Python. I find that having 2x or less code to implement the same functionality to be very refreshing and empowering. Big Rails/Django apps suffer entropy much faster than comparable Clojure apps.

Full-stack with ClojureScript. The ability to code all in one language, and to share code between client and server, is quite nice compared to Ruby/JavaScript, Ruby/CoffeeScript. It's one small reason node.js has gained momentum so quickly.

Neither Rails nor Django has a robust, mature threading story. Both achieve concurrency primarily through many processes. This can eat through memory on your hosts quickly as the footprint for Rails and Django themselves is not small and is replicated in each process. This comes up as you end up serving web requests to multiple processes to concurrently, and end up with a many processes model for working background requests. This has gotten a bit better in the Ruby/Rails world as Rails is now thread safe and so can be run multithreaded in alternative Ruby implementations (other than MRI), and Sidekiq has come on the scene for background work, and it leverages Celluloid's actor implementation, borrowed from Erlang, to provide multi-threaded Rails workers. Django has celery but I think that's still process based like resque is in the Ruby world. Not positive, but I don't think Django is threadsafe though.

Framework vs. ecosystem of interoperable libraries. The monolithic framework (see Rails and Django) is not the way web development is done in Clojure. Instead imagine the Clojure world as a set of legos. You get to build whatever you want with your legos, selecting just the blocks you need, in the colors and sizes you want, and they all snap together [1]. The benefit to this approach is simplicity. There is a LOT to know about the Rails and Django frameworks, as they are big, mature things that have grown to solve all the common web development needs, and you start your development with ALL of that framework. Your starting point with Clojure on the other hand is much simpler. It will seem too simple... at first you'll be wondering... where is all the "stuff"? You start with maybe just ring and compojure (web middleware and routing) and a few lines of your code and your first couple of stories are already complete. You add libraries as you find you need them as you build out your app over time, but at all times your app is as simple as it can be, and only has the "stuff" it needs. It's therefore much easier to understand and less a "big ball of mud".

The magic of meta-programming (Ruby) vs. magic of macros (Clojure) vs. no magic (Python). This probably comes down to personal preference, and so is a potential benefit of any of them, depending on your preference, but it's a significant difference, so it's worth pointing out. By "magic" I just mean when the system seems to be more doing something much more powerful than you can readily see just by looking at the code. A simple few lines of code are doing all these wondrous things that aren't spelled out completely in the code (unless you peek under the covers and can understand the black arts that lie there). In Ruby/Rails this is the result of a lot of metaprogramming happening, based on conventions (that must be learned). In Clojure this happens through macros that implement powerful DSLs. In Python, it doesn't happen nearly as much. Code tends to be more transparent.

I hope that was helpful. Good luck convincing your team Marcus!

Cheers,

Sean

[1] a robust set of immutable data structures, sequences and maps, and conventions about how they are used in libraries are the uniformity of the lego blocks that makes them snap together easily. With Ruby and Python you've always had some of this with hashes and dicts and arrays, but they aren't used as consistently in libraries, and you have a lot of custom objects in libraries too, so you end up with a lot more library glue code.

[Marcus Blankenship]

Interesting you suggest that, as that's exactly what we decided to do. :-)

Thanks,
Marcus

Marcus Blankenship
541-805-2736

> On Nov 14, 2013, at 12:11 AM, Bastien <bastie...@gmail.com> wrote:
>
> Marcus Blankenship <mar...@creoagency.com> writes:
>

>> Brian, that’s really interesting. I think we’re seeing something
>> similar, and are going to look at Pedestal and Caribou as options for
>> a project we’re working on. Are their others we should consider?
>

[Marcus Blankenship]

Good point!  

Thanks,

Marcus

Marcus Blankenship

541-805-2736

[Brian Craft]

On Thursday, November 14, 2013 1:01:33 AM UTC-8, Ryan Spangler wrote:

I've also used Python extensively (and we still have some projects in it), but there is a lot of complexity in just dealing with the environments and dependencies (virtualenv is a standard practice now?)  Once people have lein installed, they can download one of our Clojure projects and we know everyone is using the same version of everything without any hassle.  It just works.  

I find virtualenv/pip much easier to use than lein. The search feature in lein is unusable, and having to edit a file to pull in dependencies is awkward. I'd love to have something as easy as pip for clojure.

[Brian Craft]

On Thursday, November 14, 2013 2:28:51 AM UTC-8, Sean Johnson wrote:

Framework vs. ecosystem of interoperable libraries. The monolithic framework (see Rails and Django) is not the way web development is done in Clojure. Instead imagine the Clojure world as a set of legos. You get to build whatever you want with your legos, selecting just the blocks you need, in the colors and sizes you want, and they all snap together [1]. The benefit to this approach is simplicity. There is a LOT to know about the Rails and Django frameworks, as they are big, mature things that have grown to solve all the common web development needs, and you start your development with ALL of that framework. Your starting point with Clojure on the other hand is much simpler. It will seem too simple... at first you'll be wondering... where is all the "stuff"? You start with maybe just ring and compojure (web middleware and routing) and a few lines of your code and your first couple of stories are already complete. You add libraries as you find you need them as you build out your app over time, but at all times your app is as simple as it can be, and only has the "stuff" it needs. It's therefore much easier to understand and less a "big ball of mud".

I don't believe the legos analogy is very accurate for clojure. Or, rather, it's more of a vision than a reality. I'm unaware of any libraries in clojure that you can piece together to give you the features of django-south, django admin, and the forms/validation/db layers, for example.  Today, clojure web libraries are more like an auto parts store: your chance of putting together a complete car from the inventory is slim indeed, and your chance of doing it in a timely fashion is exactly zero.  There are about a dozen migration libraries for clojure, for example, but none of them integrate very well with the other libraries because there isn't much in the way of a data modeling framework for the different components to leverage. Also, I don't think any of them provide the simplicity of south, where your data model is declared in one place (rather than in a sequence of sql commands that you have to interpret to deduce the final data model), and migrations are derived from changes in the data model.

The putting of things together is the hard part of software engineering. This makes a huge difference in development time.  Clean algorithms are easy in comparison. Rich talks about the importance of taking things apart, but you can't take things apart that never fit together in the first place. IIRC he actually suggested that one take things apart such that they fit together again. I agree with this, but I don't think it describes much of clojure web tooling today.

Also, the overwhelming majority of db and web work is boilerplate. That's what django provides: all the mindless boilerplate that is completely uninteresting to your problem domain, but necessary to launch a web site. With the current clojure web tools, there is no solution for most of that boilerplate, which can easily add an order of magnitude to your development time.

And again, I will have to reevaluate all of this in light of caribou, which appears to address much of it. Also, I'd be very happy to hear that I'm wrong, if anyone has example projects to show, or libraries I've missed.

[Manuel Paccagnella]

Indeed, Caribou seems to fit quite nicely in all this: the data model is first class, and gives you at good amount of ready-made wiring and boilerplate code. Same thing (to a maybe lesser degree) does Luminus. I agree that taking things apart is a win, working with simple things gives you a lot of power and flexibility. On the other hand there is a tension between use and reuse. However it doesn't mean that you can only work with simple components. I think that the approach taken by Luminus (and maybe Caribou, I've just started exploring it) is the best of both worlds: a template that assembles simple libraries with in a good default structure that you can use like a normal framework, but you still have the power to dive into the guts of the system and change it to suit your unique needs if you need to.

Probaby it comes down to a stratified design: simple components, assembled in a higher level template (still a component, not so reusable but much more usable) with a reasonable wiring and boilerplate code already in place but that doesn't stop you if you want to change it. It doesn't have the disadvantages of a framework, but neither requires you to "compos[e] art out of found objects" (quoting Pedestal documentation).

Cheers,
Manuel

[gaz jones]

There is no reason not to consider using a combination of both technologies. I have found Clojure is excellent for writing http services (for example sending/receiving JSON), but is a bit weak in comparison to say the combination of ruby and haml for building UIs quickly. 

--

[James Reeves]

On 14 November 2013 16:22, Brian Craft <craft...@gmail.com> wrote:

I don't believe the legos analogy is very accurate for clojure. Or, rather, it's more of a vision than a reality. I'm unaware of any libraries in clojure that you can piece together to give you the features of django-south, django admin, and the forms/validation/db layers, for example.  Today, clojure web libraries are more like an auto parts store: your chance of putting together a complete car from the inventory is slim indeed, and your chance of doing it in a timely fashion is exactly zero.

Except people are developing web applications in Clojure, so clearly this isn't an accurate statement.

I certainly wouldn't be using Clojure for the web if it took longer to develop in. It's my opinion that every non-trivial web application I've ever written in Rails would have been faster to write in Clojure.

Also, the overwhelming majority of db and web work is boilerplate. That's what django provides: all the mindless boilerplate that is completely uninteresting to your problem domain, but necessary to launch a web site.

 

This certainly hasn't been my experience.

Can you be certain this isn't a symptom of the tool you're using? Frameworks like Django and Rails are designed to build web applications in a very specific way, and this involves generating a lot of database and HTML boilerplate.

I can't say for sure whether it's because of Clojure, or because I'm working on different problems these days, but I don't tend to deal with the same issues I did when I used Rails. Nowadays I find myself building systems out of small, isolated components, which seem to eliminate a lot of issues Rails was designed to work around.

For instance, database migrations. This was something I thought about a lot a few years ago, until I gradually realised that I wasn't finding myself in any situations where I needed them.

- James

[Brian Craft]

On Thursday, November 14, 2013 9:17:32 AM UTC-8, James Reeves wrote:

On 14 November 2013 16:22, Brian Craft <craft...@gmail.com> wrote:

I don't believe the legos analogy is very accurate for clojure. Or, rather, it's more of a vision than a reality. I'm unaware of any libraries in clojure that you can piece together to give you the features of django-south, django admin, and the forms/validation/db layers, for example.  Today, clojure web libraries are more like an auto parts store: your chance of putting together a complete car from the inventory is slim indeed, and your chance of doing it in a timely fashion is exactly zero.

Except people are developing web applications in Clojure, so clearly this isn't an accurate statement.

I haven't seen a web application in clojure with the functionality of a django app, with the exception of polyglot applications that are using non-clojure frameworks to fill the gaps. I would very much like to see one.

Also, the overwhelming majority of db and web work is boilerplate. That's what django provides: all the mindless boilerplate that is completely uninteresting to your problem domain, but necessary to launch a web site.

 

This certainly hasn't been my experience.

Can you be certain this isn't a symptom of the tool you're using? Frameworks like Django and Rails are designed to build web applications in a very specific way, and this involves generating a lot of database and HTML boilerplate.

I can't say for sure whether it's because of Clojure, or because I'm working on different problems these days, but I don't tend to deal with the same issues I did when I used Rails. Nowadays I find myself building systems out of small, isolated components, which seem to eliminate a lot of issues Rails was designed to work around.

For instance, database migrations. This was something I thought about a lot a few years ago, until I gradually realised that I wasn't finding myself in any situations where I needed them.

I don't really understand this argument. What database tables would we not require if we used clojure? User passwords? Page content? User settings? Admin privileges? Of those tables, which ones would we not need admin interfaces for if we used clojure? I don't understand how choice of framework has any bearing on what data the app must store, what admin interfaces are required, what validation layers must be present, etc. You must validate all the user input. You must store all the data required by the app. You must have admin backends for all of the data used by the app. Using clojure doesn't change any of those requirements.

And when we have changes to the data model, for example when finding that a new feature requires that we track an additional field for each user, or finding that a new clinical data set requires a more complex model of clinical samples, how would clojure allow us to avoid migrating the data model in the database, or update the model in a reliable way across several installs without using migrations? Again, I don't understand how the framework has anything to do with the necessity of updating a data model when new features or new data demand it.

[vra...@gmail.com]

Attaching nREPL to debug a live application is pretty cool.

[J Irving]

* James Reeves <ja...@booleanknot.com> [2013-11-14 17:17 +0000]:

> For instance, database migrations. This was something I thought about a lot
> a few years ago, until I gradually realised that I wasn't finding myself in
> any situations where I needed them.

I agree with much of what you write James - I'm paid to write rails and
node.js code, and I'm finding that node is encouraging me to compose
small components and basically sidestep a lot of the issues that rails
is designed to address.

But migrations, or more particularly, schema version management, is
still something I need for databases which are schema based. How do you
deal with that?

Do you tend to use schemaless databases? Or maybe keep a native schema
dump under version control?

If you use a relational db, how do you make and track changes?

I have recently encountered a large project which includes several
distinct and separately deployed components, but which shares a single
data model. A lack of schema management in this case has seriously
impeded the project's ability to move forward quickly.

Maybe the answer in that case is that those components should own their
own data models, communicate through clearly defined interfaces, and
never share a database. Maybe that's the way to deal with this -
decompose a system until data model management is trivial for any given
component.

I'm interested to know what strategies people use for this, and what
tooling (if any) is useful.

cheers, J

[Brian Craft]

On Thursday, November 14, 2013 9:42:52 AM UTC-8, Jonathan Irving wrote:

I agree with much of what you write James - I'm paid to write rails and
node.js code, and I'm finding that node is encouraging me to compose
small components and basically sidestep a lot of the issues that rails
is designed to address.

Can you give a concrete example? 

[Stanislav Sedov]

On Nov 14, 2013, at 3:28 AM, Sean Johnson <bel...@acm.org> wrote:

> Agreed. This is a huge point. And not just in lowered hosting costs, but also potentially a better user experience (lower latency).

I am actually not sure how true it is. While the raw throughput of JVM applications
is indisputably much higher than those based on non JVM ruby virtual machines,
in my experience the response latency in JVM is much more unpredictable and
have a much higher worst case. I believe this is due to the fact that ruby HTTP
servers tend to spawn multiple processes to handle requests which lead to a
per-process garbage collection, and thus to shorter pauses. At least that was true
for all the applications I built around Sinatra. I never used rails, so cannot
comment on that.

Regarding multithreading, modern ruby implementations like rubinius are
completely multithreaded and it's up to the application code to take advantage
of it. However, the lack of proper multiprocessing primitives in the language
and lack of multithreading in the ruby interpreters for a long time lead to
libraries and frameworks designed around the asynchronous abstraction
and are generally not thread-safe. So in practice, it's very hard to take
advantage of that support.

For me, the biggest advantage of using ruby vs clojure for web development
comes from the development standpoint. While I love clojure as a language,
which is arguably a much more powerful and better designed language than
ruby is, it suffers from the typical problems any JVM platform has:
1) long startup time. While there are workarounds to this problem like
using a persistent vm, it's still a hassle compared to a non-jvm platform
where one does not even have to think about it;
2) horrible dependencies management. In jvm world you have to use
maven or ant style jar artifacts, which are tarballs with compiled
java classes inside. Those have no versions in them, so when
something breaks you cannot easily debug the issue just by looking
at the corresponding line of code like one does in ruby: I routinely
have to decompile class file to look at the code to try to figure out
what broke (I even set up mc to invoke jad(1) automatically when
I view a class file). The lack of versions in classfiles lead to another
problem: you are never sure that the jars hosted on the public
artifact sites correspond to the code versions they advertise, I
found it to be not true in a lot of cases and spent hours debugging
issues which were not present in the actual source code of the
library I used. There is also no verification of who uploads those
jar files to the artifactory and you cannot AFAIK provide maven or
ant with the checksum of the artifact you want, so those are as safe
to use as running a random code downloaded from the internet.
Unfortunately, it seems that the only way to do development on
the jvm platform at the time is to set up your own artifactory and
set up an automated build for each dependency version you
are using. :-(. Which slows develoment a lot.

Another issue I often face is that class files are not namespaced
via symbol versioning like native libraries are. That is the entire
JVM application has to use a specific dependency version for
all it's code. What happens if some library you use uses a different
dependency version than you need? You end up maintaining a
fork.

Some of this is true for ruby libraries as well, but late binding,
relaxed version specifications and generally better API stability
in ruby world makes this issue rare. And if it happens, it's much
easier to submit a fix upstream and rely on it, at least in my
experience.

3) General lack of decent tools. The only jvm debugger, jdb,
while nice, does not allow you to attach to an already running
process like gdb would. Profiling tools are pretty much GUI
only so won't allow you to collect profiled data in an automated
way. The same goes to JMX counters: you can only collect
them by using a gui application, or by using java API; there are
no command line tools provided in standard distribution which
complicates ad-hoc monitoring and collection. There are no
easy to use tools or APIs to work with bytecodes, jars, manifests
and so on either.

I guess the bottom line is that jvm does not play well with the
outside world and you pretty much have to commit to JVM
platform if you want to use it for development. Any interoperation
with other platforms and OS services was a big hassle for me.
I hope that clojurejs on nodejs will make the life much easier
in that regard.

As as a side note, I used erlang/webmachine for one of my
recent projects and found the experience quite pleasant. While
the language is not as powerful as clojure or any other kind of
lisp is and metaprogramming capabilities are quite limited,
it is very easy to spawn processes to do background processing
and communicate with them, and error and failure handling is
awesome. It also plays really well with unix, and tools are
great and well documented. I wish someone would do clojure
for BEAM. :-) (although there is LFE, but it does not seem to
be popular and does not provide all those nice things clojure
does).

--
ST4096-RIPE

[Waldemar Schwan]

I just want to say that this is one of the most interesting discussions I followed on this mailing list.

Thanks to all participants.

[J Irving]

* Brian Craft <craft...@gmail.com> [2013-11-14 10:24 -0800]:

Tricky, because my work doesn't belong to me.

I have a component which collects weather forecasts from a public API
for all of the US. This is isolated from other parts of the app. It
acquires the data in one form, transcodes and strips it down, and puts
it into a mongodb.

Another component runs async map reduce ops on the data, creates
aggregated and otherwise reduced data.

Finally, there is a service endpoint which serves the product to another
component, which (irrelevantly) is a rails app.

Managing the data model for these components is trivial, because they
only touch one kind of data, and they are each very small. What the
rails app sees is a service endpoint with a well defined API. I use no
ODM or ORM, just mongodb queries.

OTOH as I mentioned further down my first post, I don't see how to avoid
schema (version/migration) management in apps which have a richer data
model. Your user management example fits that description. Hence my
questions.

BTW I realize that I'm talking about node not clojure here, but I see a
lot of parallels between the approaches taken in clojure and node
webapps. They're more similar to me than either of them are to rails or
django, for example.

Nonetheless, sorry if this feels off topic.

cheers, J

[Justin Smith]

On Thursday, November 14, 2013 10:23:40 AM UTC-8, Stanislav Sedov wrote:

3) General lack of decent tools.  The only jvm debugger, jdb,
    while nice, does not allow you to attach to an already running
    process like gdb would.  Profiling tools are pretty much GUI
    only so won't allow you to collect profiled data in an automated
    way.

Both of these statements are false. 

[James Reeves]

SQL databases complect a number of different ideas about data, and this complexity results in certain problems.

Database migrations, for instance, are a symptom of a system where data storage is tightly coupled to grouping and validation of data. In systems which separate data storage from data indexing and viewing, migrations cease to be something you need.

Let's assume that we're measuring the heart rate of subjects for a clinical trial. We might have a data structure that looks like this:

{:timestamp  #inst "..."

 :subject/id #uuid "..."

 :subject/heart-rate 80}

Let's further assume that we're only going to deal with storage and ordering for this data. So we're going to maintain an atomically updated log of incoming data from patients.

We now want to index this data in some fashion, to display the results in a table or a chart. This can be done by taking the data from the log, and copying it to a read-only database that supports querying and aggregation. The data could be copied over in bulk periodically, or pushed in real time.

After some time, the requirements for our system change, and we now have to record heart-rate at rest and during exercise. Because we don't want to complect the data by prematurely grouping it, we instead introduce two new types of data structure:

{:timestamp  #inst "..."

 :subject/id #uuid "..."

 :subject/hr-rest 80}

{:timestamp  #inst "..."

 :subject/id #uuid "..."

 :subject/hr-max 180}

We then update our indexing algorithm, so that it now indexes two different heart-rates. We might also choose to make :subject/heart-rate equivalent to :subject/hr-rest. Instead of migrating this data, we can just regenerate the database from scratch from the raw data.

Essentially we're doing away with a data model altogether, and replacing it with a log of isolated, immutable data entries combined with a set of disposable, read-only views.

Django's admin interface is nice, but it's still tied to the idea of having a monolithic, complex and mutable database. The benefits of working with isolated, simple, immutable data aren't worth giving up for that.

- James

[Brian Craft]

On Thursday, November 14, 2013 10:50:48 AM UTC-8, Jonathan Irving wrote:

* Brian Craft <craft...@gmail.com> [2013-11-14 10:24 -0800]:
> On Thursday, November 14, 2013 9:42:52 AM UTC-8, Jonathan Irving wrote:
> >
> > I agree with much of what you write James - I'm paid to write rails and
> > node.js code, and I'm finding that node is encouraging me to compose
> > small components and basically sidestep a lot of the issues that rails
> > is designed to address.
> >
>
> Can you give a concrete example?

Tricky, because my work doesn't belong to me.

I have a component which collects weather forecasts from a public API
for all of the US. This is isolated from other parts of the app. It
acquires the data in one form, transcodes and strips it down, and puts
it into a mongodb.

Another component runs async map reduce ops on the data, creates
aggregated and otherwise reduced data.

Finally, there is a service endpoint which serves the product to another
component, which (irrelevantly) is a rails app.

Managing the data model for these components is trivial, because they
only touch one kind of data, and they are each very small. What the
rails app sees is a service endpoint with a well defined API. I use no
ODM or ORM, just mongodb queries.

Doesn't this reflect more on your problem domain than on your choice of tools? The first two components are roughly ETL, and the last is data API? How would this have been different in rails? I've done this in django, and it's the same as you describe here.

But more to the point, the pieces you've mentioned can't be composed to build a web app with a UI, and you can't decompose a web app with a UI into just the service layer bits. You also need the UI bits: the forms layer, the validation layer, the admin, etc., etc. To be maintainable, those layers should be common for different components in the application: it does not scale to require devs to learn multiple template languages, multiple migration tools, multiple admin interfaces, etc.

[J Irving]

On Thu, Nov 14, 2013 at 2:53 PM, Brian Craft <craft...@gmail.com> wrote:

On Thursday, November 14, 2013 10:50:48 AM UTC-8, Jonathan Irving wrote:

* Brian Craft <craft...@gmail.com> [2013-11-14 10:24 -0800]:

> On Thursday, November 14, 2013 9:42:52 AM UTC-8, Jonathan Irving wrote:
> >
> > I agree with much of what you write James - I'm paid to write rails and
> > node.js code, and I'm finding that node is encouraging me to compose
> > small components and basically sidestep a lot of the issues that rails
> > is designed to address.
> >
>
> Can you give a concrete example?

Tricky, because my work doesn't belong to me.

I have a component which collects weather forecasts from a public API
for all of the US. This is isolated from other parts of the app. It
acquires the data in one form, transcodes and strips it down, and puts
it into a mongodb.

Another component runs async map reduce ops on the data, creates
aggregated and otherwise reduced data.

Finally, there is a service endpoint which serves the product to another
component, which (irrelevantly) is a rails app.

Managing the data model for these components is trivial, because they
only touch one kind of data, and they are each very small. What the
rails app sees is a service endpoint with a well defined API. I use no
ODM or ORM, just mongodb queries.

Doesn't this reflect more on your problem domain than on your choice of tools? The first two components are roughly ETL, and the last is data API? How would this have been different in rails? I've done this in django, and it's the same as you describe here.

But more to the point, the pieces you've mentioned can't be composed to build a web app with a UI,

True say. There is another app that would be a better example, but it is not mine - a node API service, a node web app serving an AngularJS client, and an agent which performs work tasks for the app. The latter can be exec'd by the web app, or from the command line. It is a better example, but I don't know it as well because it isn't my code. I can also imagine composing an app as described by tjholowaychuk here: http://vimeo.com/56166857 - where each component has its own middleware stack based on its specific requirements; I can imagine some being API endpoints, others being bits of template or html/js client code, etc.

But I think this is kind of off topic.

and you can't decompose a web app with a UI into just the service layer bits. You also need the UI bits: the forms layer, the validation layer, the admin, etc., etc. To be maintainable, those layers should be common for different components in the application: it does not scale to require devs to learn multiple template languages, multiple migration tools, multiple admin interfaces, etc.

I'm hoping to be convinced otherwise - though I agree that you need those different bits, I disagree with the rest of that paragraph. I certainly don't have a problem with multiple languages and multiple sets of tools. I suspect you have different criteria for "maintainable" than I do - for example, you seem to be saying that scaling out development to a group of people who can't handle multiple languages, tools, admin interfaces, is a requirement for you, which it isn't for me. I'm not really that interested in admin interfaces at all for example.

So I'll respectfully disagree and look out for some suggestions from people who have made this work. Also, remember that I already do use rails, and am quite attached to its schema management tools - but I want to discover another way, not be convinced its impossible.

cheers, J

[Bayard Randel]

In a world where increasingly we're pushing more logic, rendering to the client, what is there in the python ecosystem to support this? I'm a professional django developer and am a bit dumbstruck by the lack of interest in supporting python on the client. Clojure and ClojureScript are far more forward thinking in this regard. To my mind, any platform that is going to be successful for web development in the future needs to support for the 'JS VM'.

[Marcus Blankenship]

Me too!  Thanks to everyone who’s contributed, it’s been *very* helpful!

[Marcus Blankenship]

Thanks Sean, wonderful input!  

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Brian Craft]

Marcus -- I hope you will post updates to the list with your experiences. It would be very interesting.

This thread has drifted a bit from (roughly) "What can you do with clojure web tooling?" toward "What can you imagine some day doing with the clojure web tooling of the future?", which are both interesting questions, but have somewhat different audiences. And the answers inform each other.

[Jarrod Swart]

I think the more interesting question is: do web developers need Clojure?  If performance were the sole concern then everyone would still code web apps in C or assembler.  There is great power in abstraction.  I like this talk by Uncle Bob: http://skillsmatter.com/podcast/agile-testing/bobs-last-language  If you don't have time to watch it I will summarize: every time we get a more powerful programming environment (for developers) we reduce our programming options (in the language).  

• C     --> Java     == no pointers
• Java --> Clojure == no immutability, etc.
  

Every major advancement for developers has largely involved reducing our options and choices.  Take away register manipulation, take away pointers, take away variables, and so on.  I highly recommend the talk by Uncle Bob.

So I highly encourage you to investigate the benefits that Clojure provide to the developer, over the benefits provided to the machine.

Good luck in your search!

Jarrod

On Wednesday, November 13, 2013 5:38:49 PM UTC-5, Marcus Blankenship wrote:

Hi Folks,

We’re a Python / Django shop, and some folks are getting excited about using Clojure for building web apps.  Certainly there are numerous open-source options to assist us (Pedastal, Ring, Compojure, Caribou, etc), but I think it begs a larger question: as a rule, do web applications need the power that Clojure brings to the table?

Other folks on my team are telling me that solutions built with Python / Django (or even RubyOnRails) fully satisfy the needs of 99% of the web apps we have built, and that Clojure offers nothing new to this problem space.  

So, here’s the question: How are you are actually using Clojure, and why did you choose to use it, particularly in the “web application” space?  

Thanks,

Marcus

[Marcus Blankenship]

Thanks, Jerrod.  Great way to frame the question!

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Marcus Blankenship]

Brian, I certainly will.  I’ll type up something later this week as we progress.  The current pain point is Korma, and generally learning clojure.

[Brian Craft]

Re: korma, and sql dsls, I've been moving between korma, honeysql, and raw sql, without being satisfied with any of them. Desirable traits of the db layer in my problem domain are 1) eliminating boilerplate (e.g. setting up default keys, and indexes, and performing standard joins across relations), 2) isolating view layers from data access layers (so the view doesn't need to know if a subselect or a join is required to span some relation, for example), 3) ability to progressively optimize by dropping back to sql when required, 4) ability to safely expose a general purpose query API over the data.

korma eliminates a very, very small part of the boilerplate. It's almost not worth the effort. Falling back to raw sql smoothly is difficult in korma, and I've had to drop it entirely in places where I need performance. Honeysql eliminates no boilerplate, but representing queries with data structures does make it easy to expose a sql-like query API with db firewalling (by matching on the incoming structure). Korma appears to also represent queries as data structures, but it's not part of the documented API. You have to reverse-engineer it, and I expect it's subject to change.

[Marcus Blankenship]

Do you have a link to the UncleBob talk, Jarrod?

On Nov 16, 2013, at 4:34 PM, Jarrod Swart <jcs...@gmail.com> wrote:

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Marcus Blankenship]

Brian,

Yeah, and I realize I’m going to take darts for this, but coming from Django’s ORM / Rails ActiveRecord makes Korma and these other tools feel like stone-age tools.  I’d rather do it all in SQL than fight something to get out of my way, or reveal it’s magic.

I know I’m probably not thinking about the problem right, but here’s an example of something that I still can’t get to work in Korma.  Grrrr….

lein-repl commands

(use [`advent2.models.db] :reload-all)

(get-unlocked-videos-for-campaign 1)

models/db.clj function

(defn format-todays-date []

  (let [date (java.util.Date.)]

    (prn date)

    (str \' (.format (java.text.SimpleDateFormat. "yyyy-MM-dd") date) \' )))

(defn get-unlocked-videos-for-campaign [campaign]

  (let [c_id (:id campaign)]

    (select videos (where {:unlock_date [<= (format-todays-date)] :campaign_id c_id}))))

output

user=> (get-unlocked-videos-for-campaign 1)

#inst "2013-11-18T19:06:09.595-00:00"

Failure to execute query with SQL:

SELECT "videos".* FROM "videos" WHERE ("videos"."unlock_date" <= ? AND "videos"."campaign_id" IS NULL)  ::  ['2013-11-18']

PSQLException:

 Message: ERROR: operator does not exist: date <= character varying

  Hint: No operator matches the given name and argument type(s). You might need to add explicit type casts.

  Position: 63

 SQLState: 42883

 Error Code: 0

PSQLException ERROR: operator does not exist: date <= character varying

  Hint: No operator matches the given name and argument type(s). You might need to add explicit type casts.

  Position: 63  org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse (QueryExecutorImpl.java:2102)

[Ryan Spangler]

I just ran this in the Caribou repl:

    (caribou.model/gather :field {:where {:created-at {:<= (java.util.Date.)} :model-id 1}})

And got a bunch of results.  No string coercion necessary!  (also, dashes not underscores!)

We looked at Korma and found it lacking (similar problem as Compojure: macros).  For that reason we built an entire query engine that works on data directly, so that you can build up queries programmatically just like you would any other data structure (!)

Check out the section on retrieving content here:  http://caribou.github.io/caribou/docs/content.html

You can do a bunch of cool stuff (logic operations, "in" queries, conditions that span associations, selecting certain fields, seamless queries over join tables etc).  

Let me know if there is something you expect out of a query engine that Caribou doesn't do, we will add it (if it makes sense!)

All this stuff is in caribou-core (which you can use independently of the rest of Caribou): https://github.com/caribou/caribou-core

Maybe I should call it something else to emphasize it is a standalone library... (and replacement for korma)

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/ZxUUBlYf1ck/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

[Marcus Blankenship]

Ryan,

Awesome!  We’re playing with Caribou now, and are loving it.  At first we thought “Oh, this is going to be like Drupal, with all it’s pseudo-tables and crap.  But it’s not, and we’re having fun with it!

I’ll try the query engine now.  Thanks!

[Ryan Spangler]

Marcus,

Good to hear!  I am have never used Drupal, what do you mean by pseudo-table?  

Also, looking into this I think the one caveat to using Caribou for your query engine is that it requires that you have a table for both "model" and "field" that contain the descriptions of the model you are querying.  If you create the model using Caribou in the first place this will exist, but it probably won't if you are trying to use it on a non-caribou db.

This could be mitigated by providing the model descriptions in some kind of configuration step... that wouldn't be too hard.  I will ponder on how best to support that.

[Marcus Blankenship]

Ryan,

FYI - The date comparison syntax appears to work fine under h2, but throws a exception on PSQL.  

Here’s my function

(defn calendar

  [request]

  (controller/render

    (assoc request

      :currentdate (java.util.Date.)

      :videos (model/gather :video {:where {:unlock-date {:<= (java.util.Date.)} :campaign_id 1}}))))

Here’ the error message:

PSQLException Can't infer the SQL type to use for an instance of org.joda.time.DateTime. Use setObject() with an explicit Types value to specify the type to use.  org.postgresql.jdbc2.AbstractJdbc2Statement.setObject (AbstractJdbc2Statement.java:1801)

Any ideas?

Thanks!

Marcus

On Nov 18, 2013, at 12:31 PM, Ryan Spangler <ryan.s...@gmail.com> wrote:

[Ben Wolfson]

try (java.sql.Timestamp (.getDate (java.util.Date.)))

Ben Wolfson
"Human kind has used its intelligence to vary the flavour of drinks, which may be sweet, aromatic, fermented or spirit-based. ... Family and social life also offer numerous other occasions to consume drinks for pleasure." [Larousse, "Drink" entry]

[Marcus Blankenship]

Hmm… no luck.  Did I do it wrong?

New Code

(defn calendar

  [request]

  (controller/render

    (let [campaign (get-current-campaign)]

      (assoc request

        :videos (model/gather :video {:where {:unlock-date {:<= (java.sql.Timestamp (.getDate (java.util.Date.)))} :campaign-id campaign}})))))

New Error

ERROR  java.lang.ClassCastException: java.lang.Class cannot be cast to clojure.lang.IFn

[Ben Wolfson]

oh whoops, mistranscription on my part; there shoudl be a trailing dot at the end of Timestamp (so "Timestamp.").

[Marcus Blankenship]

Hmm.. that does’t look right either…

advent3.core=> (java.sql.Timestamp. (.getDate (java.util.Date.)))

#inst "1970-01-01T00:00:00.018000000-00:00”

Seems not to return today, but the epoch...

[Ben Wolfson]

ahhhhh .getTime not .getDate, sorry.

[Ryan Spangler]

Just saw all this.  Ben, thank you!  It is indeed .getTime, so the whole thing looks like:

    (java.sql.Timestamp. (.getTime (java.util.Date.)))

It is easy enough to write a conversion fn for this:

    (defn timestamp [date] (java.sql.Timestamp (.getTime date)))

But really we should be doing this in caribou.field.timestamp.  Thanks for the catch!  I'll fix this now.

Until then Marcus, Ben's approach should work.  Let me know.

[Ryan Spangler]

As an aside, though it does not obviate the need to understand how to navigate the many faces of time in Java, this is a good library for working with dates and times in Clojure:  https://github.com/clj-time/clj-time

[Marcus Blankenship]

Boom, works like a charm!

Thanks Ben & Ryan!

P.S. I’m *very* impressed with Caribou so far.  Nice work!

[Jarrod Swart]

The talk is linked in my initial post, and again here: http://skillsmatter.com/podcast/agile-testing/bobs-last-language its very interesting!

[Marcus Blankenship]

Thanks!

[Marcus Blankenship]

Agreed, Ryan.  I’m going to have to get me a Java book…

[Colin Yates]

this.

The ease of which data can be mangled and transformed was the primary reason I chose Clojure over Java, Groovy and Scala.

That, and the fact the language is just so darn expressive.

On Thursday, November 14, 2013 8:56:59 AM UTC, Islon Scherer wrote:

For me it's about 1 thing: Data.

A web application is about taking data from the user, transform it and store it on the database and take data from the database, transform it and show it to the user.
Clojure is the best language I used to work with data, it just gives you a composable set of tools and then get out of your way, and there's always macros for the more complex use cases.
We have a web application that serves edn data to our clojurescript frontend, our webdevelopers created a new site for mobile in backbone.js that used json, I had just to create a function (ring middleware) that transformed my edn data to json based on the accept header.

My 2¢

On Thursday, November 14, 2013 9:11:04 AM UTC+1, Bastien Guerry wrote:

Marcus Blankenship <mar...@creoagency.com> writes:

> Brian, that’s really interesting.  I think we’re seeing something
> similar, and are going to look at Pedestal and Caribou as options for
> a project we’re working on.  Are their others we should consider?

Perhaps you should consider starting from scratch, in parallel.

Maybe that's because I'm a beginner in both the language and in web
development, but so far I've found it's the best way to understand the
choices behind framaworks.  Otherwise I would confuse "web development"
with those choices, and I feel the richness of the Clojure ecosystem
is precisely to open your mind about "web development".

2 cents,

--
 Bastien