[ANN} Aleph 0.4.0 released, plus Manifold, Dirigiste, and a whole host of other libraries

[Zach Tellman]

Hey all,

In preparation for Clojure/West, I'm formally releasing the latest Aleph and the libraries that surround it.  Aleph 0.4.0 has been running in production at Factual for half a year now, and across a variety of services is handling at peak 600k HTTP requests/sec (spread across 15-20 machines).  

Since the landscape of Clojure HTTP servers is pretty crowded these days, it's worth taking some time to explain how Aleph differs.  To be clear, most Clojure deployments likely use Jetty, and should continue to do so.  However, Aleph has some unique properties:

* It uses the Netty library, which is a high-performance and very battle-tested network layer for the JVM

* It is the only HTTP server that has *ubiquitous* asynchronous streams wherever data can be received or sent (all other libraries can only represent streaming requests using InputStreams, or like http-kit don't support streaming HTTP requests at all)

* It is the only server that has a WebSocket implementation with any support for per-connection backpressure.  I won't make this post even longer by going into why this is important, but this will be a central theme of my talk at Clojure/West next week if you're interested in hearing more.

* It uses consistent abstractions to represent network connections over a variety of protocols, which makes it straightforward to use the same application logic for all of them.

Again, none of these points mean you should immediately drop whatever you're using and move over to Aleph instead.  However, I do feel it represents the only (current) good option for using core.async or a similar stream abstraction to represent network data, which is an idea a number of people seem to be playing with lately.  Some examples of this can be found at http://ideolalia.com/aleph/literate.html.

A full list of the libraries:

aleph - https://github.com/ztellman/aleph - uses the excellent Netty library to expose HTTP, TCP, and UDP using a consistent asynchronous stream representation.

manifold - https://github.com/ztellman/manifold - an unopinionated stream representation designed to cleanly interoperate with other stream representations (Clojure's seqs, core.async channels, Java's BlockingQueues, and others).  This is the base stream representation for all network sources and sinks in Aleph.

dirigiste -  https://github.com/ztellman/dirigiste - a pure-Java library that provides instrumented, dynamically sized thread and object pools.  This is used for thread pools in Aleph's HTTP server, and for connection pools in Aleph's HTTP client.

byte-streams -  https://github.com/ztellman/byte-streams - a means of translating any byte representation into another.  Want to turn a core.async channel that emits byte-arrays into an InputStream, or maybe the other way around?  Look no further.  The library's conversion mechanism is extensible, which is used in Aleph to make Netty's custom byte representations interoperable with more familiar representations.

byte-transforms -  https://github.com/ztellman/byte-transforms - a curated collection of byte compression, hashing, and encoding mechanisms, which can work on anything byte-streams can convert.

While all these libraries are used in concert to create Aleph, I've been very careful to make sure any of them can be used by themselves.  If anyone has questions about them, the best place to get my attention is the Aleph mailing list: https://groups.google.com/forum/#!forum/aleph-lib.

I will be mentioning some of these libraries at my upcoming Clojure/West talk (http://clojurewest.org/speakers#ztellman), but I've also set aside an Unsession for specifically discussing these libraries: https://github.com/clojurewest/clojurewest2015/wiki/Unsessions.  If you're interested, please add your name to the list.

Zach

[adrian...@mail.yu.edu]

You really hit the ball out of the park with Aleph 0.4.0's API. You have set the standard for simplicity and power in a Clojure API with this release as far as I'm concerned. Thank for your your contribution! 

[Dmitri]

I'd like to add Aleph to the Luminus template and I was wondering if there's an equivalent of dev mode available for other servers where it watches for changes in source and reloads them. I did a cursory look but didn't spot anything like a -dev option.

On Friday, April 17, 2015 at 5:06:30 PM UTC-4, Zach Tellman wrote:

[Zach Tellman]

Hey Dmitri,

I haven't used any sort of dev-mode before (I just update stuff in the REPL when necessary), but it seems like something like that belongs in middleware, not the server.  The server is just calling a function, it shouldn't care if something else is changing that function's behavior.

If other servers have this behavior embedded, maybe it can be extracted into a standalone library?

Zach

--
You received this message because you are subscribed to the Google

Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/KH8Js8URKwA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Dmitri]

Ah, you're right there is ring.middleware.reload middleware available and wrapping the handler with it in dev works perfectly.

[Matthias Lange]

In your examples, you put a let around the reads from timeouts.

(let [_ (a/<! (a/timeout 1000))] ... )

As far as i know, that is not neccessary. So your first example could be:

(defn delayed-hello-world-handler
  [req]
  (d/->deferred
    (a/go
      (a/<! (a/timeout 1000))
      (hello-world-handler req))))

Would that not work?

[Zach Tellman]

I suspect it would, I think I was just letting the mechanics of Manifold's let-flow macro color my judgment. Happy to accept any pull requests which make my core.async examples more idiomatic.