Haskell for devops

340 views
Skip to first unread message

Michael Snoyman

unread,
Apr 8, 2015, 1:19:49 AM4/8/15
to commerci...@googlegroups.com
Both for our own services and for our customers, we end up doing a lot of work in the devops realm at FP Complete. And since we're a Haskell shop, when custom tools need to be written, we write them in Haskell. We've written enough of these kinds of things that we feel like the advantages Haskell offers here are real, tangible, and demonstrable.

My question to the rest of the group is: do you have similar experience? I'm talking about cases of writing code in Haskell to ease devops, or taking prebuilt Haskell solutions. We're planning on starting to express some of these benefits via blog posts and new projects, and it would be very useful to talk with other companies with similar experiences. Even better would be to coordinate efforts on this messaging.

To make things concrete, here are some examples of code we've recently put together to solve specific problems:

* wai-middleware-crowd[1]: a WAI middleware to require that users are authenticated through an Atlassian Crowd server, and a standalone executable providing file serving and HTTP reverse proxying capabilities
* bouncy[2], a no-configuration web server which redirects http requests to https. Our devops team found it easier to write up this application from scratch than to manage Nginx over time (taking into account the additional domain name that this would require in the config in the future)
* wai-middleware-consul[3]: distribute webhooks across a cluster of web hosts by using the Consul config management system

The advantages I'm seeing for Haskell in this space are:

* Compiled executables do not require runtime dependencies like Python or Ruby. (It's true that we have some deps like the GMP library, but that's relatively light and easy to provide.)
* High performance: Warp is comparable to Nginx, so you're not taking a performance hit by writing a new tool, and versus other possible languages (like Ruby and Python), the performance is significantly better
* Lots of libraries are readily available, and can be easily composed to create more complicated solutions (like how wai-middleware-crowd combines Warp, authenticate, and clientsession)
* By writing simple, single-service tools instead of using more powerful systems, we can simplify configuration and lower the surface area for attack
* Multithreaded runtime provides for much easier concurrency than many other languages, which is often a requirement for devops tools

There are of course the standard Haskell benefits- like lower bug rates due to strong typing- but I'm just focusing on the specific benefits for the devops realm. Another interesting question for you all is: did you consider languages besides Haskell for this kind of work and, if so, why did you choose Haskell? Concrete comparisons to common languages in this space (Python, Ruby, Go) would be very helpful.

Arnaud Bailly | Capital Match

unread,
Apr 8, 2015, 1:50:17 AM4/8/15
to Michael Snoyman, commerci...@googlegroups.com, Willem van den Ende
Hi Michael,

Indeed we (Willem and I) at Capital Match are also going down that route, and for the same reasons you are mentioning: no runtime dependencies, fewer bug rates, encapsulated tools,... 

We are still using duct tape (aka. shell scripts) but slowly moving a lot of things in compiled Haskell:
 - configuration management is already handled in Haskell by propellor (https://propellor.branchable.com/), with lot of micro-languages for configuring things like firewall, stunnel, docker containers...
 - interaction with cloud providers (digitalocean and AWS) still handled by script shells but we plan to move asap to haskell code (only requires some time to convert JSON formats to proper types and do queries in haskell using e.g. wreq)
 - we use bake (https://github.com/ndmitchell/bake) for continuous integration, automatically deploying docker containers to dockerhub
 - we are using hs-webdriver a lot for end-to-end-testing and of course other standard testing tools in Haskell

I do not have experience with Ruby or Python so cannot compare, but at least when it comes to shell script, using Haskell is a huge improvement. I think Willem would have more things to talk about in this respect.

I would be very interested indeed in sharing experience, mutualising effort and promoting Haskell in the DevOps space. Actually, I already started speaking about what we are doing :-) https://speakerdeck.com/abailly/haskell-all-the-way-down

Best regards,

--
You received this message because you are subscribed to the Google Groups "Commercial Haskell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to commercialhask...@googlegroups.com.
To post to this group, send email to commerci...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/commercialhaskell/CAKA2JgLb%2Bp_CKfCPO6rzkfn1ipPMOi72WVJVxn2_xZM%2BQBq_1g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.



--
Arnaud Bailly

CTO | Capital Match

CapitalMatch

71 Ayer Rajah Crescent | #06-16 | Singapore 139951

(FR)  +33 617 121 978 / (SG) +65 8408 7973arn...@capital-match.com | www.capital-match.com

Disclaimer:

Capital Match Platform Pte. Ltd. (the "Company") registered in Singapore (Co. Reg. No. 201501788H), a subsidiary of Capital Match Holdings Pte. Ltd. (Co. Reg. No. 201418682W), provides services that involve arranging for multiple parties to enter into loan and invoice discounting agreements. The Company does not provide any form of investment advice or recommendations regarding any listings on its platform. In providing its services, the Company's role is limited to an administrative function and the Company does not and will not assume any advisory, fiduciary or other duties to clients of its services.

Neil Mitchell

unread,
Apr 8, 2015, 1:52:15 AM4/8/15
to Michael Snoyman, commerci...@googlegroups.com
Hi Michael,

I think of DevOps as something slightly different to you. I think of
it more as things like build systems, continuous integration systems,
installer generators etc. In that sense of the word, I gave an entire
talk about DevOps in Haskell covering Shake, Bake and NSIS:
http://www.codemesh.io/codemesh2014/neil-mitchell

I think before discussing "DevOps" we should come up with what it
means - there may be two closely related but entirely distinct areas
here (and of course, Haskell excels at both :) ).

Thanks, Neil

Arnaud Bailly | Capital Match

unread,
Apr 8, 2015, 1:54:27 AM4/8/15
to Neil Mitchell, Michael Snoyman, commerci...@googlegroups.com
Hi Neil,
The essence of DevOps as I think Patrick Debois has explained it for years is exactly to be sure there is NO separation between the two areas you are mentioning, viewing the Development to Operations as a continuum. So the tools you are mentioning are definitely part of that space, just like configuration management tools or distributed orchestration tools.

My 50 cts



For more options, visit https://groups.google.com/d/optout.

Michael Snoyman

unread,
Apr 8, 2015, 3:26:22 AM4/8/15
to Arnaud Bailly | Capital Match, Neil Mitchell, commerci...@googlegroups.com
Arnaud: This is really awesome feedback, your experience seems to align closely with ours. We haven't been quite as aggressive as you it seems: we used Propellor for a test project, but haven't used it in production, and Neil's warnings about Bake made us decide to wait a few more months before trying it (though our devops team really is itching to play with it already).

Neil: I'm glad you're raising that basic question of "what is devops." I think we treat it here the same way Arnaud does: spanning everything from the end of software development till the operations team (i.e., 24/7 sysadmin coverage) takes over. That includes: CI, creating runtime clusters, configuration management, etc. Perhaps it is a broad usage of the term, but like you say: since Haskell excels at all of it, why not? ;)

That said, we could try to focus on a specific aspect of devops when talking to the outside world if there's a good reason to. My basic thought was to use the mantra of "Haskell for devops" and fill in the specific details in each message, e.g.:

* Haskell works really well for creating simple services
* Haskell is a good replacement for crufty bash scripts in automation
* ...

They're clearly very different roles, but the point is that the receiving audience will typically be touching all of those domains, and therefore the messages will all resonate with them. That said, this is not a fully formed thought, which is exactly why I'm asking for feedback.

Alfredo Di Napoli

unread,
Apr 8, 2015, 3:52:05 AM4/8/15
to commerci...@googlegroups.com, mic...@snoyman.com
Good morning folks,

here at Iris Connect we use a mixture of Haskell and more "classic" COTS solutions.

When I say DevOps here I will embrace a broader definition, which possibly includes both Micheal's and Neil's ones.

- For continental routing (i.e. given a universal link, redirect to a geographically closer subdomain, e.g. us.irisconnect.com, ect) we developed a small webapp (I might imagine
similar in spirit to bouncy) using MaxMind's GeoIP database.
- For CI we use Jenkins (when we started we wanted something to start really quickly, and bake wasn't around by then)
- For deployments we use two solutions on two different projects:
  - For the main RESTful server my colleague Chris Dornan built a separate CLI tool (in Haskell) to take care of the management of the deployment keys (using a crypto sublayer)
    and ultimately leveraging RPMs to deploy on RedHat
  - For the project I'm the lead on, I confess I wasn't bold enough to go for Propellor: I settle on the status quo with Ansible (driven by a CLI tool I wrote in Haskell) and
    perhaps I regret it a bit: Ansible is great because it's batteries included (i.e. supports lots of AWS services) but it's Python + yaml, which means making changes is buggy
    and error prones, with the results to be assessed only once you deploy (or fail trying!).
    I'm glad to see folks at Capital Match using Propellor! If you guys could blog about it would be ace!
- For monitoring (i.e. check when a url is down) we built a Snap web app which can notify us on HipChat and email, and plus serves as a UI for the RESTful server I have mentioned.
  It's inspired, in spirit, to Better's "pinger" tool - as presented at the Haskell Exchange 2013.

I'm probably forgetting something, but on top of my mind it's pretty much it!

Alfredo

Blake Rain

unread,
Apr 8, 2015, 4:32:30 AM4/8/15
to Michael Snoyman, commerci...@googlegroups.com
Hi Michael,

Haskell really is a great tool. I don't really know what DevOps is, but we use Haskell for a lot of our "glue" and a number of simple services:

* Redirecting traffic to various places: I love the name 'bouncy' :) I've found it easier to just use Keter and/or some small WAI/Yesod applications rather than nginx.
* Various micro-services from PDF rendering and thumbnail generation to various forms of data wrangling. I'm sure that Python or something could be used for this, but Haskell is just so much less hassle.
* Some scheduled processes that were easier to write externally than shoe-horn into existing (and often broken) systems.
* We often need to facilitate integration between different systems from different parts of the group: ghastly accretions of PHP and monstrous ASP.Net behemoths (why all the interfaces and reflection?). I tend to do this in Haskell when I can.
* Numerous little tools for data cleansing and the like. 
* Shake appears in various places to replace broken or non-existent build systems. We also use Shake to automate tasks where there are quite complex dependencies. I do love Shake.

Haskell's lack of dependencies, strong typing and excellent libraries make it really quite fantastic. Other developers in the group insist that their language of choice is superior, of course, and we have a fair number of other bits and pieces written in these languages. A number of these eventually break or need extending in various ways. Generally I find Haskell to be a much better language in this regard: tools tend to work well and can be evolved very easily.

I expect there is quite a lot of "stuff" that is held by commercial users of Haskell which could be of benefit, or at least some comfort, to those who are new to the Haskell language and it's ecosystem.

--

Neil Mitchell

unread,
Apr 8, 2015, 5:59:57 AM4/8/15
to Blake Rain, Michael Snoyman, commerci...@googlegroups.com
So it seems that the examples above, while they all might be "DevOps",
fall into two distinct categories:

* You have services and parts of the system, which seems to be what
Michael does most of. These are smallish apps, often webby, which
would probably have been written in Python/Ruby/PHP or some other
scripting language. There you get advantages from static typing,
better runtime, easy deployment.

* You have the more glue like pieces, which is what I do more of.
These are often a bit bigger, and would probably have been written in
bash. There you get the same advantages as above, plus portability and
the ability to package and reuse code.

I think both are better done in Haskell. I think the fact we're
competing with bash in the second case makes it a much easier sell :).
I'm not a salesman, and whether to combine or separate them is for
others to think about. I guess it's a question of whether you say
"when thinking about deployment, think Haskell" or "when thinking
about using Bash, think Haskell".

BTW, the warnings for Bake remain in full effect - it's not ready for
prime time, even though it now has been deployed twice. It is also
designed for large teams, in the region of 30+, which is still pretty
rare for a commercial Haskell shop.

Thanks, Neil
> https://groups.google.com/d/msgid/commercialhaskell/CANUq-hEsvpxZy2UKY9yKwkLwrx8OX%3D7ajnRmYjBCVeafRW2crA%40mail.gmail.com.

Jonathan Lange

unread,
Apr 8, 2015, 6:06:36 AM4/8/15
to Michael Snoyman, commerci...@googlegroups.com
I'm not using Haskell commercially, but I'm employed in devops and wanted to do a correspondingly good job of deploying my own hobby code.

I initially explored Nix & Nixops due to their claims at pure functionality, easy rollbacks, and composition. The offering for Haskell is going through a bit of a change right now, and there are quite a few bugs in their OS X builds, so I gave up on that after two or three weeks of playing. Nix's ideas are very Haskelly, which makes it attractive. OTOH, no one is being paid to issue security fixes for NixOps, which makes it unsuitable for hosting other people's data.

I also briefly explored using Heroku, but that would have required I learned & migrated to Halcyon, and I wasn't in the mood for learning a new build system that day. (Especially given that the instructions assume you're starting a new project with Halcyon, rather than migrating an existing one).

Because I actually wanted to get something deployed, I chose Ansible. It has modules for almost all of the things I've wanted to do, and plenty of examples on the net for me to crib from. Also, I'm very comfortable with Python, it's implementation language. It wasn't all smooth sailing, but now I have a one-click build step and a one-click deploy step.

I don't have any CI. The closest I get to that is my build script that runs on vagrant with a minimal build env and produces a deployable Linux binary.

The downside is that Ansible doesn't compose well. I'd like to explore Propellor, but it doesn't ship with EC2 functionality, so I'd have to write that myself, which is time when I'm not delivering features to customers. (Even though it's a hobby project, I still want to do a good job).

Summary:
  • I chose tools based on what I could get to work with minimal effort
  • Future maintainability is a low priority for one person working alone
  • Being able to find examples on the web matters a great deal
  • Evaluating and comparing devops technologies is time consuming: once you have something that works, there's little motivation to switch
  • Haskell is poorly situated for various PaaS providers
jml

--

Jonathan Lange

unread,
Apr 8, 2015, 6:26:46 AM4/8/15
to Michael Snoyman, commerci...@googlegroups.com
I really like Haskell, and think that if you have to write code then you should strongly consider writing that code in Haskell. However, when I scanned this email, a number of objections arose in my mind. I've shared them here in the hope that they'll contribute positively to the discussion.

On Wed, 8 Apr 2015 at 06:19 Michael Snoyman <mic...@snoyman.com> wrote:
* Compiled executables do not require runtime dependencies like Python or Ruby. (It's true that we have some deps like the GMP library, but that's relatively light and easy to provide.)

While that's great (seriously, it's the only good way to deploy anything), it's only really an advantage if *everything* is in Haskell. If there's just one Python thing that you need to run, then you need to solve the problem of "how to deploy a Python app with weirdo runtime dependencies". Because it's a computer, solving it once is almost as expensive as solving it for all time. 

Most devops environments I've heard of are at least mildly heterogeneous, because there's almost always one thing that's a category killer that was written in a regrettable language.

* High performance: Warp is comparable to Nginx, so you're not taking a performance hit by writing a new tool, and versus other possible languages (like Ruby and Python), the performance is significantly better

Yeah, but why would I use Warp over Nginx? My presently deployed Haskell uses Nginx for SSL termination and serving static binaries (following your helpful documentation, thanks!). It's a little bit of management overhead, but not that much. And the net result is that my users get things like SPDY and OCSP stapling, and I get timely security updates from a widely used open source project.
 
* Lots of libraries are readily available, and can be easily composed to create more complicated solutions (like how wai-middleware-crowd combines Warp, authenticate, and clientsession)

Yeah, but not as many as Python or Ruby.
 
* By writing simple, single-service tools instead of using more powerful systems, we can simplify configuration and lower the surface area for attack

I'm all in favour of single-service tools. I'm less in favour of writing your own thing when something already exists.

Rewriting existing software is costly, and it's really difficult to evaluate the benefit of doing so vs using an existing system. Sometimes the resulting improvement in simplicity really does help, other times you have a business-critical but non-core service that no one's maintaining and is holding the rest of the business back while the standard, more powerful services that you initially avoided are getting better and better.

That said, if I were developing a new simple, single-service tool, I'd totally use Haskell. Or maybe Rust. Probably Haskell because there'd be more libraries and thus less code to write. 


Willem van den Ende

unread,
Apr 8, 2015, 8:54:54 AM4/8/15
to commerci...@googlegroups.com, mic...@snoyman.com, wil...@qwan.eu
Hi Michael, Arnaud,

- Propellor vs Puppet / Chef:
I was an early adopter of Puppet (when they changed part of the language without proper announcement / error messages and then were surprised at our feedback I gave up, and decided not to use external DSLs designed by sysadmins again ;)) and then Chef. Until Chef released an APT package I could set to a specific version, both gave me the feeling: "You've got N configuration management problems. You add configuration management, now you have N+1 configuration management problems". Chef was an improvement because I could more or less write Ruby, which I was also using for other things. Main downsides : peculiarities in the internal DSL meant that I made mistakes by writing 'plain' ruby instead of 'chef ruby'. Some errors only appeared halfway through a Chef run, which is time consuming. Both Chef and Puppet moved from 'tools for hackers' to 'tools to sell to enterprises', so not a very good fit for 1-3 person shops. I used to run them with a git repo on the target host, instead of the central server both propose (N-1 configuration problems ;)).

The other thing (and this might pop up with propellor later as well) was that reuse and modularity in Chef and Puppet were less than I hoped for. There were tons of cool recipes, but most were overcomplicated (install nginx and this_complicated_monitoring_tool_I_dont_want instead of 'install nginx'). I also had a hard time convincing my fellow developers to use it in anger. Both just seem to be too big if you don't use them day in day out.

And that is the model Propellor uses. It is made by Joey Hess, so more on the scale of 1-3 devs. He uses it differently to how we use it, but has been very helpful in making changes for us, or accepting our changes. We're still running into some issues now we are modularizing, but having the entire config compiled, and being able to choose how stringly typed we work (i.e. copy-paste a bash script quickly, or write a decent haskell function a bit more slowly at first) is very powerful, as is getting good feedback the moment you press 'save'. Reusing our emacs setup and getting completion etc: very useful.

I find Joeys idea of distributing Propellor as source, or as a library where the 'main' application is your configuration is very empowering.

So although Propellor (like chef and puppet before) took some time to get used to, the experience is overwhelmingly. Especially in combination with Docker containers - Propellor sets the context, and then we build or pull some Docker images to do the rest, in relative isolation.

The one thing I'm struggling a bit with Propellor at the moment is using it to create a dev environment that also uses ghc and cabal. Propellor uses the distribution ghc (7.6.3 in ubuntu trusty) and efficiently setting up a more recent ghc for development can be a bit tricky. We also experimented with a dev environment in a Docker container, but that seems to stretch beyond what they were meant for.



On Wednesday, April 8, 2015 at 6:50:17 AM UTC+1, Arnaud Bailly wrote:
Hi Michael,

Indeed we (Willem and I) at Capital Match are also going down that route, and for the same reasons you are mentioning: no runtime dependencies, fewer bug rates, encapsulated tools,... 

We are still using duct tape (aka. shell scripts) but slowly moving a lot of things in compiled Haskell:Mi

Willem van den Ende

unread,
Apr 8, 2015, 8:55:58 AM4/8/15
to commerci...@googlegroups.com, blake...@gmail.com, mic...@snoyman.com
Even though Bake is designed for large teams, it seems to scale down (to two or three developers) fairly well for us :-).

Michael Snoyman

unread,
Apr 9, 2015, 3:37:08 AM4/9/15
to commerci...@googlegroups.com
Thanks everyone for the responses, this was stronger support for the idea than I expected. Given that there's a lot of agreement that we can make a good case to the world for Haskell in this domain, I'd like to figure out a way to move forward constructively. Here's a proposal:

* Let's start off with a central page in the Commercial Haskell repo dedicated to being a central place for information on this topic, targeted non-Haskellers. It should have at least the following:
    * A short explanation of why Haskell is good for devops, with a "read more" link to something more in-depth
    * A link to a getting started with Haskell guide
    * A recommendation to participate in this mailing list for more information
* Companies with code worth demoing should move ahead with:
    * Open sourcing the code
    * Writing blog posts/doing screencasts/etc
    * Linking to this content from that devops homepage
* Start talking about this in non-Haskell areas, like proggit or hacker news, and try to elicit feedback from others, specifically on how we can make this more attractive
* If anyone has ideas on new ways to move forward, discuss it on this mailing list

This is just a proposed next set of steps, please provide feedback if you have other ideas.

Arnaud Bailly | Capital Match

unread,
Apr 9, 2015, 3:39:45 AM4/9/15
to Michael Snoyman, commerci...@googlegroups.com
Devopsdays is a (un)conference that's organised at various locations all over the world and would be an ideal audience for exposing how Haskell can help for DevOps, and what is currently done. I proposed a talk to this year's Paris edition but it was rejected, although they offered a slot in lightning talks section instead. 

My 50 cts

--
You received this message because you are subscribed to the Google Groups "Commercial Haskell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to commercialhask...@googlegroups.com.
To post to this group, send email to commerci...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

Michael Snoyman

unread,
Apr 9, 2015, 3:46:13 AM4/9/15
to Jonathan Lange, commerci...@googlegroups.com
On Wed, Apr 8, 2015 at 1:26 PM Jonathan Lange <j...@mumak.net> wrote:
I really like Haskell, and think that if you have to write code then you should strongly consider writing that code in Haskell. However, when I scanned this email, a number of objections arose in my mind. I've shared them here in the hope that they'll contribute positively to the discussion.


It's definitely good to challenge these ideas.
 
On Wed, 8 Apr 2015 at 06:19 Michael Snoyman <mic...@snoyman.com> wrote:
* Compiled executables do not require runtime dependencies like Python or Ruby. (It's true that we have some deps like the GMP library, but that's relatively light and easy to provide.)

While that's great (seriously, it's the only good way to deploy anything), it's only really an advantage if *everything* is in Haskell. If there's just one Python thing that you need to run, then you need to solve the problem of "how to deploy a Python app with weirdo runtime dependencies". Because it's a computer, solving it once is almost as expensive as solving it for all time. 

Most devops environments I've heard of are at least mildly heterogeneous, because there's almost always one thing that's a category killer that was written in a regrettable language.


While this is true, I see a few responses:

* There still is an advantage to Haskell here, it's just mitigated somewhat.
* Someone with a Python environment set up would rightfully be hesitant to add Ruby to the mix due to an entire extra runtime environment. That concern wouldn't transfer to Haskell.
* With the rise of containerization (ala Docker), I think the argument is stronger: imagine having a Docker image based on busybox that just adds a single Haskell executable to do some action. There are few moving parts (making it easy to validate) and relatively lightweight for downloading (sans the admittedly large GHC binary sizes).
    * Actually, maybe releasing some Docker images like that is a good way to get the ball rolling here...
 
* High performance: Warp is comparable to Nginx, so you're not taking a performance hit by writing a new tool, and versus other possible languages (like Ruby and Python), the performance is significantly better

Yeah, but why would I use Warp over Nginx? My presently deployed Haskell uses Nginx for SSL termination and serving static binaries (following your helpful documentation, thanks!). It's a little bit of management overhead, but not that much. And the net result is that my users get things like SPDY and OCSP stapling, and I get timely security updates from a widely used open source project.
 

Because of composability. For simple cases of serving static files, I probably *wouldn't* recommend using Warp over Nginx, as the latter is more well supported by existing tooling (I actually told our devops team explicitly *not* to use Warp for that case). However, once you have to do things moderately complex- like requiring some kind of OpenID authentication or downloading on demand from S3- having the full power of Haskell readily available is much more liberating than dealing with Nginx's config system.
 
* Lots of libraries are readily available, and can be easily composed to create more complicated solutions (like how wai-middleware-crowd combines Warp, authenticate, and clientsession)

Yeah, but not as many as Python or Ruby.
 
Yes, it's a fair point we need to acknowledge. However, the practical gap between us and them is shrinking on a daily basis. I rarely run into situations where there isn't a library in Haskell already written these days.

 
* By writing simple, single-service tools instead of using more powerful systems, we can simplify configuration and lower the surface area for attack

I'm all in favour of single-service tools. I'm less in favour of writing your own thing when something already exists.

Rewriting existing software is costly, and it's really difficult to evaluate the benefit of doing so vs using an existing system. Sometimes the resulting improvement in simplicity really does help, other times you have a business-critical but non-core service that no one's maintaining and is holding the rest of the business back while the standard, more powerful services that you initially avoided are getting better and better.

That said, if I were developing a new simple, single-service tool, I'd totally use Haskell. Or maybe Rust. Probably Haskell because there'd be more libraries and thus less code to write. 



Two points:

* See my comments about composability above. For things that don't fit nicely into a preexisting tool's configuration, there's a big win here.
* There's an interesting case where the Haskell code ends up shorter than the necessary configuration for a more powerful tool. In that case, you could argue that the "rewriting" argument no longer applies.

Neil Mitchell

unread,
Apr 9, 2015, 4:05:29 AM4/9/15
to Michael Snoyman, commerci...@googlegroups.com
> * Companies with code worth demoing should move ahead with:
> * Open sourcing the code
> * Writing blog posts/doing screencasts/etc

The one "devops" thing people keep coming to me with is writing Shake
build systems to generate Docker containers. I think 5 companies have
all told me they are doing that, but none of them have yet said so
publicly. Is that some kind of unfilled niche that Haskell should be
getting in to? Is there some compelling reason that using
Shake/Haskell to generate the Docker containers is better than
anything else? I've got no idea, since I don't know what any of these
companies using it have come up with, but there certainly seems to be
overlap.

Thanks, Neil

Michael Snoyman

unread,
Apr 9, 2015, 5:03:07 AM4/9/15
to Neil Mitchell, commerci...@googlegroups.com
That's interesting, and something we may want to try out as well. Our Docker scripts are currently mostly bash, which we've internally acknowledged as being Really Bad(tm). Does anyone have some examples of Shake-powered Dockerfiles? I'd be really interested in seeing that.

Michael 

Arnaud Bailly | Capital Match

unread,
Apr 9, 2015, 5:10:45 AM4/9/15
to Michael Snoyman, Neil Mitchell, commerci...@googlegroups.com
Hi Michael,
Don't know if Neil counted us among the 5 but we are actually orchestrating our docker containers build using shake, tracking dependencies by using ID of containers. Here is our Build.hs file:


The idea is that we want dont' want to have to rebuild all dependencies all the time, so we are building a stack of containers: base OS, base build env (ghc+clojure), project's dependencies then actual build. When only source code changes, it does a full build in 9 minutes (including tests and clojurescript compiling).

 HTH

--
You received this message because you are subscribed to the Google Groups "Commercial Haskell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to commercialhask...@googlegroups.com.
To post to this group, send email to commerci...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Michael Snoyman

unread,
Apr 9, 2015, 5:12:55 AM4/9/15
to Arnaud Bailly | Capital Match, Neil Mitchell, commerci...@googlegroups.com
Thanks! I'll pass this around at FP Complete, may lead to some inspiration :)

Arnaud Bailly | Capital Match

unread,
Apr 9, 2015, 5:14:36 AM4/9/15
to Michael Snoyman, Neil Mitchell, commerci...@googlegroups.com
And do not hesitate to provide us your feedback: We struggled a bit with getting shake dependencies right, and all this stuff is still WIP. We are all learning :-)

I recently heard people released a project to interacat with docker server in python. Doing this in haskell would rock, and remove the need to mess with Dockerfile...

Cheers,

Deni Bertović

unread,
Apr 9, 2015, 5:33:32 AM4/9/15
to Arnaud Bailly | Capital Match, Michael Snoyman, Neil Mitchell, commerci...@googlegroups.com
Hi Everyone,

Arnaud, if you're talking about docker-py that's been around almost as docker itself I think.

I've stared writing a Haskell port: https://github.com/denibertovic/docker-hs

One of these days I may actually finish and cover the entire API.

Anyway, I find this conversation very inspiring. I'm not using Haskell for anything commercial (either devops or software dev) but I'm starting to see problems with a lot of
devops tools I'm using (looking at you Ansible) so I'm eager to learn about new things.

-Deni

Arnaud Bailly | Capital Match

unread,
Apr 9, 2015, 5:35:36 AM4/9/15
to Deni Bertović, Michael Snoyman, Neil Mitchell, commerci...@googlegroups.com
Nice! 

Deni Bertović

unread,
Apr 9, 2015, 7:18:10 AM4/9/15
to Arnaud Bailly | Capital Match, Michael Snoyman, Neil Mitchell, commerci...@googlegroups.com
Currently there's stuff in the dev branch that I've been working on. Any advice and/or help
on the code is welcome.

Also, I have an extensive knowledge on all things devops, docker, and (not that relevant here) Python software development, so anything I can do to help, let me know.


-Deni

Arnaud Bailly | Capital Match

unread,
Apr 9, 2015, 7:57:24 AM4/9/15
to Deni Bertović, Michael Snoyman, Neil Mitchell, commerci...@googlegroups.com
Thats indeed pretty cool. I can see how I would like to integrate that into our propellor setup to manage containers !

Deni Bertović

unread,
Apr 9, 2015, 8:33:00 AM4/9/15
to Arnaud Bailly | Capital Match, Michael Snoyman, Neil Mitchell, commerci...@googlegroups.com
From what I saw about propellor it already has something built in for managing containers right?

I think they built their own API wrapper perhaps.

Luke Hoersten

unread,
Apr 10, 2015, 11:11:39 AM4/10/15
to commerci...@googlegroups.com, mic...@snoyman.com, ndmit...@gmail.com
Using Shake to automate docker building is a great idea. I've also heard of a few companies doing this.

If you'd be willing to pull out some Shake rules which you feel may be general to anyone working with docker that'd be awesome. I'm trying to push to build the external rule ecosystem around Shake because its helpful and I think has lead to the success of other task runners/build systems of other languages.

Make sure to use the new Shake Hackage category to ensure your rules easy to find.

This will hopefully provide some convergence on how we all use Shake to manage docker and let the best ideas from everyone bubble up.

Great discussion guys! Been lurking for a while.

Thanks,
Luke

Michael Snoyman

unread,
Apr 14, 2015, 8:30:54 AM4/14/15
to Luke Hoersten, commerci...@googlegroups.com, ndmit...@gmail.com
I've put together a skeleton page on the SIG repo at:

https://github.com/commercialhaskell/commercialhaskell/blob/master/taskforce/devops.md

I'm hoping others will expand on this (just ping me if you need write access to the repo) with more information. And hopefully people will start creating blog posts that we can reference from there.

Arnaud Bailly | Capital Match

unread,
Apr 14, 2015, 3:03:42 PM4/14/15
to Michael Snoyman, Luke Hoersten, commerci...@googlegroups.com, Neil Mitchell
Thanks a lot Michael for triggering this discussion and moving things forward. 
I am on holidays next week so I might find some time to write a blog post or two on what we are doing.

If some people looking at this list are in Singapore first half of May, I would be happy to discuss Haskell for DevOps in person :-)

Best regards,


For more options, visit https://groups.google.com/d/optout.

Deni Bertović

unread,
Apr 15, 2015, 4:00:37 AM4/15/15
to Arnaud Bailly | Capital Match, Michael Snoyman, Luke Hoersten, commerci...@googlegroups.com, Neil Mitchell
Yes, thanks Michael. I will be sure to include any written documentation/blogpost to the github page you linked.

I think this is a worthwhile effort and we should all to our best to help.

-Deni

Alexandr Kurilin

unread,
Apr 15, 2015, 1:44:42 PM4/15/15
to commerci...@googlegroups.com, j...@mumak.net, mic...@snoyman.com
Let me play the devil's advocate here for a second.

I also had a similar reaction to Jonathan's here. Tools such as nginx have massive adoption, a lot of people's interests depending on it working well, they're a good choice for the vast majority of people. I honestly haven't run into custom scenarios such as the OpenID / S3 ones you mention, so I might be the wrong demographic here. For configuration management, tools like ansible (which I personally am a big fan of) are a large community effort spanning many years and > 10k commits: they don't have to perfect, but they have to have a large # of people keeping them polished, working, and expanding its capabilities so that you can do most work within the project with the least # of bash scripts.

One big concern for me is that it's not clear that we as a community have enough manpower to support building an entire ecosystem of devops tools that are always up to date with the latest changes, and also support enough scenarios to be a clear win over existing tooling out there. The question is if this is the best area for folks to be putting a ton of effort into. It's somewhat reminiscent of the situation with nagios: it's old, kind of nasty to work with, but there's so much stuff out there written over the years to make everything possible through existing plugins that it's almost impossible to replace with a modern cleaner tool.

Again, if we had the community size of a Ruby/Python, it'd make sense to have part of the community take a stab at the problem, but it's not clear if we can pull that off with our numbers. Thoughts?

Greg Weber

unread,
Apr 15, 2015, 3:01:06 PM4/15/15
to Alexandr Kurilin, commerci...@googlegroups.com, j...@mumak.net, Michael Snoyman
I agree but I also think there is a false dichotomy here. For example, people are using shake to build docker containers even though there is at least 1 Haskell code project for creating linux containers.

So ops is always going to use established tools. There are 2 primary questions
1) for dev-ops, which is largely a customization of established tools, to what extent will the customization code be written in Haskell?
2) what established tools make sense for Haskell?

Actually replacing tools (for example using bake for CI) is always going to have a lot of potential down-side because there is a lot of competition and established players. On the other hand, you could argue we are replacing make with shake. I guess I am trying to make a distinction between where lots of custom code is written. Build systems and scripts end up being thousands of lines of custom code.

--
You received this message because you are subscribed to the Google Groups "Commercial Haskell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to commercialhask...@googlegroups.com.
To post to this group, send email to commerci...@googlegroups.com.

Miëtek Bak

unread,
May 5, 2015, 7:55:46 PM5/5/15
to Jonathan Lange, Michael Snoyman, commerci...@googlegroups.com
On 2015-04-08, at 11:06, Jonathan Lange <j...@mumak.net> wrote:

> I also briefly explored using Heroku, but that would have required I learned & migrated to Halcyon, and I wasn't in the mood for learning a new build system that day. (Especially given that the instructions assume you're starting a new project with Halcyon, rather than migrating an existing one).

Thanks for your feedback. Can you point me to the instructions which gave you this impression, so that I can correct the problem?

I intended to demonstrate working with a small existing project in the Halcyon tutorial (and the Haskell on Heroku tutorial). I’ve also made sure the Halcyon website includes 7 examples of real-world Haskell applications, and 12 examples of simple “Hello, world!” projects (one for each of the reasonably popular Haskell web frameworks).

> I'd like to explore Propellor, but it doesn't ship with EC2 functionality, so I'd have to write that myself, which is time when I'm not delivering features to customers.

If you’re happy with using another tool to provision EC2 instances, Halcyon can do the rest.

As part of my regular testing, I use Halcyon to bootstrap fresh EC2 instances (and DigitalOcean droplets, and Linode… linodes), provisioned using Fabric. The bootstrap process consists of executing two commands — the first command installs Halcyon; the second command installs an application using Halcyon.

I’ll be happy to answer your questions in detail. If you use IRC, we can chat in #haskell-deployment on Freenode.


--
Miëtek
https://mietek.io





Reply all
Reply to author
Forward
0 new messages