Supporting Libraries (a la OTP) for Cloud Haskell

[Tim Watson]

It seems that things have come a long way since I first watched SPJ on a podcast talking about Cloud Haskell a little under a year ago. Now the distributed-process implementation has matured quite a bit and after seeing some impressive presentations at the recent Haskell Exchange in London this month, it appears that the time has come for some community participation to help keep things moving.

I'm an Erlang programmer by day, so it seemed sensible to volunteer to look at porting some of the *ideas* from Erlang's Open Telecom Platform to (Cloud) Haskell. I am not a Haskell expert (though not quite a virgin either) and my background is mainly in C and ML, so whilst I can code in Haskell, my ability to design a beautiful API for the language is probably somewhat lacking. Nevertheless, I have started a repository at https://github.com/hyperthunk/distributed-process-platform and would like to request feedback on the issue tracker about what features from OTP are of interest to Cloud Haskell users, and (I hope) some useful dialogue that will help shape the implementation itself.

Perhaps the most central concept in OTP is that of the supervision tree, and it is here that I would like to start: https://github.com/hyperthunk/distributed-process-platform/issues/1. There are some challenges however, that arise when thinking about how to bring over a concept from Erlang, which is dynamically typed! What type, for example, should the process specification(s) for a supervisor have? How best do we represent error conditions in such a way that a supervisor (which it seems in Cloud Haskell, must monitor its children rather than linking, in order to prevent the supervisor itself dying) can deduce whether or not a restart is required, and how to perform that restart?

If you are a keen Cloud Haskell user then please do come and give some feedback. I've already come to the conclusion that distributed-process-platform should look *quite* different to OTP, and I'd really like to see what kind of beast rears its head once the ideas start flowing.

Tim

[Tim Watson]

Oh and I'll be at the LHUG meetup this Wednesday if anyone wants to catch up for a beer and some brain storming afterwards!

[Jeff Epstein]

This is a great idea. I'm not an expert in OTP, but I'd love to
discuss some approaches with you.

Jeff

[Tim Watson]

That would be much appreciated Jeff!

Cheers
Tim

[Simon Peyton-Jones]

Tim

Excellent!  Do talk to Duncan Coutts and Edsko d Vries who have been doing much of the heavy lifting recently. 

 

In general, Cloud Haskell needs people willing to roll up their sleeves and get stuck in.  What we have so far is very much a jumping off point, not a finished product  All the code is on Github, and I would love to see a community of people emerge, who work on developing the libraries and infrastructure of Cloud Haskell.   It’s just too big a job for any of us to tackle alone... let’s work together. 

Simon

[Tim Watson]

Hi Simon,

Yes that's exactly what I'm hoping for! I'm going to do some 'idea collecting' first and then bring some (hopefully) coherent plans to Duncan and Edsko for further discussion. I'm always too keen to jump in and start coding, but I think this one needs a definite chunk of design and planning.

Cheers,
Tim

On 22 Oct 2012, at 13:31, Simon Peyton-Jones wrote:

> Tim
>
> Excellent! Do talk to Duncan Coutts and Edsko d Vries who have been doing much of the heavy lifting recently.
>
> In general, Cloud Haskell needs people willing to roll up their sleeves and get stuck in. What we have so far is very much a jumping off point, not a finished product All the code is on Github, and I would love to see a community of people emerge, who work on developing the libraries and infrastructure of Cloud Haskell. It’s just too big a job for any of us to tackle alone... let’s work together.
>
> Simon
>

[Jiansen He]

Hi Tim,

I am also working on using OTP principles in typed settings.  I think the design of the Akka Library (http://akka.io/) may help your work, especially the supervision trees.  However, messages to Akka actors are dynamically typed.  If you would like to have stronger typed APIs, you may want to have a look at my TAkka library (http://homepages.inf.ed.ac.uk/s1024484/takka/), where actors are parameterized with the type of their expecting messages.

Cheers

Jiansen

On Monday, 22 October 2012 01:43:39 UTC+1, Tim Watson wrote:

[Tim Watson]

Hi

That's pretty much exactly what we came up with (passing the input and output type) but only for a single input message. This is fine if your process is only dealing with a narrow input domain but in my experience that's rarely the case.

Anyway, I'll take a better look and see what ideas I can learn from your library, so thanks for mentioning it.

Cheers

Tim

[Jiansen He]

Hi Tim,

Actor in my TAkka library is parameterized only on the input type because message passing in TAkka is always asynchronous.  

I examined the expressiveness by porting examples built on top of Erlang/OTP and the Akka library.  In terms of program size and code structure, I didn't see any significant differences between the version using type-parameter and the version without using type-parameter.

In a system that supports sub-typing, I don't think a type-parametrized actor can "only deal with a narrow input domain"; However, parametrized on the type of output message does not help much.

If you know examples to which passing the input type will result to awkward or inappropriate results, I will be appreciate if you could send them to me.

Cheers

Jiansen

[Tim Watson]

Hiya

On 2 Dec 2012, at 00:40, Jiansen He <jian...@gmail.com> wrote:

Hi Tim,

Actor in my TAkka library is parameterized only on the input type because message passing in TAkka is always asynchronous.  

The same is true for cloud Haskell, though the gen server API introduces concepts like rpc and promises built on top of basic message passing.

I examined the expressiveness by porting examples built on top of Erlang/OTP and the Akka library.  In terms of program size and code structure, I didn't see any significant differences between the version using type-parameter and the version without using type-parameter.

When we got down to the bones of implementing gen server, this turned out to be true for us as well. It would still be easier to deal with if the operations on the abstract message type were all available however.

In a system that supports sub-typing, I don't think a type-parametrized actor can "only deal with a narrow input domain"; However, parametrized on the type of output message does not help much.

Well, what do we mean by sub-typing here? As you know the type system of languages like Haskell and ML is very different to OO languages like scala and type classes are a completely different beast to OO classes.

But I might've over stated the restrictions here. The point I was making is that CH has all its basic operations for abstract messages (usin the internal data type) hidden and whilst this is a good bit of APi design it does complicate low level interactions like the ones we make when implementing very generic processes like gen server that deal with just about any dod with regards input/output types.

If you know examples to which passing the input type will result to awkward or inappropriate results, I will be appreciate if you could send them to me.

I'm not really sure what you're asking for here, but suffice to say that parametric polymorphism is the balm we've applied to this probl, making the client specify the functions used to translate a very abstract input domain into the specific types it requires. We've also been able to harness existentials to make the API a bit more users friendly, but my argument remains and it's not an argument about the use of types or type parameters per se.

Cloud Haskell does support an abstract message API, most of which is still internal, and used the Typeble class to ensure that all data is transmitted with a proper type fingerprint. The APIs in CH that deal with matching (aka selective receive) and so on, use this type fingerprint to handle the choice about where to dispatch at runtime, so if we already have this kind of

[Tim Watson]

Sorry, trigger happy send button on my phone there!

On 3 Dec 2012, at 08:47, Tim Watson <watson....@gmail.com> wrote:

Cloud Haskell does support an abstract message API, most of which is still internal, and used the Typeble class to ensure that all data is transmitted with a proper type fingerprint. The APIs in CH that deal with matching (aka selective receive) and so on, use this type fingerprint to handle the choice about where to dispatch at runtime, so if we already have this kind of

... Capability, then why not expose it? Otherwise we risk making the gen server author implement so much boilerplate code just to deal with mapping messages to its expected input domains, it becomes more work and not less to do that than just use a Plain Old Process. It is my belief that the gen server API should allow the user to write callback functions as simply as possible, even pure functions that take the state and input domain and then return an action tag and new state. Of course we need to allow callbacks to execute in the IO monad if they want, but that's unavoidable transformation, whereas having to write functions just to map out input types seems like something the underlying infrastructure should do for you.

Of course some of these problems are solved now and there are open issues and discussions on the CH GitHub issue tracker, where these questions and ideas at being thrashed out now. One of my co-conspirators has even started contributing to the CH base framework, so I think our OTP over CH will make in interesting comparison with other approaches to making something like OTP once it's able to stand up and walk. :)

[Tim Watson]

Just a quick note to let everyone know that we are gradually moving forward with this project. Do please submit any ideas or issues on the bug tracker and, as our list of todo items grows feel free to come along and help out!

I'd also like to remind everyone that the Cloud Haskell initiative is still alive and well, but that project too depends on community input in order to thrive so any input there would also be great fully received I'm sure.

Its not just code contributions that would help. Both projects would benefit, for example, from having some infrastructure available on which to run automated performance and regression tests. We're trying to get travis-ci set up to handle our automate unit tests but it's unlikely to suffice for more demanding tasks.

http://haskell-distributed.github.com/distributed-process-platform

Cheers
Tim