Which functional programming languages are you comfortable with?

[Kevin Van Horn]

I'm playing around with some ideas for transforming Stan programs and perhaps doing auto-differentiation via source code transformation instead of via operator overloading. C++ isn't the best language for that sort of symbolic computing, so I'm looking at other languages more suited to the task. At some point I'll probably want to contribute the code to Stan, which means that accessibility and familiarity to the Stan development team are important criteria in choice of language.

Mathematica is probably the best language for implementing these kinds of transformations, but it fails on accessibility -- it has a hefty licensing fee. The possibilities I'm considering are Haskell, Common Lisp, Racket (a dialect of Scheme), F#, Clojure, OCaml, Standard ML, and Scala. My question to you all is this: which of these languages, if any, are you comfortable with?

[Michael Betancourt]

Fair warning -- there is but an epsilon probability that we would ever support such an addition, even if you had a complete package that didn't require any development in our part. It's hard enough supporting a single language.

> On Aug 14, 2015, at 9:56 AM, Kevin Van Horn <ke...@ksvanhorn.com> wrote:
>
> I'm playing around with some ideas for transforming Stan programs and perhaps doing auto-differentiation via source code transformation instead of via operator overloading. C++ isn't the best language for that sort of symbolic computing, so I'm looking at other languages more suited to the task. At some point I'll probably want to contribute the code to Stan, which means that accessibility and familiarity to the Stan development team are important criteria in choice of language.
>
> Mathematica is probably the best language for implementing these kinds of transformations, but it fails on accessibility -- it has a hefty licensing fee. The possibilities I'm considering are Haskell, Common Lisp, Racket (a dialect of Scheme), F#, Clojure, OCaml, Standard ML, and Scala. My question to you all is this: which of these languages, if any, are you comfortable with?
>

> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Bob Carpenter]

Being a programming language geek from way back, I'm familiar
at a conceptual level with all of the functional programming
language concepts. But I have zero real experience programming
any of them.

If you want to do source code transformed autodiff, you need
to restrict the language so that it never autodiffs through
iterative algorithms (unless they're fixed iteration numbers)
and disallows parameters in conditionals.

I'd start with a proof of concept. But the real target for us
would be the kind of ODE models and big hierarchical models with
multivariate priors that tend to be our bread and butter in applied
settings.

Would the idea be to generate a C++ class that would be compatible
with our MLE and MCMC (and maybe VB) tools?

It's always going to be possible to have a standalone repo that
generates code in a different way. But Michael's right that we're
unlikely to want to try to support anything other than C++ given the
compatiblity demands of operating systems and language front ends
and our user base.

- Bob

[Ben Goodrich]

On Friday, August 14, 2015 at 12:56:53 PM UTC-4, Kevin Van Horn wrote:

Mathematica is probably the best language for implementing these kinds of transformations, but it fails on accessibility -- it has a hefty licensing fee. The possibilities I'm considering are Haskell, Common Lisp, Racket (a dialect of Scheme), F#, Clojure, OCaml, Standard ML, and Scala. My question to you all is this: which of these languages, if any, are you comfortable with?

I don't think it matters much what we are comfortable with, but if I were to go down this road I would try SymPy / SymEngine and attempt to hook into PyStan.

[Bob Carpenter]

That'd be a rather different approach of doing a symbolic derivative.
That is, you'd generate an expression, then differentiate using SymPy,
then code up the resulting expression. Theano uses symbolic math for
derivatives, and it's in Python, so you could also look there.

You'll have much more luck finding people comfortable with Python than
any of the others you mention.

The alternative tha I thought Kevin was suggesting was to just
code generate the autodiff. There are a bunch of autodiff systems
that do that for (subsets of) Fortran and C (not C++). You could
even write a C++ system that ran once to record the expression then
took a pass over the abstract syntax tree to code generate the
derivative. That's how I'd approach the problem if we needed to do
it within Stan.

Matt Hoffman was urging us to look at both this kind of approach
and the Theano symbolic differentiation approach all along. It's much
more work than autodiff, though, because you can't autodiff through
complex iterative functions like we used to do.

- Bob

[Rob J. Goedman]

Kevin,

One other approach might be to try Stan.jl and Mamba.jl, both in Julia.

Regards,
Rob

> On Aug 14, 2015, at 9:56 AM, Kevin Van Horn <ke...@ksvanhorn.com> wrote:
>

> I'm playing around with some ideas for transforming Stan programs and perhaps doing auto-differentiation via source code transformation instead of via operator overloading. C++ isn't the best language for that sort of symbolic computing, so I'm looking at other languages more suited to the task. At some point I'll probably want to contribute the code to Stan, which means that accessibility and familiarity to the Stan development team are important criteria in choice of language.
>
> Mathematica is probably the best language for implementing these kinds of transformations, but it fails on accessibility -- it has a hefty licensing fee. The possibilities I'm considering are Haskell, Common Lisp, Racket (a dialect of Scheme), F#, Clojure, OCaml, Standard ML, and Scala. My question to you all is this: which of these languages, if any, are you comfortable with?
>

[Kevin Van Horn]

I have two cases in mind. The first is Stan-to-Stan transformations, where you start with a Stan model written for simplicity and obvious correctness, and you transform it into a form that is more efficient.

The second is what you said, to generate the code that computes the log joint density and its gradient and plug that into the existing Stan framework. The motivation is to look for opportunities for code optimization, and eventually be able to generate GPU code.

[Ben Goodrich]

On Friday, August 14, 2015 at 4:49:58 PM UTC-4, Kevin Van Horn wrote:

The second is what you said, to generate the code that computes the log joint density and its gradient and plug that into the existing Stan framework. The motivation is to look for opportunities for code optimization, and eventually be able to generate GPU code.

We have talked about an option to execute with doubles only but allow the .stan program to write to the gradient vector analytically.

Ben

[Bob Carpenter]

We've also talked about using a more limited BUGS-like
graphical modeling language which would be much easier
to optimize than the free-for-all that is a Stan program.

- Bob

[Bob Carpenter]

On Aug 14, 2015, at 5:07 PM, Ben Goodrich <goodri...@gmail.com> wrote:
>
>
> We have talked about an option to execute with doubles only but allow the .stan program to write to the gradient vector analytically.

I still think the easiest way to do this would be to allow
a general syntax for writing functions with gradients, then
to plug those into a Stan model.

There are all kinds of problems that will be introduced because
MLE and VB are going to assume second-order derivatives. So we
might want to template out definitions like the above, or introduce
yet another syntax for also specifying Hessians.

- Bob

[Ben Goodrich]

On Friday, August 14, 2015 at 5:14:09 PM UTC-4, Bob Carpenter wrote:

On Aug 14, 2015, at 5:07 PM, Ben Goodrich <goodri...@gmail.com> wrote:
> We have talked about an option to execute with doubles only but allow the .stan program to write to the gradient vector analytically.

I still think the easiest way to do this would be to allow
a general syntax for writing functions with gradients, then
to plug those into a Stan model.  

What might this look like? I guess you would say that user-defined functions that end in _grad are "gradient aware" and called with doubles. But unless we return tuples or pass-by-reference a gradient vector, I'm not seeing how Stan would associate the gradient with the parameters.

Ben

[Bob Carpenter]

I haven't worked out a concrete syntax. But the underlying
semantics would be to generate a function with gradients by
implementing a vari instance involving precomputed gradients.
We'd almost certainly have to pass a mutable reference in for
the gradients since we don't want to separate the computation
of the function from the computation of the gradient.

And if we ever want to do higher-order, we'd have to deal
with more than just vari.

In other words, this is a pretty big project.

- Bob