Edward: (another) library for probabilistic modeling, inference, and criticism
718 views
Skip to first unread message
Dustin Tran
May 5, 2016, 10:41:10 AM5/5/16
to stan development mailing list
Hey all,
Just wanted to let you know that a few of us in Dave Blei's group have released a Python library (https://github.com/blei-lab/edward) a few days ago, which does variational inference among other things.
We were motivated primarily by research: there's a lack of software in the machine learning community for developing new probability models and inference algorithms. While Bayesian practices (and more generally, statistical practices) in machine learning is rising, it is by far a minority in the field. I think part of this reason is because of the lack of software to readily apply these tools, where a student may not have the foresight to see how a hierarchical linear model connects to an SVM or neural network. We hope to bridge this gap by appealing to deep learners with ""state-of-the-art"" results for example while pointing them to foundational questions such as validating their models and inference.
Also I'd like to dispel any potential misunderstandings: the software distinguishes itself from Stan by first and foremost being a library for methodological research rather than a library for applied research. This means that we're leveraging a lot of the recent tools such as TensorFlow as the backend for GPU support and computational graphs, data subsampling, BBVI for discrete latent variable models and Bayesian nonparametric models, and neural network tools for composing deep generative models. To do this we sacrifice a lot of the robustness and stability that Stan does well. After the research has matured—just like how Rayleigh has a tool for benchmarking inference algorithms—then we can think about how to put it into Stan.
Dustin
Just wanted to let you know that a few of us in Dave Blei's group have released a Python library (https://github.com/blei-lab/edward) a few days ago, which does variational inference among other things.
We were motivated primarily by research: there's a lack of software in the machine learning community for developing new probability models and inference algorithms. While Bayesian practices (and more generally, statistical practices) in machine learning is rising, it is by far a minority in the field. I think part of this reason is because of the lack of software to readily apply these tools, where a student may not have the foresight to see how a hierarchical linear model connects to an SVM or neural network. We hope to bridge this gap by appealing to deep learners with ""state-of-the-art"" results for example while pointing them to foundational questions such as validating their models and inference.
Also I'd like to dispel any potential misunderstandings: the software distinguishes itself from Stan by first and foremost being a library for methodological research rather than a library for applied research. This means that we're leveraging a lot of the recent tools such as TensorFlow as the backend for GPU support and computational graphs, data subsampling, BBVI for discrete latent variable models and Bayesian nonparametric models, and neural network tools for composing deep generative models. To do this we sacrifice a lot of the robustness and stability that Stan does well. After the research has matured—just like how Rayleigh has a tool for benchmarking inference algorithms—then we can think about how to put it into Stan.
Dustin
Andrew Gelman
May 5, 2016, 7:06:59 PM5/5/16
to stan...@googlegroups.com
Cool! But is it Turing-complete? (just kidding)
> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Dustin Tran
May 5, 2016, 7:39:29 PM5/5/16
to stan...@googlegroups.com
Hi Andrew, it serves the same purpose as the Stan math library but due to the developers’ interests, it has different features. For example, probability distributions aren’t in TensorFlow so I had to write a mini-library doing precisely that. But it has GPU support, which is important for the large-scale problems that machine learning engineers are interested in.
Dustin
Dustin
Andrew Gelman
May 5, 2016, 7:41:08 PM5/5/16
to stan...@googlegroups.com
Any possibility of there being an integration of Stan math library and Tensorflow? I could talk with the Google people about this. Or would the technical challenges be too overwhelming?
Dustin Tran
May 5, 2016, 7:58:09 PM5/5/16
to stan...@googlegroups.com
The technical challenge is writing a wrapper function, so that each function in the Stan math library fits into TensorFlow’s C++ API. (See https://www.tensorflow.org/versions/r0.8/how_tos/adding_an_op/index.html). This would expose all of Stan math to TensorFlow’s interface in both C++ and Python. So short answer: yes, it’s possible, and I hope someone does it!
Alp Kucukelbir
May 6, 2016, 9:30:27 AM5/6/16
to stan development mailing list
+1 for easier access to stan math. we frequently end up reinventing the wheel, which is a shame when such a finely engineered math library already exists.
Bob Carpenter
May 6, 2016, 12:32:05 PM5/6/16
to stan...@googlegroups.com
What kind of access for what kind of client code?
We're certainly open to suggestions on making it easier
to use. I don't have any ideas myself how to do that beyond
more user-facing doc --- having a header-only library
with functional access to autodiff is our best shot so far.
The Edward GitHub page says it integrates Stan, but I
haven't looked into it closely enough to know what that
means.
-Bob
> On May 6, 2016, at 9:30 AM, Alp Kucukelbir <akucu...@gmail.com> wrote:
>
> +1 for easier access to stan math. we frequently end up reinventing the wheel, which is a shame when such a finely engineered math library already exists.
>
We're certainly open to suggestions on making it easier
to use. I don't have any ideas myself how to do that beyond
more user-facing doc --- having a header-only library
with functional access to autodiff is our best shot so far.
The Edward GitHub page says it integrates Stan, but I
haven't looked into it closely enough to know what that
means.
-Bob
> On May 6, 2016, at 9:30 AM, Alp Kucukelbir <akucu...@gmail.com> wrote:
>
> +1 for easier access to stan math. we frequently end up reinventing the wheel, which is a shame when such a finely engineered math library already exists.
>
Krzysztof Sakrejda
May 6, 2016, 12:32:29 PM5/6/16
to stan development mailing list
On Friday, May 6, 2016 at 9:30:27 AM UTC-4, Alp Kucukelbir wrote:
> +1 for easier access to stan math. we frequently end up reinventing the wheel, which is a shame when such a finely engineered math library already exists.
> +1 for easier access to stan math. we frequently end up reinventing the wheel, which is a shame when such a finely engineered math library already exists.
What could be easier? (serious question). Krzysztof
Alp Kucukelbir
May 6, 2016, 12:52:53 PM5/6/16
to stan development mailing list
sorry, i now realize that my message was obscure.
as far as "ease of use" in c++, stan math is heaven.
what i meant was: easier to use from a scripting language, such as python.
e.g. i wanted to draw from a skewnormal in python. it's not in scipy
(0.18) yet. but it's in stan. so i had to write a small c++ program to
wrap around stan math and draw from a skewnormal.
e.g.2. stan has multiple parameterization of distributions, like the
negative binomial. scipy doesn't. i had to interface the two and it
was tedious.
> --
> You received this message because you are subscribed to a topic in the Google Groups "stan development mailing list" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/stan-dev/2SwPrMOUCQ0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to stan-dev+u...@googlegroups.com.
as far as "ease of use" in c++, stan math is heaven.
what i meant was: easier to use from a scripting language, such as python.
e.g. i wanted to draw from a skewnormal in python. it's not in scipy
(0.18) yet. but it's in stan. so i had to write a small c++ program to
wrap around stan math and draw from a skewnormal.
e.g.2. stan has multiple parameterization of distributions, like the
negative binomial. scipy doesn't. i had to interface the two and it
was tedious.
> You received this message because you are subscribed to a topic in the Google Groups "stan development mailing list" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/stan-dev/2SwPrMOUCQ0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to stan-dev+u...@googlegroups.com.
Allen B. Riddell
May 6, 2016, 1:37:56 PM5/6/16
to stan...@googlegroups.com
It looks like tensorflow's C++/Python bindings are hand-written, e.g.,
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/lib/core/py_func.cc
It would be wonderful to have bindings to stan-math along these lines.
Best,
Allen
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/lib/core/py_func.cc
It would be wonderful to have bindings to stan-math along these lines.
Best,
Allen
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
Krzysztof Sakrejda
May 6, 2016, 1:38:51 PM5/6/16
to stan development mailing list
On Friday, May 6, 2016 at 12:52:53 PM UTC-4, Alp Kucukelbir wrote:
> sorry, i now realize that my message was obscure.
>
> as far as "ease of use" in c++, stan math is heaven.
>
> what i meant was: easier to use from a scripting language, such as python.
>
> e.g. i wanted to draw from a skewnormal in python. it's not in scipy
> (0.18) yet. but it's in stan. so i had to write a small c++ program to
> wrap around stan math and draw from a skewnormal.
>
> e.g.2. stan has multiple parameterization of distributions, like the
> negative binomial. scipy doesn't. i had to interface the two and it
> was tedious.
> sorry, i now realize that my message was obscure.
>
> as far as "ease of use" in c++, stan math is heaven.
>
> what i meant was: easier to use from a scripting language, such as python.
>
> e.g. i wanted to draw from a skewnormal in python. it's not in scipy
> (0.18) yet. but it's in stan. so i had to write a small c++ program to
> wrap around stan math and draw from a skewnormal.
>
> e.g.2. stan has multiple parameterization of distributions, like the
> negative binomial. scipy doesn't. i had to interface the two and it
> was tedious.
I worked with Python a while back and had similar problems with scipy
where I ended up writing random number generators for various things
which seemed to defeat the point of using a high-level language. It does
look like you're describing a well-defined niche that a Stan-math interface
could fill, really more of an issue of somebody having the motivation
to do it than anything else.
Krzysztof
Dustin Tran
May 6, 2016, 2:07:38 PM5/6/16
to stan...@googlegroups.com
Here’s what we had to write to get distributions in TensorFlow:
It’s very bare bones. The log density only supports scalar arguments with limited support for vectorized arguments. The random number generation also resorts to SciPy, which is not very useful for TensorFlow because it doesn’t work well with the symbolic computations. The current hack, without using Stan math, is to build our own random number generation from TensorFlow’s uniform sampler: https://github.com/blei-lab/edward/issues/51. You can see how much reinventing the wheel there is.
Dustin
You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
Bob Carpenter
May 6, 2016, 5:12:04 PM5/6/16
to stan...@googlegroups.com
Isn't 32-bit floating point dangerous from a precision point
of view? Is it there because it's more amenable to GPU parallelization?
How does the GPU integration work with Python code like this?
Are function calls so expensive in Python that you can't pull
out all the cut-and-paste code like the RHS of these:
x = tf.cast(tf.squeeze(x), dtype=tf.float32)
a = tf.cast(tf.squeeze(a), dtype=tf.float32)
b = tf.cast(tf.squeeze(b), dtype=tf.float32)
Does Python have anything like pass-by-reference? If so, then
you could write all these as single functions.
I would strongly recommend putting the operators first on a newline
C++/Java-style rather than at the end R-style. That \ at the end
of lines is the price you pay for no semicolons or brackets; nothing
to be done about that.
Also, there's a lot you can do to optimize all this code by rearranging
terms.
- Bob
of view? Is it there because it's more amenable to GPU parallelization?
How does the GPU integration work with Python code like this?
Are function calls so expensive in Python that you can't pull
out all the cut-and-paste code like the RHS of these:
x = tf.cast(tf.squeeze(x), dtype=tf.float32)
a = tf.cast(tf.squeeze(a), dtype=tf.float32)
b = tf.cast(tf.squeeze(b), dtype=tf.float32)
Does Python have anything like pass-by-reference? If so, then
you could write all these as single functions.
I would strongly recommend putting the operators first on a newline
C++/Java-style rather than at the end R-style. That \ at the end
of lines is the price you pay for no semicolons or brackets; nothing
to be done about that.
Also, there's a lot you can do to optimize all this code by rearranging
terms.
- Bob
Dustin Tran
May 6, 2016, 8:13:33 PM5/6/16
to stan...@googlegroups.com
Yeah we require 32-bit for the GPU speedup. You can see how TensorFlow does this in
re:pass-by-reference. I’m not sure. It’s worth looking into. I haven't spent much time trying to optimize this code and so far only ensured it took the right inputs and gave the right outputs.
Dustin
Reply all
Reply to author
Forward
0 new messages