Complex scalar codes

[Matthew Rocklin]

What is the general wisdom about Theano's practicality when it comes to complex scalar codes?  I have a computation that involves several thousand nodes on about a hundred distinct scalar inputs.  An example c++ program is here 

https://github.com/hazelnusse/bicycle.model/blob/master/src/bicycle_generated.cc

What kinds of Theano compile times should I expect?

What kind of runtime performance can I expect from Theano?  

I know that scalar codes are not a common application for Theano users.  Is this domain well supported?  Can I expect runtime performance to be similar to direct code like what is linked to above or is there some substantial overhead?

[Frédéric Bastien]

Hi,

complex is a corner of Theano that isn't much used and tested. We have a old start of C code for it, but it isn't well developed. So most of the code will be in python. Also, automatic symbolic differentiation don't work on complex.

Complex only affect optimization in a few cases: no GPU as this isn't float32 dtype, and a few dot22/gemm optimization aren't being applied for complex, but it shouldn't be hard to add support for that.

The compilation time isn't influenced directly with the use of complex/no-complex. It is indirectly influenced by the fact that some optimization like the fusion of elemwise work only for Op with C code, so this is probably not applied (except in a few cases) to complex number.

As we don't have much c code, we won't be faster then NumPy I think, but I didn't benchmark it.

If your code is only on scalar, theano can't be faster then C here, as we have an overhead for each apply node in the graph.

As you want complex+scalar, I wouldn't try Theano on that especially as you already have C++ code, I don't see any hope of speed gain without significant work on Theano.

Just a question, what you where expecting Theano to do again you C++ code here? Better optimization?

If you have questions, don't hesitate.

Fred

--
 
---
You received this message because you are subscribed to the Google Groups "theano-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theano-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Matthew Rocklin]

Sorry, I meant complex as in "very large and complicated expressions" not as in "elements with both real and imaginary components".  I'll state a few explicit questions below

How do Theano compilation times scale with problem size?  I have computations with hundreds of thousands of nodes.  What will this break in Theano?  For example are there optimizations that I should exclude when working with FunctionGraphs of this size?

These are only scalar codes so I don't gain much of what Theano has to offer for array computation.  Is Theano still a useful tool?  One might guess that Theano would introduce overhead that is insignificant for array computations but is significant for scalar computations.  Is this the case?

I've provided a link to a pre-existing code.  I'm pretty sure it's near optimal and don't expect an automated solution to beat it.  Will Theano be able to come close to this code in terms of performance?  If not why not?

> Just a question, what you where expecting Theano to do again you C++ code here? Better optimization?

Generating code like what I've linked to is apparently somewhat inconvenient.  We're raising the possibility of depending on Theano rather than further develop an in-house code generation solution.  I don't expect Theano to reduce runtime, I do expect it to reduce development time.  I'm concerned about compile time.

[Frédéric Bastien]

On Mon, May 6, 2013 at 3:17 PM, Matthew Rocklin <mroc...@gmail.com> wrote:

Sorry, I meant complex as in "very large and complicated expressions" not as in "elements with both real and imaginary components".  I'll state a few explicit questions below

How do Theano compilation times scale with problem size?  I have computations with hundreds of thousands of nodes.  What will this break in Theano?  For example are there optimizations that I should exclude when working with FunctionGraphs of this size?

We have super linear scaling compilation time vs number of node in the graph. If you disable inplace optimization, we can make it linear I think, but I haven't tested it. I don't know the memory usage with such big graph.

 

These are only scalar codes so I don't gain much of what Theano has to offer for array computation.  Is Theano still a useful tool?  One might guess that Theano would introduce overhead that is insignificant for array computations but is significant for scalar computations.  Is this the case?

sadly, Yes, but if the elemwise fusion optimization do much fusion, this could aliviate some/much of the overhead. But I wouldn't bet on that.

 

I've provided a link to a pre-existing code.  I'm pretty sure it's near optimal and don't expect an automated solution to beat it.  Will Theano be able to come close to this code in terms of performance?  If not why not?

No, due to the overhead, except if the elemwise fusion do the work. He can't fusion elemwise that will make an elemwise generate multiple output. The only way to know if it work enough in your case is to benchmark it.

 

> Just a question, what you where expecting Theano to do again you C++ code here? Better optimization?

Generating code like what I've linked to is apparently somewhat inconvenient.  We're raising the possibility of depending on Theano rather than further develop an in-house code generation solution.  I don't expect Theano to reduce runtime, I do expect it to reduce development time.  I'm concerned about compile time.

 

About the compilation time, I would guess the best to do is to do a graph, compile time vs graph size for the default optimization and with the inplace optimization disabled. I wouldn't be surprised we can make some small modification to disable more inplace optimization. In Theory, you should be able to use this Theano flag to do this:

optimizer_excluding=inplace,

but I haven't tested it.

If the graph show that this is too slow, there is also the option that you help us make Theano compilation faster (we have ideas and direction for that) instead of spending time on duplicating work in another project. As always, we can help. There is people I know would love to have faster compilation, but I don't know if they have the time to help.

Is that for SymPy?

Fred

[Matthew Rocklin]

Thanks for the answers.  They'll give a really good starting place as I work on this.  

> Is that for SymPy?

It's for a mechanics project, PyDy, that depends on SymPy. 

http://www.pydy.org/

--

[Frédéric Bastien]

ok.

thanks

Fred