Re: [theano-dev] For loop vs scan loop in Theano

[Olivier Delalleau]

Oops, forgot to make sure the mailing list was included... and actually moving to theano-users since this is not a dev question.

-=- Olivier

2013/10/14 Olivier Delalleau <olivier....@gmail.com>

So, I'm not a scan expert, so hopefully someone else will correct me if I say something wrong, but here are the sources of overhead I can think of when using a "for" loop instead of scan:

1. By default, there are checks made when you call a function, to ensure the input arguments are compatible, which scan can bypass.

2. I believe there is a Cython & a GPU implentations of scan, to iterate faster.

3. If you're not careful, you might be providing to your function input arguments that need to be transferred to GPU, while scan could keep everything on GPU.

The for-loop won't make the graph bigger, actually scan will (but it shouldn't matter except for compilation).

Note that there are many situations where the 3 overheads above are meaningless compared to the cost of the function call, so they don't matter at all. For instance, iterating over decently sized minibatches in a large enough neural network (assuming the minibatch data is stored on GPU and the function call only provides the minibatch index and not the data itself, to avoid problems with #3).

NB: Some tips to speed-up function calls can be found at http://deeplearning.net/software/theano/tutorial/faq.html#faster-theano-function (and the section "Faster Small Theano Function" below).

-=- Olivier

2013/10/14 Mateusz Malinowski <m4l...@gmail.com>

Thanks Olivier for the answer, but can you elaborate a bit more about the overhead?

Is it because for-loop results in bigger computational graph which impact the execution?

On Monday, October 14, 2013 12:19:37 AM UTC+2, Olivier Delalleau wrote:

The main advantage of scan is that it's symbolic, so it can be part of the graph.

When using for, you will pay an overhead at each function call, which may be meaningful if these calls are very fast.

-=- Olivier

Le 2013-10-13 à 17:53, Mateusz Malinowski <m4l...@gmail.com> a écrit :

Hi,

Is there any performance penalty (apart from larger computational graph) when using for loop instead of scan loop?

What are benefits of using the latter over the former if we want to loop only over a small set of elements?

Best,

M.M.

--
 
---
You received this message because you are subscribed to the Google Groups "theano-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to theano-dev+...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Razvan Pascanu]

+1 to what Olivier said.

I would add that scan can compute things "inplace", i.e. reusing the same memory of some of the inputs and does some optimizations between the inner graph and outer graph (like pushing computations outside the loop and so on)

In the end you can (if you know your way around theano) get the same effects yourself by carefully compiling the function over which you loop. For all of these things we mentioned, it also matters a lot what computations happen inside of scan, in certain situation scan might not give you any advantages. The best way to decide is to do a little benchmark and see which approach fairs best.

Razvan

--
 
---
You received this message because you are subscribed to the Google Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to theano-users...@googlegroups.com.

[Pascal Lamblin]

There seems to be some confusion about what the "for loop" means. There
are two ways a for loop can be used:

- Compile a Theano function that does one step of the loop, and call
this function repeatedly in a for loop. I think that is what Olivier's
replies are about.

- Use a for loop in Python while building the function graph. This will
indeed unroll the loop (the loop will be executed once, creating new
nodes at each iteration). The main drawback of this method is that the
size of the graph can become quite big, leading to two main problems:
possible busting of the Python stack (which can be avoided to some
extent by raising the stack limit), and increased graph compilation time
(it scales super-linearly wrt the number of nodes in the graph. Another
drawback is that you cannot have the number of loops to execute be a
symbolic variable: it has to be known when you build the graph (this can
be somewhat mitigated by building a large loop and use the ifelse lazy
conditional to skip the remaining iterations).

Hope this helps,

> >>> Le 2013-10-13 � 17:53, Mateusz Malinowski <m4l...@gmail.com> a �crit :

> >>>
> >>> Hi,
> >>> Is there any performance penalty (apart from larger computational graph)
> >>> when using for loop instead of scan loop?
> >>> What are benefits of using the latter over the former if we want to loop
> >>> only over a small set of elements?
> >>>
> >>> Best,
> >>> M.M.
> >>>
> >>> --
> >>>
> >>> ---
> >>> You received this message because you are subscribed to the Google
> >>> Groups "theano-dev" group.
> >>> To unsubscribe from this group and stop receiving emails from it, send

> >>> an email to theano-dev+...@**googlegroups.com.
> >>>
> >>> For more options, visit https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out>
> >>> .
> >>>
> >>>
> >
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups "theano-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to theano-users...@googlegroups.com.

> For more options, visit https://groups.google.com/groups/opt_out.

--
Pascal