Delaying compilation of cutils_ext until we actually need it

[Olivier Delalleau]

Currently, when we do "import theano", cutils_ext is compiled if it doesn't exist already. This is done in the "import cutils" statements.

I would like to delay this import until we actually show some intent to use it... at first sight it would only be in ProfileMode.__setstate__ and CLinker._execute.

The motivation is to be able to get a compiled cutils_ext module from some place else and copy it into Theano's compiledir: the problem is I need to import theano to obtain theano.config.compiledir, and thus it's already too late... (it's on a computer where the compilation does not work).

Can anyone foresee issues with such a change? The main drawbacks I can see are:
1. Just doing "import theano" would not let us do a quick check to make sure compilation seems to work ok, we would have to manually import cutils
2. It is against the general guideline "imports should come at the top of the file"
3. The extra import statements in ProfileMode.__setstate__ and CLinker._execute might slow things down. Anyone knows how costly it is when one tries to re-import an already imported module? (I know it has no effect, but I don't know if it's slow). Alternatively, the code could be a bit smarter and just do a boolean comparison with a module-wide flag that would be set after the first import, to avoid further import attempts.

Thoughts?

-=- Olivier

[James Bergstra]

I really like the idea of not compiling anything immediately on
import. How about checking/compiling cutils_ext when calling a
relevant Linker.accept? That happens once during function creation, of
a function that would actually use cutils_ext. (Am I remembering
correctly that cutils_ext is used to call c thunk objects?)

An alternative would be to simply delete cutils_ext as cruft, and
create proper callable Types in the auto-generated C files. It's not
hard to do that, but it takes some familiarity with the Python C API.
That is a cleaner solution, and makes nice progress towards being able
to use Theano compiled modules on their own... but it's a bigger
change.

James
--
http://www-etud.iro.umontreal.ca/~bergstrj

[Olivier Delalleau]

2011/7/1 James Bergstra <james.b...@gmail.com>

Yes you are correct, cutils is used for its run_cthunk() function.
I'll try your .accept suggestion for now. I added ticket #715 for your second option.

-=- Olivier

[Olivier Delalleau]

2011/7/1 Olivier Delalleau <dela...@iro.umontreal.ca>

Actually I can't do the import only in accept, since otherwise cutils will not be in the scope of the method that actually uses it.
So I think I'll go with my import in the functions that use c_thunk, but following the strategy described in http://wiki.python.org/moin/PythonSpeed/PerformanceTips#Import_Statement_Overhead
(which basically corresponds to the flag idea I mentioned in my first email).

-=- Olivier