Lazy import advice needed

[Stefan van Zwam]

Hi all,

The matroid code we want to submit to Sage soon has a reasonably complex web of imports: the base class Matroid is abstract, but invokes subclasses of itself (DualMatroid, MinorMatroid, sometimes BasisMatroid) for several of the standard implementations of methods. Of course each of these, in its own file, inherits from the abstract class.

For the end user, we only want to auto-import a few classes and a constructor function. We want to use lazy_import to play nice with Sage's startup time. After a little struggle I got it to work fine (autocomplete, runtime docstrings and all), but that gives trouble with building the reference manual. I get errors like

[matroids ] /Users/svanzwam/sage-5.8.beta1/devel/sage/doc/en/reference/matroids/sage/matroids/basis_exchange_matroid.rst:11: WARNING: autodoc can't import/find module 'sage.matroids.basis_exchange_matroid', it reported error: "'module' object has no attribute 'BasisExchangeMatroid'", please check your spelling and sys.path

Not only that, but the corresponding documentation files will be basically empty (or, depending on the command issued, it fails to build)

With normal imports these warnings vanish and the documentation builds as expected. Should I give up on lazy_import, or is there another way out?

Cheers,

Stefan.

[Stefan]

Another documentation issue: a whole bunch of functions reference the same standard work: "For more information, see [X]_, page 5"

If I include a reference block for each function, the documentation builder will complain about duplicate references. If I don't, the on-the-fly documentation doesn't contain the reference. What's the preferred way?

[Travis Scrimshaw]

Hey Stefan,

The matroid code we want to submit to Sage soon has a reasonably complex web of imports: the base class Matroid is abstract, but invokes subclasses of itself (DualMatroid, MinorMatroid, sometimes BasisMatroid) for several of the standard implementations of methods. Of course each of these, in its own file, inherits from the abstract class.

   My first thought with this is that there is an issue with design. In my belief, implementations of abstract/base classes should never rely on specific subclasses begin called/created/etc.. However if there is a definite need for this (i.e. you can't refactor your code to adjust for this for some reason), you can always import what you need directly in the method. For example, in Matroid, you have a method foo() which needs DualMatroid, you could do the following:

    def foo(self, arg):

        from sage.matrioid.dual_matroid import DualMatroid

        return DualMatriod(arg)

By my understanding, this occurs a small but noticeable time penalty (more so than lazy importing). However it avoids the circular import problem, and I recall lazy imports can have problems with caching (e.g. UniqueRepresentation; I believe there is a ticket about this).

For the end user, we only want to auto-import a few classes and a constructor function. We want to use lazy_import to play nice with Sage's startup time. After a little struggle I got it to work fine (autocomplete, runtime docstrings and all), but that gives trouble with building the reference manual. I get errors like

[matroids ] /Users/svanzwam/sage-5.8.beta1/devel/sage/doc/en/reference/matroids/sage/matroids/basis_exchange_matroid.rst:11: WARNING: autodoc can't import/find module 'sage.matroids.basis_exchange_matroid', it reported error: "'module' object has no attribute 'BasisExchangeMatroid'", please check your spelling and sys.path

Not only that, but the corresponding documentation files will be basically empty (or, depending on the command issued, it fails to build)

Have you tried deleting and rebuilding the entire (matroid) doc? 

With normal imports these warnings vanish and the documentation builds as expected. Should I give up on lazy_import, or is there another way out?

See first paragraph. 

Hope that helps,

Travis

[Volker Braun]

Autodoc will resolve the lazy imports when it tries to get at the docstrings, there are various lazily imported objects for which the documentation builds just fine. Your problem is most likely elsewhere. Whats the trac ticket?

Also, how about moving the matroid code into the sage.geometry.matroid subdirectory? There is already some related code in there, including code to compute the circuits of a point configuration in sage.geometry.triangulation.point_configuration

[Robert Bradshaw]

How about lazily importing the whole thing, rather than trying to have
part of it lazy and part of it not? Lazy import works best with
top-level constructors than individual objects anyways.

- Robert

[Stefan]

The Trac ticket is 7477, but so far we've been developing our code separately, at

https://bitbucket.org/matroid/sage_matroids/overview

To get lazy imports, modify the last two lines of all.py

I think the goals and viewpoints of matroid theorists are different enough from finite geometry that we can be sage.matroids. I like to see our code as a sibling of sage.graphs.

By the way, I did all my tests against Sage 5.8.beta1, which has fooled with the documentation a bit.

--Stefan.

[Stefan]

Here's what I was trying to do (lazy version)

In sage.all.py:
====
from sage.matroids.all import *

In sage.matroids.all.py:
====
lazy_import('sage.matroids.importer', ['Matroid', 'matroids', 'LinearMatroid', 'BasisMatroid', ...])

In sage.matroids.importer.py:
====
from basis_matroid import BasisMatroid
from linear_matroid import LinearMatroid, ...
from constructor import Matroid
import matroids_catalog as matroids

I somehow seemed to need the extra layer of 'importer.py' to make it work; previously I just had the list of non-lazy import statements in all.py. Note that everything imported into Sage is done lazily. I import classes (LinearMatroid, BasisMatroid), functions (Matroid), and modules (matroids). There's a mix of Python and Cython files.

Most files in the code do a fair number of imports of each other (none lazy). Notably, the following files (which contain abstract classes) are used all the time by the code (LinearMatroid derives from BasisExchangeMatroid), but not imported by Sage:

sage.matroids.matroid.pyx
sage.matroids.basis_exchange_matroid.pyx

I hope I am making a little sense; it's getting late here...

--Stefan.

[Robert Bradshaw]

On Tue, Feb 26, 2013 at 9:15 PM, Stefan <stefan...@gmail.com> wrote:
> Here's what I was trying to do (lazy version)
>
> In sage.all.py:
> ====
> from sage.matroids.all import *
>
> In sage.matroids.all.py:
> ====
> lazy_import('sage.matroids.importer', ['Matroid', 'matroids',
> 'LinearMatroid', 'BasisMatroid', ...])
>
> In sage.matroids.importer.py:
> ====
> from basis_matroid import BasisMatroid
> from linear_matroid import LinearMatroid, ...
> from constructor import Matroid
> import matroids_catalog as matroids
>
> I somehow seemed to need the extra layer of 'importer.py' to make it work;
> previously I just had the list of non-lazy import statements in all.py. Note
> that everything imported into Sage is done lazily. I import classes
> (LinearMatroid, BasisMatroid), functions (Matroid), and modules (matroids).
> There's a mix of Python and Cython files.

Were you perchance importing stuff from all itself?

> Most files in the code do a fair number of imports of each other (none
> lazy). Notably, the following files (which contain abstract classes) are
> used all the time by the code (LinearMatroid derives from
> BasisExchangeMatroid), but not imported by Sage:
>
> sage.matroids.matroid.pyx
> sage.matroids.basis_exchange_matroid.pyx
>
> I hope I am making a little sense; it's getting late here...
>
> --Stefan.
>

> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sage-devel?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Robert Bradshaw]

On Tue, Feb 26, 2013 at 9:55 PM, Robert Bradshaw
<robe...@math.washington.edu> wrote:
> On Tue, Feb 26, 2013 at 9:15 PM, Stefan <stefan...@gmail.com> wrote:
>> Here's what I was trying to do (lazy version)
>>
>> In sage.all.py:
>> ====
>> from sage.matroids.all import *
>>
>> In sage.matroids.all.py:
>> ====
>> lazy_import('sage.matroids.importer', ['Matroid', 'matroids',
>> 'LinearMatroid', 'BasisMatroid', ...])
>>
>> In sage.matroids.importer.py:
>> ====
>> from basis_matroid import BasisMatroid
>> from linear_matroid import LinearMatroid, ...
>> from constructor import Matroid
>> import matroids_catalog as matroids
>>
>> I somehow seemed to need the extra layer of 'importer.py' to make it work;
>> previously I just had the list of non-lazy import statements in all.py. Note
>> that everything imported into Sage is done lazily. I import classes
>> (LinearMatroid, BasisMatroid), functions (Matroid), and modules (matroids).
>> There's a mix of Python and Cython files.
>
> Were you perchance importing stuff from all itself?

E.g. sage/matroids/basis_exchange_matroid.pyx

- Robert