Recommendations for a new Parent/Element pair

[R. Andrew Ohana]

Hi,

Reading through the documentation (from the latest beta) on the coercion framework and category framework, it isn't exactly clear to me what the recommended way to implement a new Parent/Element pair as part of the Sage ecosystem.

For instance, what is the recommended class I should inherit from for my parent class? I see at the top of sage.rings.rings, that it is not recommended that I inherit from the Ring class anymore (maybe?), but should use the category framework. In which case, what should I inherit from? Parent directly? If I was looking to add a group, there is no such warning in sage.groups.groups, so should I inherit from Group in that case?

Same goes for my elements, what is the correct base class to use?

Moreover, the two frameworks documentation don't really mention anything about each other, or how they interact. From my understanding, they were built up independently, so it would make some sense if they don't interact. However, the two frameworks do try to solve similar issues, so it seems like there would be at least some interactions between them. Anything in particular I should look out for? And should I prefer one framework over the other if they both solve some issue that I'm encountering?

I don't really have much in the way of specifics at the moment as I'm just starting out, but I would like to go down the recommended path from the start, rather than going off my old knowledge which is certainly dated by a few years in this area.

(P.S. For those interested, I'm looking to flesh out the algebraic function field functionality over the summer)

--

Andrew

[Simon King]

Hi Andrew,

On 2014-05-28, R. Andrew Ohana <andrew...@gmail.com> wrote:
> For instance, what is the recommended class I should inherit from for my
> parent class? I see at the top of sage.rings.rings, that it is not
> recommended that I inherit from the Ring class anymore (maybe?),

REALLY? Why is that?

Personally, I would not hesitate to use these existing base classes
(sage.rings.ring.Ring for example) for things that are guaranteed to be
rings. Of course, if the actual algebraic structure depends on
parameters, then one must use a more basic base class.

That said, it is certainly suspicious that sage.rings.ring.Ring inherits
from sage.structure.parent_old(!).Parent. But this also inherits from
sage.structure.parent.Parent, and all should be good.

Similarly for elements. I'd say, if there is a suitable cythoned base
class, then use it.

By the way, this is what I recommended in the thematic tutorial on
categories and coercion.

> but should use the category framework.

This should certainly be done, too. Personally, I'd not like to let the
category framework do all the work. After all, it is mainly written in
Python, whereas the old base classes are written in Cython. I suppose
this will matter when it comes to speed.

> In which case, what should I inherit from?
> Parent directly?

This is a possibility.

> If I was looking to add a group, there is no such warning
> in sage.groups.groups, so should I inherit from Group in that case?

This is another possibility. To me, it seems that
sage.groups.group.Group just provides a couple of placeholder methods
(is_abelian and so on).

OMG, I just see that sage.groups.group.Group.__contains__ overrides
sage.structure.parent.Parent.__contains__ in a very bad way!! Ouch, that
is not nice. So, after seeing this, I'd say I would stick with
sage.structure.parent.Parent in this case, and as soon as I have time
I will open a ticket to remove the __contains__.

> Same goes for my elements, what is the correct base class to use?

In the case of elements, I would tend more clearly towards using cythoned
base classes than in the case of parents. After all, Parents are containers,
while elements do the actual work and thus need speed.

> Moreover, the two frameworks documentation don't really mention anything
> about each other, or how they interact. From my understanding, they were
> built up independently, so it would make some sense if they don't interact.

I believe that I explain in the thematic tutorial on categories and
coercion how they interact.

Best regards,
Simon

[Eric Gourgoulhon]

Dear Andrew (and Simon),

I had exactly the same questions as yours (!) when trying to implement algebraic structures related to fields on manifolds, for the SageManifolds project. After reading the thematic tutorial by Simon and discussing with Nicolas Thiery and other Sage experts in Paris, I came up with the following solution. I am showing it here, not only to provide some concrete example (maybe not a good one!) but to get comments from experts, before going further in the actual implementation.

Let us take the example of the set C^oo(U) of smooth real-valued functions (scalar fields) U --> R, where U is some open subset of a differentiable manifold M. This set is a commutative algebra over R. Therefore, to represent it in Sage, we declare

class ScalarFieldAlgebra(UniqueRepresentation, Parent):

    Element = ScalarField

    def __init__(self, domain):
        Parent.__init__(self, base=SR, category=CommutativeAlgebras(SR))
        self.domain = domain
        self._populate_coercion_lists_()

       
In the constructor, domain represents the open set U. Note that the field R (over which the algebra is based) is "represented" by the symbolic ring SR.
Then the class ScalarFieldAlgebra is equipped with the methods _element_constructor_, _an_element_ and _coerce_map_from_, defined as follows:

    def _element_constructor_(self, coord_expression=None, name=None,
                              latex_name=None):
        if coord_expression == 0:
            return ZeroScalarField(self.domain)
        if isinstance(coord_expression, ScalarField):
            if self.domain.is_subdomain(coord_expression.domain):
                # restriction of the scalar field to self.domain:
                sexpress = {}
                for chart, funct in coord_expression.express.items():
                    for schart in self.domain._atlas:
                        if schart in chart.subcharts:
                            sexpress[schart] = funct.expr()
                resu = self.element_class(self.domain,
                                          coord_expression=sexpress, name=name,
                                          latex_name=latex_name)
            else:
                raise TypeError("Cannot coerce the " + str(coord_expression) +
                                "to a scalar field on the " + str(self.domain))
        else:
            resu = self.element_class(self.domain,
                                      coord_expression=coord_expression,
                                      name=name, latex_name=latex_name)
        return resu

    def _an_element_(self):
        return self.element_class(self.domain, coord_expression=2)
           
           
    def _coerce_map_from_(self, other):
        r"""
        Determine whether coercion to self exists from other parent
        """
        if other is SR:
            return True  # coercion from the base ring (multiplication by the
                         # algebra unit, i.e. self.one())
        elif other is ZZ:
           return True   # important to define self(1) (for self.one())
        elif other is QQ:
            return True
        elif isinstance(other, ScalarFieldAlgebra):
            return self.domain.is_subdomain(other.domain)
        else:
            return False

The elements, i.e. the smooth functions U --> R, are implemented as follows:

class ScalarField(CommutativeAlgebraElement):
    def __init__(self, domain, coord_expression=None, name=None,
                 latex_name=None):
        CommutativeAlgebraElement.__init__(self, domain.scalar_field_algebra())
        self.manifold = domain.manifold
        self.domain = domain
        self.tensor_type = (0,0)
        self.name = name
        if latex_name is None:
            self.latex_name = self.name
        else:
            self.latex_name = latex_name
        self.express = {}
        if coord_expression is not None:
            if isinstance(coord_expression, FunctionChart):
                self.express[coord_expression.chart] = coord_expression
            elif isinstance(coord_expression, dict):
                for chart, expression in coord_expression.items():
                    if isinstance(expression, FunctionChart):
                        self.express[chart] = expression
                    else:
                        self.express[chart] = FunctionChart(chart, expression)
            elif coord_expression == 0:
                for chart in self.domain._atlas:
                    self.express[chart] = chart.zero_function
            else:
                for chart in self.domain._atlas:
                    self.express[chart] = FunctionChart(chart,
                                                        coord_expression)
        self._init_derived()   # initialization of derived quantities

This element class is equipped with the following methods:

    def __nonzero__(self):
        res = True
        for funct in self.express.values():
            res = res and funct.is_zero()
        return not res

    def __eq__(self, other):

        if not isinstance(other, ScalarField):
            try:
                other = self.parent()(other)    # conversion to a scalar field
            except TypeError:
                return False
        if other.domain != self.domain:
            return False
        if other.is_zero():
            return self.is_zero()
        com_charts = self.common_charts(other)
        if com_charts is None:
            raise ValueError("No common chart for the comparison.")
        resu = True
        for chart in com_charts:
            resu = resu and (self.express[chart] == other.express[chart])
        return resu

    def __ne__(self, other):
        r"""
        Non-equality operator.
        """
        return not self.__eq__(other)

The class ScalarField is also equipped with the following single-underscore arithmetic operators:

 

   def _add_(self, other):
        if isinstance(other, ZeroScalarField):
            return self.copy()
        com_charts = self.common_charts(other)
        if com_charts is None:
            raise ValueError("No common chart for the addition.")
        result = ScalarField(self.domain)
        for chart in com_charts:
             result.express[chart] = self.express[chart] + other.express[chart]
        if result.is_zero():
            return self.domain.zero_scalar_field
        return result

    def _sub_(self, other):
        # code similar to _add_

    def _mul_(self, other):
        # code similar to _add_

    def _div_(self, other):
        # code similar to _add_

    def _lmul_(self, number):
        if number == 0:
            return self.domain.zero_scalar_field
        result = ScalarField(self.domain)
        for chart, expr in self.express.items():
            result.express[chart] = number * expr
        return result

    def _rmul_(self, number):
        return self._lmul_(number) # since the algebra is commutative

Note that we are in the case described by Simon in his reply, namely the parent class inherits directly from Parent (not from some commutative algebra class), with the category set to CommutativeAlgebras(SR), while the element class inherits from the Cythoned class CommutativeAlgebraElement.

In the above, I've skipped some pieces of code and the doctests. The full sources are available at
https://github.com/sagemanifolds/sage
Tensor fields on U are implemented as modules over the ring (algebra) C^oo(U) described above.

Any comment / suggestion is most welcome.

Best wishes,

Eric.

[Volker Braun]

Some comments:

By not hiding attributes (not using underscores like self._manifold) you implement a mutable interface, essentially inviting users to change them. This of course will generally not work (e.g. reach into ZeroScalarField and making it non-zero). It also prevents you from ever using caching. In Sage, attributes are generally hidden (leading underscore convention) and accessor functions ensure immutabliity. One could also use read-only attributes but that is generally not done in Sage.

Having an extra ScalarField -> ZeroScalarField element subclass unnecessarily duplicates information about the value in the type. It makes the implementation awkward in that you always have to check if the result of operations is not zero so you don't accidentally construct a zero ScalarField. If its important to find out quickly if a field is zero or not just cache .is_zero()

                for chart in self.domain.</spa

...

[Volker Braun]

Some comments:

Non-underscore attributes implement a mutable interface. You are inviting your users to modify field.tensor_type = (1,2), which will almost certanly lead to inconsistent objects. In Sage, the convention is generally to use underscore attributes and ensure immutability through accessor methods. An alternative would be read-only attributes, but this is generally not used in Sage.

The split ScalarField  <-> ZeroScalarField duplicates value information in the type. This is awkward for the implementation where you always have to check and make sure that algebraic operations don't accidentally create a ScalarField that happens to be zero. If you really need to be able to tell quickly if a field is zero just cache .is_zero()

[Eric Gourgoulhon]

Thank you for these comments. I'll modify the code accordingly!

Eric.

[Travis Scrimshaw]

At some point, someone(s) should sit down and make all of the old parents into the new Parent class. This might have to be done all at once (at least the one time I tried to change homset IIRC) and will likely take a full day plus.

Best,

Travis

[Nils Bruin]

On Wednesday, May 28, 2014 7:23:20 AM UTC-7, Volker Braun wrote:

Non-underscore attributes implement a mutable interface.

I wasn't aware of that convention. Is it documented somewhere, with rationale?

I've indeed seen "constant" instance attributes accessed via accessors and was aware of the following reasons:

 - An accessor method allows a docstring to be attached, so it provides a documentation hook. This is something we could find a solution to.

 - For a while access via calling something was faster (once CachedMethodNoArgs was optimized), but this was due to cython classes not participating in Pythons attribute lookup cache. This has since been corrected.

 - In cython, it's often desirable to have a cdef attribute store the actual data, for optimized access. If declaring that "public" for some reason doesn't work propery (or because the code predates that feature being available in cython), there will be an underscore attribute and routines to make it available to the python world.

Python's primary mechanism for storing information on instances is via attributes stored in the instance __dict__. Do we really want to dissuade people from using the most straightforward storing method by default? The "consenting adults" doctrine of Python suggests we shouldn't do that just to enforce immutability.

In fact, the dynamic class machinery may store bound methods in instance dictionaries! Those are mutable and reassigning those can wreak real havoc.

[Volker Braun]

On Wednesday, May 28, 2014 5:35:18 PM UTC+1, Nils Bruin wrote:

Non-underscore attributes implement a mutable interface.

I wasn't aware of that convention. Is it documented somewhere, with rationale?

Python provides @property and @foo.setter to make this convention work safely, so clearly its a common pattern.

 

 - An accessor method allows a docstring to be attached, so it provides a documentation hook. This is something we could find a solution to.

Docstrings can be attached to any attribute, no?

Python's primary mechanism for storing information on instances is via attributes stored in the instance __dict__. Do we really want to dissuade people from using the most straightforward storing method by default? The "consenting adults" doctrine of Python suggests we shouldn't do that just to enforce immutability. 

The underscore is a convention, nobody stops you from ignoring it. But at least you have been warned. There is a difference between "we are consenting adults" and having everything filled with glass shards.

[William Stein]

On Wed, May 28, 2014 at 12:59 PM, Volker Braun <vbrau...@gmail.com> wrote:
> On Wednesday, May 28, 2014 5:35:18 PM UTC+1, Nils Bruin wrote:
>>>
>>> Non-underscore attributes implement a mutable interface.
>>
>> I wasn't aware of that convention. Is it documented somewhere, with
>> rationale?
>
>
> Python provides @property and @foo.setter to make this convention work
> safely, so clearly its a common pattern.

For the record, I wrote a ton of code in 2004-2006 that followed the
pattern Volker describes exactly, following standard Python convention
back then. At the time there were no decorators or properties in
Python. So using underscore to keep things private was the only
option.

>>
>> - An accessor method allows a docstring to be attached, so it provides a
>> documentation hook. This is something we could find a solution to.
>
>
> Docstrings can be attached to any attribute, no?

When properties were added to Python, and Sage got them, I was not
convinced and didn't want to switch all the code to them. Why?
Because Sage (unlike most Python code) is usually used interactively,
and I've spent a lot of time watching people use Sage. People can
easily learn the following Ipython-invented way of working:

sage: f = Foo()
sage: f.[tab]
sage: f.something?
docstring about how to call something
sage: f.something(...)

With properties, guess what happens? Suppose f.something is a
property that's an immutable number, e.g., the determinant of a
matrix.

sage: a = matrix(...)
sage: a.det?
docstring about integer!

And the above just makes absolutely no sense. The user expects to get
a docstring about a function that computes the determinant, but
instead they get the same docstring as "5?" would give. Yes, there is
a function behind the scenes (that computes det), and it's possibly to
get a docstring for it. I don't know how to make "a.det?" give that
doctring though.

Another issue is that frequently in math we have algorithms/options to
functions, e.g.,

sage: a.det(algorithm="padic")

Expressing the above with properties is awkward and hard to discover.
What's worse is maybe you can only think of one way to compute det, so

sage: a.det

to compute det is fine. But then somebody *later* things of another
algorithm, and you have to completely break all code that used det.
Also, computable properties are annoying since it makes it harder to
see why code is slow, e.g.,

sage: a.det * b.det

... no function calls, so fast, right? No.

So we didn't end up embracing properties in Sage, purely because they
aren't friendly toward the above use cases.

For non-interactive more library oriented Python programming I think
properties can be extremely useful (are they used even once in all of
Sage anywhere?). But for interactive Sage use, I was dubious when
they were introduced long ago.

-- William

>
>> Python's primary mechanism for storing information on instances is via
>> attributes stored in the instance __dict__. Do we really want to dissuade
>> people from using the most straightforward storing method by default? The
>> "consenting adults" doctrine of Python suggests we shouldn't do that just to
>> enforce immutability.
>
>
> The underscore is a convention, nobody stops you from ignoring it. But at
> least you have been warned. There is a difference between "we are consenting
> adults" and having everything filled with glass shards.
>

> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.



--
William Stein
Professor of Mathematics
University of Washington
http://wstein.org

[Eric Gourgoulhon]

Thanks for these clear explanations.
The case of IPython tab mechanism for getting documentation is very convincing!

Coming from the C++ world, I was used to have the attributes private, but was not sure about the Python way of dealing with this (having read different opinions about it). I did not underscored the "light" attributes, like tensor_type, as noticed by Volker, to avoid the cost of calling some method tensor_type() (in C++, thanks to the inline mechanism, there is no cost issue for such access methods). But I am fully convinced now and I am turning all the attributes of SageManifolds classes to private and introduce the corresponding access methods, with their docstrings.

Eric.

[Simon King]

Hi William,

On 2014-05-28, William Stein <wst...@gmail.com> wrote:
> With properties, guess what happens? Suppose f.something is a
> property that's an immutable number, e.g., the determinant of a
> matrix.
>
> sage: a = matrix(...)
> sage: a.det?
> docstring about integer!

That's not true, if det is a property:
sage: class MyMatrix(object):
....: @property
....: def det(self):
....: "Determinants are computed by..."
....: return 5
....:
sage: m = MyMatrix()
sage: m.det
5
sage: m.det?
Type: property
String Form:<property object at 0x904b2fc>
Docstring: Determinants are computed by...

> And the above just makes absolutely no sense. The user expects to get

> a docstring about a function that computes the determinant, ...

She in fact does. That's the difference between attributes and
properties.

However:

> Another issue is that frequently in math we have algorithms/options to
> functions, e.g.,
>
> sage: a.det(algorithm="padic")
>
> Expressing the above with properties is awkward and hard to discover.

+1

Best regards,
Simon

[Volker Braun]

On Thursday, May 29, 2014 12:03:54 AM UTC+1, William wrote:

Another issue is that frequently in math we have algorithms/options to
functions, e.g.,
  sage: a.det(algorithm="padic")

And even if there is no argument I would prefer a.det() over a.det to make it clear that this is invoking a computation and not just accessing one of the defining attributes of a. This is _the_ difference between an argument-less method and a readonly @property: If you have to call() it, it is clear that there is a computation involved. Whereas a single property invites you to tab-complete your way through it a.det.is_prime, and you don't want to run a computation every time you press tab.

There is an argument to be made for using @property to access the defining data of an object, but since we historically don't do it I would prefer consistency over that syntactic gadget. The benefit of not having to always type () vs. the continual uncertainty about whether you need parentheses... In Sage, there are generally a lot of methods computing stuff compared to the simple accessor methods needed, so there isn't much need for @property to start with. 

[William Stein]

Please try to use a single underscore, e.g., self._tensor, instead of
double underscore self.__tensor. The latter is the official Python
way of making things (more) private. It has technical implications
beyond just being a convention, in that it mangles things, e.g.,
self.__tensor works, but if T is a tensor, then T.__tensor does *not*
work...

The general convention we use these days is do NOT bother with the
double underscore attributes, since it's just kind of annoying and
we're consenting adults enough to just honor the convention that a
single underscore means private. There is a lot of very old code in
Sage that uses double underscore that we wrote during the first two
years, but most newer code uses single underscore.

-- William

>
> Eric.

[William Stein]

Simon -- for the record, that the above works now is some clever and
*relatively new* trickery in IPython.
For example, in SageMathCloud (and sagenb and straight Python), which
don't use IPython for introspection, m.det? or

sage: help(m.det)

work exactly like I said -- they return help on Integer.
Obviously, the IPython devs got fed up with this and implemented some
trick to get around this for ?, which we're automatically inheriting
in command-line Sage (and which I should figure out and implement for
notebooks...). I can't see any possible way to patch help to deal
with this though, since it's just a function that gets the integer 5
as input. It doesn't know where 5 came from.

Thanks for the clarification, but note that this wan't my only
argument against properties for interactive Sage use, and it's
definitely one that can be worked around in some cases.

>
> However:
>
>> Another issue is that frequently in math we have algorithms/options to
>> functions, e.g.,
>>
>> sage: a.det(algorithm="padic")
>>
>> Expressing the above with properties is awkward and hard to discover.
>
> +1
>
> Best regards,
> Simon
>
>

[Eric Gourgoulhon]

Thanks for the advice.
I've just finished the changes, setting all attribute names to single underscore (2280 substitutions, thank you sed !).

Best wishes,

Eric.

[Nicolas M. Thiery]

On Wed, May 28, 2014 at 04:03:11PM -0700, William Stein wrote:
> When properties were added to Python, and Sage got them, I was not
> convinced and didn't want to switch all the code to them. Why?

> ...

Thanks William for this synthetic answer on that matter. We should
definitely add this to the developers guide! Suggestions about where?
Volunteers?

Cheers,
Nicolas
--
Nicolas M. Thiéry "Isil" <nth...@users.sf.net>
http://Nicolas.Thiery.name/

[Nicolas M. Thiery]

On Wed, May 28, 2014 at 09:57:43AM +0000, Simon King wrote:
> Personally, I would not hesitate to use these existing base classes
> (sage.rings.ring.Ring for example) for things that are guaranteed to be
> rings. Of course, if the actual algebraic structure depends on
> parameters, then one must use a more basic base class.

> ...

>
> In the case of elements, I would tend more clearly towards using cythoned
> base classes than in the case of parents. After all, Parents are containers,
> while elements do the actual work and thus need speed.

> ...

Funny, I had exactly the same discussion yesterday. I am apparently
not lazy enough, and getting bitten by it: I should have written
something about this in the documentation a long time ago. I started
doing this today, and will post a ticket probably tomorrow.

[Nicolas M. Thiery]

On Thu, May 29, 2014 at 11:30:36PM +0200, Nicolas M. Thiery wrote:
> Funny, I had exactly the same discussion yesterday. I am apparently
> not lazy enough, and getting bitten by it: I should have written
> something about this in the documentation a long time ago. I started
> doing this today, and will post a ticket probably tomorrow.

A first draft is on #16427. The final "recommendations" shall depend
on serious benchmarks. We are working on this with Florent.

Comments/suggestions/reviews are welcome even now though!

[R. Andrew Ohana]

Ok, follow up question. Is there much documentation on morphisms, or am I just blind and missing it?

At least as far as I can tell, the documentation on the coercion seems to recommend implementing morphisms for _coerce_map_from_, but in the toy examples, only boolean return values are used (which is great for getting started, but doesn't seem like a wise long term plan).

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To post to this group, send email to sage-...@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

--

Andrew

[Travis Scrimshaw]

I don't think there's much documentation beyond what's in _coerce_map_from_() in Parent.

If it returns re, then it uses _element_constructor_ to construct the morphism. Otherwise you return a morphism. I think there's some examples in src/algebras which returns morphisms (I'm pretty sure I've seen it, definitely wrote some code on it, but I don't remember off-hand for what).

Best,

Travis

[Vincent Delecroix]

2014-06-06 17:12 UTC+02:00, Travis Scrimshaw <tsc...@ucdavis.edu>:

Two other examples are ZZ -> QQ and QQ->ZZ which are respectively a
Morphism and a Map (see sage/rings/rational.pyx at the very end of the
file).

[R. Andrew Ohana]

Yeah, I also found some morphisms in the polynomial ring code, but like the QQ <-> ZZ code, it seemed dated (and hence might not follow recommended practices).

As for sage.algebras.*, I didn't really see anything other than Boolean return values.

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To post to this group, send email to sage-...@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

--

Andrew