RFC: draft PEP for adding @ as a matrix multiplication operator to Python

[n...@vorpus.org]

Greetings, Sage Ones,

Some of you may have already seen this, but I've started working on a draft PEP for adding a dedicated operator for matrix multiplication to Python:
  https://github.com/numpy/numpy/pull/4351
  https://github.com/njsmith/numpy/blob/matmul-pep/doc/neps/return-of-revenge-of-matmul-pep.rst

I'm not sure how this would affect Sage, but since I know you have both matrix-like types and your own ideas about what various unused tokens should mean, I figure you'll probably have opinions :-). (Also, there's some discussion of sage.misc.decorators.infix_operator in the final section - hopefully I'm not mischaracterizing anything there.) The overall goal is to come up with a proposal that the whole Python numeric community can support, and that ideally in the long run will help reduce fragmentation in matrix APIs, so I definitely want to hear your feedback.

-n

[John Cremona]

I find it very hard to imagine Sage using anything other than * (as in
A*B) for normal matrix multiplication, as anything else would alienate
all of its mathematical users. I would have no reason at all ever to
have an element-wise matrix product!

John Cremona

> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.

[Jeroen Demeyer]

On 2014-03-09 16:09, n...@vorpus.org wrote:
> I definitely want to hear your feedback.

I completely agree with John Cremona: please keep * for matrix
multiplication. Why not add a new dedicated operator for elementwise
multiplication and use * for matrix multiplication?

In your PEP, you say that using * for matrix multiplication is a bad
idea, but without any justification (the only justification is
variations on "it's a bad idea" without reasons).

Jeroen.

[Jeroen Demeyer]

On 2014-03-09 18:19, Jeroen Demeyer wrote:
> In your PEP, you say that using * for matrix multiplication is a bad
> idea, but without any justification (the only justification is
> variations on "it's a bad idea" without reasons).

From reading your PEP, it's clear that you don't like numpy.matrix but
you don't say what's wrong with it.

[Dima Pasechnik]

In fact the PEP has a justification along the lines that elementwise
matrix multiplication and scalar multiplication pops up 4 times more
often in numpy code. But I doubt that the count was made correctly.
Even if it was the case, still this would be truly horrible idea to use *
for entrywise matrix multiplication...

I've posted a comment here:
https://github.com/numpy/numpy/pull/4351#issuecomment-37135112

Dima

>
> Jeroen.
>

[Jeroen Demeyer]

I think the following piece should be made more clear, I don't
understand what you're trying to say here:

The problem is that the presence of two different duck-types for numeric
data -- one where * means matrix multiply, and one where * means
elementwise multiplication -- make it impossible to write generic
functions that can operate on arbitrary data.

[n...@vorpus.org]

Indeed, it's clear from everyone's responses here that I at least need to add a new section talking about these things explicitly, and also about why elementwise-* is actually used so often in practice in numeric computation (as opposed dto symbolic comptuation), and why np.matrix is so loathed. (It really is though, I didn't just make that up :-) I doubt you can find a single developer of any numpy-using package who has anything good to say about it.) I'll try to get to write such a section over the next few hours; at least it might make a better basis for discussion.

In brief, the issue is that elementwise-* is a fine convention and you can use it to write useful code, and matrix-multiply-* is a fine convention and you can use it to write useful code, but if you then try to glue those two pieces of code together into a larger system you will quickly find yourself in hell as you have to cast objects back and forth at every function call. Or similarly, if you try to write a function that works regardless of whether it receives an elementwise-*-object or a matrix-multiply-*-object, then this is also very painful (suddenly you need lots of type-checking and if statements scattered around every single function you write, it's horrible and no-one is willing to do it).

This doesn't get into the reasons why numpy goes with the elementwise-* convention, but it's why it seems important to have *some* convention.

(Some of the reasons for why numpy goes with elementwise-* are here: https://github.com/njsmith/numpy/blob/matmul-pep/doc/neps/return-of-revenge-of-matmul-pep.rst#choice-of-operation ; the other important reason that that section doesn't talk about as much as it could is that numpy has n-dimensional arrays, while matrix multiplication is only really natural for 2d arrays.)

-n

[William Stein]

On Sun, Mar 9, 2014 at 12:40 PM, <n...@vorpus.org> wrote:
> On Sunday, March 9, 2014 7:20:50 PM UTC, Jeroen Demeyer wrote:
>>
>> I think the following piece should be made more clear, I don't
>> understand what you're trying to say here:
>>
>> The problem is that the presence of two different duck-types for numeric
>> data -- one where * means matrix multiply, and one where * means
>> elementwise multiplication -- make it impossible to write generic
>> functions that can operate on arbitrary data.
>
>
> Indeed, it's clear from everyone's responses here that I at least need to
> add a new section talking about these things explicitly, and also about why
> elementwise-* is actually used so often in practice in numeric computation
> (as opposed dto symbolic comptuation), and why np.matrix is so loathed. (It
> really is though, I didn't just make that up :-) I doubt you can find a
> single developer of any numpy-using package who has anything good to say
> about it.) I'll try to get to write such a section over the next few hours;
> at least it might make a better basis for discussion.
>
> In brief, the issue is that elementwise-* is a fine convention and you can
> use it to write useful code, and matrix-multiply-* is a fine convention and

I'm not convinced * is a fine convention for element-wise
multiplication of matrices, since there is no software I know of that
does that. Even Matlab uses * to mean matrix multiplication:
http://www.math.utah.edu/~eyre/computing/matlab-intro/math.html

But * is set in stone for numpy. The motivation for your proposal is
to make your example code involving matrix multiplication of numpy
arrays easier to read/write. I wonder if there is any way to do this
using a with statement, e.g.,


c = a*b # element-wise multiplication

with numpy.mul_as_dot():
c = a*b # matrix multiplication


Then the exact part of the code that involves some expressions with
matrix multiplies of numpy arrays can be cleanly written with *
instead of of numpy.dot, and without having to introduce another
operator. Is this at least technically possible? If so, it is worth
addressing in the PEP. It would solve the problem of coding up
certain algorithms cleanly, and it is extremely clear and explicit
what is going on when you just jump into the code.

Another issue -- imagine jumping into code and seeing @'s everywhere
-- you would have to google and figure out what is going on, which
could be hard for "@ operator". For example, I use the @ operator
all the time in CoffeeScript, so it's a useful thing, but googling "@
operator" yields basically nothing useful about anything.

-- William

> you can use it to write useful code, but if you then try to glue those two
> pieces of code together into a larger system you will quickly find yourself
> in hell as you have to cast objects back and forth at every function call.
> Or similarly, if you try to write a function that works regardless of
> whether it receives an elementwise-*-object or a matrix-multiply-*-object,
> then this is also very painful (suddenly you need lots of type-checking and
> if statements scattered around every single function you write, it's
> horrible and no-one is willing to do it).
>
> This doesn't get into the reasons why numpy goes with the elementwise-*
> convention, but it's why it seems important to have *some* convention.
>
> (Some of the reasons for why numpy goes with elementwise-* are here:
> https://github.com/njsmith/numpy/blob/matmul-pep/doc/neps/return-of-revenge-of-matmul-pep.rst#choice-of-operation
> ; the other important reason that that section doesn't talk about as much as
> it could is that numpy has n-dimensional arrays, while matrix multiplication
> is only really natural for 2d arrays.)
>
> -n
>

> --
> You received this message because you are subscribed to the Google Groups
> "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.

--
William Stein
Professor of Mathematics
University of Washington
http://wstein.org

[Dima Pasechnik]

On 2014-03-09, William Stein <wst...@gmail.com> wrote:
> On Sun, Mar 9, 2014 at 12:40 PM, <n...@vorpus.org> wrote:
>> On Sunday, March 9, 2014 7:20:50 PM UTC, Jeroen Demeyer wrote:
>>>
>>> I think the following piece should be made more clear, I don't
>>> understand what you're trying to say here:
>>>
>>> The problem is that the presence of two different duck-types for numeric
>>> data -- one where * means matrix multiply, and one where * means
>>> elementwise multiplication -- make it impossible to write generic
>>> functions that can operate on arbitrary data.
>>
>>
>> Indeed, it's clear from everyone's responses here that I at least need to
>> add a new section talking about these things explicitly, and also about why
>> elementwise-* is actually used so often in practice in numeric computation
>> (as opposed dto symbolic comptuation), and why np.matrix is so loathed. (It
>> really is though, I didn't just make that up :-) I doubt you can find a
>> single developer of any numpy-using package who has anything good to say
>> about it.) I'll try to get to write such a section over the next few hours;
>> at least it might make a better basis for discussion.
>>
>> In brief, the issue is that elementwise-* is a fine convention and you can
>> use it to write useful code, and matrix-multiply-* is a fine convention and
>
> I'm not convinced * is a fine convention for element-wise
> multiplication of matrices, since there is no software I know of that
> does that.

Mathematica does this. (and it uses '.' for the usual matrix
multiplication).
IIRC Maple used to have something like *& to denote matrix
multiplication, don't know if this is still the case,

[Jeroen Demeyer]

On 2014-03-09 20:40, n...@vorpus.org wrote:
> In brief, the issue is that elementwise-* is a fine convention and you
> can use it to write useful code, and matrix-multiply-* is a fine
> convention and you can use it to write useful code, but if you then try
> to glue those two pieces of code together into a larger system you will
> quickly find yourself in hell as you have to cast objects back and forth
> at every function call. Or similarly, if you try to write a function
> that works regardless of whether it receives an elementwise-*-object or
> a matrix-multiply-*-object, then this is also very painful (suddenly you
> need lots of type-checking and if statements scattered around every
> single function you write, it's horrible and no-one is willing to do it).

My impression is that a lot of this confusion is because you confuse
matrix and ndarray. They represent very different data structures and
purposes (they might be very similar in implementation and may share
many methods, but that doesn't matter here) . They are simply different
kinds of objects with different multiplication operations. Duck typing
should treat * as a "black box" operation, when you want to
differentiate between different kinds of multiplication you're probably
doing it wrong.

[Jeroen Demeyer]

On 2014-03-09 20:40, n...@vorpus.org wrote:

> Indeed, it's clear from everyone's responses here that I at least need
> to add a new section talking about these things explicitly, and also
> about why elementwise-* is actually used so often in practice in numeric
> computation (as opposed dto symbolic comptuation), and why np.matrix is
> so loathed.

Perhaps a better solution would be to fix numpy.matrix then?

[William Stein]

Thanks for pointing that out. I looked it up and Mathematica does not
even have a Matrix data type at all: "Matrices are represented in
Mathematica with lists."

https://reference.wolfram.com/mathematica/howto/CreateAMatrix.html

> IIRC Maple used to have something like *& to denote matrix
> multiplication, don't know if this is still the case,

Hey, you're right-ish:

http://kb.iu.edu/data/afbm.html

They don't use * either, they use "." (like in Mathematica). They
deprecated *&.

So I stand corrected.

[Nathaniel Smith]

On Sun, Mar 9, 2014 at 8:44 PM, William Stein <wst...@gmail.com> wrote:
> On Sun, Mar 9, 2014 at 12:40 PM, <n...@vorpus.org> wrote:
>> On Sunday, March 9, 2014 7:20:50 PM UTC, Jeroen Demeyer wrote:
>>>
>>> I think the following piece should be made more clear, I don't
>>> understand what you're trying to say here:
>>>
>>> The problem is that the presence of two different duck-types for numeric
>>> data -- one where * means matrix multiply, and one where * means
>>> elementwise multiplication -- make it impossible to write generic
>>> functions that can operate on arbitrary data.
>>
>>
>> Indeed, it's clear from everyone's responses here that I at least need to
>> add a new section talking about these things explicitly, and also about why
>> elementwise-* is actually used so often in practice in numeric computation
>> (as opposed dto symbolic comptuation), and why np.matrix is so loathed. (It
>> really is though, I didn't just make that up :-) I doubt you can find a
>> single developer of any numpy-using package who has anything good to say
>> about it.) I'll try to get to write such a section over the next few hours;
>> at least it might make a better basis for discussion.
>>
>> In brief, the issue is that elementwise-* is a fine convention and you can
>> use it to write useful code, and matrix-multiply-* is a fine convention and
>
> I'm not convinced * is a fine convention for element-wise
> multiplication of matrices, since there is no software I know of that
> does that. Even Matlab uses * to mean matrix multiplication:
> http://www.math.utah.edu/~eyre/computing/matlab-intro/math.html

R alsos use * for elementwise mul. And really, it does work fine and
is useful in many application areas, I promise! :-) I've sent similar
heads-up emails to 13 projects now, and so far you guys are the only
ones who have blinked at this part; numpy has ~25x more PyPI downloads
than sympy (a terrible metric I know, but perhaps it's at least some
vague noisy estimate of relative user bases) and I guess those users
are pretty much all happy with elementwise-*. My point isn't that the
majority is always right or anything like that. I'm just saying, if
you think elementwise-* is not just sub-optimal but also bizarre and
unthinkable, then you might be missing something :-) (just like I was
missing something when I thought that elementwise-* was so obvious it
needed no justification).

It may well be that the answer is that sage (and perhaps sympy) is
just not in the target audience for this PEP, because you only have
one multiplication operation that matters and the status quo works
fine for you. If so that's great, though it'd still be good to
coordinate to the extent it makes sense, and the feedback is still
really useful.

> But * is set in stone for numpy. The motivation for your proposal is
> to make your example code involving matrix multiplication of numpy
> arrays easier to read/write. I wonder if there is any way to do this
> using a with statement, e.g.,
>
>
> c = a*b # element-wise multiplication
>
> with numpy.mul_as_dot():
> c = a*b # matrix multiplication
>
>
> Then the exact part of the code that involves some expressions with
> matrix multiplies of numpy arrays can be cleanly written with *
> instead of of numpy.dot, and without having to introduce another
> operator. Is this at least technically possible? If so, it is worth
> addressing in the PEP. It would solve the problem of coding up
> certain algorithms cleanly, and it is extremely clear and explicit
> what is going on when you just jump into the code.

This is a really interesting suggestion, and it is technically
possible. Unfortunately, though, after thinking for a bit I see some
problems that strike me as insurmountable -- here's a potential
discussion in PEP-ese (to be slotted into the list of "rejected
alternatives"):

**Use ``with`` to switch the meaning of ``*`` within a single code
block**: E.g., numpy could define a special context object so that
we'd have::

c = a * b # element-wise multiplication
with numpy.mul_as_dot:
c = a * b # matrix multiplication

However, this has two serious problems: first, it requires that every
matrix-like object ``__mul__`` method know how to check some global
state (``numpy.mul_is_currently_dot`` or whatever). This is fine if
``a`` and ``b`` are numpy objects, but the world contains many
non-numpy matrix-like objects. So this either requires non-local
coupling -- every numpy competitor library has to import numpy and
then check ``numpy.mul_is_currently_dot`` on every operation -- or
else it breaks duck-typing, with the above code doing radically
different things depending on whether ``a`` and ``b`` are numpy
objects or some other sort of object. Second, and worse, ``with``
blocks are dynamically scoped, not lexically scoped; i.e., any
function that gets called inside the ``with`` block will suddenly find
itself executing inside the mul_as_dot world, and crash and burn
horribly (if you're lucky). So this is a construct that could only be
used safely in rather limited cases (no function calls), and which
makes it very easy to shoot yourself in the foot without warning.

What do you think?

> Another issue -- imagine jumping into code and seeing @'s everywhere
> -- you would have to google and figure out what is going on, which
> could be hard for "@ operator". For example, I use the @ operator
> all the time in CoffeeScript, so it's a useful thing, but googling "@
> operator" yields basically nothing useful about anything.

I'm not really that worried about this, because it's the same problem
that *every* builtin operator faces. How do you know what << or // or
^ or | or % or ** do when you first see them? You flip to the table at
the back/front of your Python reference that lists all the operators,
and it tells you what they are. If this PEP is accepted, then @ will
be listed there right next to the rest of them. Even * is pretty
arbitrary the first time you see it -- real mathematical notation uses
\cdot or occasionally \times, not \star -- but we forget this because
we already learned it a while ago :-).

-n

[Simon King]

Hi Nathaniel,

are you sure that you talk about actual matrices? Or do you just talk
about 2-dimensional arrays?

By the way, how could your PEP possibly be relevant to Python? IIRC,
there neither is a matrix type nor an array type in Python, thus, it
makes no sense whatsoever to theoretise about special symbols for matrix
multiplication and component-wise action on arrays in *Python*.

Such discussion sure makes sense in a CAS, though..

On 2014-03-09, Nathaniel Smith <n...@pobox.com> wrote:
> ...

> R alsos use * for elementwise mul. And really, it does work fine and
> is useful in many application areas, I promise! :-) I've sent similar
> heads-up emails to 13 projects now, and so far you guys are the only
> ones who have blinked at this part;

Quite bizarre indeed, as I'd thought that your suggestion will seem rather
odd to most mathematicians and computer scientists.

> Even * is pretty
> arbitrary the first time you see it -- real mathematical notation uses
> \cdot or occasionally \times, not \star -- but we forget this because
> we already learned it a while ago :-).

In mathematics, a lot of different multiplication symbols are in use.
For example, usually \cdot denotes pointwise multiplication, whereas
\star denotes convolution and \circ denotes composition, and there
is also cartesian product \times and tensor product \otimes. In
principal, all these operations can occur at the same time, thus, there
is no way around using a variety of symbols, to avoid confusion.

Thus, it would certainly be a reasonable PEP to provide a framework in
Python to define custom infix operators (say, operator.compose), by
providing a character (say: '@') and the name of a custom "magical
method" (say: '__composition__') that is called on the operator's
arguments, in the same way as operator.mul uses '*' and results in
calling '__mul__'.

However, in all algebraic textbooks I am aware of, \cdot is consistently
used to denote...
- multiplication in fields,
- the componentwise action of a field on vectors resp. matrices, aka
scalar multiplication,
- the dot product of vectors (resulting in a field element)
- matrix multiplication,
- generally multiplication in a ring, unless different kinds of
multiplication occur

It may seem natural to multiply two 2-dimensional ARRAYS component-wise.
However, if you have MATRICES that deserve such a title (say, they
encode a linear map) then "Matrix \cdot Matrix" clearly is matrix
multiplication and nothing else.

Python does not know about rings and vector spaces, whereas Sage does:
Sage uses '*' for multiplication in rings and for module actions. In
particular, by consistency, Sage must use '*' to denote multiplication
of square matrices (since this is multiplication in a matrix ring)

Python uses * to denote multiplication of numbers, which is \cdot in
textbooks. Hence, according to the "principle of least surprise", one
has to use * for what is commonly denoted by \cdot in textbooks. And
this includes matrix multiplication.

In conclusion:
1) Python has no array or matrix types, thus adding a special symbol for
matrix multiplication makes no sense in Python.
2) It makes sense to support adding custom infix operators to Python.
3) Python could of course add an array type with component-wise
multiplication and a matrix type with matrix multiplication. But
please keep "arrays" and "matrices" apart.
4) It is common to use the same operator symbol for various types of
operands. There is 'ab'+'cd' and 'ab'*3 in contrast to 3+4 and 4*3.
So, why should one suddenly start using new symbols if the operand
types happens to be "matrix"?

Best regards,
Simon

[Simon King]

On 2014-03-10, Simon King <simon...@uni-jena.de> wrote:
> In mathematics, a lot of different multiplication symbols are in use.
> For example,

Here I did a wrong edit. Sorry. I meant to say:

FOR FUNCTIONS, \cdot usually denotes pointwise multiplication

> ..., whereas

> \star denotes convolution and \circ denotes composition, and there
> is also cartesian product \times and tensor product \otimes. In
> principal, all these operations can occur at the same time, thus, there
> is no way around using a variety of symbols, to avoid confusion.

For matrices, \cdot clearly is NOT pointwise.

Cheers,
Simon

[Nathaniel Smith]

On Mon, Mar 10, 2014 at 12:09 AM, Simon King <simon...@uni-jena.de> wrote:
> Hi Nathaniel,
>
> are you sure that you talk about actual matrices? Or do you just talk
> about 2-dimensional arrays?

Sure -- this is a useful distinction, and as far as it goes, I'm
talking about arrays. But people do in fact want to do the operation
commonly referred to as "matrix multiplication" when working with
arrays, whether we like it or not. Many many people. Github code
search produces >23,000 Python files containing the string "np.dot",
and every one of those is a violation of the "keep matrices and
arrays" separate principle. So this is the reality we live in...

> By the way, how could your PEP possibly be relevant to Python? IIRC,
> there neither is a matrix type nor an array type in Python, thus, it
> makes no sense whatsoever to theoretise about special symbols for matrix
> multiplication and component-wise action on arrays in *Python*.

There are a dozen or more array and matrix types in Python. I use them
all the time. So do thousands of other people. You don't download them
from http://python.org, but this doesn't make them nonsense...

> Such discussion sure makes sense in a CAS, though..
>
> On 2014-03-09, Nathaniel Smith <n...@pobox.com> wrote:
>> ...
>> R alsos use * for elementwise mul. And really, it does work fine and
>> is useful in many application areas, I promise! :-) I've sent similar
>> heads-up emails to 13 projects now, and so far you guys are the only
>> ones who have blinked at this part;
>
> Quite bizarre indeed, as I'd thought that your suggestion will seem rather
> odd to most mathematicians and computer scientists.

Well, not ones who crunch numbers all day :-). It's got a long history
-- e.g. Fortran also uses * to mean elementwise mul.

>> Even * is pretty
>> arbitrary the first time you see it -- real mathematical notation uses
>> \cdot or occasionally \times, not \star -- but we forget this because
>> we already learned it a while ago :-).
>
> In mathematics, a lot of different multiplication symbols are in use.
> For example, usually \cdot denotes pointwise multiplication, whereas
> \star denotes convolution and \circ denotes composition, and there
> is also cartesian product \times and tensor product \otimes. In
> principal, all these operations can occur at the same time, thus, there
> is no way around using a variety of symbols, to avoid confusion.
>
> Thus, it would certainly be a reasonable PEP to provide a framework in
> Python to define custom infix operators (say, operator.compose), by
> providing a character (say: '@') and the name of a custom "magical
> method" (say: '__composition__') that is called on the operator's
> arguments, in the same way as operator.mul uses '*' and results in
> calling '__mul__'.

I do not believe that it is possible to write such a PEP that would be
acceptable to the broader Python community. (I don't even think I
could come up with such a PEP that I would accept if I were BDFL.)
It's a huge change to how Python works, and runs into a lot of
practical problems (defining precedence, defining scoping, creating
nasty couplings between parse time and eval time, etc.).

That's just my best judgement, though; if someone comes along with a
convincing proposal then I'll certainly advocate it.

> However, in all algebraic textbooks I am aware of, \cdot is consistently
> used to denote...
> - multiplication in fields,
> - the componentwise action of a field on vectors resp. matrices, aka
> scalar multiplication,
> - the dot product of vectors (resulting in a field element)
> - matrix multiplication,
> - generally multiplication in a ring, unless different kinds of
> multiplication occur
>
> It may seem natural to multiply two 2-dimensional ARRAYS component-wise.
> However, if you have MATRICES that deserve such a title (say, they
> encode a linear map) then "Matrix \cdot Matrix" clearly is matrix
> multiplication and nothing else.
>
> Python does not know about rings and vector spaces, whereas Sage does:
> Sage uses '*' for multiplication in rings and for module actions. In
> particular, by consistency, Sage must use '*' to denote multiplication
> of square matrices (since this is multiplication in a matrix ring)
>
> Python uses * to denote multiplication of numbers, which is \cdot in
> textbooks. Hence, according to the "principle of least surprise", one
> has to use * for what is commonly denoted by \cdot in textbooks. And
> this includes matrix multiplication.

Sure, fair enough. If you're lucky enough to be dealing only with pure
mathematical matrices, then the problem this PEP is addressing doesn't
arise. Practical number crunching OTOH isn't so pretty, but it is
useful :-).

-n

[Marc Mezzarobba]

William Stein wrote:
>> IIRC Maple used to have something like *& to denote matrix
>> multiplication, don't know if this is still the case,
>
> Hey, you're right-ish:
>
> http://kb.iu.edu/data/afbm.html
>
> They don't use * either, they use "." (like in Mathematica). They
> deprecated *&.

Actually Maple has no real matrix multiplication operator. They use "*"
to denote "commutative multiplication", and "." for "non-commutative
multiplication" (and "*~" for elementwise multiplication, "@" for
composition, "&*" for user-defined multiplication). Both "*" and "." can
be used as part of symbolic expressions (a*b-b*a is immediately
simplified to 0, while a.b-b.a is not), and are interpreted as
multiplication of more concrete objects where it makes sense.

--
Marc

[Nicolas M. Thiery]

On Mon, Mar 10, 2014 at 12:51:46AM +0000, Nathaniel Smith wrote:
> > Thus, it would certainly be a reasonable PEP to provide a framework in
> > Python to define custom infix operators (say, operator.compose), by
> > providing a character (say: '@') and the name of a custom "magical
> > method" (say: '__composition__') that is called on the operator's
> > arguments, in the same way as operator.mul uses '*' and results in
> > calling '__mul__'.
>
> I do not believe that it is possible to write such a PEP that would be
> acceptable to the broader Python community. (I don't even think I
> could come up with such a PEP that I would accept if I were BDFL.)
> It's a huge change to how Python works, and runs into a lot of
> practical problems (defining precedence, defining scoping, creating
> nasty couplings between parse time and eval time, etc.).
>
> That's just my best judgement, though; if someone comes along with a
> convincing proposal then I'll certainly advocate it.

For whatever it's worth, I'd love to have more infix operators in
Python. Ideally ones that you can define dynamically. Ideally UTF-8
ones like ⊗ (typically for tensor products) and ⊕ (direct sums) which
would not create syntax conflicts. Now I can see the difficulty you
point at, so the following would already be progress:

- More infix operators defined statically. E.g. @ calling a custom
method __composition__ as suggested by Simon.

- The possibility to define such operators in the ipython console,
even if they don't work in code.

Other than that, I see the importance of both multiplications,
definitely prefer * for matrix multiplication, and see the conflict
with the large numerical communities where * is often used for
pointwise multiplication on arrays. Good luck!

Cheers,
Nicolas
--
Nicolas M. Thiéry "Isil" <nth...@users.sf.net>
http://Nicolas.Thiery.name/

[Travis Scrimshaw]

I think having the ability to define additional infix operators would be great. Yet I agree that it would be near impossible to deal with in practice; in particular, changing the parser language (I could see even a static change triggering a recompile of python) and sorting out the precedence. However since * is used in python (other languages) to denote multiplication of the left and right sides, I think with matrices it should denote the usual multiplication.

[Niles Johnson]

I actually just have a minor complaint about the very last sentence.  In the last section you write

Use overloading hacks to define a "new infix operator" like *dot*, as in a well-known Python recipe: (See: [2]) Beautiful is better than ugly. This solution is so ugly that most developers will simply refuse to consider it for use in serious, reusable code. This isn't just speculation -- a variant of this recipe is actually distributed as a supported part of a major Python mathematics system [3], so it's widely available, yet still receives minimal use. OTOH, the fact that people even consider such a 'solution', and are supporting it in shipping code, could be taken as further evidence for the need for a proper infix operator for matrix product.

[2]	http://code.activestate.com/recipes/384122-infix-operators/

[3]	http://www.sagemath.org/doc/reference/misc/sage/misc/decorators.html#sage.misc.decorators.infix_operator

 

First, I didn't know Sage had that and I doubt I will ever use it!

Second, I think your last sentence is too much of a stretch.  It's fair to say that Sage shipping an infix hack is (possibly) evidence that people love infix operators. (Although the fact that it's not used much would suggest that they don't love them that much.)  But -- as you rightly point out earlier -- Sage has no element-wise multiplication, using * only for matrix multiplication.  So it's a bit unfair to say that Sage's infix hack is further evidence that Python needs a new operator for matrix multiplication.

If anything, it would be more accurate (and a stronger argument) to point out that the entire Sage library uses an infix operator for matrix multiplication and doesn't even define element-wise multiplication -- that's how highly Sage thinks of matrix multiplication.  This point is implicit in your discussion of Partial- or Non-Adoption, and if you think it's important I would suggest expanding that instead of trying to rework the last sentence.

[rjf]

Maxima, part of Sage, has had an extensible (at run time) parser for perhaps 35 years.
You could ask about the experience there, maybe read about the pros and cons. 
Or you could be more conventional and ignore  others' past experience. :)

What's  "PEP" stand for?  I assume that at least one of the P's is python...
RJF

[Robert Bradshaw]

First off, to start with a funny anecdote, I remember working with
some of the early Sage devs way back in the day optimizing matrix
multiplication and we were baffled by the fact that numpy somehow
managed to have O(n^2) behavior into the thousand by thousand matrix
range :).

More serious responses below.

On Sun, Mar 9, 2014 at 5:51 PM, Nathaniel Smith <n...@pobox.com> wrote:
> On Mon, Mar 10, 2014 at 12:09 AM, Simon King <simon...@uni-jena.de> wrote:
>> Hi Nathaniel,
>>
>> are you sure that you talk about actual matrices? Or do you just talk
>> about 2-dimensional arrays?
>
> Sure -- this is a useful distinction, and as far as it goes, I'm
> talking about arrays. But people do in fact want to do the operation
> commonly referred to as "matrix multiplication" when working with
> arrays, whether we like it or not. Many many people. Github code
> search produces >23,000 Python files containing the string "np.dot",
> and every one of those is a violation of the "keep matrices and
> arrays" separate principle. So this is the reality we live in...

I think this arrays vs. matrices is *the* key distinction to keep in
mind. Element-wise is the only sane way to extend binary operations to
arrays of arbitrary dimension.

By this logic, we should define multiplication of polynomials
element-wise as well: after all polynomials are typically defined as
arrays of coefficients....

There are certainly times when one wants to work with arrays like this
in Sage, and it's perfectly fine to invoke numpy to do so. In general,
however, x*y (where x and y are of the same type, or coercible to the
same type) denotes multiplication as defined in the parent (Group,
Ring, Monoid, whatever) of x and y. This isn't special to matrices: it
applies to rationals, algebraic numbers, polynomials, permutations,
etc. The * operator is also used to denote an action, e.g. on scalar
multiplication of a matrix, a matrix acting on a vector, or the
integer ring on a abelian group like an elliptic curve. Pretty much
whatever would be represented on the blackboard by a \cdot or simply
placing symbols adjacent to each other.

IMHO it seems odd to try to define a new infix operator that
specifically has "matrix multiplication" semantics. That seems out of
the domain of the Python language. If it were to be accepted, there's
no way Sage would start using * to do element-wise multiplication of
matrices.

- Robert

[William Stein]

On Tue, Mar 11, 2014 at 5:29 PM, Robert Bradshaw <robe...@gmail.com> wrote:
> First off, to start with a funny anecdote, I remember working with
> some of the early Sage devs way back in the day optimizing matrix
> multiplication and we were baffled by the fact that numpy somehow
> managed to have O(n^2) behavior into the thousand by thousand matrix
> range :).

I remember precisely that happening to me. I was extremely
impressed.... for a few minutes.

> More serious responses below.

>
>> Sure, fair enough. If you're lucky enough to be dealing only with pure
>> mathematical matrices, then the problem this PEP is addressing doesn't
>> arise. Practical number crunching OTOH isn't so pretty, but it is
>> useful :-).
>
> There are certainly times when one wants to work with arrays like this
> in Sage, and it's perfectly fine to invoke numpy to do so. In general,
> however, x*y (where x and y are of the same type, or coercible to the
> same type) denotes multiplication as defined in the parent (Group,
> Ring, Monoid, whatever) of x and y. This isn't special to matrices: it
> applies to rationals, algebraic numbers, polynomials, permutations,
> etc. The * operator is also used to denote an action, e.g. on scalar
> multiplication of a matrix, a matrix acting on a vector, or the
> integer ring on a abelian group like an elliptic curve. Pretty much
> whatever would be represented on the blackboard by a \cdot or simply
> placing symbols adjacent to each other.

Relevant link is [1].

>
> IMHO it seems odd to try to define a new infix operator that
> specifically has "matrix multiplication" semantics. That seems out of
> the domain of the Python language. If it were to be accepted, there's
> no way Sage would start using * to do element-wise multiplication of
> matrices.

As founder of the project, I second Robert's assertion that no matter
what, "there's no way Sage would start using * to do element-wise
multiplication of matrices."


[1] http://www.sagemath.org/doc/reference/coercion/

[Nathaniel Smith]

On Tue, Mar 11, 2014 at 12:43 PM, Niles Johnson <nil...@gmail.com> wrote:

Second, I think your last sentence is too much of a stretch.  It's fair to say that Sage shipping an infix hack is (possibly) evidence that people love infix operators. (Although the fact that it's not used much would suggest that they don't love them that much.)  But -- as you rightly point out earlier -- Sage has no element-wise multiplication, using * only for matrix multiplication.  So it's a bit unfair to say that Sage's infix hack is further evidence that Python needs a new operator for matrix multiplication.

Yeah, you're right, that was definitely me getting carried away on a first draft there. Thanks for catching it. I rewrote that section to just leave sage out of it entirely -- it's not really the main point.

-n

[Nathaniel Smith]

On Tue, Mar 11, 2014 at 10:26 PM, rjf <fat...@gmail.com> wrote:

Maxima, part of Sage, has had an extensible (at run time) parser for perhaps 35 years.
You could ask about the experience there, maybe read about the pros and cons. 
Or you could be more conventional and ignore  others' past experience. :)

Alas, there's pretty much no chance of getting an extensible parser into Python at this point. (Among other things, Python uses a strict two-stage parse-then-run system!)

 

What's  "PEP" stand for?  I assume that at least one of the P's is python...

"Python Enhancement Proposal" -- it's the standard process for proposing changes to the Python language:
  https://en.wikipedia.org/wiki/Python_%28programming_language%29#Development

-n

[Nathaniel Smith]

On Mon, Mar 10, 2014 at 10:15 AM, Nicolas M. Thiery
<Nicolas...@u-psud.fr> wrote:
> On Mon, Mar 10, 2014 at 12:51:46AM +0000, Nathaniel Smith wrote:
>> > Thus, it would certainly be a reasonable PEP to provide a framework in
>> > Python to define custom infix operators (say, operator.compose), by
>> > providing a character (say: '@') and the name of a custom "magical
>> > method" (say: '__composition__') that is called on the operator's
>> > arguments, in the same way as operator.mul uses '*' and results in
>> > calling '__mul__'.
>>
>> I do not believe that it is possible to write such a PEP that would be
>> acceptable to the broader Python community. (I don't even think I
>> could come up with such a PEP that I would accept if I were BDFL.)
>> It's a huge change to how Python works, and runs into a lot of
>> practical problems (defining precedence, defining scoping, creating
>> nasty couplings between parse time and eval time, etc.).
>>
>> That's just my best judgement, though; if someone comes along with a
>> convincing proposal then I'll certainly advocate it.
>
> For whatever it's worth, I'd love to have more infix operators in
> Python. Ideally ones that you can define dynamically. Ideally UTF-8
> ones like ⊗ (typically for tensor products) and ⊕ (direct sums) which
> would not create syntax conflicts. Now I can see the difficulty you
> point at, so the following would already be progress:
>
> - More infix operators defined statically. E.g. @ calling a custom
> method __composition__ as suggested by Simon.

Well, *cough*, you didn't hear it from me, but just because a method
is called __matmul__, doesn't mean it actually has to multiply any
matrices... I don't think we'll ever see BDFL approval for adding
random infix operators with no predetermined use case, but once an
operator is added the use case is just a convention.

> - The possibility to define such operators in the ipython console,
> even if they don't work in code.

Doesn't sage already do something like this? In any case, it's outside
the scope of a PEP; luckily getting changes into ipython is *much*
easier than getting them into cpython :-)

> Other than that, I see the importance of both multiplications,
> definitely prefer * for matrix multiplication, and see the conflict
> with the large numerical communities where * is often used for
> pointwise multiplication on arrays. Good luck!

Thanks!

-n

[Nils Bruin]

On Sunday, March 9, 2014 8:09:58 AM UTC-7, n...@vorpus.org wrote:

Greetings, Sage Ones,

Some of you may have already seen this, but I've started working on a draft PEP for adding a dedicated operator for matrix multiplication to Python

I've seen many examples of complicated matrix expressions that really benefit from having an infix notation for matrix multiplication. I haven't seen many examples of complicated expressions that involve element-wise multiplication, and none that mix the two. Therefore, in the absence of two infix multiplication operators, would it make sense to have "A*B" denote matrix product and "A.hadamard(B)" or "A.elementwise_product(B)" element-wise product? This is what sage already does.

[David Roe]

I think their objection is that hadamard is too long, given the ubiquity of element-wise product in the numerical computing world.  From Sage's perspective, I don't think we want anything to change.
David

[William Stein]

> David.

In retrospect, it appears that the OP should have asked the following
question: From the perspective of Sage, if Python were to have
another arithmetic operator (denoted @) with identical precedence
rules to *, would we use it for anything? Would we extend the
coercion model, etc.? Are there any algebraic structures at all that
have two different "multiplications"?

If the answers are "yes!" and there are interesting examples, it would
provide support for his proposal.

My personal answer to the above is: no, no, no... My reasoning is
that if the answer had been "YES!", then we most certainly would have
added such a thing via the preparser. However, I can't remember even
once in 8 years when anybody ever brought up this as a proposed
addition to the preparser (both on lists or in person). His actual
PEP had something claiming we had added something to the preparser,
but that was not a multiplication operator, but simply a backslash
shorthand for matrix inversion, like in Matlab, that nobody uses or
cares about -- I just added it one day for fun in the early days --
based on no demand.

-- William

-- William

[Nathaniel Smith]

I replied to Jeroen asking a similar question a few days ago, but I
guess it was on the sympy list instead of here. Anyway, this gives
more details on why elementwise products are important for
number-crunching (though not so much for mathematical analysis), and
has an example of a natural expression that uses both * and @
together:
https://groups.google.com/d/msg/sympy/4tGlBGTggZY/JHK74QnGdMYJ

-n

--
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org

[Travis Scrimshaw]

In retrospect, it appears that the OP should have asked the following
question:   From the perspective of Sage, if Python were to have
another arithmetic operator (denoted @) with identical precedence
rules to *, would we use it for anything?  Would we extend the
coercion model, etc.?   Are there any algebraic structures at all that
have two different "multiplications"?

If the answers are "yes!" and there are interesting examples, it would
provide support for his proposal.

 

My personal answer to the above is:  no, no, no...   My reasoning is
that if the answer had been "YES!", then we most certainly would have
added such a thing via the preparser.  However, I can't remember even
once in 8 years when anybody ever brought up this as a proposed
addition to the preparser (both on lists or in person).     His actual
PEP had something claiming we had added something to the preparser,
but that was not a multiplication operator, but simply a backslash
shorthand for matrix inversion, like in Matlab, that nobody uses or
cares about  -- I just added it one day for fun in the early days --
based on no demand.

   I think of a tensor algebra of an algebra (I could probably come up with others with some thought that might be somewhat contrived but practical). Yet I don't want to modify the preparser to add (for example) # to denote something as specific as a tensor product, must less that we can't override it in subclasses. On a slightly related note, I would like a non-commutative and/or non-associative infix operator.

I had no idea such a shorthand was available in Sage. *Goes off to play with it*

Best,

Travis

[Nathaniel Smith]

On Fri, Mar 14, 2014 at 8:48 PM, William Stein <wst...@gmail.com> wrote:
> In retrospect, it appears that the OP should have asked the following
> question: From the perspective of Sage, if Python were to have
> another arithmetic operator (denoted @) with identical precedence
> rules to *, would we use it for anything? Would we extend the
> coercion model, etc.? Are there any algebraic structures at all that
> have two different "multiplications"?

Indeed, this probably would have been the correct question, if only
I'd known! But the end result is that I learned a lot about how Sage
approaches things that I didn't know before, and the PEP got much
stronger from getting this feedback from a perspective I hadn't
understood before -- in fact I rewrote large parts of it in response
to comments here and on the sympy list. Thank you very much to
everyone who took the time to comment.

The current status, in case anyone is curious, is that it looks like
'@' will in fact appear in Python 3.5, once some details are worked
out:
https://github.com/numpy/numpy/pull/4351#issuecomment-37717330
If anyone here does have any opinions on those details then I'd love
to hear them.

> His actual
> PEP had something claiming we had added something to the preparser,
> but that was not a multiplication operator, but simply a backslash
> shorthand for matrix inversion, like in Matlab, that nobody uses or
> cares about -- I just added it one day for fun in the early days --
> based on no demand.

Hmm, I don't remember saying this, and the text changed quite a bit
from what I originally sent around, but if you see any remaining
inaccuracies then please let me know so I can fix them! :-):
http://legacy.python.org/dev/peps/pep-0465/
(In fact I know there's one inaccuracy in that version -- it says that
sage & sympy don't implement elementwise multiplication at all. My
local draft changes that to "Projects which currently use * for matrix
multiplication, and
which don't really care about elementwise multiplication at all",
which I think is closer to the truth?)

[Robert Bradshaw]

It's not that we don't care about elementwise multiplication of
*arrays*, it's that we don't care about elementwise multiplication of
*matrices* (and I've yet to find any use of the latter at all).

I think the problem of numpy.matrix is
http://en.wikipedia.org/wiki/Composition_over_inheritance There's
currently only a single example (after much searching) where the two
need to be mixed--an explicit conversion would be fine here.

I am also opposed to this PEP as it singles out one (or two) special
operators that behave differently than any other operator w.r.t. *not*
being elementwise. The current behavior is much easier to understand.
(I'm also not a fan of @ being both an operator and being used for
decorators... but that's a different issue.)

Also, this is not just about multiplication (or exponentiation). Note
that for a matrix M and a scalar a the sum M + a is M + I a (where I
is the identity matrix), again not elementwise.

sage: import numpy as np
sage: a = np.array([[1,2],[3,4]])
sage: a
array([[1, 2],
[3, 4]])
sage: a + 100
array([[101, 102],
[103, 104]])

sage: M = matrix([[1,2],[3,4]]); M
[1 2]
[3 4]
sage: M + 100
[101 2]
[ 3 104]

- Robert