problem with matrix over finite field

[Hal Snyder]

This works on sage-8.5:

sage: a = var('a')

....: matrix(GF(25, a), [[1,0], [0, 1]])

....: 

[1 0]

[0 1]

but not on sage-8.6 or later:

sage: a = var('a')

....: matrix(GF(25, a), [[1,0], [0, 1]])

....: 

p025.zzz: No such file or directory

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

<ipython-input-1-337d96d24b11> in <module>()

      1 a = var('a')

----> 2 matrix(GF(Integer(25), a), [[Integer(1),Integer(0)], [Integer(0), Integer(1)]])

/ext/sage/sage-8.8_1804/local/lib/python2.7/site-packages/sage/matrix/constructor.pyx in sage.matrix.constructor.matrix (build/cythonized/sage/matrix/constructor.c:2417)()

    623       :class:`MatrixArgs`, see :trac:`24742`

    624     """

--> 625     return MatrixArgs(*args, **kwds).matrix()

    626 

    627 

/ext/sage/sage-8.8_1804/local/lib/python2.7/site-packages/sage/matrix/args.pyx in sage.matrix.args.MatrixArgs.matrix (build/cythonized/sage/matrix/args.c:7765)()

    656                     break

    657         else:

--> 658             M = self.space(self, coerce=convert)

    659 

    660         # Also store the matrix to support multiple calls of matrix()

/ext/sage/sage-8.8_1804/local/lib/python2.7/site-packages/sage/matrix/matrix_space.pyc in __call__(self, entries, coerce, copy)

    815             [t]

    816         """

--> 817         return self.element_class(self, entries, copy, coerce)

    818 

    819     def change_ring(self, R):

/ext/sage/sage-8.8_1804/local/lib/python2.7/site-packages/sage/matrix/matrix_gfpn_dense.pyx in sage.matrix.matrix_gfpn_dense.Matrix_gfpn_dense.__init__ (build/cythonized/sage/matrix/matrix_gfpn_dense.c:5245)()

    427         cdef long nc = ma.ncols

    428 

--> 429         self.Data = MatAlloc(fl, nr, nc)

    430         self._converter = FieldConverter(ma.base)

    431 

RuntimeError: Cannot select field GF(25) in file matcore.c (line 130)

This is on Ubuntu 18.04, on CoCalc.

[Dima Pasechnik]

it works for me:

$ ./sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 8.9.beta1, Release Date: 2019-07-03 │
│ Using Python 2.7.15. Type "help()" for help. │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable. ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
sage: var('a'); matrix(GF(25,a), [[1,0], [0, 1]])
a
[1 0]
[0 1]

as well as

sage: a=var('a'); matrix(GF(25,a), [[1,0], [0, 1]])
[1 0]
[0 1]

> --
> You received this message because you are subscribed to the Google Groups "sage-support" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-support...@googlegroups.com.
> To post to this group, send email to sage-s...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-support.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-support/edfa5067-b473-4242-8879-1a35a93cca33%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Simon King]

Hi Hal,

Apparently this Sage installation uses MeatAxe as a backend for matrices
over finite non-prime fields. The file p025.zzz is supposed to be
automatically written to $DOT_SAGE/meataxe

So, I wonder: What is DOT_SAGE in your Sage environment? Is there
trouble with writing permissions? Or perhaps blank-space in the path
name?

For comparison (the value of DOT_SAGE can be obtained inside a Sage
shell or alternatively in a usual Sage session):

sage: DOT_SAGE
'/home/king/.sage/'
sage: !ls /home/king/.sage/meataxe
p002.zzz p005.zzz p013.zzz p025.zzz p064.zzz p125.zzz p243.zzz
p003.zzz p007.zzz p017.zzz p027.zzz p081.zzz p127.zzz
p004.zzz p009.zzz p019.zzz p049.zzz p121.zzz p169.zzz

Best regards,
Simon

[Harald Schilly]

I've no idea why this is happening on CoCalc. My first guess is there is an optional package installed, which is causing this.

Interestingly, this works:

matrix(GF(16), [[1,0], [0, 1]]) while matrix(GF(25), [[1,0], [0, 1]]) fails with the above error. 

[Harald Schilly]

On Friday, July 12, 2019 at 11:48:40 AM UTC+2, Simon King wrote: 

sage: DOT_SAGE
'/home/king/.sage/'

It's

sage: DOT_SAGE

'/home/user/.sage/' 

Which I bet is the wrong place to look. (all cocalc projects are essentially linux docker containers, where /home/user is $HOME, though)

[Harald Schilly]

[Dima Pasechnik]

Hi Simon,
isn't MeatAxe interfaced via a library, rather than via files?!
Doing arithmetics on small matrices storing them on a disk is insanely
inefficient...

Dima

> --
> You received this message because you are subscribed to the Google Groups "sage-support" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-support...@googlegroups.com.
> To post to this group, send email to sage-s...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-support.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-support/qg9l1e%241n6b%241%40blaine.gmane.org.

[Vincent Delecroix]

Le 12/07/2019 à 11:56, Harald Schilly a écrit :
> I've no idea why this is happening on CoCalc. My first guess is there is an
> optional package installed, which is causing this.

As Simon already said, there is meataxe installed. While

> Interestingly, this works:
>
> matrix(GF(16), [[1,0], [0, 1]]) while matrix(GF(25), [[1,0], [0, 1]]) fails
> with the above error.

This makes sense as their are tons of various implementations for
matrices.

sage: type(matrix(GF(16),[1]))
<class 'sage.matrix.matrix_gf2e_dense.Matrix_gf2e_dense'>
sage: type(matrix(GF(25),[1]))
<class 'sage.matrix.matrix_generic_dense.Matrix_generic_dense'>

The last one would be different with meataxe installed.

[Harald Schilly]

Ok... To me, this sounds like we have to uninstall MeatAxe on CoCalc.

The CoCalc setup is quite easy to explain, and so far I wasn't aware of any issues. Sage is in a globally shared read-only directory /ext/sage/sage-<version> and permissions are set such that a user "user" can read/exec it. Hence, only what's installed into this directory is available to a user/project. I checked, and even on the machine where Sage is built for CoCalc, there is neither a subdirectory in $DOT_SAGE/meataxe, nor can I find that missing file in the sources. 

I'm wondering which other packages depend on it...

I'll try to install MeatAxe again, though. Maybe that reveals a problem … but in that case I think it's weird that while the installation is broken, the branching for selecting the matrix implementation is still trying to pick it.

-- h

[Dima Pasechnik]

by the way, libgap will get a direct interface to GAP matrices, which
might be a useful matrix backend for various purposes. See

https://github.com/gap-system/gap/pull/3554

> --
> You received this message because you are subscribed to the Google Groups "sage-support" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-support...@googlegroups.com.
> To post to this group, send email to sage-s...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-support.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-support/cd0c9b80-48d0-4bea-b27b-6d1f0b265ccf%40googlegroups.com.

[Simon King]

Hi Harald,

On 2019-07-12, Harald Schilly <harald....@gmail.com> wrote:
> On Friday, July 12, 2019 at 11:48:40 AM UTC+2, Simon King wrote:
>>
>> sage: DOT_SAGE
>> '/home/king/.sage/'
>
> It's
>
> sage: DOT_SAGE
> '/home/user/.sage/'

Do you literally mean "user", not the specific name of a single user?
Does that perhaps mean that some users have no write permission to
it? That would explain why the files are missing (because they are
created when missing, which requires write permission).

Cheers,
Simon

[Vincent Delecroix]

Dear Hal,

For the time being, you can use the following workaround

matrix(GF(25, a), [[1,0], [0, 1]], implementation='generic')

[Simon King]

Hi Dima,

On 2019-07-12, Dima Pasechnik <dim...@gmail.com> wrote:
> Hi Simon,
> isn't MeatAxe interfaced via a library, rather than via files?!
> Doing arithmetics on small matrices storing them on a disk is insanely
> inefficient...

Maybe. Do you mean one should instead compute a new multiplication table
in memory in each individual Sage session, rather than reloading
previously computed tables? Would that be more efficient?

Reloading is the design choice of MeatAxe, which I preserved when I
forked it to create SharedMeatAxe.

Of course, it might be possible to diverge even further from upstream
and change SharedMeatAxe (which is used in Sage) to not use these files.

Best regards,
Simon

[Simon King]

Hi Harald,

On 2019-07-12, Harald Schilly <harald....@gmail.com> wrote:

> Ok... To me, this sounds like we have to uninstall MeatAxe on CoCalc.
>
> The CoCalc setup is quite easy to explain, and so far I wasn't aware of any
> issues. Sage is in a globally shared read-only directory
> /ext/sage/sage-<version> and permissions are set such that a user "user"
> can read/exec it.

But not write? Then there is the underlying reason why meataxe doesn't
work. I created the SharedMeatAxe wrapper under the assumption that a
user has write permissions in DOT_SAGE.

> I'm wondering which other packages depend on it...

p_group_cohomology. I guess that's all. And of course matrix arithmetic
over non-prime finite fields is a lot faster with MeatAxe as backend
than with Sage's generic matrix implementation (that is used when
MeatAxe is not installed).

> I'll try to install MeatAxe again, though. Maybe that reveals a problem …
> but in that case I think it's weird that while the installation is broken,
> the branching for selecting the matrix implementation is still trying to
> pick it.

I don't believe the installation is broken. It is just that the
installation relies on a contract that is broken on CoCalc (if missing
write permission to DOT_SAGE is the problem).

Best regards,
Simon

[Simon King]

Hi Harald,

On 2019-07-12, Harald Schilly <harald....@gmail.com> wrote:

> Interestingly, this works:
>
> matrix(GF(16), [[1,0], [0, 1]]) while matrix(GF(25), [[1,0], [0, 1]]) fails
> with the above error.

MeatAxe is only used for finite non-prime fields of order <255, and I
forgot to add: Only in odd characteristic (in characteristic 2, M4RIE is
used).

Best regards,
Simon

[Harald Schilly]

On Fri, Jul 12, 2019 at 3:38 PM Simon King <simon...@uni-jena.de> wrote:
> > sage: DOT_SAGE
> > '/home/user/.sage/'
>
> Do you literally mean "user", not the specific name of a single user?

yes, all cocalc projects run under the same user "user" in their own container.

$ echo $USER
user
$ id
uid=2001(user) gid=2001(user) groups=2001(user)

> Does that perhaps mean that some users have no write permission to
> it?

No, /home/user is a read/write mounted volume and ~/.sage is a
directory inside of it. i.e.

$ ls -ld ~/.sage
drwx------ 14 user user 20 Aug 19 2018 /home/user/.sage

$ touch ~/.sage/foo
$ ls -l ~/.sage/foo
-rw-r--r-- 1 user user 0 Jul 12 17:38 /home/user/.sage/foo

> That would explain why the files are missing (because they are
> created when missing, which requires write permission).

Is there a way to explicitly trigger the creation of such a file?
Maybe some error pops up that reveals more information.

> > Sage is in a globally shared read-only directory
> > /ext/sage/sage-<version> and permissions are set such that a user "user"
> > can read/exec it.
>
> But not write? Then there is the underlying reason why meataxe doesn't
> work. I created the SharedMeatAxe wrapper under the assumption that a
> user has write permissions in DOT_SAGE.

No, users cannot write to /ext. But DOT_SAGE points to
/home/user/.sage, which is read-write, though.

$ sage -c 'print DOT_SAGE'
/home/user/.sage/

$ sage -c 'matrix(GF(25), [[1]])'

p025.zzz: No such file or directory

Traceback (most recent call last):
[...]

[Simon King]

Hi Harald,

On 2019-07-12, Harald Schilly <har...@schil.ly> wrote:
>> Do you literally mean "user", not the specific name of a single user?
>
> yes, all cocalc projects run under the same user "user" in their own container.

> No, /home/user is a read/write mounted volume and ~/.sage is a
> directory inside of it. i.e.
>
> $ ls -ld ~/.sage
> drwx------ 14 user user 20 Aug 19 2018 /home/user/.sage
>
> $ touch ~/.sage/foo
> $ ls -l ~/.sage/foo
> -rw-r--r-- 1 user user 0 Jul 12 17:38 /home/user/.sage/foo

So, users ARE able to write to /home/user/.sage/meataxe, right? Then I
don't see what goes wrong here. Is there a symbolic link involved? I
could imagine that MeatAxe needs to be given an absolute path.

>> That would explain why the files are missing (because they are
>> created when missing, which requires write permission).
>
> Is there a way to explicitly trigger the creation of such a file?
> Maybe some error pops up that reveals more information.

In very old MeatAxe versions, there was an executable called maketab
that was responsible for table creation. But now it is just a library
function. The trigger is: Create a matrix over a non-prime finite field
of size <255 in odd characteristic.

Anyway, I created #28188 and would appreciate getting some explanation
on the specific situation on CoCalC and some feedback on the potential
ways out (such as: Keep the arithmetic tables in memory, not on disk).

Best regards,
Simon