Cannot load session create in another version of sage

[SteveJJ]

I saved a session when using sage 8.0.

When I try to load that session into sage 8.8, I get errors such as:

• RuntimeError: invalid pickle data
  
• ValueError: unknown matrix implementation 'flint' over Symbolic Ring
  

The full error output is below.

Am I missing some packages or do I need to somehow convert the session file?

Attempt 1

sage: load_session("/home/user/sage/sage_session_1")
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-1-6224abcd403e> in <module>()
----> 1 load_session("/home/user/sage/sage_session_1")

/usr/lib64/python3.7/site-packages/sage/misc/session.pyx in sage.misc.session.load_session (build/cythonized/sage/misc/sessio
n.c:3687)()                                                                                                                   
   390             if os.path.exists(nm):
   391                 name = nm
--> 392     D = load(name)
   393     for k, x in D.items():
   394         state[k] = x

/usr/lib64/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2838
)()                                                                                                                           
   149  
   150     ## Load file by absolute filename
--> 151     with open(filename, 'rb') as fobj:
   152         X = loads(fobj.read(), compress=compress)
   153     try:

/usr/lib64/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2788
)()                                                                                                                           
   150     ## Load file by absolute filename
   151     with open(filename, 'rb') as fobj:
--> 152         X = loads(fobj.read(), compress=compress)
   153     try:
   154         X._default_filename = os.path.abspath(filename)

/usr/lib64/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:735
3)()                                                                                                                          
   967  
   968     unpickler = SageUnpickler(io.BytesIO(s))
--> 969     return unpickler.load()
   970  
   971  

/usr/lib64/python3.7/site-packages/sage/matrix/matrix0.pyx in sage.matrix.matrix0.unpickle (build/cythonized/sage/matrix/matr
ix0.c:38713)()                                                                                                                
  5709     A._cache = cache
  5710     if version >= 0:
-> 5711         A._unpickle(data, version)
  5712     else:
  5713         A._unpickle_generic(data, version)

/usr/lib64/python3.7/site-packages/sage/matrix/matrix_integer_dense.pyx in sage.matrix.matrix_integer_dense.Matrix_integer_de
nse._unpickle (build/cythonized/sage/matrix/matrix_integer_dense.c:8264)()                                                    
   541                 self._unpickle_matrix_2x2_version0(data)
   542             else:
--> 543                 raise RuntimeError("invalid pickle data")
   544         else:
   545             raise RuntimeError("unknown matrix version (=%s)"%version)

RuntimeError: invalid pickle data

Attempt 2

sage: load_session("/home/user/sage/sage_session_2")
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-dbd66d1d5c33> in <module>()
----> 1 load_session("/home/user/sage/sage_session_2")

/usr/lib64/python3.7/site-packages/sage/misc/session.pyx in sage.misc.session.load_session (build/cythonized/sage/misc/sessio
n.c:3687)()                                                                                                                   
   390             if os.path.exists(nm):
   391                 name = nm
--> 392     D = load(name)
   393     for k, x in D.items():
   394         state[k] = x

/usr/lib64/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2838
)()                                                                                                                           
   149  
   150     ## Load file by absolute filename
--> 151     with open(filename, 'rb') as fobj:
   152         X = loads(fobj.read(), compress=compress)
   153     try:

/usr/lib64/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2788
)()                                                                                                                           
   150     ## Load file by absolute filename
   151     with open(filename, 'rb') as fobj:
--> 152         X = loads(fobj.read(), compress=compress)
   153     try:
   154         X._default_filename = os.path.abspath(filename)

/usr/lib64/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:735
3)()                                                                                                                          
   967  
   968     unpickler = SageUnpickler(io.BytesIO(s))
--> 969     return unpickler.load()
   970  
   971  

/usr/lib64/python3.7/site-packages/sage/structure/unique_representation.py in unreduce(cls, args, keywords)
  1179  
  1180     """
-> 1181     return cls(*args, **keywords)
  1182  
  1183  

/usr/lib64/python3.7/site-packages/sage/misc/classcall_metaclass.pyx in sage.misc.classcall_metaclass.ClasscallMetaclass.__ca
ll__ (build/cythonized/sage/misc/classcall_metaclass.c:1714)()                                                                
   333         """
   334         if cls.classcall is not None:
--> 335             return cls.classcall(cls, *args, **kwds)
   336         else:
   337             # Fast version of type.__call__(cls, *args, **kwds)

/usr/lib64/python3.7/site-packages/sage/matrix/matrix_space.py in __classcall__(cls, base_ring, nrows, ncols, sparse, impleme
ntation)                                                                                                                      
   474             raise OverflowError("number of rows and columns may be at most %s" % sys.maxsize)
   475  
--> 476         matrix_cls = get_matrix_class(base_ring, nrows, ncols, sparse, implementation)
   477         return super(MatrixSpace, cls).__classcall__(
   478                 cls, base_ring, nrows, ncols, sparse, matrix_cls)

/usr/lib64/python3.7/site-packages/sage/matrix/matrix_space.py in get_matrix_class(R, nrows, ncols, sparse, implementation)
   306         # generic fallback
   307         if implementation != 'generic' and implementation is not None:
--> 308             raise ValueError("unknown matrix implementation %r over %r" % (implementation, R))
   309         else:
   310             return matrix_generic_dense.Matrix_generic_dense

ValueError: unknown matrix implementation 'flint' over Symbolic Ring

[Dima Pasechnik]

On Fri, Aug 30, 2019 at 12:48 PM 'SteveJJ' via sage-support
<sage-s...@googlegroups.com> wrote:
>
> I saved a session when using sage 8.0.
>
> When I try to load that session into sage 8.8, I get errors such as:
>
> RuntimeError: invalid pickle data
> ValueError: unknown matrix implementation 'flint' over Symbolic Ring
>
> The full error output is below.
>
> Am I missing some packages or do I need to somehow convert the session file?

I don't think that sage/load_session is guaranteed w.r.t. any kind of
backward/forward compatibility.
It relies on internals being unchanged, which is certainly not the
case from one Sage version to another.

> --
> You received this message because you are subscribed to the Google Groups "sage-support" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-support...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-support/8302eb8a-a1c4-4000-9c55-0fafa70ed9c8%40googlegroups.com.

[Simon King]

Hi Steve,

On Friday, August 30, 2019 at 11:55:45 AM UTC+2, Dima Pasechnik wrote:

On Fri, Aug 30, 2019 at 12:48 PM 'SteveJJ' via sage-support
<sage-s...@googlegroups.com> wrote:
>
> I saved a session when using sage 8.0.
>
> When I try to load that session into sage 8.8, I get errors such as:
>
> RuntimeError: invalid pickle data
> ValueError: unknown matrix implementation 'flint' over Symbolic Ring
>
> The full error output is below.
>
> Am I missing some packages or do I need to somehow convert the session file?

I don't think that sage/load_session is guaranteed w.r.t. any kind of
backward/forward compatibility.
It relies on internals being unchanged, which is certainly not the
case from one Sage version to another.

If I understand correctly, saving/loading individual objects is supposed to be backwards compatible. But saving whole sessions (which I never tried and even didn't know that it exists in Sage, btw.) probably not. There are systems (such as GAP), in which one can save the whole session but cannot save individual objects; that's a flawed approach, IMHO.

Best regards,
Simon

[Vincent Delecroix]

Le 30/08/2019 à 12:54, Simon King a écrit :
> Hi Steve,
>
> On Friday, August 30, 2019 at 11:55:45 AM UTC+2, Dima Pasechnik wrote:
>>
>> On Fri, Aug 30, 2019 at 12:48 PM 'SteveJJ' via sage-support
>> <sage-s...@googlegroups.com <javascript:>> wrote:
>>>
>>> I saved a session when using sage 8.0.
>>>
>>> When I try to load that session into sage 8.8, I get errors such as:
>>>
>>> RuntimeError: invalid pickle data
>>> ValueError: unknown matrix implementation 'flint' over Symbolic Ring
>>>
>>> The full error output is below.
>>>
>>> Am I missing some packages or do I need to somehow convert the session
>> file?
>>
>> I don't think that sage/load_session is guaranteed w.r.t. any kind of
>> backward/forward compatibility.
>> It relies on internals being unchanged, which is certainly not the
>> case from one Sage version to another.
>>
>
> If I understand correctly, saving/loading individual objects is supposed to
> be backwards compatible.

This is not correct. Saving individual object is not supposed to
be backward compatible. See

https://trac.sagemath.org/ticket/28302

Vincent

[Simon King]

Hi Vincent,

On 2019-09-03, Vincent Delecroix <20100.d...@gmail.com> wrote:
>> If I understand correctly, saving/loading individual objects is supposed to
>> be backwards compatible.
>
> This is not correct. Saving individual object is not supposed to
> be backward compatible. See
>
> https://trac.sagemath.org/ticket/28302

Thank you for the pointer.

I believe a CAS which doesn't even *attempt* to offer a way to store user
data permanently and reliably is a failure. I'll rant more on it on the
ticket.

Best regards,
Simon

[Vincent Delecroix]

I strongly agree that this is a needed feature! But, I was told on the
ticket that pickling is not the way to do it... I am not Python expert
enough to judge this.

Vincent

[Simon King]

Then I misunderstood. By your comment on the ticket, you found that
Python actually states that the pickle protocol IS supposed to be
compatible.

In fact it isn't compatible, but that of course means it must be
attempted to make it compatible.

Best regards,
Simon

[Vincent Delecroix]

Le 03/09/2019 à 11:28, Simon King a écrit :
> Hi Vincent,
>
> On 2019-09-03, Vincent Delecroix <20100.d...@gmail.com> wrote:
>>> I believe a CAS which doesn't even *attempt* to offer a way to store user
>>> data permanently and reliably is a failure. I'll rant more on it on the
>>> ticket.
>>
>> I strongly agree that this is a needed feature! But, I was told on the
>> ticket that pickling is not the way to do it... I am not Python expert
>> enough to judge this.
>
> Then I misunderstood. By your comment on the ticket, you found that
> Python actually states that the pickle protocol IS supposed to be
> compatible.

compatible up to pickling protocol versions and datastructure changes
between Python2 and Python3. See also the related Julien Puydt answer
on sage-devel.

> In fact it isn't compatible, but that of course means it must be
> attempted to make it compatible.

It is already compatible. You just need to specify which kind of
data has been pickled ('bytes' vs 'str') and which protocol to use
(at pickling and unpickling time).

[Simon King]

Hi Vincent,

On 2019-09-03, Vincent Delecroix <20100.d...@gmail.com> wrote:
>> In fact it isn't compatible, but that of course means it must be
>> attempted to make it compatible.
>
> It is already compatible. You just need to specify which kind of
> data has been pickled ('bytes' vs 'str')

I disagree. When you pickle a Python-2 str and unpickle with Python-3,
you get a Python-3 str. And that's a problem, as Python-2 str
corresponds to Python-3 bytes, whereas Python-3 str corresponds to
Python-2 unicode. Moreover, the attempt to unpickle a Python-2 str as a
Python-3 str can fail.

Of course it is possible that you tell Python-3 to unpickle Python-2 str
as bytes (namely by pickle.load(<file>, encoding="bytes")). But can we
really say that things are *compatible* if an apple (Python-2 str) is
unpickled as a banana (Python-3 str)? And can we really say that things
*are* compatible when compatibility can only be achieved by providing an
optional keyword argument?

Anyway. After I learned about the `encoding` keyword, I opened #28444,
where I propose to add the `encoding` keyword to Sage's `load` function,
so that it can be passed to pickle.load.

Best regards,
Simon