I can't load objets in Sage 3.2.3 that were created in Sage 3.1.1
Alex Lara
3.1.1. I got the following message:
DeprecationWarning: Your data is stored in an old format. Please use
the save() function to store your data in a more recent format.
Mike Hansen
They should still load. The warning just tells you that the format
has changed so you should resave them to get the new format. If they
don't actually load, could you please post a link to one of the .sobj
files?
--Mike
Alex Lara
open it with sage 3.2.3. Next I open test.sobj (this was created by a
sage program) in sage 3.1.1, saved it again, and when I tried to open
it on sage 3.2.3. Here you will find that object: http://math.arizona.edu/~alara/test.sobj
----Alex
Mike Hansen
>
> From command line of Sage 3.1.1, I saved some object, and then I could
> open it with sage 3.2.3. Next I open test.sobj (this was created by a
> sage program) in sage 3.1.1, saved it again, and when I tried to open
> it on sage 3.2.3. Here you will find that object: http://math.arizona.edu/~alara/test.sobj
>
> ----Alex
Thanks. I'll take a look at it. I've also CC'd Craig Citro who most
likely knows what is going on here.
--Mike
Craig Citro
>> open it with sage 3.2.3. Next I open test.sobj (this was created by a
>> sage program) in sage 3.1.1, saved it again, and when I tried to open
>> it on sage 3.2.3. Here you will find that object: http://math.arizona.edu/~alara/test.sobj
>>
>> ----Alex
>
> Thanks. I'll take a look at it. I've also CC'd Craig Citro who most
> likely knows what is going on here.
>
> --Mike
>
Hi Alex,
Yeah, this one is going to be a little bit annoying to fix. Here's
what's going on: in between 3.1.1 and 3.2.3, fraction fields got moved
over to the new coercion model. This is good, but as it happens, the
switch was done in such a way that it caused the pre-existing pickles
to not load correctly (which is exactly what you're seeing). There's
probably a clean and classy way to fix this, and I'll be happy to look
at it carefully in a few weeks if no one beats me to it (I'm busy
trying to finish up my thesis). This is now trac #5419.
In the interim, you probably want to hack something together so that
you can load your pickles. I managed to make sage 3.2.3 load the
pickles, but it's a pretty ugly hack, and you'll still need to re-save
the objects as newer pickles (which actually has one or two more
wrinkles), so that you don't have to deal with this in the short term.
How many of these files are there? If it's just a few, e-mail them to
me off-list, and I'll just re-pickle them in a newer format and send
them back to you. If there are more than that, I'll carefully explain
how you can fix them yourself. :)
-cc
Burcin Erocal
On Sun, 1 Mar 2009 22:41:08 -0800
Craig Citro <craig...@gmail.com> wrote:
>
> >> From command line of Sage 3.1.1, I saved some object, and then I
> >> could open it with sage 3.2.3. Next I open test.sobj (this was
> >> created by a sage program) in sage 3.1.1, saved it again, and
> >> when I tried to open it on sage 3.2.3. Here you will find that
> >> object: http://math.arizona.edu/~alara/test.sobj
> >>
> >> ----Alex
> >
> > Thanks. I'll take a look at it. I've also CC'd Craig Citro who
> > most likely knows what is going on here.
> >
> > --Mike
> >
>
> Hi Alex,
>
> Yeah, this one is going to be a little bit annoying to fix. Here's
> what's going on: in between 3.1.1 and 3.2.3, fraction fields got moved
> over to the new coercion model. This is good, but as it happens, the
> switch was done in such a way that it caused the pre-existing pickles
> to not load correctly (which is exactly what you're seeing). There's
> probably a clean and classy way to fix this, and I'll be happy to look
> at it carefully in a few weeks if no one beats me to it (I'm busy
> trying to finish up my thesis). This is now trac #5419.
I was the one who moved fraction fields to the new coercion model. This
was #4360 on trac. Since I also moved the element class to cython,
pickles broke with this change. This led to a fix for the general
pickling strategy in Sage, which used the make_element method in
sage.structure.element for all classes which didn't specify anything
else. This fix was #4698. The pickles for Fraction Fields worked after
these changes.
The deprecation message is just a warning. It's just there to
encourage the users to save their data in a newer format.
The error I get when I try to load the .sobj file linked above in
Sage-3.3 is:
...
/home/burcin/sage/sage-3.3/local/lib/python2.5/site-packages/sage/structure/sage_object.so
in sage.structure.sage_object.loads
(sage/structure/sage_object.c:6156)()
RuntimeError: (None, <function PolynomialRing at 0x1355e60>, (Fraction
Field of Univariate Polynomial Ring in t over Finite Field of size 13,
'T', None, False)) invalid data stream
invalid load key, 'x'.
Unable to load pickled data.
This looks a lot like the error we saw with your pickles at SD12. I
thought that was because we were trying to unpickle compressed data at
first?
> In the interim, you probably want to hack something together so that
> you can load your pickles. I managed to make sage 3.2.3 load the
> pickles, but it's a pretty ugly hack, and you'll still need to re-save
> the objects as newer pickles (which actually has one or two more
> wrinkles), so that you don't have to deal with this in the short term.
> How many of these files are there? If it's just a few, e-mail them to
> me off-list, and I'll just re-pickle them in a newer format and send
> them back to you. If there are more than that, I'll carefully explain
> how you can fix them yourself. :)
I don't have so much time to investigate this either, but if you
already found the cause then I could probably write a clean patch. Can
you explain how you managed to load the old files, or send your hackish
patch to me?
Cheers,
Burcin
Craig Citro
> The error I get when I try to load the .sobj file linked above in
> Sage-3.3 is:
>
> ...
> /home/burcin/sage/sage-3.3/local/lib/python2.5/site-packages/sage/structure/sage_object.so
> in sage.structure.sage_object.loads
> (sage/structure/sage_object.c:6156)()
>
> RuntimeError: (None, <function PolynomialRing at 0x1355e60>, (Fraction
> Field of Univariate Polynomial Ring in t over Finite Field of size 13,
> 'T', None, False)) invalid data stream
> invalid load key, 'x'.
> Unable to load pickled data.
>
Yep, this is the same error we had with the other pickles. This is a
wildly uninformative error message, which I think I know enough to fix
at this point (someone should file a ticket and assign it to me). All
this error really means is that cPickle ran into an error trying to
unpickle the object.
In more detail: Both Pickle and cPickle keep a stream of arguments as
they load an object, which is what gets stored in a pickle. They just
run in a loop, that basically says: if there's anything left in the
stream, pop off an argument, which is a key telling it what kind of
object is next, dispatch to a method that knows how to recreate that
object, pop the appropriate number of arguments off the stream, use
those to reconstruct the object, and then loop. I'm pretty sure what's
happening here is that a correct load key gets popped off the stream,
and the attempt to reconstruct that part of the object fails, at which
point the *key* is gone from the stream, but not all of the arguments,
and the exception we raise gets caught, so the attempt to unpickle
continues. However, now the stream has nonsense (specifically leftover
arguments) at the front of it, so on the next loop, we see the
"Invalid load key: 'x'" error. So all this error means is that
something went wrong unpickling.
Here's what's happening with Alex's 3.1.1 pickle. Actually, in the
process of explaining what happens in this case, I've realized it's
slightly more subtle than I thought (and in particular, some of these
details should be added to the trac ticket).
- The FractionField object Alex pickled in 3.1.1 was a pre-coercion
FractionField. So it doesn't have _element_class and
_element_constructor attributes, and it was created when FractionField
had no __reduce__ method -- so it just calls __new__ on the class, and
tries to start filling in the dict for the object. This goes fine,
actually -- the FractionField unpickles okay. Keep in mind, though,
that it's now an instance of the *new* FractionField class (i.e. one
with the new coercion stuff in place) that happens to be missing some
key attributes (like _element_class and _element_constructor -- which
are set to None).
- The next thing Alex's code tries to unpickle is a polynomial ring
over the FractionField. This has always had a __reduce__ method, which
stores a call to the PolynomialRing function in
sage.rings.polynomial.polynomial_ring_constructor.
- The polynomial ring constructor always creates a polynomial in the
process of initializing, namely its generator. This is accomplished by
calling self([0,1], is_gen=True), which ultimately calls [ R(z) for z
in [0,1] ], where R is self.base_ring().
- Since self.base_ring() is the fraction field, we try to dispatch
__call__ on that. Of course, the new FractionField class doesn't have
a __call__ method -- that's in Parent now. So it goes up the class
hierarchy and dispatches that __call__.
- The first thing that method does is say "if
self._element_constructor is None: raise NotImplementedError." This is
the error we hit, because our 3.1.1 FractionField object has this set
to None. Boom.
This could explain why you didn't hit this in testing the new code --
did you create some polynomial rings over fraction fields with the old
class, then try to unpickle them with the new one? I bet this would
fail.
> I don't have so much time to investigate this either, but if you
> already found the cause then I could probably write a clean patch. Can
> you explain how you managed to load the old files, or send your hackish
> patch to me?
>
So that's a pretty longwinded explanation for what's happening. Now,
the fix was easy: I commented out the two lines in Parent.__call__
which raise the NotImplementedError, and I copy-pasted the code for
FractionField.__call__ back in place. This is clearly a hack. :) In
fact, it's bad -- the new objects I create don't pickle correctly, or
even if they did, we'd have the same problem trying to load them.
However, they loaded their coefficients just fine -- so I'm going to
write something that will instantiate correctly formed classes, coerce
the old values over via lists, and then finally save the new,
correctly structured objects. A mild pain, but it seems
straightforward enough, and I didn't think of anything easier.
Hopefully that makes sense -- but it's 2:30AM here, so feel free to
write back and say "that didn't make any sense ... try again." :)
-cc
Alex Lara
Thanks for explanation, it was clear. Now I have six files, but the
program is still running and will generate more files. So, I would
love you can explain me how to fix the problem.
Thanks,
----Alex
Craig Citro
think it'll be easier to give you a patch against 3.1.1 than a patch
against 3.2.3 ...
-cc
Alex Lara
I have 3.2.3 in my laptop, but I use the machines of the University
(which have 3.1.1) to do computations that take long time.
---Alex
On 2 mar, 11:19, Craig Citro <craigci...@gmail.com> wrote:
> Ah, does this mean you have a running copy of 3.1.1 still? Because I
> think it'll be easier to give you a patch against 3.1.1 than a patch
> against 3.2.3 ...
>
> -cc
>
Craig Citro
Well, I have one question first before deciding the right way to
proceed. Are all the objects you're creating the same general shape as
the ones in the test.sobj you posted -- collections of polynomials
over rational function fields in one variable over finite fields? If
so, then we have two options; if not, we probably only have option
(2).
(1) We can patch 3.2.3, so you can load the objects in 3.2.3. I'll
then write a function that will take the objects, recreate
corresponding objects in 3.2.3, coerce the old ones over, and re-save
them. This isn't so bad, but it depends on me knowing exactly what
kind of objects you've got. It's slightly more error-prone, I think.
(2) We build and patch 3.1.1, and you just load and save each object.
Then they should load fine in 3.2.3.
Which seems better to you? (I think (2) is slightly easier for me, but
nominally more work for you -- you have to build 3.1.1 on your
laptop.)
-cc
Alex Lara
All the objects have the same general shape as the one posted. The
option (2) it is ok. I do not want you spend much time with my
problem.
Alex
On 2 mar, 12:21, Craig Citro <craigci...@gmail.com> wrote:
> Hi Alex,
>
> Well, I have one question first before deciding the right way to
> proceed. Are all the objects you're creating the same general shape as
> the ones in the test.sobj you posted -- collections of polynomials
> over rational function fields in one variable over finite fields? If
> so, then we have two options; if not, we probably only have option
> (2).
>
> (1) We can patch 3.2.3, so you can load the objects in 3.2.3. I'll
> then write a function that will take the objects, recreate
> corresponding objects in 3.2.3, coerce the old ones over, and re-save
> them. This isn't so bad, but it depends on me knowing exactly what
> kind of objects you've got. It's slightly more error-prone, I think.
>
> (2) We build and patch 3.1.1, and you just load and save each object.
> Then they should load fine in 3.2.3.
>
> Which seems better to you? (I think (2) is slightly easier for me, but
> nominally more work for you -- you have to build 3.1.1 on your
> laptop.)
>
> -cc
>
Alex Lara
All the objects have the same general shape as the one posted. The
option (2) it is ok. I do not want you spend much time with my
problem.
Alex
On 2 mar, 12:21, Craig Citro <craigci...@gmail.com> wrote:
>
> Well, I have one question first before deciding the right way to
> proceed. Are all the objects you're creating the same general shape as
> the ones in the test.sobj you posted -- collections of polynomials
> over rational function fields in one variable over finite fields? If
> so, then we have two options; if not, we probably only have option
> (2).
>
> (1) We can patch 3.2.3, so you can load the objects in 3.2.3. I'll
> then write a function that will take the objects, recreate
> corresponding objects in 3.2.3, coerce the old ones over, and re-save
> them. This isn't so bad, but it depends on me knowing exactly what
> kind of objects you've got. It's slightly more error-prone, I think.
>
> (2) We build and patch 3.1.1, and you just load and save each object.
> Then they should load fine in 3.2.3.
>
> Which seems better to you? (I think (2) is slightly easier for me, but
> nominally more work for you -- you have to build 3.1.1 on your
> laptop.)
>
> -cc
>
Craig Citro
> All the objects have the same general shape as the one posted. The
> option (2) it is ok. I do not want you spend much time with my
> problem.
>
Excellent -- (2) was definitely the easier plan for me. :) I've posted
a patch here:
http://sage.math.washington.edu/home/craigcitro/fix-frac-pickle-3.1.1.patch
So here are the steps you need to take to use this (I'm happy to
explain any of these in more detail, if you want):
1. Build sage-3.1.1 on your machine. It's here:
http://sage.math.washington.edu/sage/src/sage-3.1.1.tar
2. In your sage-3.1.1 build directory, type: ./sage -clone pickle
3. From that same directory: cd devel/sage-pickle
4. Copy the above patch file into that directory, and type: hg patch
fix-frac-pickle-3.1.1.patch
5. cd ../..
6. ./sage -br
You should now be running a copy of sage 3.1.1 with the patch applied.
Then you can just update the pickles like this:
sage: foo = load('test.sobj')
sage: save(foo, 'test2.sobj')
or even
sage: save(load('test.sobj'), 'test-new.sobj')
where it's going to look for the patches in the sage-3.1.1 directory.
(Of course, you could just run the copy of 3.1.1 you've built from
anywhere by giving the full path.) In fact, if you have a bunch of
files in the same directory, you could just do something like:
sage: for name in ['file1', 'file2', 'file3']:
...: save(load(name + '.sobj'), name + '-new.sobj')
and that should work. It takes a few seconds for each pickle, but it
seems to work for me.
Now, the new pickles you produce will load just fine in sage 3.2.3 (or
at least, they did for me) -- however, they'll still give you a
DeprecationWarning when you load them. You can ignore this, or just
load and re-save them again in sage 3.2.3.
Let me know if you run into any trouble ...
-cc
Alex Lara
Every thing works ok. I already did a proof with one object and there
was no problem. There was no DeprecationWarning.
Thanks a lot, you're a genius man.
I'll abuse of your amability, and ask a new question. Please, if you
do not have time, don't worry. I know you are very busy with your
thesis (I am too, so I understand)
Do you know the sagetex package? I followed the directions to use the
sagetex package, without successful. I tried with the example and I
got the following messages:
RuntimeError: Unable to start maxima
**** Running Sage on example.sage failed! Fix example.tex and try
again.
Thanks,
---Alex
On Mar 3, 6:02 pm, Craig Citro <craigci...@gmail.com> wrote:
> Hi Alex,
>
> > All the objects have the same general shape as the one posted. The
> > option (2) it is ok. I do not want you spend much time with my
> > problem.
>
> Excellent -- (2) was definitely the easier plan for me. :) I've posted
> a patch here:
>
Craig Citro
> Every thing works ok. I already did a proof with one object and there
> was no problem. There was no DeprecationWarning.
>
> Thanks a lot, you're a genius man.
>
Awesome! I'm glad you got those objects loaded. :)
> Do you know the sagetex package? I followed the directions to use the
> sagetex package, without successful. I tried with the example and I
> got the following messages:
>
Honestly, I've never used sagetex myself -- I hear it's really cool.
Dan Drake is the man to ask -- I'm pretty sure he reads sage-support,
but he may have stopped reading this thread at some point. I'd
recommend starting a new thread to ask him, and you'll probably hear
something really quick. If that doesn't work, he hangs out in the sage
IRC channel a lot -- #sage-devel on irc.freenode.net ... you can
surely find him there. (He's in Korea, so he keeps different hours.)
-cc
Jason Grout
When you start that new thread, it would be really helpful if you could
post a minimal tex document that gives you the error and exactly the
commands you used when you received the error.
Sagetex is great!
Jason