[ARM] pickling problem : help needed
Julien Puydt
I ran "make ptestlong" again ; this time with the SAGE_PICKLE_JAR
environment variable.
Now, here is what I have for example in ptestlong.log :
* unpickle failure:
load('/home/jpuydt/.sage/temp/hecke/26329/dir_2/pickle_jar/_class__sage_coding_linear_code_LinearCode__.sobj')
and here is what happens when I try to load from the pickle jar:
sage:
load('/home/jpuydt/sage-4.8/tmp/pickle_jar/_class__sage_coding_linear_code_LinearCode__.sobj')
Linear code of length 7, dimension 4 over Finite Field of size 2
And that situation is the same for all pickling errors we have on ARM!
So how does one explain that the same thing fails within doctest, but
works from inside sage? Is it possible to get a more explicit error
message than just "unpickle failure"?
Snark on #sagemath
Dima Pasechnik
Dima Pasechnik
Willem Jan Palenstijn
Displaying the actual unpickle exception, I get:
('value too large to convert to char', <built-in function unpickle_matrix_mod2_dense_v1>, (4, 7, [137, 80, 78, 71, 13, 10, 26, 10, 0, 0, 0, 13, 73, 72, 68, 82, 0, 0, 0, 7, 0, 0, 0, 4, 1, 3, 0, 0, 0, 120, 8, 183, 62, 0, 0, 0, 6, 80, 76, 84, 69, 0, 0, 0, 255, 255, 255, 165, 217, 159, 221, 0, 0, 0, 16, 73, 68, 65, 84, 8, 153, 99, 144, 99, 72, 99, 88, 197, 160, 3, 0, 5, 2, 1, 91, 3, 248, 139, 52, 0, 0, 0, 0, 73, 69, 78, 68, 174, 66, 96, 130], 91))
So this is the same issue with char being unsigned on your platform.
Specifically, it seems the corresponding __reduce__ function converts a char
array into a list of python ints, which has range 0-255 for you, instead of the
-128-127 on x86. And the unpickle function has the opposite issue.
I wonder how many of these are around, and if there is any general fix for
them.
-Willem Jan
Dima Pasechnik
Willem Jan Palenstijn
> Are you saying it's actually a Python bug, not a Sage bug?
(Aside: please quote at least part of the message you're replying to.
I'll assume you're asking me...)
No, not python, but the custom reduce/unpickle functions we have for
matrix_mod2_dense.
-Willem Jan
Julien Puydt
> So this is the same issue with char being unsigned on your platform.
> Specifically, it seems the corresponding __reduce__ function converts a char
> array into a list of python ints, which has range 0-255 for you, instead of the
> -128-127 on x86. And the unpickle function has the opposite issue.
>
> I wonder how many of these are around, and if there is any general fix for
> them.
Well, if there is a nice inheritance graph, there's hope fixing the base
class will maket the rest just go smooth.
Now, to fix the base class... If the goal is to take a "char" on one
platform and turn it into a "char" on another, then it's a lost battle.
What are those "char" really encoding? If they are really "foo", then
turning a "foo" on one platform into a "foo" on another might not be
impossible!
Snark on #sagemath
Willem Jan Palenstijn
> Le 29/01/2012 12:13, Willem Jan Palenstijn a �crit :
> >So this is the same issue with char being unsigned on your platform.
> >Specifically, it seems the corresponding __reduce__ function converts a char
> >array into a list of python ints, which has range 0-255 for you, instead of the
> >-128-127 on x86. And the unpickle function has the opposite issue.
> >
> >I wonder how many of these are around, and if there is any general fix for
> >them.
>
> Well, if there is a nice inheritance graph, there's hope fixing the
> base class will maket the rest just go smooth.
>
> Now, to fix the base class... If the goal is to take a "char" on one
> platform and turn it into a "char" on another, then it's a lost
> battle.
Which base class do you mean exactly?
> What are those "char" really encoding? If they are really "foo",
> then turning a "foo" on one platform into a "foo" on another might
> not be impossible!
In this particular case, it seems Matrix_mod2_dense's __reduce__ serializes its
data by compressing it as a black/white PNG image. So it's really just a raw
byte stream.
-Willem Jan
Dima Pasechnik
On Sunday, 29 January 2012 20:05:26 UTC+8, Willem Jan Palenstijn wrote:
On Sat, Jan 28, 2012 at 04:36:17PM +0100, Julien Puydt wrote:
> Le 29/01/2012 12:13, Willem Jan Palenstijn a �crit :
> >So this is the same issue with char being unsigned on your platform.
> >Specifically, it seems the corresponding __reduce__ function converts a char
> >array into a list of python ints, which has range 0-255 for you, instead of the
> >-128-127 on x86. And the unpickle function has the opposite issue.
> >
> >I wonder how many of these are around, and if there is any general fix for
> >them.
>
> Well, if there is a nice inheritance graph, there's hope fixing the
> base class will maket the rest just go smooth.
>
> Now, to fix the base class... If the goal is to take a "char" on one
> platform and turn it into a "char" on another, then it's a lost
> battle.Which base class do you mean exactly?
> What are those "char" really encoding? If they are really "foo",
> then turning a "foo" on one platform into a "foo" on another might
> not be impossible!In this particular case, it seems Matrix_mod2_dense's __reduce__ serializes its
data by compressing it as a black/white PNG image. So it's really just a raw
byte stream.
-Willem Jan
Julien Puydt
> and then it does (line 1750):
>
> cdef char *buf = <char*>gdImagePngPtr(im, &size)
> data = [buf[i] for i in range(size)]
>
> and this data goes into the pickle. No wonder it gets different on
> different platforms!
Would that also be a case of just using uint8_t and everything will be ok?
Snark on #sagemath
Martin Albrecht
The new version of M4Ri has PNG support built in directly, but I'm not sure it
makes things better or worse. It uses libPNG directly though.
for(rci_t i=0; i<m; i++) {
png_read_row(png_ptr, row, NULL);
for(j=0; j<A->width-1; j++) {
tmp = ((word)row[8*j+7])<<56 | ((word)row[8*j+6])<<48 \
| ((word)row[8*j+5])<<40 | ((word)row[8*j+4])<<32 \
| ((word)row[8*j+3])<<24 | ((word)row[8*j+2])<<16 \
| ((word)row[8*j+1])<< 8 | ((word)row[8*j+0])<< 0;
A->rows[i][j] = ~tmp;
}
tmp = 0;
switch((n/8 + ((n%8) ? 1 : 0))%8) {
case 7: tmp |= ((word)row[8*j+7])<<56;
case 6: tmp |= ((word)row[8*j+6])<<48;
case 5: tmp |= ((word)row[8*j+5])<<40;
case 4: tmp |= ((word)row[8*j+4])<<32;
case 3: tmp |= ((word)row[8*j+3])<<24;
case 2: tmp |= ((word)row[8*j+2])<<16;
case 1: tmp |= ((word)row[8*j+1])<< 8;
case 0: tmp |= ((word)row[8*j+0])<< 0;
};
A->rows[i][j] |= (~tmp & bitmask_end);
}
It's not in Sage yet, but I could cut a release if it helps.
Cheers,
Martin
--
name: Martin Albrecht
_pgp: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x8EF0DC99
_otr: 47F43D1A 5D68C36F 468BAEBA 640E8856 D7951CCF
_www: http://martinralbrecht.wordpress.com/
_jab: martinr...@jabber.ccc.de
Dima Pasechnik
On Sunday, 29 January 2012 00:34:28 UTC+8, Snark wrote:
Le 29/01/2012 13:38, Dima Pasechnik a �crit :
> and then it does (line 1750):
>
> cdef char *buf = <char*>gdImagePngPtr(im, &size)
> data = [buf[i] for i in range(size)]
>
> and this data goes into the pickle. No wonder it gets different on
> different platforms!Would that also be a case of just using uint8_t and everything will be ok?
Snark on #sagemath
Julien Puydt
>
>
> On Sunday, 29 January 2012 00:34:28 UTC+8, Snark wrote:
>
> > and then it does (line 1750):
> >
> > cdef char *buf = <char*>gdImagePngPtr(im, &size)
> > data = [buf[i] for i in range(size)]
> >
> > and this data goes into the pickle. No wonder it gets different on
> > different platforms!
>
> Would that also be a case of just using uint8_t and everything will
> be ok?
>
> rather, int8_t (char = unsigned char on ARM, right?)
> Cause int8_t would retain the compatibility with the existing pickles
> created on x86.
Fair point. Are you working on it?
> I guess we should write a sage-devel post asking people to look at other
> cases of this they are aware about....
Well, that won't hurt. At least the doctest didn't seem to uncover other
such cases.
Snark on #sagemath
Volker Braun
Cause int8_t would retain the compatibility with the existing pickles created on x86.I guess we should write a sage-devel post asking people to look at other cases of this they are aware about....