Problem when trying to use shedskin

[Piotr Tarsa]

Hi,

I'm the author of https://github.com/tarsa/TarsaLZP project and I want to test ShedSkin on my code. Unfortunately I'm not able to use it because of error during using shedskin. The log of errors shown during translation is here: http://pastebin.com/SbC2abWJ

Regards,

Piotr

[Mark Dufour]

Hi Piotr,

Thanks a lot for reporting, and sharing your code. I had a quick look, and got the program to compile and run with a few minor fixes:

-make Lg2.lgLut and Common.CostScale globals, rather than class attributes
-in Common.__init__, pass list comprehensions to array.array instead of generator expressions

I will have a look at fixing these problems, or at least have Shedskin output warnings for when this is impossible..

Btw, your program might be interesting to add to shedskin/examples.. :-) I will play a bit with it in any case when I have more time.

Thanks again,
Mark.

--
You received this message because you are subscribed to the Google Groups "shedskin-discuss" group.
To view this discussion on the web visit https://groups.google.com/d/msg/shedskin-discuss/-/0nUO418e6xcJ.
To post to this group, send email to shedskin...@googlegroups.com.
To unsubscribe from this group, send email to shedskin-discu...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/shedskin-discuss?hl=en.

--
http://www.youtube.com/watch?v=E6LsfnBmdnk

[Mark Dufour]

oh, another thing, in Streams.py there is a call to 'open', open(self.filename, mode='w+b', buffering=65536), which needs to be changed to open(self.filename, 'w+b') to work correctly. I will also try and fix this before the next release of shedskin, or if anyone else is interested..

mark.

--
http://www.youtube.com/watch?v=E6LsfnBmdnk

[Piotr Tarsa]

Thank you for your response. Unfortunately I couldn't get shedskin to work, even after fixes. Could you send me the project after your fixes?

FYI two tests from test suite are failing on my system.

[Mark Dufour]

hi piotr,

On Wed, Dec 5, 2012 at 9:44 PM, Piotr Tarsa <piotr...@gmail.com> wrote:

Thank you for your response. Unfortunately I couldn't get shedskin to work, even after fixes. Could you send me the project after your fixes?

please see attached patch.
 

FYI two tests from test suite are failing on my system.

where can I find these tests..? :)

thanks,
mark.

--
http://www.youtube.com/watch?v=E6LsfnBmdnk

[Piotr Tarsa]

Hi Mark,

I've meant the tests that are bundled with ShedSkin itself. Two of them failed.

I've applied the patch but still the same AttributeError happens (ie the same that I've posted in first post). I have Ubuntu 12.04 64-bit and I've installed Boehm GC and PCRE from Ubuntu repository.

[Mark Dufour]

hi piotr,

that's strange, are you sure? does the following give the exact same error..?

git clone https://github.com/tarsa/TarsaLZP.git
cd TarsaLZP
git checkout 7314a114ecbed24ff1917b376df2227eefd86078
git apply ../0001-workarounds-for-shedskin.patch
cd python/TarsaLZP/
shedskin TarsaLZP
make
./TarsaLZP

if so, please send the backtrace, so we can figure out where exactly it's failing now..

thanks!
mark.

--
You received this message because you are subscribed to the Google Groups "shedskin-discuss" group.

To view this discussion on the web visit https://groups.google.com/d/msg/shedskin-discuss/-/bFzPMoN44cIJ.

To post to this group, send email to shedskin...@googlegroups.com.
To unsubscribe from this group, send email to shedskin-discu...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/shedskin-discuss?hl=en.

--
http://www.youtube.com/watch?v=E6LsfnBmdnk

[Mark Dufour]

hi piotr,

I profiled the compiled TarsaLZP a bit, and found that I had forgotten about an important optimization in array.array.. using shedskin GIT, TarsaLZP should be about twice as fast now. I still have one more optimization up my sleeve, but not sure how much that will help.

thanks in any case for triggering!

btw, thanks also for mentioning the failing tests. which ones are failing exactly, and can you see why? (difference between python version and compiled version)

mark.

To view this discussion on the web visit https://groups.google.com/d/msg/shedskin-discuss/-/I17krt-Wj1oJ.

To post to this group, send email to shedskin...@googlegroups.com.
To unsubscribe from this group, send email to shedskin-discu...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/shedskin-discuss?hl=en.

--
http://www.youtube.com/watch?v=E6LsfnBmdnk

[Piotr Tarsa]

I've followed exactly the steps you've provided and shedskin managed to translate the program successfully. Unfortunately the resulting program produces wrong compressed output (ie correct filesize, wrong content). When doing decoding also following warning appears:

GC Warning: Repeated allocation of very large block (appr. size 268439552):

May lead to memory leak and poor performance.

But if I provide the correct input then the resulting binary decodes it to correct output.

(so to recap: when encoding resulting binary produces wrong output; when decoding output is fine, but there is warning from GC)

after running run.py there are two errors listed in summary:

*** tests failed: 2
[172, 199]

After rerunning:

piotrek@p5q-pro:~/devel/shedskin/shedskin-0.9.2/tests$ ./run.py -l 172 > 172.txt
Traceback (most recent call last):
  File "./run.py", line 104, in run_test
    check_output(command, 'python %d.py' % test_nr)
  File "./run.py", line 171, in check_output
    raise AssertionError
AssertionError
piotrek@p5q-pro:~/devel/shedskin/shedskin-0.9.2/tests$ ./run.py -l 199 > 199.txt
Traceback (most recent call last):
  File "199.py", line 12, in <module>
    decompressor(open('testdata/binas', 'rb'))
IOError: [Errno 2] No such file or directory: 'testdata/binas'
Traceback (most recent call last):
  File "./run.py", line 104, in run_test
    check_output(command, 'python %d.py' % test_nr)
  File "./run.py", line 171, in check_output
    raise AssertionError
AssertionError

172.txt: http://pastebin.com/CrjkL5wK

199.txt: http://pastebin.com/Lxq4hzTw

[Mark Dufour]

hi piotr,

program produces wrong compressed output (ie correct filesize, wrong content). When doing decoding also following warning appears:

thanks! I will see if I can find the problem.. note that the GC warning can be turned off with shedskin -g. this problem occurs sometimes, and often it's not really a problem (though the program may run a bit slower).

172.txt: http://pastebin.com/CrjkL5wK

199.txt: http://pastebin.com/Lxq4hzTw

ah, I see. hmm sorry I should have probably noted this sooner: 172 are tests for the random module, which iirc works a bit differently since 2.7 or so. if anyone would like to look into this.. :P the problem with 199 has been fixed in git some time after 0.9.2. I simply forgot to commit the file it is trying to open.. so in any case, the ecoding problem is not related to these.


thanks again,
mark.
--
http://www.youtube.com/watch?v=E6LsfnBmdnk

[Mark Dufour]

hi piotr,

it seems the program assumes arbitrary-size or unsigned arithmetic, while shedskin uses 32-bit signed arithmetic by default. so for example, adding 1946648576 and 270548992 becomes negative. using shedskin -l (64-bit signed arithmetic) it seems to work better. though that makes it twice as slow here..

cheers,
mark.

To view this discussion on the web visit https://groups.google.com/d/msg/shedskin-discuss/-/HxIlN5oIrMMJ.

To post to this group, send email to shedskin...@googlegroups.com.
To unsubscribe from this group, send email to shedskin-discu...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/shedskin-discuss?hl=en.

--
http://www.youtube.com/watch?v=E6LsfnBmdnk

[Piotr Tarsa]

hi,

thanks for information. after changing:

if self.rcBuffer > 0x7fffffff:

to

if self.rcBuffer > 0x7fffffff or self.rcBuffer < 0:

in Encoder.addWithCarry program works correctly (at least a quick test shows that).

is there a chance that shedskin would work without moving constants and constant tables to outside of classes? also i think that using list comprehensions instead of generators can have negative impact on memory usage when it comes to CPython, Jython and maybe PyPy.

[Mark Dufour]

hi piotr,

is there a chance that shedskin would work without moving constants and constant tables to outside of classes? also i think that using list

well, the class variable CostScale doesn't actually have to be made global, if you replace all usages of 'self.CostScale' by 'Common.CostScale'. the problem with Lg2.lgLut is also not that it's a class attribute, but that a for loop is used to fill it at the 'class level'.

it will be some work, but I think it's quite doable to support for loops and other constructs at this level.. I will have a look at this for 0.9.3. if I don't improve the situation before then, I should at least add some proper warning messages (both for 'self.CostScale' and the for loop at least).
 

comprehensions instead of generators can have negative impact on memory usage when it comes to CPython, Jython and maybe PyPy.

yes, this is just working around a bug in shedskin. I will have a look at fixing this properly for 0.9.3.

thanks!
mark.
 

To view this discussion on the web visit https://groups.google.com/d/msg/shedskin-discuss/-/TlI9E7sHONwJ.

To post to this group, send email to shedskin...@googlegroups.com.
To unsubscribe from this group, send email to shedskin-discu...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/shedskin-discuss?hl=en.

--
http://www.youtube.com/watch?v=E6LsfnBmdnk

[Piotr Tarsa]

hi mark,

I'm waiting for 0.9.3 then :)

As for the loop, I can move the initialization code to some static method and call that method manually at startup. It will be less elegant, but maybe that would save you headaches.

[Mark Dufour]

hi piotr,

As for the loop, I can move the initialization code to some static method and call that method manually at startup. It will be less elegant, but maybe that would save you headaches.

I just made some improvements so the for loop in class Lg2 now works unmodified. in fact a lot more things now work at this level. fortunately it turned out not to be very hard. thanks for triggering this!

I also fixed the generator bug. generator expressions are still a bit slower than lists here, but array.extend (used under water) can still be made a lot faster for iterators I think.

while I was at it I made 'self.CostScale' display a warning message. the existing warning mechanism did not get triggered because of the surrounding inheritance.

so if I clone your repo now and run shedskin, it gives three 'CostScale' warnings, and if I fix those, everything seems to compile fine.

thanks again for the nice test case (I will probably add it to shedskin/examples for 0.9.3),
mark.
--
http://www.youtube.com/watch?v=E6LsfnBmdnk

[Piotr Tarsa]

hi mark,

i've checked shedskin-0.9.3 and it indeed works. you've forgot to apply the workaround for the overflow problem. i've pushed a different (from the one presented in this discussion) workaround to my github repository. it does produce correct output (as well as the old workaround does). now i have to retest everything (ie redo timings for my readme file on github) - this can somewhat delayed, as it takes some time, especially the browser testing.

thanks for the interest!

piotr

[srepmub]

(workaround merged in the meantime)

[Piotr Tarsa]

FYI, I've redone the benchmarks finally. They are still under the same address: https://github.com/tarsa/TarsaLZP

[Mark Dufour]

On Tue, Feb 26, 2013 at 11:05 PM, Piotr Tarsa <piotr...@gmail.com> wrote:

FYI, I've redone the benchmarks finally. They are still under the same address: https://github.com/tarsa/TarsaLZP

great, thanks for the update!

mark.

--
http://www.youtube.com/watch?v=E6LsfnBmdnk