ANN: unpyc3 - a python bytecode decompiler for Python3

[Arnaud Delobelle]

Hi all,

Unpyc3 can recreate Python3 source code from code objects, function
source code from function objects, and module source code from .pyc
files. The current version is able to decompile itself successfully
:). It has been tested with Python3.2 only.

It currently reconstructs most of Python 3 (see TODO below) constructs
but probably needs to be tested more thoroughly. All feedback welcome.

Unpyc3 is a single file and is available at http://code.google.com/p/unpyc3/

Example:

>>> from unpyc3 import decompile
>>> def foo(x, y, z=3, *args):
... global g
... for i, j in zip(x, y):
... if z == i + j or args[i] == j:
... g = i, j
... return
...
>>> print(decompile(foo))
def foo(x, y, z=3, *args):
global g
for i, j in zip(x, y):
if z == i + j or args[i] == j:
g = i, j
return

TODO:

* Support for keyword-only arguments
* Handle assert statements
* Show docstrings for functions and modules
* Nice spacing between function/class declarations

Have fun!

Note: unpyc3 is totally unrelated to another project called "unpyc"
which I discovered when I tried to register the same project name on
google code.

--
Arnaud

[Arnaud Delobelle]

On 14 September 2011 06:53, Vincent Vande Vyvre
<vincent.v...@swing.be> wrote:
>
> Hi, trying your code, I have had numbers of errors:

Hi Vincent, thanks for trying it.

>   File "unpyc3.py", line 55, in <module>
>     SETUP_WITH,
> NameError: name 'SETUP_WITH' is not defined
>
> commented it
>
>   File "unpyc3.py", line 58, in <module>
>     STORE_DEREF, DELETE_DEREF,
> NameError: name 'DELETE_DEREF' is not defined
>
> commented it

What version of Python are you running this on? This is module is
written for Python 3. It looks like you're using an old version of
Python (before the with statement was introduced - 2.5?)

>   File "unpyc3.py", line 96, in dec_module
>     stream = open(pyc_path, "rb")
> UnboundLocalError: local variable 'pyc_path' referenced before assignment
>
> change pyc_path to path

Thanks, I've fixed that.

--
Arnaud

[Arnaud Delobelle]

On 14 September 2011 09:44, Vincent Vande Vyvre
<vincent.v...@swing.be> wrote:
>   File "unpyc3.py", line 211, in __init__
>     for v in code_obj.co_cellvars + code_obj.co_freevars]
> AttributeError: 'NoneType' object has no attribute 'co_cellvars'

Could you show me what you do to get this error? Thank you,

Arnaud

PS: I've checked; DELETE_DEREF was introduced in Python3.2

[Arnaud Delobelle]

On 14 September 2011 11:03, Vincent Vande Vyvre
<vincent.v...@swing.be> wrote:
> Le 14/09/11 11:31, Arnaud Delobelle a écrit :
[...]

> Could you show me what you do to get this error? Thank you,

> [vincent@myhost unpyc3]$ python
> Python 3.2.1 (default, Jul 11 2011, 12:37:47)
> [GCC 4.6.1] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> from unpyc3 import decompile
>>>> print (decompile("shredder.pyc"))
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "unpyc3.py", line 110, in decompile
>     return dec_module(obj)
>   File "unpyc3.py", line 99, in dec_module
>     code = Code(code_obj)

>   File "unpyc3.py", line 211, in __init__
>     for v in code_obj.co_cellvars + code_obj.co_freevars]
> AttributeError: 'NoneType' object has no attribute 'co_cellvars'

>>>> print (decompile("unpyc3.pyc"))
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "unpyc3.py", line 110, in decompile
>     return dec_module(obj)
>   File "unpyc3.py", line 99, in dec_module
>     code = Code(code_obj)

>   File "unpyc3.py", line 211, in __init__
>     for v in code_obj.co_cellvars + code_obj.co_freevars]
> AttributeError: 'NoneType' object has no attribute 'co_cellvars'

>>>> import os
>>>> os.path.isfile("shredder.pyc")
> True
>>>> os.path.isfile("unpyc3.pyc")
> True
>>>>
>
> it seems the return of marshal.load(stream) is None

I think the reason may be that your unpyc3.pyc and shredder.pyc files
were compiled with a different version of python and the read_code
function returns None because the magic number in the .pyc file is
incorrect because of these two lines:

if magic != imp.get_magic():
return None

I have now changed this so that it loads the file anyway but prints a
warning. I guess this may break badly though.

In Python 3.2, .pyc files are "hidden" in a __pycache__ directory. So
the Python 3.2 specific unpyc3.pyc file for example is probably
located at .../unpyc3/__pycache__/unpyc3-cpython-32.pyc

The easiest way to find the path of the .pyc file if you know the path
of the .py file is probably as follows:

>>> import imp
>>> imp.cache_from_source("unpyc3.py")
'__pycache__/unpyc3.cpython-32.pyc'

Here's an example decompiling the dis module from the standard library:

>>> import dis
>>> dis.__file__
'/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/dis.py'
>>> imp.cache_from_source(dis.__file__)
'/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/__pycache__/dis.cpython-32.pyc'
>>> print(decompile(_))
__doc__ = 'Disassembler of Python byte code into mnemonics.'
import sys
import types
from opcode import *
from opcode import __all__ as _opcodes_all
__all__ = ['code_info', 'dis', 'disassemble', 'distb', 'disco',
'findlinestarts', 'findlabels', 'show_code'] + _opcodes_all
del _opcodes_all
_have_code = types.MethodType, types.FunctionType, types.CodeType, type
def _try_compile(source, name):
try:
c = compile(source, name, 'eval')
except SyntaxError:
c = compile(source, name, 'exec')
return c
[... many more lines ...]

I hope this will work for you,

--
Arnaud

PS: I've also added support for the IMPORT_STAR opcode which I had overlooked.

[n.pop...@xs4all.nl]

Last night I accidentally deleted a group of *.py files (stupid-stupid-stupid!).

Thanks to unpyc3 I have reconstructed all but one of them so far from the *.pyc files that were in the directory __pycache__. Many thanks!!!

-- Nico

[Devin Jeanpierre]

FWIW I put all my source code inside Dropbox so that even things I
haven't yet committed/pushed to Bitbucket/Github are backed up. So far
it's worked really well, despite using Dropbox on both Windows and
Linux. (See also: Google Drive, etc.)

(Free) Dropbox has a 30 day recovery time limit, and I think Google
Drive has a trash bin, as well as a 29 day recovery for emptied trash
items.

That said, hindsight is easier than foresight. I'm glad you were able
to recover your files!

-- Devin

> --
> https://mail.python.org/mailman/listinfo/python-list

[Chris Kaynor]

On Wed, Jan 28, 2015 at 10:42 AM, Devin Jeanpierre
<jeanpi...@gmail.com> wrote:
> FWIW I put all my source code inside Dropbox so that even things I
> haven't yet committed/pushed to Bitbucket/Github are backed up. So far
> it's worked really well, despite using Dropbox on both Windows and
> Linux. (See also: Google Drive, etc.)
>
> (Free) Dropbox has a 30 day recovery time limit, and I think Google
> Drive has a trash bin, as well as a 29 day recovery for emptied trash
> items.
>
> That said, hindsight is easier than foresight. I'm glad you were able
> to recover your files!

I use Google Drive for it for all the stuff I do at home, and use SVN
for all my personal projects, with the SVN depots also in Drive. The
combination works well for me, I can transfer between my desktop and
laptop freely, and have full revision history for debugging issues.

>
> -- Devin
>
> On Wed, Jan 28, 2015 at 10:09 AM, <n.pop...@xs4all.nl> wrote:

[Chris Angelico]

On Thu, Jan 29, 2015 at 5:47 AM, Chris Kaynor <cka...@zindagigames.com> wrote:
> I use Google Drive for it for all the stuff I do at home, and use SVN
> for all my personal projects, with the SVN depots also in Drive. The
> combination works well for me, I can transfer between my desktop and
> laptop freely, and have full revision history for debugging issues.

I just do everything in git, no need for either Drive or something as
old as SVN. Much easier. :)

Using a more modern source control system (I'd normally recommend
people use either git or Mercurial, though there are a few others that
are also viable) saves you the trouble of "oops, I'm offline and can't
reach the Subversion server" and other issues.

ChrisA

[Devin Jeanpierre]

On Wed, Jan 28, 2015 at 1:40 PM, Chris Angelico <ros...@gmail.com> wrote:
> On Thu, Jan 29, 2015 at 5:47 AM, Chris Kaynor <cka...@zindagigames.com> wrote:
>> I use Google Drive for it for all the stuff I do at home, and use SVN
>> for all my personal projects, with the SVN depots also in Drive. The
>> combination works well for me, I can transfer between my desktop and
>> laptop freely, and have full revision history for debugging issues.
>
> I just do everything in git, no need for either Drive or something as
> old as SVN. Much easier. :)

Git doesn't help if you lose your files in between commits, or if you
lose the entire directory between pushes.

-- Devin

[Chris Angelico]

On Thu, Jan 29, 2015 at 8:52 AM, Devin Jeanpierre
<jeanpi...@gmail.com> wrote:
> Git doesn't help if you lose your files in between commits, or if you
> lose the entire directory between pushes.

So you commit often and push immediately. Solved.

ChrisA

[Devin Jeanpierre]

I distrust any backup strategy that requires explicit action by the
user. I've seen users fail too often. (Including myself.)

-- Devin

> --
> https://mail.python.org/mailman/listinfo/python-list

[Chris Kaynor]

On Wed, Jan 28, 2015 at 3:01 PM, Devin Jeanpierre
<jeanpi...@gmail.com> wrote:
> On Wed, Jan 28, 2015 at 2:02 PM, Chris Angelico <ros...@gmail.com> wrote:
>> On Thu, Jan 29, 2015 at 8:52 AM, Devin Jeanpierre
>> <jeanpi...@gmail.com> wrote:
>>> Git doesn't help if you lose your files in between commits, or if you
>>> lose the entire directory between pushes.
>>
>> So you commit often and push immediately. Solved.

> I distrust any backup strategy that requires explicit action by the
> user. I've seen users fail too often. (Including myself.)

That tends to be my opinion and experience as well :)

And that is where Drive is quite nice: its an automatic backup to an
off-site backup that requires no user action. Having some form of
source control is still needed however, as you don't get all the nice
history with Drive, and don't have the atomic updates - typically,
every save will be uploaded, even if that change itself will break
everything as you haven't made the required changes to other files.

Chris K

[sohca...@gmail.com]

I'd definitely store all of my programming projects in a Google Drive if I wasn't already using Dropbox.

I recently finished my CS degree, and I had more than one professor say that they won't take "My computer crashed and I lost everything!" as an excuse for not being able to turn in homework. Dropbox and Google Drive are both free, easy to use, and will keep several versions of your files so you can even use the excuse that your most recent save got corrupted.

Also, it was really nice to easily be able to save my work on my laptop, finish it on my desktop, and then print it from a school computer without dealing with a thumb drive.

[Steven D'Aprano]

sohca...@gmail.com wrote:

> I recently finished my CS degree, and I had more than one professor say
> that they won't take "My computer crashed and I lost everything!" as an
> excuse for not being able to turn in homework.  

How about "My computer crashed and died and now I can't get to Dropbox to
access my files"?

"My computer got infected by ransomware which encrypted all my data files
and blocks access to Dropbox."

"One of my housemates torrented a Linux tarball, and the MPAA wrongly
identified it as a movie file. Purely on their say-so, my ISP disabled my
account and banned me from the Internet. But all is not lost, if I move 45
miles away, I can sign up with a different ISP!"

"Some dude I've never seen before gate-crashed our party and was smoking
pot, and the police raided us and seized my computer and everybody's
phones. My lawyer tells me the raid was illegal and if spend two or three
hundred thousand dollars in legal fees, I'll probably be able to get my
computer back within eight years or so."

"My dog ate my USB stick."

:-)




--
Steven

[Steven D'Aprano]

Sure it does? You just lose the changes made since the previous commit, but
that's no different from restoring from backup. The restored file is only
as up to date as the last time a backup was taken.

> or if you
> lose the entire directory between pushes.

Then restore from wherever you are pushing to.

But as Devin says, any backup strategy that requires the user to make a
backup is untrustworthy. I'm hoping that the next generation of DVCSs will
support continuous commits, the next generation of editors support
continuous saves, and the only time you need interact with the editor
(apart from, you know, actual editing) is to tell it "start a new branch
now".



--
Steven

[Mark Lawrence]

On 28/01/2015 23:12, Chris Kaynor wrote:
> On Wed, Jan 28, 2015 at 3:01 PM, Devin Jeanpierre
> <jeanpi...@gmail.com> wrote:
>> On Wed, Jan 28, 2015 at 2:02 PM, Chris Angelico <ros...@gmail.com> wrote:
>>> On Thu, Jan 29, 2015 at 8:52 AM, Devin Jeanpierre
>>> <jeanpi...@gmail.com> wrote:

>>>> Git doesn't help if you lose your files in between commits, or if you

>>>> lose the entire directory between pushes.
>>>

>>> So you commit often and push immediately. Solved.
>> I distrust any backup strategy that requires explicit action by the
>> user. I've seen users fail too often. (Including myself.)
>
> That tends to be my opinion and experience as well :)
>

s/any backup strategy/anything/

--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

[Cem Karan]

On Jan 28, 2015, at 5:02 PM, Chris Angelico <ros...@gmail.com> wrote:

> On Thu, Jan 29, 2015 at 8:52 AM, Devin Jeanpierre
> <jeanpi...@gmail.com> wrote:
>> Git doesn't help if you lose your files in between commits, or if you
>> lose the entire directory between pushes.
>
> So you commit often and push immediately. Solved.
>

> ChrisA

Just to expand on what Chris is saying, learn to use branches. I use git flow ([1][2]), but you don't need it, plain old branches are fine. Then you can have a feature branch like 'Joes_current', or something similar which you and only you push/pull from. Whenever you're done with it, you can merge the changes back into whatever you & your group see as the real branch. That is the model I use at work, and it works fairly well, and its saved me once already when the laptop I was working on decided to die on me.

Thanks,
Cem Karan

[1] http://nvie.com/posts/a-successful-git-branching-model/
[2] https://github.com/nvie/gitflow

[Devin Jeanpierre]

On Wed, Jan 28, 2015 at 4:34 PM, Steven D'Aprano
<steve+comp....@pearwood.info> wrote:

> Devin Jeanpierre wrote:
>> Git doesn't help if you lose your files in between commits,
>
> Sure it does? You just lose the changes made since the previous commit, but
> that's no different from restoring from backup. The restored file is only
> as up to date as the last time a backup was taken.

Yeah. My point here is that Drive/Dropbox take snapshots at much
shorter intervals than any reasonable person will commit with a DVCS,
so you lose much less.

-- Devin

[Dave Angel]

In emacs, bnd the git add, git commit to Ctrl-x - s, and saving also
means committing.

My backup system uses MD5's to decide which files need backing up, so
theoretically it shouldn't cost too much to backup the git archives once
daily. It's all still under development, however. (I've been offline
for two weeks, developing and running the backup system, and preparing
for a complete reinstall of a corrupted system)



--
DaveA

[sohca...@gmail.com]

On Wednesday, January 28, 2015 at 4:30:11 PM UTC-8, Steven D'Aprano wrote:
> sohca...@gmail.com wrote:
>
> > I recently finished my CS degree, and I had more than one professor say
> > that they won't take "My computer crashed and I lost everything!" as an
> > excuse for not being able to turn in homework.  
>
> How about "My computer crashed and died and now I can't get to Dropbox to
> access my files"?

If you have access to a web browser, you have access to your Dropbox files.

I don't own a printer. If I needed to print something for school, I just printed it from my Dropbox at school.

> "My computer got infected by ransomware which encrypted all my data files
> and blocks access to Dropbox."

Dropbox saves previous versions of files. Just restore them from the last version you had before they got encrypted by the ransomware. This can be done from any browser.

> "One of my housemates torrented a Linux tarball, and the MPAA wrongly
> identified it as a movie file. Purely on their say-so, my ISP disabled my
> account and banned me from the Internet. But all is not lost, if I move 45
> miles away, I can sign up with a different ISP!"

Not exactly a likely scenario, and still not a problem unless you're not allowed to access the internet from someone's WiFi.

> "Some dude I've never seen before gate-crashed our party and was smoking
> pot, and the police raided us and seized my computer and everybody's
> phones. My lawyer tells me the raid was illegal and if spend two or three
> hundred thousand dollars in legal fees, I'll probably be able to get my
> computer back within eight years or so."

They can take your computer and it doesn't matter if you've got your files on Dropbox.

> "My dog ate my USB stick."
>
> :-)

I never used a USB stick for school work.

At this point, I'm probably sounding like a shill for Dropbox, but I'm really not. I imagine Google Drive offers the same features. Access to your files from the web, synchronization of local files among computers with access to it, and the ability to retrieve and restore files from previous versions.

[Michael Torrie]

On 02/04/2015 05:19 PM, sohca...@gmail.com wrote:
> They can take your computer and it doesn't matter if you've got your files on Dropbox.
>
>> "My dog ate my USB stick."
>>
>> :-)
>
> I never used a USB stick for school work.
>
> At this point, I'm probably sounding like a shill for Dropbox, but I'm really not. I imagine Google Drive offers the same features. Access to your files from the web, synchronization of local files among computers with access to it, and the ability to retrieve and restore files from previous versions.

In my mind, they are all tools. And no one tool should be used and
trusted above all others.

Anyone that's programming should be using version control, period. But
that's not for backup, and backup can and should be used as well as
version control. Everything I work on I commit to git regularly because
of the utility git gives me. If I end up trying something that doesn't
pan out, I can retrace my steps (that's what branches are for). I don't
have to dig through two weeks of hourly backups to find out where it was
when I started making a change. Backup and git serve two complementary
but different purposes.

As well as regularly committing code to Git, I run CrashPlan and on a
regular schedule (hourly perhaps) it copies all changes, committed or
not, and including the git repo itself to the cloud, and also to my
other computer, as well as my parents' computer. CrashPlan makes this
stuff easy, so there's no reason not have redundancy. As well, I
semi-regularly run a manual rsync backup to three different USB hard
drives on a rotating backup.

Is this overkill? I don't believe so. It requires virtually no work on
my part.

I don't see any one cloud service as the best product. Why not use them
all? Encrypt if you need to, and sync hourly snapshots to google drive,
drop box, and any other free cloud service.