OS X Code Signing and Pyinstaller

[Nicholas Nassar]

Mac OS X 10.8 "Mountain Lion" is going to require that all applications are signed by default. Users will need to disable gatekeeper functionality in System Preferences in order to run unsigned apps.

Has anyone had success signing pyinstaller app bundles? In theory, it should be easy, however, I get the following error when I try it with an app created with pyinstaller 1.5.1.

$ codesign -s "Developer ID Application" TestApp.app
codesign_allocate: the __LINKEDIT segment does not cover the end of the file (can't be processed) in: TestApp.app/: object file format unrecognized, invalid, or unsuitable


Any help is appreciated.

-------------------------------------------------
Nick Nassar nickn...@genarts.com
Software Engineer Tel: 617-492-2888x1014
GenArts, Inc. www.genarts.com

[Giovanni Bajo]

Il giorno 11/lug/2012, alle ore 16:15, Nicholas Nassar ha scritto:

> Mac OS X 10.8 "Mountain Lion" is going to require that all applications are signed by default. Users will need to disable gatekeeper functionality in System Preferences in order to run unsigned apps.
>
> Has anyone had success signing pyinstaller app bundles? In theory, it should be easy, however, I get the following error when I try it with an app created with pyinstaller 1.5.1.
>
> $ codesign -s "Developer ID Application" TestApp.app
> codesign_allocate: the __LINKEDIT segment does not cover the end of the file (can't be processed) in: TestApp.app/: object file format unrecognized, invalid, or unsuitable

A PyInstaller binary is a standard Mac binary (the bootloader), with appended some data files at the end. If codesign makes the wrong assumption, it will not parse it correctly. Under Windows, it works flawlessly.

If you can find me a technical description on how codesigning works on Mac (how it parses the binary, where the signature is put, etc.), I can see if there's a way to workaround it.
--
Giovanni Bajo :: ra...@develer.com
Develer S.r.l. :: http://www.develer.com

My Blog: http://giovanni.bajo.it

[Martin Zibricky]

Jeff Keacher píše v Po 10. 12. 2012 v 14:40 -0800:
> The problem appears to be that when append_pkg is True (the default,
> at least on pyinstaller 1.6), the archive is directly appended to the
> executable. Since codesign expects the LINKEDIT segment to be at the
> end of a MachO executable, codesign wasn't happy. From looking at the
> pyinstaller source, it appeared that append_pkg should default to
> False for non-ELF executables, but that doesn't appear to be happening
> on OS X for some reason.

Hi Jeff,

having append_pkg False would break some other features (like --onefile)
of pyinstaller.

Do you have any idea how to append archive to the executable and still
having LINKEDIT segment at the end of executable?

[Jeff Keacher]

Hi Martin,

With the pyinstaller bootstrap code as currently written, I don't see a clean way of building the executable with append_pkg true while still keeping the app signable with codesign. The reason is that codesign expects the LINKEDIT segment to be at the end of the executable, but pyinstaller expects a magic number (MEI\014\013\012\013\016, specifically) to be at the end of the archive, which also forces it to be at the end of the file when append_pkg is true. 

Now, pyinstaller does include the ability to look backwards from the end of the executable a little ways in an effort to find the magic number (a feature that was apparently added to support digital signatures on Windows), so a relatively small change might add support for sticking the archive further back in the executable. One possibility would be sticking the archive in a TEXT segment of the executable (caveat: I haven't investigated that thoroughly), but then the pyinstaller build script would need to be able to intelligently reconstruct the executable with a valid MachO structure, all without disrupting the pointers in the "load commands" section of the file.

Another possibility (that I haven't looked into much) would be to build the pyinstaller bootloader for OS X 10.5 or earlier. Apple changed the MachO format for OS X 10.6, and I know that codesign treats executables built for the earlier MachO format differently than it does for the MachO format present in 10.6 and later.  Some quick Googling suggests that codesign *might* be able to sign those older MachO executables even with data present after the LINKEDIT segment, but there are some rumors that doing so could impact the developer's ability to distribute the app in Apple's MacOS store.

Jeff

[Martin Zibricky]

Jeff Keacher píše v Út 11. 12. 2012 v 09:55 -0800:

> Now, pyinstaller does include the ability to look backwards from the
> end of the executable a little ways in an effort to find the magic
> number (a feature that was apparently added to support digital
> signatures on Windows), so a relatively small change might add support
> for sticking the archive further back in the executable.

I'm going to work more on the bootloader code so this could be done as
part of that.

> One possibility would be sticking the archive in a TEXT segment of the
> executable (caveat: I haven't investigated that thoroughly), but then
> the pyinstaller build script would need to be able to intelligently
> reconstruct the executable with a valid MachO structure, all without
> disrupting the pointers in the "load commands" section of the file.

Do you think that library macholib would be able to do this 'intelligent
reconstructions'?

We already use that library for some manipulation with MachO headers.

[Jeff Keacher]

I took a look at macholib just now, and it seems like it would be able to get us most of the way there, at least as far keeping the headers updated.

Here's a post on StackOverflow that describes embedding arbitrary data at link time:

http://stackoverflow.com/questions/1604673/how-do-i-embed-data-into-a-mac-os-x-mach-o-binary-files-text-section?rq=1

Obviously, we don't want to force the user to rebuild the bootloader, but experimentation with the -sectcreate flag on ld should provide some illumination about how the data is supposed to be embedded in the executable.

Jeff

[Martin Zibricky]

Maybe we could look at how the upx is doing compression of macho files.
Upx could contain any ideas how to properly modify macho executables.

http://upx.sourceforge.net/

We could use machoview to examine bootloader and macho headers.

http://sourceforge.net/projects/machoview/

Jeff Keacher píše v Út 11. 12. 2012 v 11:41 -0800:

[zw g]

Dear list,


I have problem to 'codesign' my osx bundle created by PyInstaller(from git master).

My osx bundle created by pyinstaller works great on OSX 10.9 Mavericks.
But it failed to pass the Apple's codesign step.

I was stuck for couple of days, and still have no clue how to work it out.
Any help would be warmly appreciated.

PyInstaller:  master branch of git repo.
OS:  Mac OS X 10.9 Mavericks
Python: 2.7.5  (installed by homebrew)
PyQt: 4.10.3   (installed by homebrew)
Qt: 4.8.5         (installed by homebrew)

Here below is the 'codesign' steps i did:

=============================================
HandsomeGui-on-MBP:dist zhenweigui$ pwd
/Users/zhenweigui/pyinstaller-git/Events/dist

HandsomeGui-on-MBP:dist zhenweigui$ ll
total 0
drwxr-xr-x  68 zhenweigui  staff  2312 Nov  8 00:17 Events
drwxr-xr-x   3 zhenweigui  staff   102 Nov  8 00:17 Events.app

HandsomeGui-on-MBP:dist zhenweigui$ codesign -s "3rd Party Mac Developer Application: GUI ZHENWEI" -fv ./Events.app
./Events.app: code object is not signed at all
In subcomponent: /Users/zhenweigui/apps/pyinstaller/Events/dist/Events.app/Contents/MacOS/_AE.so

HandsomeGui-on-MBP:dist zhenweigui$ codesign -s "3rd Party Mac Developer Application: GUI ZHENWEI" -fv ./Events.app --deep
./Events.app: bundle format unrecognized, invalid, or unsuitable
In subcomponent: /Users/zhenweigui/apps/pyinstaller/Events/dist/Events.app/Contents/MacOS/include/python2.7

HandsomeGui-on-MBP:dist zhenweigui$ ll /Users/zhenweigui/apps/pyinstaller/Events/dist/Events.app/Contents/MacOS/include/python2.7
total 72
-rw-r--r--  1 zhenweigui  staff  36328 Aug 30 16:09 pyconfig.h

HandsomeGui-on-MBP:dist zhenweigui$
=============================================

[zw g]

Could anyone help me on this? I am stuck.
Thanks in advance.

[Jeff Keacher]

Have you tried the technique I described earlier in the thread, involving changing append_pkg to False in the EXE section of your app's .spec file and signing with 'codesign -f -s "Developer ID" yourappname.app'?

Jeff

--
You received this message because you are subscribed to a topic in the Google Groups "PyInstaller" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pyinstaller/Rp6UUSJ7dP4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pyinstaller...@googlegroups.com.
To post to this group, send email to pyins...@googlegroups.com.
Visit this group at http://groups.google.com/group/pyinstaller.
For more options, visit https://groups.google.com/groups/opt_out.

[zw g]

Thank you, Jeff.

Changing append_pkg to False didn't help.

Here below is my output:

=======================

$ ls -l

total 8

drwxr-xr-x  70 zhenweigui  staff  2380 Nov  9 16:16 Events

-rw-r--r--   1 zhenweigui  staff   240 Nov  9 16:19 Events-Entitlements.plist

drwxr-xr-x   3 zhenweigui  staff   102 Nov  9 16:15 Events.app

$ codesign -f -s "3rd Party Mac Developer Application: GUI ZHENWEI" Events.app/

Events.app/: bundle format unrecognized, invalid, or unsuitable

In subcomponent: /Users/zhenweigui/apps/pyinstaller/Events/dist/Events.app/Contents/MacOS/include/python2.7

$ codesign -f -s "3rd Party Mac Developer Application: GUI ZHENWEI" Events.app/ --deep

Events.app/: bundle format unrecognized, invalid, or unsuitable

In subcomponent: /Users/zhenweigui/apps/pyinstaller/Events/dist/Events.app/Contents/MacOS/include/python2.7

$ ls -l /Users/zhenweigui/apps/pyinstaller/Events/dist/Events.app/Contents/MacOS/include/python2.7

total 72

-rw-r--r--  1 zhenweigui  staff  36328 Aug 30 16:09 pyconfig.h

$ cat ../../Events.spec

# -*- mode: python -*-

a = Analysis(['/Users/zhenweigui/events/Events.py'],

             pathex=['/Users/zhenweigui/apps/pyinstaller/Events'],

             hiddenimports=[],

             hookspath=None,

             runtime_hooks=None)

pyz = PYZ(a.pure)

exe = EXE(pyz,

          a.scripts,

          exclude_binaries=True,

          name='Events',

          debug=False,

          strip=None,

          upx=True,

          append_pkg=False,

          console=False , icon='events.icns')

coll = COLLECT(exe,

               a.binaries,

               a.zipfiles,

               a.datas,

               strip=None,

               upx=True,

               name='Events')

app = BUNDLE(coll,

             name='Events.app',

             icon='events.icns')

$

=======================

Any thing i did wrong?

Gui

[zw g]

Oh, I am using PyInstall from the git master branch.

Gui

[wins...@stratolab.com]

I'm having this problem on Mavericks too. I feel like there are several problems going on. One is the "LINKEDIT segment" problem which it sounds like "append_pkg=True" fixes. But another problem seems to be codesign thinking that any other files in the .app/Contents/MacOS/ folder is a bundle and it needs to be signed too. I think this because I'm getting messages like:

Electropocalypse.app/Contents/MacOS/electropocalypse: code object is not signed at all

In subcomponent: Electropocalypse.app/Contents/MacOS/eggs/setuptools-2.0.2-py2.7.egg

And I would get similar messages for .tcl and .html files in the MacOS folder. Perhaps we can move all the PyInstaller files from the MacOS folder to the Resources folder and that would help?

-Winston

[Giovanni Bajo]

Actually, it works for me if I put append_pkg=False, and then use codesign with —deep and -f. This is with the standard Python framework in OSX Mavericks.

You received this message because you are subscribed to the Google Groups "PyInstaller" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyinstaller...@googlegroups.com.

To post to this group, send email to pyins...@googlegroups.com.
Visit this group at http://groups.google.com/group/pyinstaller.
For more options, visit https://groups.google.com/groups/opt_out.

-- 

[wins...@stratolab.com]

So I've figured out that on OSX, codesign --deep will definitely complain when there are folders like "Foo.app/Contents/MacOS/site-packages/gst-1.0". It seems to think that gst-1.0 is a framework or something because it ends in a .digit. If I rename it to gst-1_0 then codesign doesn't complain. Similarly if I just move gst-1.0 to the Foo.app/Contents/Resources folder, it is also happy.

[wins...@stratolab.com]

Is there a way we could put all the files in the Contents/Resources folder and setup the boot loader (or whatever sets things up) to look in Resources instead of Contents/MacOS? E.g. would setting _MEIPASS2 to Resources be all that's needed?

[Simon Lightstone]

Hi everyone,

I'm also working on this problem. Just wanted to point out a helpful post on StackOverflow. It's related to mono, but it comes to some of the same conclusions and I think it's handy:

http://stackoverflow.com/questions/21736367/signing-code-for-os-x-application-bundle

[Simon Lightstone]

I think you're using the wrong certificate to sign for GateKeeper.

"3rd Party Mac Developer Application" is used for the App Store, not for Gatekeeper, I think.

What you want is something like: codesign --force --sign "Developer ID Application: [your name]" [app_path]

Again, I'm a newbie when it comes to Mac's code sign certs/GateKeeper, but I think that is a start.

[Daniel Caspi]

Found this thread while facing a similar Mach-O binary issue with a different (not PyInstaller) binary, and it seems I've located the most relevant thread on Google to this situation. It seems you've already solved this problem with the PyInstaller, and since there is such limited information out there... I would be really grateful if you'd be able to help me out with this one. I attached a sample binary to check out.

I'm convinced it's an issue with the headers, and I've used MachOView but can't seem to figure out why it's not working.

There is a ZIP archive stored starting at offset 4623041 that goes until the end of the file.

Any help you can give would be GREATLY appreciated... this is a very frustrating problem with the Apple codesigning.

Cheers & Thanks,

Dan

[Daniel Caspi]

Nevermind all... was able to solve it in the meantime using the info in this thread. Appreciate all the great work that everyone has done here to figure this out, it was invaluable for me. The same two Mach-O headers had to be edited in my case to make it work, and the Mach-O viewer was key to figure out the correct ones to edit.

I'm corresponding with the writer of this package and sending him the fix.

--

You received this message because you are subscribed to a topic in the Google Groups "PyInstaller" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pyinstaller/Rp6UUSJ7dP4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pyinstaller...@googlegroups.com.
To post to this group, send email to pyins...@googlegroups.com.
Visit this group at http://groups.google.com/group/pyinstaller.

For more options, visit https://groups.google.com/d/optout.

[Martin Zibricky]

Hi,

 

I collected some information about codesigning and PyInstaller on this wiki page:

 

https://github.com/pyinstaller/pyinstaller/wiki/Recipe-OSX-Code-Signing

[Daniel Caspi]

Thanks Martin, great collection of info. The work you've put together in solving this has saved me endless hours of frustration. 

One tip I could share:

I think VM size should not equal file size but should be byte aligned to nearest 16 byte value. Codesign also fixes headers it seems -- provided it doesn't fail when running. 

--