Mac OS X Mavericks codesigning and --onefile Unix executables

[Jeremy Weinstein]

Hello all,

I'm using PyInstsaller to package a command line application into a single executable binary file. I'd like to codesign the application so that when a user double clicks the executable, Terminal opens and runs the command and the program exits, leaving the command output up in Terminal. This is the default behavior of a Unix executable binary on Mac OS and that's how my app works on my computer when it's created with PyInstaller.

When I transfer the app to another computer, double clicking the app results in a "This file is damaged and should be moved to the Trash." message. When I run the program from the command line using Terminal, it works great.

Basically, I'd like to give other users the same ability to double click the app and see the output, just like I have on my computer – and I thought I'd have to codesign the binary to do that.

I found a thread or two on this list with similar questions but no conclusive answers. It seems one person was able to codesign a Mac application bundle, but their program was not a command line tool, and it also used the "--onedir" flag and not the "--onefile" flag like I am.

Here's my question: Is there a conclusive way to codesign a "--onefile" Unix executable application for Mac? I am unable to do this. If not, would anyone have a recommendation on how to distribute a Python command line executable tool on Mac?

Here are my steps that fail:

$ pyinstaller grow.spec

$ codesign -f -s "Developer ID Application: Jeremy Weinstein (<developer id here>)" --entitlements ./entitlements.plist --deep dist/grow

dist/grow: replacing existing signature

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/codesign_allocate: the __LINKEDIT segment does not cover the end of the file (can't be processed) in: /Users/jeremydw/git/macgrow/dist/grow

dist/grow: the codesign_allocate helper tool cannot be found or used

grow.spec:

# -*- mode: python -*-

a = Analysis(['pygrow/bin/grow'],

             pathex=['pygrow/', '/Users/jeremydw/git/macgrow', '/Library/Python/2.7/site-packages/'],

             hiddenimports=['markdown', 'markdown.extensions'],

             hookspath=None,

             runtime_hooks=None)

a.datas += [

    ('VERSION', '/Users/jeremydw/git/macgrow/pygrow/grow/VERSION', 'DATA'),

    ('server/templates/error.html', '/Users/jeremydw/git/macgrow/pygrow/grow/server/templates/error.html', 'DATA'),

]

pyz = PYZ(a.pure)

exe = EXE(pyz,

          a.scripts,

          a.binaries,

          a.zipfiles,

          a.datas,

          name='grow',

          debug=False,

          strip=None,

          upx=True,

          console=True,

          icon='macgrow/icon.icns')

entitlements.plist

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">

<plist version="1.0">

  <dict>

    <key>com.apple.security.app-sandbox</key>

    <true/>

    <key>com.apple.security.inherit</key>

    <true/>

  </dict>

</plist>

Thanks for any tips you might have!

-- 

jeremydw

[Martin Zibricky]

Hi Jeremy,

recently someone created a pull request with potential fix to codesigning on
osx:

https://github.com/pyinstaller/pyinstaller/pull/113

Could you please try it if that works for you?

On Thursday 13 of March 2014 23:05:51 Jeremy Weinstein wrote:
> Hello all,
>
> I'm using PyInstsaller to package a command line application into a single
> executable binary file. I'd like to codesign the application so that when a
> user double clicks the executable, Terminal opens and runs the command and
> the program exits, leaving the command output up in Terminal. This is the
> default behavior of a Unix executable binary on Mac OS and that's how my
> app works on my computer when it's created with PyInstaller.
>
> When I transfer the app to another computer, double clicking the app
> results in a "This file is damaged and should be moved to the Trash."
> message. When I run the program from the command line using Terminal, it
> works great.
>
> Basically, I'd like to give other users the same ability to double click
> the app and see the output, just like I have on my computer – and I thought
> I'd have to codesign the binary to do that.
>
> I found a thread or two on this list with similar questions but no
> conclusive answers. It seems one person was able to codesign a Mac
> application bundle, but their program was not a command line tool, and it
> also used the "--onedir" flag and not the "--onefile" flag like I am.
>

> Here's my question: *Is there a conclusive way to codesign a "--onefile"
> Unix executable application for Mac?* I am unable to do this. If not, would

> anyone have a recommendation on how to distribute a Python command line
> executable tool on Mac?
>
> Here are my steps that fail:
>
> $ pyinstaller grow.spec
> $ codesign -f -s "Developer ID Application: Jeremy Weinstein (<developer id
> here>)" --entitlements ./entitlements.plist --deep dist/grow
> dist/grow: replacing existing signature
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolcha
> in/usr/bin/codesign_allocate: the __LINKEDIT segment does not cover the end
> of the file (can't be processed) in: /Users/jeremydw/git/macgrow/dist/grow
> dist/grow: the codesign_allocate helper tool cannot be found or used
>

> *grow.spec:*

>
> # -*- mode: python -*-
> a = Analysis(['pygrow/bin/grow'],
> pathex=['pygrow/', '/Users/jeremydw/git/macgrow',
> '/Library/Python/2.7/site-packages/'],
> hiddenimports=['markdown', 'markdown.extensions'],
> hookspath=None,
> runtime_hooks=None)
> a.datas += [
> ('VERSION', '/Users/jeremydw/git/macgrow/pygrow/grow/VERSION', 'DATA'),
> ('server/templates/error.html',
> '/Users/jeremydw/git/macgrow/pygrow/grow/server/templates/error.html',
> 'DATA'),
> ]
> pyz = PYZ(a.pure)
> exe = EXE(pyz,
> a.scripts,
> a.binaries,
> a.zipfiles,
> a.datas,
> name='grow',
> debug=False,
> strip=None,
> upx=True,
> console=True,
> icon='macgrow/icon.icns')
>

> *entitlements.plist*

[Burak Nehbit]

Hi,

I have a problem I have been trying to solve throughout the last month to no avail, I am hoping you will be able to help.

I am the creator of a free, open source peer–to–peer and encrypted forum application called Aether. ( www.getaether.net ) I have recently released v1.1 version, but I am having significant problems packaging the app with PyInstaller. 

The first version of the app also used PyInstaller, which can be obtained at the website. The new version is however much more performant mainly because I have split the networking backend from the front–end GUI into two separate processes, which is the reason I am having the problem with packaging.

Ampoule is a twisted plugin that in essence creates another Python process in which to run code. It’s fairly straightforward, and it uses Twisted’s own spawnProcess API, which seems to be supported by PyInstaller as far as my Google–fu tells me. 

When I package my app using PyInstaller, the main process works flawlessly. When it tries to instantiate the second process, the instantiated process breaks and the main process starts to create an infinite amount of subprocesses.

1) It is specified Ampoule behaviour to restart dead processes, that’s where the repeated new processes come from. The second process dies very early in its lifetime.

2) When I try to instantiate an empty process, which is a process with no code provided to it, it still crashes. Ditto when I use the default Ampoule stub. So whatever is throwing an exception or crashing is not in my code. 

3) I believe I am providing the imports and it imports the modules required.

Here is the source code where Ampoule instantiates a worker process:

def spawnProcess(processProtocol, bootstrap, args=(), env={},

                 path=None, uid=None, gid=None, usePTY=0,

                 packages=()):

    env = env.copy()

    pythonpath = []

    for pkg in packages:

        p = os.path.split(imp.find_module(pkg)[1])[0] 

        # imp.find_module: look to the modules to find it,

        # get the address [1] and get the former path. [0]

        if p.startswith(os.path.join(sys.prefix, 'lib')): 

        # If it starts with usr/local or any other

        # system dir, its already importable, no worries. ignore.

            continue

        pythonpath.append(p) 

        # if not, appendt to pythonpath array.

    pythonpath = list(sets.Set(pythonpath)) 

    # ensure everything is unique and convert to list.

    pythonpath.extend(env.get('PYTHONPATH', '').split(os.pathsep))

    env['PYTHONPATH'] = os.pathsep.join(pythonpath)

    args = (sys.executable, '-c', bootstrap) + args

    # childFDs variable is needed because sometimes child processes

    # misbehave and use stdout to output stuff that should really go

    # to stderr. Of course child process might even use the wrong FDs

    # that I'm using here, 3 and 4, so we are going to fix all these

    # issues when I add support for the configuration object that can

    # fix this stuff in a more configurable way.

    if IS_WINDOWS:

        return reactor.spawnProcess(processProtocol, sys.executable, args,

                                    env, path, uid, gid, usePTY)

    else:

        return reactor.spawnProcess(processProtocol, sys.executable, args,

                                    env, path, uid, gid, usePTY,

                                    childFDs={0:"w", 1:"r", 2:"r", 3:"w", 4:"r"})

As you can see, it’s mostly a wrapper to Twisted’s spawnProcess, with a few additions to resolve the imports. 

I have almost no visibility into this process as the child crashes before it can log back anything or even return an exception. The process starter at the main process does not return any exceptions. 

What approach would you recommend to fix this issue? I have been stuck at this phase for about a month, I am willing to give a shot at almost anything. It’s also a project very dear to me—I’ve been working on this for one and a half year now—so, whatever it takes…

Thank you,

Burak / @nehbit

[Hartmut Goebel]

Am 05.04.2014 23:57, schrieb Burak Nehbit:

1) It is specified Ampoule behaviour to restart dead processes, that’s where the repeated new processes come from. The second process dies very early in its lifetime.

So at first you have to find out, why this process dies.

Unfortunately you say nothing about the second program. Does it run if started manually? Did you try with a *simple* second program like "import time; time.sleep(300)"?

        p = os.path.split(imp.find_module(pkg)[1])[0]

I doubt this is helping anything, since most modules' path will be pointing into the executable.

        return reactor.spawnProcess(processProtocol, sys.executable, args,

                                    env, path, uid, gid, usePTY)

Are you sure about this? See <http://pythonhosted.org/PyInstaller/#id60>

"sys.executable ... In a bundled app, it is the path to the bundled executable."

BTW: The git-repo of aether is called "aether-public". Is this the complete source, or is there also a private part?

--
Schönen Gruß
Hartmut Goebel
Dipl.-Informatiker (univ), CISSP, CSSLP
Information Security Management, Security Governance, Secure Software Development

Goebel Consult, Landshut
http://www.goebel-consult.de

Blog: http://www.goebel-consult.de/blog/kurzanalyse-disconnect.me
Kolumne: http://www.cissp-gefluester.de/2011-09-kommerz-uber-recht-fdp-die-gefaellt-mir-partei

Goebel Consult ist Mitglied bei http://www.7-it.de/

[Burak Nehbit]

So at first you have to find out, why this process dies.

Unfortunately you say nothing about the second program. Does it run if started manually? 

The second program, which is the child process, runs on its own if I run it alone. It’s a GUI using PyQt and Qt. It sets up a second Twisted loop in the child process, which allows that process and the main one to communicate with each other over pipes in an asynchronous fashion over pipes. 

Did you try with a *simple* second program like "import time; time.sleep(300)”?

Yes. It also does not work. Neither does the stub program provided by Ampoule, nor an empty second program. All of these work in case they’re not packaged. The symptom is the same, Ampoule is trying to spawn an infinite number of processes. I also tested Twisted’s own spawnProcess, which does not have this issue and runs perfectly. So whatever is creating the problem is inside Ampoule. 

Are you sure about this? See <http://pythonhosted.org/PyInstaller/#id60>

"sys.executable ... In a bundled app, it is the path to the bundled executable.”

This might be the culprit. I’ll take a deeper look into this part. 

BTW: The git-repo of aether is called "aether-public". Is this the complete source, or is there also a private part?

Yes, it’s entirety of the source code. I do have a private repo, but the the few differences are that it has my full git work history, i.e. breaking commits that get reverted etc, and my own feature tracker. I push to the public repo on every new version release. The current version there is 1.1, the one in question here. If you want to give a shot running it on your local machine, you can grab the code and follow the ‘Dependencies’ part of this blog post: http://blog.getaether.net/post/79723395145/aether-v1-1-is-here If you have PyQt5 / Qt installed, it should be fairly straightforward.

Best,

Burak

[Hartmut Goebel]

Am 06.04.2014 23:55, schrieb Burak Nehbit:

The second program, which is the child process, runs on its own if I run it alone. It’s a GUI using PyQt and Qt. It

Have you bundled it using PyInstaller, too? Are you using MERGE?

If you want to give a shot running it on your local machine, you can grab the code and follow the ‘Dependencies’ part of this blog post: http://blog.getaether.net/post/79723395145/aether-v1-1-is-here If you have PyQt5 / Qt installed, it should be fairly straightforward.

Well, I'd prefer having a requirements.txt and a PyInstaller .spec-file, so I can immediately start looking into the problem. :-)

[Burak Nehbit]

Have you bundled it using PyInstaller, too? Are you using MERGE?

No, I haven’t. I haven’t had taken a look at Merge—up until now my belief was that multiprocessing did not need such tools. I can run Twisted’s own spawnProcess without bundling the subprocess or without merge, so I would think Ampoule would not change anything. It’s nothing but a wrapper over spawnProcess.

Well, I'd prefer having a requirements.txt and a PyInstaller .spec-file, so I can immediately start looking into the problem. :-)

My apologies, here’s the last working spec file I had. 

https://dl.dropboxusercontent.com/u/5815330/aether.spec

Requirements.txt does not work in this case, because PyQt needs to be compiled against the Qt installation you have. It handles everything, but it’s not automated into pip. 

I have tried to replace the sys.executable in the code I have sent:

return reactor.spawnProcess(processProtocol, sys.executable, args,

                                    env, path, uid, gid, usePTY,

                                    childFDs={0:"w", 1:"r", 2:"r", 3:"w", 4:"r”})

And replaced it with the directory of my virtual environment. This at least started to show some import errors. It seems that it was trying to invoke the PyInstaller bundled executable as if it was the Python executable, which was the reason I was seeing a million of the same app being spawned. When I pointed it to a real Python executable, it started to at least show something.

My worst case scenario is that I can just put my entire virtual environment into the packaged folder, which I would find via _MEIPASS and use the Python executable there. A rather roundabout way to do this, but I can’t seem to find any other. 

Should I be packaging the second process on its own? Can Merge handle this?

Thanks for the help. I have had more progress yesterday than I had in entire last month.

Best,

Burak

[Hartmut Goebel]

Am 08.04.2014 07:54, schrieb Burak Nehbit:

Have you bundled it using PyInstaller, too? Are you using MERGE?

No, I haven’t. I haven’t had taken a look at Merge—up until now my belief was that multiprocessing did not need such tools. I can run Twisted’s own spawnProcess without bundling the subprocess or without merge, so I would think Ampoule would not change anything. It’s nothing but a wrapper over spawnProcess.

Did I understand right? You haven't bundled the second program using PyInstaller at all?

If you second program is a Python-Program, and you are *not* shipping the Python interpreter and you are *not* bundling that program, how should this program run?

Well, I'd prefer having a requirements.txt and a PyInstaller .spec-file, so I can immediately start looking into the problem. :-)

My apologies, here’s the last working spec file I had. 

https://dl.dropboxusercontent.com/u/5815330/aether.spec

This should be something like:

a1 = Analysis(['/Volumes/Sky/Repos/aether/Code/aether/main.py'])
pyz1 = PYZ(a1.pure)
exe1 = EXE(pyz1, a1.scripts, ...)

a2 = Analysis(['/Volumes/Sky/Repos/aether/second-program.py'])
pyz2 = PYZ(a2.pure)
exe2 = EXE(pyz2, a2.scripts, ...)

coll = COLLECT(exe1, exe2
               a1.binaries, a1.zipfiles, a1.datas,
               a2.binaries, a2.zipfiles, a2.datas,
               strip=None,
               upx=True,
               name='Aether'
               )

I strongly suggest testing this with two very simple applications, both of which only printing some text.

Should I be packaging the second process on its own? Can Merge handle this?

Yes. No, Merge is the advanced usage. You should first get an simple app running, then your app, them you may try merge.

--
Schönen Gruß
Hartmut Goebel
Dipl.-Informatiker (univ), CISSP, CSSLP
Information Security Management, Security Governance, Secure Software Development

Goebel Consult, Landshut
http://www.goebel-consult.de

Blog: http://www.goebel-consult.de/blog/sag-nein-zum-routerzwang
Kolumne: http://www.cissp-gefluester.de/2010-09-mut-zur-beschraenkung

[Burak Nehbit]

Hi there,

Per the discussion copied below, as Hartmut have suggested I have split the app into two self-sufficient pieces, and packaged them separately, and voila, it works! 

Though, I am having an interesting problem packaging the final executable. When I package, the resulting Mac OS bundle immediately quits with this console error:

6/8/14 5:03:07.051 PM com.apple.launchd.peruser.501[267]: ([0x0-0x42d42d].Aether[16525]) Exited with code: 255

This is also not local to this particular app. I created a test case, which you can find below. I cannot package this test case into a bundle: or rather, I cannot produce a working bundle. When I go into the bundle, find the executable and run it, it works perfectly. It doesn’t work when clicked on the bundle however, resulting in the console log given above. The test code is below. You need Twisted and PyQt5, but otherwise it’s dependency-free. 

My impression is that this console log happens when the process crashes very early in the startup, but I do not know of any environment differences between launching an app directly from the executable (test-case) deep in the bundle, or outside the bundle by clicking onto it (test.case.app)

Can someone tell me whether this works for them? Is this a known bug, am I making a mistake? Or do I have a system-wide issue?

Here is the package with spec file and qt5reactor.py. Spec file is pretty vanilla, exactly as generated by Pyinstaller with the bundling request added to the bottom.

https://dl.dropboxusercontent.com/u/5815330/test-pyqt.zip

Here is the entire code of test-pyqt.py only:

from PyQt5.QtWidgets import QApplication, QMainWindow, QFileDialog, QWidget, QLabel

from PyQt5 import QtCore

import sys, datetime

class Example(QWidget):

    def __init__(self):

        super(Example, self).__init__()

        lbl1 = QLabel('Test Label', self)

        lbl1.move(15, 10)

        self.setGeometry(300, 300, 250, 150)

        self.show()

def main():

    app = QApplication(sys.argv)

    w = Example()

    w.setWindowTitle(str(datetime.datetime.now()))

    w.show()

    import qt5reactor

    qt5reactor.install()

    from twisted.internet import reactor

    sys.exit(reactor.run())

if __name__ == '__main__':

    main()

Respectfully,

Burak