PyInstaller not bringing imports of hidden imports?

[Michael Sverdlik]

Hey guys,

I've noticed this behaviour and I'm not sure if this is by design or a bug.

I've added two cases in my Github gist: https://gist.github.com/m1keil/8605617be9dd473e056a

In the good case, when I have static import, PyInstaller run will detect all submodules (with my custom hook) and will also detect all the dependencies of these submodules.

In the bad case, when I have dynamic import, I add --hiddenimport flag, PyInstaller run detects all submodules (again, with the hook) but will not detect submodule's dependencies.

You can see that there's a huge difference in the number of modules went into each one of the binaries.

I thought that hiddenimport behaviour is the same as if I had explicit import statement in my script?

Thank you!

[Martin Zibricky]

On Tuesday 04 of November 2014 03:41:40 Michael Sverdlik wrote:
> PyInstaller run detects all submodules (again, with the hook) but will not
> detect submodule's dependencies.
> You can see that there's a huge difference in the number of modules went
> into each one of the binaries.

It's a bug. PyInstaller should detect dependencies of submodules.

[Hartmut Goebel]

Am 04.11.2014 um 12:41 schrieb Michael Sverdlik:

I thought that hiddenimport behaviour is the same as if I had explicit import statement in my script?

Yes, it is. Both ways use the same code.

--
Schönen Gruß
Hartmut Goebel
Dipl.-Informatiker (univ), CISSP, CSSLP
Information Security Management, Security Governance, Secure Software Development

Goebel Consult, Landshut
http://www.goebel-consult.de

Blog: http://www.goebel-consult.de/blog/in-need-for-an-enhanced-git-url-scheme
Kolumne: http://www.cissp-gefluester.de/2010-08-scheingefechte-um-rim

Goebel Consult ist Mitglied bei http://www.7-it.de/

[Michael Sverdlik]

Oh, ok good to know I wasn't losing my mind for nothing :-)

Is it open on Github or would you like me to report it?

Is there a known workaround for this? Maybe in dev branch or any other smart way to bypass this? (My only idea is to scan each module for *.py and add these to the scripts list in the Analysis phase..)

[Martin Zibricky]

On Tuesday 04 of November 2014 03:52:51 Michael Sverdlik wrote:
> Oh, ok good to know I wasn't losing my mind for nothing

> Is it open on Github or would you like me to report it?

I don't think this is reported. You can report it on github.

[Steve Barnes]

On 04/11/14 11:52, Michael Sverdlik wrote:
> Oh, ok good to know I wasn't losing my mind for nothing :-)
> Is it open on Github or would you like me to report it?
>
> Is there a known workaround for this? Maybe in dev branch or any other
> smart way to bypass this? (My only idea is to scan each module for
> *.py and add these to the scripts list in the Analysis phase..)
>

In the past when I had some imports that were not being picked up this
way that I did/could not have directly imported I found that
pyInstaller/py2exe would find them if I included in the module, if
False: import some_module - a bit of a hack but it worked.

Gadget/Steve

[Michael Sverdlik]

Thanks Steve, 

I've checked it and indeed this works. I'm not sure I'll solve my problem this way (other devs won't be thrilled about it), but good to know.

[Steve Barnes]

On 04/11/14 12:13, Michael Sverdlik wrote:
> Thanks Steve,
> I've checked it and indeed this works. I'm not sure I'll solve my
> problem this way (other devs won't be thrilled about it), but good to
> know.

The other "hack" or "work around" that I have used on occasion is to
have a variation of the __import__ class that, when run from source,
i.e. not Frozen, produces a file that "just" imports the relevant
modules and twist the exe builders tail into also building this, for
shared libraries this will ensure that all the libraries that I need are
included in the share, then don't ship the "dummy" executable, or of
course have it simply display a message. One sneaky is to have a dummy
program that is generated and has as it's main method a display package
information function.

That way the other developers don't need to know about the manual task
of adding if False statements and my build process can run a command to
(re-)generate the dummy file.

Gadget/Steve

[Michael Sverdlik]

Just posting this in case someone else might bump into it down the road:

Since we use the entry point hack, I decided to extend it a bit and add code that will create a dummy file with explicit imports:

def Entrypoint(dist, group, name, distributions,
               scripts=None, pathex=None, hiddenimports=None,
               hookspath=None, excludes=None, runtime_hooks=None):
    import pkg_resources

    def get_toplevel(dist):
        distribution = pkg_resources.get_distribution(dist)
        return list(distribution.get_metadata('top_level.txt').split())

    packages = []
    for distribution in distributions:
        packages += get_toplevel(distribution)

    scripts = scripts or []
    pathex = pathex or []
    # get the entry point
    ep = pkg_resources.get_entry_info(dist, group, name)
    # insert path of the egg at the verify front of the search path
    pathex = [ep.dist.location] + pathex
    # script name must not be a valid module name to avoid name clashes on import
    script_path = os.path.join(WORKPATH, name + '-script.py')
    print "creating script for entry point", dist, group, name
    with open(script_path, 'w') as fh:
        fh.write("import {0}\n".format(ep.module_name))
        fh.write("{0}.{1}()\n".format(ep.module_name, '.'.join(ep.attrs)))
        for package in packages:
            fh.write("import {0}\n".format(package))

    return Analysis([script_path] + scripts, pathex, hiddenimports, hookspath, excludes, runtime_hooks)

[Martin Zibricky]

On Sunday 09 of November 2014 22:51:29 Michael Sverdlik wrote:

> Since we use the entry point hack 

> <http://www.pyinstaller.org/wiki/Recipe/SetuptoolsEntryPoint>, I decided to 

> extend it a bit and add code that will create a dummy file with explicit

> imports:

 

Hi Michael,

 

I moved the recipe to github and put there your example:

https://github.com/pyinstaller/pyinstaller/wiki/Recipe-Setuptools-Entry-Point

 

Could you please update this wiki page if necessary?

 

Thanks

 

Martin

 

[Michael Sverdlik]

Sure Martin, 

will take a look tomorrow.