Freezing Gourmet / Hidden imports and hooks

[Bernhard Reiter]

Hi,

I'm the Windows version maintainer of Gourmet Recipe Manager [1]. As I
haven't done any release for more than a year due to problems with
py2exe not working with PyGTK, I thought I'd give pyInstaller a shot. My
code is in the pyinstaller branch in Gourmet's git repository [2,3].

I've created a basic spec file (windows/Gourmet.spec) and already
included some stuff that Gourmet needs on startup, but I'm now stuck
with some hidden imports (of some localized default values which are
imported only for the current locale). I've tried to add a hook as
described in the manual, but my resulting Gourmet.exe still complains
about not finding defaults_en.

As I'd really like to roll out another Windows release soon, I'd be
grateful for any help in order to make this happen!

Kind regards
Bernhard Reiter

[1] http://grecipe-manager.sf.net
[2]
http://grecipe-manager.git.sourceforge.net/git/gitweb.cgi?p=grecipe-manager/grecipe-manager;a=shortlog;h=refs/heads/pyinstaller
[3]
git://grecipe-manager.git.sourceforge.net/gitroot/grecipe-manager/grecipe-manager

[Hartmut Goebel]

Hi,

welcome to PyInstaller.

Am 27.12.2011 15:55, schrieb Bernhard Reiter:
> I've tried to add a hook as
> described in the manual, but my resulting Gourmet.exe still complains
> about not finding defaults_en.

So you already did find your way to the hooks :-) Did you wollow the
instructions at
<http://www.pyinstaller.org/wiki/HowtoReportBugs#Beforesubmittingareport:Makesureeverythingispackaged>
to find out why your stuff is missing? Ep. logdict*.log and warn*.txt
should help you.

In your spec-file I discovered there are absolute, user and system
dependant paths (C:\\Users\\Bernie\\workspace\). You should replace
these by independent (and relative) paths.

Further remarks: Gourmet.pyw imports a lot of stuff which is not
necessary to be imported. PyInstaller should resolve most of it
automatically. See the first lines and the encoding.* stuff. If you are
going to keep it, I suggest order the imports according to PEP 8 and
keep line length below 80 chars.

I just added Gourmet Recipe Manager to
http://www.pyinstaller.org/wiki/ProjectsUsingPyInstaller. Okay?

--
Schönen Gruß - Regards
Hartmut Goebel
Dipl.-Informatiker (univ.), CISSP, CSSLP

Goebel Consult
Spezialist für IT-Sicherheit in komplexen Umgebungen
http://www.goebel-consult.de

Monatliche Kolumne: http://www.cissp-gefluester.de/
Goebel Consult ist Mitglied bei http://www.7-it.de

[Bernhard Reiter]

Am Mittwoch, den 28.12.2011, 13:56 +0100 schrieb Hartmut Goebel:
> Hi,
>
> welcome to PyInstaller.

Hi,

and thanks for the welcome and reply!

> [...]

> So you already did find your way to the hooks :-) Did you wollow the
> instructions at
> <http://www.pyinstaller.org/wiki/HowtoReportBugs#Beforesubmittingareport:Makesureeverythingispackaged>
> to find out why your stuff is missing? Ep. logdict*.log and warn*.txt
> should help you.

I've taken a look at those, but I think my problem is quite clear --
Gourmet.exe issues

[...]
File ".\build\pyi.win32\Gourmet
\outPYZ1.pyz/gourmet.defaults.defaults", line 27, in <module>
File "C:\Python27\Tools\pyinstaller-1.5.1\iu.py", line 455, in
importHook
raise ImportError, "No module named %s" % fqname
ImportError: No module named defaults_en

What's not so clear if the solution: AFAICS, the proper way to fix this
sort of hidden imports is to add a hook. I've experimented with a lot of
different names for the hooks that seemed to make sense (e.g.
hooks-gourmet.defaults.defaults.py or
hooks-gourmet.defaults.defaults.lang.py), but no success so far.
gourmet.defaults.defaults_en isn't in Gourmet.exe's outPYZ1.pyz archive,
either; the only modules starting with gourmet.defaults are

'gourmet.defaults': (True, 2511381L, 141),
'gourmet.defaults.defaults': (False, 1515441L, 1469),

I've added the warnGourmet.txt and logdict*.txt files now to the git
repo (pyinstaller branch again, of course), so maybe you (or someone
else) can point me to what I'm missing...

> In your spec-file I discovered there are absolute, user and system
> dependant paths (C:\\Users\\Bernie\\workspace\). You should replace
> these by independent (and relative) paths.

Hm, I removed those with a commit before I wrote my last mail -- see
http://grecipe-manager.git.sourceforge.net/git/gitweb.cgi?p=grecipe-manager/grecipe-manager;a=blob;f=windows/Gourmet.spec;h=b5b35e110ef1ab1115a3c5a76650ac42bcbab6fc;hb=refs/heads/pyinstaller
Could it be that you used a revision that dates before that commit?

> Further remarks: Gourmet.pyw imports a lot of stuff which is not
> necessary to be imported. PyInstaller should resolve most of it
> automatically. See the first lines and the encoding.* stuff. If you are
> going to keep it, I suggest order the imports according to PEP 8 and
> keep line length below 80 chars.

That's part of my heritage as Gourmet's Windows maintainer ;-) but I'd
like to tackle that separately after I've successfully built a working
Gourmet.exe, unless it's absolutely necessary.

Anyway, it's pretty damn cool that I'm now able to do this type of
diagnosing from within Linux, using pyinstaller from within Wine... No
more need to reboot like I had to with py2exe. (BTW, the error was the
same under Windows, so that doesn't seem to make a difference.)

> I just added Gourmet Recipe Manager to
> http://www.pyinstaller.org/wiki/ProjectsUsingPyInstaller. Okay?

Okay if I really can get it to work ;-)

Regards
Bernhard

[Hartmut Goebel]

Am 28.12.2011 20:56, schrieb Bernhard Reiter:
> 'gourmet.defaults': (True, 2511381L, 141),
> 'gourmet.defaults.defaults': (False, 1515441L, 1469),
>
> I've added the warnGourmet.txt and logdict*.txt files now to the git
> repo (pyinstaller branch again, of course), so maybe you (or someone
> else) can point me to what I'm missing...

Thanks for the files.

Gourmet obviously does some ugly code. This `defaults.defaults` modules
is a crab. Best would be to clean up the source.

Your hook must be named `defaults.defaults`, since default.defaults.lang
does not exit but is faked.

A hard-coded hidden import may not solve your problem. I suggest using a
glob to scan for the files. See hooks/hook-django.core.cache.py for how
it is done there.

You may also selt the attribute `lang`, like done for search_function in
`hook-encodings.py`.

HTH.

--
Sch�nen Gru� - Regards

Hartmut Goebel
Dipl.-Informatiker (univ.), CISSP, CSSLP

Goebel Consult
Spezialist f�r IT-Sicherheit in komplexen Umgebungen

[Bernhard Reiter]

Am Donnerstag, den 29.12.2011, 11:05 +0100 schrieb Hartmut Goebel:
> Your hook must be named `defaults.defaults`, since default.defaults.lang
> does not exit but is faked.
>
> A hard-coded hidden import may not solve your problem. I suggest using a
> glob to scan for the files. See hooks/hook-django.core.cache.py for how
> it is done there.

I've just added hooks/hooks-gourmet.defaults.defaults.py and pushed it
to the pyinstaller branch. It's just hooks/hook-django.core.cache.py
with very basic modifications -- so it's probably not as much of a
surprise that it doesn't work yet. But in order to get it to work, I'd
need some way of debugging it, but just e.g. adding "print modpath" to
the hook doesn't have any effect, neither while running Build.py, nor
when running the generated exe. So can you tell me how to achieve this?

> You may also selt the attribute `lang`, like done for search_function in
> `hook-encodings.py`.

But that would only work if I was dealing with only one language to
include, right?

Regards
Bernhard Reiter

[Hartmut Goebel]

Am 21.01.2012 15:45, schrieb Bernhard Reiter:
> I've just added hooks/hooks-gourmet.defaults.defaults.py and pushed it
> to the pyinstaller branch. It's just hooks/hook-django.core.cache.py
> with very basic modifications -- so it's probably not as much of a

You should at least have removed the copyright-message :-)

> surprise that it doesn't work yet. But in order to get it to work, I'd
> need some way of debugging it, but just e.g. adding "print modpath" to
> the hook doesn't have any effect, neither while running Build.py, nor

This definitely means the, the hook is not triggered,

a) Run pyinstaller with --log=DEBUG

b) Try hooking on gourmet.defaults.

c) If this hook does not get triggered, too, try using the full path in
the .spec file to avoid problems with the current working directory not
being the project dir.

hookspath='hooks')

d) Try a recent version of PyInstaller development branch and use the
new "hiddenimport" argument. List the modules you find in FreezeGourmet.bat