how to use find_module in my code when module is in outPYZ1.pyz

[David Lai]

Hi Hartmut,
    This issue bother me for almost 2 weeks. i try the release version with --onefile or --onedir, it's not work. and i also try the development version with --onefile or --onedir, still no work.
   I am just wondering about  how should i use find_module() when application run. is there something different between the executable which Pyinstaller produce   and the normal execution method which running application directly using "python myapp" ?
  The following is all the information you need to locate the problem:

my own app has the directory structure like the following:
    application.py (the main entry)
    server.py
    app (a directory)
      |__common 
      |            |____commonservice.py
      |__helpers
      |            |____...(many .py files and some sub-directory)
      |__controllers
      |            |____request_login.py
      |            |____redirect.py
      |            |____...(many .py files and some sub-directory)
      |__....

1. release version 
[problem description] 
can not import the  modules in the "app/controllers"  directory of my own app, both --onefile and --onedir got the same problem.
The modules in the controllers is import by load_source() in the source code application.py  when the app is running but if pyinstaller can't support this, import all of it during the building period of exectuable is another good idea but i don't know how to do it using pyinstaller
except for the find_module problem, the executable file(--onefile mode) can run successfully !

[python version] 2.7.2  http://www.python.org/download/releases/2.7.2/
[pyinstaller version] 1.5.1 http://www.pyinstaller.org/
[platform version] SuSE Linux 10 SP2 x86_64(2.6.16.60-0.21-smp)

[sample code]
 g = globals()
 modules = [] 
 for strPath, lstDirs, lstFiles in os.walk(controllers_dir):

        if lstFiles == []:
            continue
    
        modules.extend([(f[:-3], os.path.join(strPath, f)) for f in lstFiles if f.endswith('.py')])

for (name, file) in modules:
   if name not in g:
            import_module(name, file)

def import_module(name, file):
        m = load_source(name, file)
    g[name] = m

[hook import]
  i tried make some hook file in the hooks directory in pyinstaller directory which filename like hook-app.controllers.py, hook-app.controllers.redirect.py and etc. the hook file content like hiddenimports = ['app.controllers.redirect']

whether i choose --onefile or --onedir, the outPYZ1.pyz does not include any of my modules in the app/controllers directory
? O outPYZ1.pyz
 Name: (ispkg, pos, len)
{'BaseHTTPServer': (False, 862035, 8577),
 'ConfigParser': (False, 592334, 9029),
 'Cookie': (False, 1248832, 8924),
 ......
 'app': (True, 2203686, 106),
 'app.common': (True, 298627, 113),
 'app.common.bzCompress': (False, 169204, 692),
 'app.common.commonservice': (False, 911797, 6360),

2.development version
[problem description] 
i try development version , supprisingly, by adding some config item in .spec  app, both --onefile and --onedir can import the modules in "app/controllers"! 
 hiddenimports = ['app.controllers.redirect', 'app.controllers.request_login'] make the import work!
but if i call find_module in main entry , it still not found the module which has been import succesfully in outPYZ1.pyz
but the most worst thing is when i run the exectuable produced by development version, it throw some exception which is OK in release version .

Traceback (most recent call last):
  File "<string>", line 181, in <module>
  File "<string>", line 99, in config_session
  File "/opt/buildengr/build/pyi.linux2/application/out01-PYZ.pyz/web.db", line 1142, in database
  File "/opt/buildengr/build/pyi.linux2/application/out01-PYZ.pyz/web.db", line 1011, in __init__
  File "/opt/buildengr/build/pyi.linux2/application/out01-PYZ.pyz/web.db", line 1007, in import_driver
ImportError: Unable to import sqlite3 or pysqlite2.dbapi2 or sqlite


[python version] 2.7.2  http://www.python.org/download/releases/2.7.2/
[pyinstaller version] 1.5.1 http://www.pyinstaller.org/
[platform version] SuSE Linux 10 SP2 x86_64(2.6.16.60-0.21-smp)


so the issue is so weird now, if i used release version, running executable is OK but can not import some modules and find_module;  if i used development version, running executable throws exception but can import modules, still can't fine_module.

your help will be fully appreciate , if you need any other thing pls let me know, thanks in advance.
 :)

[Hartmut Goebel]

Hi David,

    This issue bother me for almost 2 weeks.

If you'd follow your debugging guide line you'd have solved it already. SCNR. I suggest reading the manual

I still have to guess, what is the problem. I assume:

1) PyInstaller does not bundle you modules in `app/controllers`. It looks as if you solved this by using the development branch and hiddenimports.

2) Now you frozen application does not find the modules. This a *very* different problem.

[sample code]
 g = globals()
 modules = [] 
 for strPath, lstDirs, lstFiles in os.walk(controllers_dir):

        if lstFiles == []:
            continue
    
        modules.extend([(f[:-3], os.path.join(strPath, f)) for f in lstFiles if f.endswith('.py')])

for (name, file) in modules:
   if name not in g:
            import_module(name, file)

def import_module(name, file):
        m = load_source(name, file)
    g[name] = m

This code can not work in your frozen app for several reasons:

• modules are held within th PYZ archive and are not extracted to the filesystem (neither in --onefile nor in --onedir mode)
• module in the PYZ are name like `app.controllers.foo` (nut `app/controllers/foo.py`
  

You are talking about `find_module`, but there is no `find_module` in this code?! What `find_module` do you mean at all?

  File "/opt/buildengr/build/pyi.linux2/application/out01-PYZ.pyz/web.db", line 1007, in import_driver
ImportError: Unable to import sqlite3 or pysqlite2.dbapi2 or sqlite

This seams to be yet another, unrelated problem: sqlite is missing. You you need to check if its in the PYZ and how to get it there.

so the issue is so weird now, if i used release version, running executable is OK but can not import some modules and find_module;  if i used development version, running executable throws exception but can import modules, still can't fine_module.

To be frank: the issue is not weird at all. You are trying to solve quite different problem at once. This is why we wrote the guideline at http://www.pyinstaller.org/wiki/FAQ#IfThingsGoWrong.

-- 
Schönen Gruß - Regards
Hartmut Goebel
Dipl.-Informatiker (univ.), CISSP, CSSLP

Goebel Consult 
Spezialist für IT-Sicherheit in komplexen Umgebungen
http://www.goebel-consult.de

Monatliche Kolumne: http://www.cissp-gefluester.de/
Goebel Consult ist Mitglied bei http://www.7-it.de

[David Lai]

Hi，
the faq I read so many times and also the manule too.
I post the sample code because it's my original code and it runs OK not using pyinstaller . Why I use "for loop " to load_source is because there are so many .py
Files which need to import and may increase in the future.
First, I use the --onefile mode and it fails to add the .py files ,and I try to add this as collect data files into pyz ,it' ok. but load_source run failed.
Then I chang my mind to modify my original code. If I can hidden import the module,I don' need to add the original .py files in PYZ,just hidden import it and use find_module() to get the module instance and my code can run without so many change.but hidden import can't take effect in release version,using development version the hidden import is OK but I faced another error like sqlite3 can not find.

After many tries
I finally resolved this problem today by using so many explicitly import sentence in the main entry code and the release version 1.5.1 can recognize all the modules using sys.modules, and I filter the sys.modules to get all the modules within app/controllers. I don't need any find_module() then.
Now the executable produced by release version 1.5.1 with -onefile mode runs successfuly. But I still not know how the import things work with py installer.
Anyway, pyinstaller is a very usefull tool to package and release our python project because it can run independently in the product enviorment, we don't have to consider about the different python enviorment our app runs.
Thanks for your reply.

[Hartmut Goebel]

Am 27.03.2012 16:39, schrieb David Lai:

Hi，
the faq I read so many times and also the  manule too. 

So we need to overhaul the text as it seams you did not get the key message. Thanks for this feedback.

There are different areas of typical problems which can occur when using PyInstaller. For fixing problems in your application, it is *critical* to understand this.

1. Some required modules, shared libs or data files are not frozen (packaged, bundled) into the resulting exec. This kind of problems are to be solved within the "freeze" phase by

• using `hiddenimport` in the `.spec`-file. (this is new feature in current development branch)
• collecting files or modules within the `.spec`-file
• implementing a new hook (or fixing an existing one)
  
• fixing a bug in PyInstaller

To check if all your files are frozen, use `utils/Archiveviewer.py`.

2. You application can not import some module, list modules, find data files, etc. This kind of problems has to be solved within our application by:

• adapting your application to be frozen (see the manual)
• If generic parts of your application are effected (e.g. commonly used packages like Tkinter, GTK, QT), adding a runtime-hook will help others users

Prior to solving "Type 2" problems, yo need to ensure there are no "Type 1" problems. You first of all you need to find out which type of problem you are facing. See the FAQ for how to do this.

Please note: If your application is complex, there may be several "Type 1" and several "Type 2" problems.

Okay yo far?

Meanwhile you solved (some of) your "Type 1" problems by using explicitly import statements in your main entry code.

I post the sample code because it's my original code and it runs OK not using pyinstaller . Why 

So this sample code id in your application? It can nor work, since all modules are packaged into the executable. So scanning the directory does not work.

A possible solution to your problem may be:

In your .spec file, collect all modules (some like as in your sample code), write the module names into a file and and this file to your frozen app (see the manual for "adding data files"). In addition all all these modules as `hiddenimport`. Thus you'll get all modules into the frozen app and you'll get a list of all available modules.

The in your application, if it is frozen (see the manual), read this data-file to get a list of all modules and import them.

but I faced another error like sqlite3 can not find.

I assume, this is yet another problem.

But I still not know how the import things work with py installer.

Paron? I don't understand this.