Is there any way to specify which modules to import?

[abhishek bhatta]

I am creating an exe with minimal modules yet the pyinstaller exe size grows large. 

Is there any way to import only specific modules and exclude the rest. 

I came across the exclude feature in the spec file but there is an overhead that becomes difficult to maintain.

Thanks,

Abhishek

[Amir Rossert]

Hi, try to look at —hidden-import and —exclude-module flags
https://pythonhosted.org/PyInstaller/usage.html#what-to-bundle-where-to-search

[abhishek bhatta]

Thanks Amir,

I looked into that but there are too many packages that needs to be in the exclude list. That is going to be an overhead, is there some way to just point which packages to import.

[Eric Fahlgren]

But that's basically what you are doing by feeding PyInstaller your script.  PyInstaller scans your source for imports and only includes what it finds by searching the dependencies exhaustively.  I'm guessing if you try to do this manually, you are going to run into grief pretty quickly as it's a very tricky task to know exactly what underlying packages and SOs/DLLs each package needs to run.

--
You received this message because you are subscribed to the Google Groups "PyInstaller" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyinstaller...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyinstaller/7052d4a3-3cf6-4c12-99bc-ad0cf1b97b34%40googlegroups.com.

[Chris Barker]

On Fri, Mar 27, 2020 at 2:13 AM abhishek bhatta <bhatta...@gmail.com> wrote:

I looked into that but there are too many packages that needs to be in the exclude list.

you might take a step back and try to figure out *why* PyInstller thinks you need all those packages.

Perhaps you can do your imports differently, and get a more compact install.

-CHB

 

That is going to be an overhead, is there some way to just point which packages to import.

On Friday, March 27, 2020 at 12:50:07 AM UTC+5:30, Amir Rossert wrote:

Hi, try to look at —hidden-import and —exclude-module flags
https://pythonhosted.org/PyInstaller/usage.html#what-to-bundle-where-to-search

--

You received this message because you are subscribed to the Google Groups "PyInstaller" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyinstaller...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyinstaller/7052d4a3-3cf6-4c12-99bc-ad0cf1b97b34%40googlegroups.com.

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Steve Barnes]

My favourite technique to identify which dependencies I need is to create a clean venv and then try testing my code with it active. If I get an import error I can either look at why I need that import or, if it really is needed, simply pip install within the venv.

 

Of course running pylint and addressing any unneeded/unused imports helps a lot as well.

 

Steve Barnes

To view this discussion on the web visit https://groups.google.com/d/msgid/pyinstaller/CALGmxE%2BKtD5z29pxk2ncf2OLW4g9XAt19iCuM5UF9whmfnpOVQ%40mail.gmail.com.

[abhishek bhatta]

Thanks Eric and Chris,

I understand that Pyinstaller is checking and importing the dependencies but there must either be some bug with that part or I am missing something.

Here is the full scenario.

Here is all the imports I am using in my original code:

#>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

#Import

#>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

import sys

import time

import socket, errno

import struct

import datetime as dt

from PyQt5.QtCore import *

from PyQt5.QtGui import *

from PyQt5.QtWidgets import *

from collections import OrderedDict

from functools import reduce

from math import *

from statistics import mean

from threading import Thread, Lock

import os

import fnmatch

import paramiko

import shutil

import subprocess

import win32process

import psutil

import winreg

import numpy as np

print("This is a test")

#>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

#End

#>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

I saved it with the name "test.py". 

If I try to compile this into an exe with the following command "pyinstaller --win-private-assemblies --win-no-prefer-redirects -w -F -n test.exe test.py"

The size of the exe comes up to 61.1 MB

Now if I just exclude scipy and execute the following command "pyinstaller --win-private-assemblies --win-no-prefer-redirects --exclude scipy -w -F -n test.exe test.py"

The size of the exe goes to 52.9 MB

Ques: Why does Pyinstaller consider scipy as a necessary import?

Thanks Steve,

I am going to look into the venv setup and only install the necessary packages there to find out about any other issues. 

Also, pylint is a good way to go as well.

Regards,

Abhishek Bhatta 

[Steve Barnes]

Abhishek,

 

Creating a venv and installing the packages that you list in the example below I found that SciPy was not installed in the process, so it is not a declared dependency of any of them!  However, running `grin scipy` in the venvs library directory I found a number of conditional imports of scipy – so basically if you have it available it will be considered a, possibly conditional, import and so will be included by pyinstaller and most similar packages but if it is missing it almost certainly will not cause an issue.

 

You can see why I like venv for building my code into executables/deliverables.

 

Steve

--

You received this message because you are subscribed to the Google Groups "PyInstaller" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyinstaller...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pyinstaller/551c94ee-d187-47f8-8127-06e4273c3080%40googlegroups.com.

[Chris Barker - NOAA Federal]

Optional imports are the bane of systems like this:-(

Also, note that SciPy imports most of its sub-modules automatically, so if you need one function, you get the whole thing. 

Though in this case, it looks like you don’t need it at all, so that’s good.

Meanwhile, QT is huge, and you do need that, so you may have done all you can. 

I have always thought that a good way to do module finding would be to not try to parse the source, but rather, run a test suite, and then dump sys.modules and you’d have a list of every module that was actually used by your application.

-CHB

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

On Mar 28, 2020, at 8:36 AM, Steve Barnes <Gadge...@live.co.uk> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/pyinstaller/VI1PR03MB4479FA5E85CDC5A4E5F55CB69BCD0%40VI1PR03MB4479.eurprd03.prod.outlook.com.

[Eric Fahlgren]

On my system (Win 10 Pro 64,  Py 3.8.1) numpy itself is almost all of that and SciPy 1.5x that, again.  Throw in Qt and I'm surprised you're not well over 100 meg.  Our installer is over 350 Mb with numpy, scipy, wx and vtk...  Just because you don't use something directly doesn't mean it's not necessary to resolve dependencies.

     57,630,720  T:/Python38/Lib/site-packages/numpy/

    109,117,440  T:/Python38/Lib/site-packages/scipy/

[Chris Barker]

yup -- a lot of this is BIG.

numpy is 23MB on my Mac -- yours seems big -- maybe using MKL?

You may not be using all of numpy, but it's really hard to tease out what you are really using mostly it doesn't matter: disk space is really cheap these days. Bandwidth not always, but still, folks are used to big downloads.

-CHB

To view this discussion on the web visit https://groups.google.com/d/msgid/pyinstaller/CAP2Qz%2BVruN_ekZUOkSpCMOkO_Og6Xw8rFhc6z6286n__%2BzKXMA%40mail.gmail.com.

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Eric Fahlgren]

We did use MKL, back before numpy was distributed as a PyPi package (big thumbs up to Christoph Gohlke for his long-time support there), but I'm pretty sure the "standard" distro does not use it.  Most of the space consumed by current Windows CPython numpy is one library:

     33,124,352  libopenblas.PYQHXLVVQ7VESDPUVUADXEVJOBGHJPAY.gfortran-win_amd64.dll

We force all of numpy and scipy into our installation framework which does multi-body dynamics simulation, since our end users could write scripts that do virtually anything (optimizations, DOE, linear regression, on and on), and we don't want to leave something out that might be useful.

To view this discussion on the web visit https://groups.google.com/d/msgid/pyinstaller/CALGmxEK-b-rJprJ%2B8jOWj53JG3tUgB0_FzR%2BcVxtn2pO60T6Nw%40mail.gmail.com.

[Adriano Gil]

Hi guys, how can I see the size of each module embedded in executable?

[abhishek bhatta]

Thank you everyone for the very useful inputs, I believe the best solution is going the virtual environment way with python and only installing the bare minimum necessary modules there. 

I will consider this post as a closed issue unless someone else comes up with an even better solution

Adriano,

Please start a new discussion with the question as I don't think you'll get the response here.