making executables smaller

[Carter Temm]

Hi,
I’m writing a couple different projects at the moment, and when I compile it into a single executable using pyinstaller, it becomes extremely large. I’m guessing this is because of the modules used. Because I’m not that skilled at python, I put stuff like for example, import sys. I imagine the final project could be made smaller by specifying from something import something_else. but the thing is, I don’t know what smaller I could import with these set of modules. Is there a program that could tell me this. Sorry if this question is really basic, but it’d be helpful.

[Wildman]

Try importing only the modules you actually use.
For example, instead of:

import os

Try using:

import os.path
import os.environ
import os.whatever

--
<Wildman> GNU/Linux user #557453
The cow died so I don't need your bull!

[Carter Temm]

OK. So I guess the question should be, how can I make these executables smaller in general?

Sent from my iPhone

> On Jul 26, 2016, at 5:13 PM, Dennis Lee Bieber <wlf...@ix.netcom.com> wrote:
>
> On Tue, 26 Jul 2016 12:22:16 -0500, Carter Temm <crtbr...@gmail.com>
> declaimed the following:

>
>> Hi,
>> I’m writing a couple different projects at the moment, and when I compile it into a single executable using pyinstaller, it becomes extremely large. I’m guessing this is because of the modules used. Because I’m not that skilled at python, I put stuff like for example, import sys. I imagine the final project could be made smaller
> by specifying from something import something_else. but the thing is, I don’t know what smaller I could import with these set of modules. Is there a program that could tell me this. Sorry if this question is really basic, but it’d be helpful.
>

> "from module import name" still has to include the entire module --
> since that is the file. It is effectively the same as doing
>
> import module
> name = module.name
> del module
>
> Also -- anything that creates an executable file for Python will be
> including the Python interpreter ITSELF. That may be a lot of the bloat you
> see.
> --
> Wulfraed Dennis Lee Bieber AF6VN
> wlf...@ix.netcom.com HTTP://wlfraed.home.netcom.com/
>
> --
> https://mail.python.org/mailman/listinfo/python-list

[Steven D'Aprano]

On Wed, 27 Jul 2016 03:22 am, Carter Temm wrote:

> Hi,
> I’m writing a couple different projects at the moment, and when I compile
> it into a single executable using pyinstaller, it becomes extremely large.

What do you consider "extremely large"? Ten gigabytes? 500 kilobytes? Give
us a clue.

> I’m guessing this is because of the modules used.

Maybe yes, maybe no.

On my system, Python 3.3 is a little bit less than 6 MB, and the std lib
under 130MB:


[steve@ando ~]$ du -hs /usr/local/bin/python3.3
5.7M /usr/local/bin/python3.3
[steve@ando ~]$ du -hs /usr/local/lib/python3.3/
129M /usr/local/lib/python3.3/

But nearly 50MB of that is the test suite:

[steve@ando test]$ du -hs /usr/local/lib/python3.3/test/
48M /usr/local/lib/python3.3/test/

I expect that on Windows or Mac OS X the sizes will be roughly the same.

I'm not an expert on the pyinstaller internals, but I would expect that it
would be able to drop the test suite, but will need to include the rest of
the std lib as well as the interpreter, plus whatever files you have
written. So I expect that a frozen executable will be of the order of 80MB,
give or take. There's probably a bit of overhead needed to hold it all
together, so let's say 100MB in round figures.

If you're getting less than 100MB, that doesn't sound too bad to me, not for
an interpreted language like Python. Anything less than that sounds really
good to me. If you are getting under 20MB, that's *fantastic*. What's the
problem with that? You can fit close to forty of these on a 4GB DVD-RW or
USB stick. You're not hoping to fit your application on a floppy disk are
you? Its 2016, not 1980.

If you're working with really constrained environments, like an embedded
device, then check out µPy or PyMite:

https://micropython.org/

http://deanandara.com/PyMite/

although I fear PyMite may not be under active development.

> Because I’m not that
> skilled at python, I put stuff like for example, import sys. I imagine the
> final project could be made smaller by specifying from something import
> something_else.

No, it doesn't work like that. For starters, sys is built into the
interpreter, so it's unlikely you'll be able to remove it. But more
generally, if you use anything from a module, the entire module must be
included.

Even if *you* don't use a module, perhaps one of the modules you do use will
in turn use that module.


Of course, you can avoid all this overhead completely by *not* freezing the
executable down to a single file. Just distribute your Python code as a .py
file, or a .pyc file. There are other options as well, such as distributing
it bundled into a zip file. What benefit do you get from using PyInstaller?

> but the thing is, I don’t know what smaller I could import
> with these set of modules. Is there a program that could tell me this.
> Sorry if this question is really basic, but it’d be helpful.

--
Steven
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

[Thomas 'PointedEars' Lahn]

Carter Temm wrote:

> I’m writing a couple different projects at the moment, and when I compile
> it into a single executable using pyinstaller, it becomes extremely large.
> I’m guessing this is because of the modules used. Because I’m not that
> skilled at python, I put stuff like for example, import sys. I imagine the
> final project could be made smaller by specifying from something import
> something_else. but the thing is, I don’t know what smaller I could import
> with these set of modules. Is there a program that could tell me this.

I recommend to comment out all “import” statements (for later reference) and
then use a Python editor like PyDev to generate step by step “from … import
…” statements for all used symbols that are not yet defined.

If it is a rather simple program where you can easily make no-ops out of
modifying statements by defining mockups, you can just run it before you
compile it to see where you get errors with commented-out “import”
statements.

In theory, you could look for code of the form “foo.bar” using the regular
expression "(?:"")?(?:[^"\\]|\\"(?:"")?)+"(?:"")?|'(?:'')?(?:[^'\\]|
\\'(?:'')?)+'(?:'')?)|((?:\w+\.)+)(\w+)¹. If $1 (e.g., “foo”) is not
defined in the file, then it is likely that $2 (e.g., “bar”) is a symbol
from module $1 and you could write

#-----------------
from $1 import $2
…
$2
#-----------------

instead of

#-----------------
import $1
…
$1.$2
#-----------------

But there can be false positives and duplicate symbols this way.

_________
¹ the first two alternatives exclude Python string literals
--
PointedEars

Twitter: @PointedEars2
Please do not cc me. / Bitte keine Kopien per E-Mail.

[Steven D'Aprano]

On Wednesday 27 July 2016 14:52, Thomas 'PointedEars' Lahn wrote:

> Carter Temm wrote:
>
>> I’m writing a couple different projects at the moment, and when I compile
>> it into a single executable using pyinstaller, it becomes extremely large.
>> I’m guessing this is because of the modules used. Because I’m not that
>> skilled at python, I put stuff like for example, import sys. I imagine the
>> final project could be made smaller by specifying from something import
>> something_else. but the thing is, I don’t know what smaller I could import
>> with these set of modules. Is there a program that could tell me this.
>
> I recommend to comment out all “import” statements (for later reference) and
> then use a Python editor like PyDev to generate step by step “from … import
> …” statements for all used symbols that are not yet defined.

What benefit do you think you will gain from changing (let's say):

import collections
x = collections.deque(foo)

to:

from collections import deque
x = deque(foo)


as far as executable size goes? Do you think that by using from...import... the
module, and all of its dependencies, won't need to be included?

Or are you just trying to save a handful of bytes ("collections.deque" in UTF-8
is 17 bytes, compared to "deque" is just 5 bytes)?


Since the OP is talking about saving space, how much space do you expect to be
able to save using this technique?


--
Steve

[Daniel Bradburn]

A couple of things you can try:

* Generate a directory rather than onefile, on the directory you can apply du
-hs * | sort -h -r (or treesize if you are using windows
https://www.jam-software.com/treesize_free) to see which folders / files
are taking up a lot of space. Then once you see what is taking up a lot of
space you can try and figure out why it is being included, maybe you have a
load of unused imports? With pyinstaller you can explicitly exclude modules
you know you won't need with --exclude-module once you've optimised the
directory build you can of course switch back to onefile

* If all else fails you can use upx to compress your binary files (dlls +
exe) which can help reduce the overall size:
https://pythonhosted.org/PyInstaller/usage.html#using-upx

Hope this helps.


2016-07-27 10:20 GMT+02:00 Steven D'Aprano <
steve+comp....@pearwood.info>:

> --
> https://mail.python.org/mailman/listinfo/python-list
>