Smaller WinPython

[Gökhan Sever]

Hello,

WinPython is super easy to install and run on Windows. I was wondering if there is a way to have a similar distribution only including core packages (Python, numpy, scipy, and matplotlib). Currently my WinPython-64bit-3.3.5.7 folder occupies 1.3 GB and has about 40k files. It would be practical to have a distribution around ~100 MB and with a couple thousand files most. Thanks.

[Iván Pulido]

Hi

A way of doing this (though probably not the best one) is using the
winpython package manager to uninstall all the components you don't need
and then use some packager (like innoscript for example) to create your
own installer of the stripped down version of Winpython.

The other way (probably the better way) is creating your own
distribution of Winpython, I think this has been already covered in the
mailing list and on the repository there is information on how to
accomplish this.

hope this helps.

[stonebig]

HeavyWeights are, on 64bit:

93Mo =  the compiler toolchain (clear the "tools" directory to get rid of it),
75Mo (guess) =  the "MKL" optimization effect on numpy+scipy (look at recent "tiny" numpy package from Christoph Gohlke)
==> You zip the resulting Winpython directory and voilà.

[stonebig]

the today problem is :
- to get a "tiny" scipy,
- you're curious request of a Python3.3 and a 64 bit (a PAST and a FUTURE characteristics)

[Gökhan Sever]

I am wondering would that be simpler to install binaries from Christoph Gohlke's page rather than removing packages from the WinPython folder. Also, what do you mean by tiny-scipy? Could you release a WinPython-tiny release (with the core libraries) in addition to the current big releases?

[stonebig]

tiny-scipy, or "not-MKL but compiled" scipy, doesn't exist yet.

Your dream  is not possible today.

That's nevertheless an interesting dream, I keep it in mind.

[stonebig]

as small steps in that direction:
- mingw compiler is now available as a wheel,
   (pip install -i https://pypi.binstar.org/carlkl/simple mingwpy)
- the Winpython build process is being somewhat revisited to allow installations from requirements.txt files and whatever download site you want.
- there is also this project that could act as a complement to a minimal winpython (or anaconda): https://pypi.python.org/pypi/pymyinstall/

[carlkl]

A preview of the upcoming OpenBLAS based numpy and scipy is available from anaconda.org:

pip install -i https://pypi.anaconda.org/carlkl/simple numpy
pip install -i https://pypi.anaconda.org/carlkl/simple scipy

It is however not useful for daily work yet, as a lot of further 3rd party libraries depends on the presence of numpy-MKL. Take a look at the list of libraries at http://www.lfd.uci.edu/~gohlke/pythonlibs which requires numpy+MKL. I guess some of winpython's packages needs numpy+MKL for that reason.

This may change in near future; since scipy-0.16.0 python packages may use scipy's BLAS/Lapack as underlying engine instead of explicit linking to a specific BLAS/Lapack implemenation.

[stonebig]

Hi Göhkan,

I'm now more at ease to experiment a 'slim' Winpython.

I'm down to 103 Mo in 20K files with Python 3.4 64 bit, with these conditions:
- using numpy 'vanilla' of Christoph Gohlke,
- not including 'scipy', which is itself '99' Mo, (I don't have non-mkl version in Christoph Gohlke's shop
- droping cython, numba, .. maybe a little too much things already


==> is '64' bit a true requirement ?
==> any suggestions ?

current package list (I notice pandoc, 8go zipped, is present but not on the list):
## WinPython 3.4.3.7Slim

The following packages are included in WinPython v3.4.3.7Slim build1.

### Tools

Name | Version | Description
-----|---------|------------
[SciTE](http://www.scintilla.org/SciTE.html) | 3.3.7 | SCIntilla based Text Editor - Multilanguage, powerful and light-weight text editor

### Python packages

Name | Version | Description
-----|---------|------------
[Python](http://www.python.org/) | 3.4.3 | Python programming language with standard library
[astroid](http://pypi.python.org/pypi/astroid) | 1.3.6 | Rebuild a new abstract syntax tree from Python's ast (required for pylint)
[backports-abc](http://pypi.python.org/pypi/backports-abc) | 0.4 | A backport of recent additions to the 'collections.abc' module.
[colorama](http://pypi.python.org/pypi/colorama) | 0.3.3 | Cross-platform colored terminal text
[cx-freeze](http://cx-freeze.sourceforge.net) | 4.3.4 | Deployment tool which converts Python scripts into stand-alone Windows executables (i.e. target machine does not require Python or any other library to be installed)
[cycler](http://pypi.python.org/pypi/cycler) | 0.9.0 | Composable style cycles
[decorator](http://pypi.python.org/pypi/decorator) | 4.0.4 | Better living through Python with decorators
[ipykernel](http://pypi.python.org/pypi/ipykernel) | 4.1.1 | IPython Kernel for Jupyter
[ipyparallel](http://pypi.python.org/pypi/ipyparallel) | 4.1.0 | Interactive Parallel Computing with IPython
[ipython](http://pypi.python.org/pypi/ipython) | 4.0.0 | Enhanced Python shell
[ipython-genutils](http://pypi.python.org/pypi/ipython-genutils) | 0.1.0 | Vestigial utilities from IPython
[ipywidgets](http://pypi.python.org/pypi/ipywidgets) | 4.1.1 | IPython HTML widgets for Jupyter
[jedi](http://pypi.python.org/pypi/jedi) | 0.8.1 | An autocompletion tool for Python that can be used for text editors
[jinja2](http://pypi.python.org/pypi/jinja2) | 2.8 | Sandboxed template engine (provides a Django-like non-XML syntax and compiles templates into executable python code)
[jsonschema](http://pypi.python.org/pypi/jsonschema) | 2.5.1 | An implementation of JSON Schema validation for Python
[jupyter](http://pypi.python.org/pypi/jupyter) | 1.0.0 | Jupyter metapackage. Install all the Jupyter components in one go.
[jupyter-client](http://pypi.python.org/pypi/jupyter-client) | 4.1.1 | Jupyter metapackage. Install all the Jupyter components in one go.
[jupyter-console](http://pypi.python.org/pypi/jupyter-console) | 4.0.3 | Jupyter metapackage. Install all the Jupyter components in one go.
[jupyter-core](http://pypi.python.org/pypi/jupyter-core) | 4.0.6 | Jupyter metapackage. Install all the Jupyter components in one go.
[logilab-common](http://pypi.python.org/pypi/logilab-common) | 1.1.0 | Collection of low-level Python packages and modules used by Logilab projects (required for pylint)
[markupsafe](http://pypi.python.org/pypi/markupsafe) | 0.23 | Implements a XML/HTML/XHTML Markup safe string for Python
[matplotlib](http://pypi.python.org/pypi/matplotlib) | 1.5.0 | 2D plotting library (embeddable in GUIs created with PyQt)
[mistune](http://pypi.python.org/pypi/mistune) | 0.7.1 | The fastest markdown parser in pure Python, inspired by marked.
[nbconvert](http://pypi.python.org/pypi/nbconvert) | 4.1.0 | Converting Jupyter Notebooks
[nbformat](http://pypi.python.org/pypi/nbformat) | 4.0.1 | The Jupyter Notebook format
[networkx](http://pypi.python.org/pypi/networkx) | 1.10 | Python package for creating and manipulating graphs and networks
[nose](http://somethingaboutorange.com/mrl/projects/nose) | 1.3.7 | nose is a discovery-based unittest extension (e.g. NumPy test module is using nose)
[notebook](http://pypi.python.org/pypi/notebook) | 4.0.6 | # Jupyter Notebook
[numpy](http://numpy.scipy.org/) | 1.9.3 | NumPy: multidimensional array processing for numbers, strings, records and objects (SciPy''s core module)
[pandas](http://pypi.python.org/pypi/pandas) | 0.17.0 | Powerful data structures for data analysis, time series and statistics
[pandas-datareader](http://pypi.python.org/pypi/pandas-datareader) | 0.2.0 | Data readers extracted from the pandas codebase,should be compatible with recent pandas versions
[path.py](http://pypi.python.org/pypi/path.py) | 8.1.2 | A module wrapper for os.path
[pep8](http://pypi.python.org/pypi/pep8) | 1.6.2 | Python style guide checker
[pickleshare](http://pypi.python.org/pypi/pickleshare) | 0.5 | Tiny 'shelve'-like database with concurrency support
[pillow](http://pypi.python.org/pypi/pillow) | 3.0.0 | Python Imaging Library (fork)
[pip](http://pypi.python.org/pypi/pip) | 7.1.2 | A tool for installing and managing Python packages
[psutil](http://code.google.com/p/psutil) | 3.2.2 | Provides an interface for retrieving information on all running processes and system utilization (CPU, disk, memory, network) in a portable way
[py](http://pypi.python.org/pypi/py) | 1.4.30 | library with cross-python path, ini-parsing, io, code, log facilities
[pyaudio](http://pypi.python.org/pypi/pyaudio) | 0.2.9 | Bindings for PortAudio v19, the cross-platform audio input/output stream library.
[pyflakes](http://pypi.python.org/pypi/pyflakes) | 0.9.2 | passive checker of Python programs
[pygments](http://pygments.org) | 2.0.2 | Generic syntax highlighter for general use in all kinds of software
[pylint](http://www.logilab.org/project/pylint) | 1.4.3 | Logilab code analysis module: analyzes Python source code looking for bugs and signs of poor quality
[pyparsing](http://pyparsing.wikispaces.com/) | 2.0.5 | A Python Parsing Module
[PyQt5](http://www.riverbankcomputing.co.uk/software/pyqt/intro) | 5.5.1 | Python bindings for the Qt cross platform GUI toolkit
[pyserial](http://pypi.python.org/pypi/pyserial) | 2.7 | Library encapsulating the access for the serial port
[pytest](http://pypi.python.org/pypi/pytest) | 2.8.2 | pytest: simple powerful testing with Python
[python-dateutil](http://labix.org/python-dateutil) | 2.4.2 | Powerful extensions to the standard datetime module
[pythonqwt](http://pypi.python.org/pypi/pythonqwt) | 0.5.1 | Qt plotting widgets for Python
[pytz](http://pypi.python.org/pypi/pytz) | 2015.7 | World Timezone Definitions for Python
[pywin32](http://pypi.python.org/pypi/pywin32) | 219 | Python library for Windows
[pyzmq](http://pypi.python.org/pypi/pyzmq) | 15.0.0 | Lightweight and super-fast messaging based on ZeroMQ library (required for IPython Qt console)
[qtconsole](http://pypi.python.org/pypi/qtconsole) | 4.1.0 | Jupyter Qt console
[rope-py3k](http://pypi.python.org/pypi/rope-py3k) | 0.9.4.post1 | a python refactoring library...
[seaborn](http://pypi.python.org/pypi/seaborn) | 0.7.0.dev0 | statistical data visualization
[setuptools](http://pypi.python.org/pypi/setuptools) | 18.5 | Download, build, install, upgrade, and uninstall Python packages - easily
[simplegeneric](http://pypi.python.org/pypi/simplegeneric) | 0.8.1 | Simple generic functions (similar to Python's own len(), pickle.dump(), etc.)
[six](http://pypi.python.org/pypi/six) | 1.10.0 | Python 2 and 3 compatibility utilities
[spyder](http://pypi.python.org/pypi/spyder) | 3.0.0.dev0 | Scientific PYthon Development EnviRonment: designed for interactive computing and data visualisation with a simple and intuitive user interface
[tornado](http://pypi.python.org/pypi/tornado) | 4.3 | Scalable, non-blocking web server and tools (required for IPython notebook)
[traitlets](http://pypi.python.org/pypi/traitlets) | 4.0.0 | Traitlets Python config system
[wheel](http://pypi.python.org/pypi/wheel) | 0.26.0 | A built-package format for Python.
[winpython](http://winpython.github.io/) | 1.2 | WinPython distribution tools, including WPPM (package manager)

[stonebig]

An experiment of a Smaller WinPython is out

Supposed market/need:
- education where multiple installations have to be done quickly, and disk space is sparse,
- nota: to install via command line you cand do : 'WinPython-32bit-3.4.3.7Slimbuild2.exe /S /DIR=you_target_directory

Main features:
- 50% smaller (650Mo, 23k files)

Know issue:
- shipped with Winpython_checker.ipynb instead of WinpythonSlim_checker.ipynb (so shipped in this message)

MD5                              | SHA-1                                    | Binary                                | Size
---------------------------------|------------------------------------------|---------------------------------------|------------------
166ddb1152e31eb18d62e94de0f35bb6 | dc8c146b0ad4ba528944815840bb196501dcc85e | winpython-32bit-3.4.3.7slimbuild2.exe | 131 632 995 Bytes


details:

https://github.com/winpython/winpython/issues/190#issuecomment-155512897

[Pär Ola Lenhoff]

Hi, I usually run Winpython from a USB and would also like to see a slimmer version, when you started talking about a slim version I initially thought you would do something like miniconda. (E.g. python and installer, but nothing else., i.e. let people then install what they like).  For my usecase I think this would be the best (if it is still possible to have it portable) but I am not sure how other people feel...

Anyway, I will be greatful for whatever slimmer version you end up with (and hopefully have them updated as part of the releases), just wondering if a bare bones version wouldn't be easier to maintain. 

[stonebig]

Hi Selepo,

your use-case is interesting:
- what package have you got on this theorical "WinPythonBarebone" after 2 weeks ?
- what will you use it for ?

[stonebig]

or more simply:
- take WinPythonSlim,
- remove what you don't want,
- tell me what remains after in ('pip list')

[selepo]

I think what I am proposing is bascially no packages besides python itself, your packaging e.g. control panel, etc., pip (I think it's already included in python3) and pipwin (for installing pre- compiled packages) nothing more. 

I do besides python often use jupyter / ipykernel /ipython console ; pandas/numpy; requests / beautifulsoup4; flask / verkzeug / jinja2; ply / python-prompt-toolkit / xonsh 

My point is that if it was just portability / winpython-packaging and python itself along with pip/pipwin then I can just run a pip install -r requirements.txt (and have the above packages in my requirements.txt) after each update of Winpython....

[stonebig]

So you don't use the "Qt" part of WinPython ? no Spyder ? no qtconsole ? no graphical WinPython Control Panel ?

[selepo]

Correct, I use it as a portable python distro, although I do use qt console...

With that said, my usage is possibly an edge case ?

[stonebig]

you may not be an edge case, but until know these demands were managed by miniconda.

Why miniconda doesn't fit you ?

[stonebig]

python3.4.3.msi 32bit is 24 Mo
miniconda3 32bit  is 29 Mo

a barebone winpython 32bit:
- with PyQt is 56 Mo,
- with Ipython and qtconsole is 63 Mo.

==> WinPython is not currently designed not-to-have PyQt.

[selepo]

miniconda isn't portable, or I would use it.
it isn't really important if it's <50 mb or <200mb, but it's currently ~ 1,2 GB decompressed on my usb drive. I guess the slim build would be about half, which is a huge improvement in itself.

[stonebig]

ok,

if you think in uncompressed:
- 'Slim' is indeed minded to be half-the-size: about 23k files, 650 Mo in 32bit
- 'barebone' with PyQt without qt console, is about 5k files / 200 Mo
- 'error' without PyQt is about 4k file , 80 Mo

[stonebig]

Hi selepo,

WinPython can now be build with nearly 'nothing'.
To my surprise, it seems smaller than the Official Python Installer.

Playing with https://github.com/winpython/winpython/issues/201 , you may be able to construct the exact stack you want.
(with a certain amount of effort)

MD5                              | SHA-1                                    | Binary                         | Size
---------------------------------|------------------------------------------|--------------------------------|------------------

cbd3e9b69dad7f60b38a2be350dd26ce | c2918bf7be98cb8d2d38625137d61875c76840dd | winpython-32bit-3.4.3.7zerobuild5.exe | 22 498 788 Bytes

see https://github.com/winpython/winpython/issues/190#issuecomment-158535811

[Gökhan Sever]

Hello stonebig,

Sorry for my late reply. Thanks for providing these small installations. They indeed come handy in some instances where one doesn't need a big setup file. 

While I am here, I have another question about migrating libraries to newer WinPython versions. There are few libraries I installed externally (via either pip or the control panel) in an older WinPython version (for example, OpenCV and Mayavi). I would like use a newer version of your bundles and be able to work with these libraries. Is there a pip trick to make this transition or do I need to install these libraries manually? Conda (which I use on Linux) has one main advantage that I can update packages in-place without moving to newer Conda versions. Is this by design in WinPython which you release a new bundle everytime the packages get enough updates as well as Python. Thanks.

[stonebig]

Getting wheels from Cgohlke not manually:
*************************************************
- I didn't try,  but you can experiment with "pipwin" (problem is that I'm not sure cgohlke is favorable to that, as maybe he only has 1 server, he is not pypi.org)

- otherwise, let suppose you have your "new" binary wheels on "D:\my_wheels", would this do what you want (from WinPython Command Line:
  pip uninstall OpenCV
  pip install OpenCV --upgrade --no-index --trusted-host=None --find-links=D:\my_wheels
  pip uninstall Mayavi
  pip install Mayavi--upgrade --no-index --trusted-host=None --find-links=D:\my_wheels

- the shortcut version may work also, but the "upgrade" keyword propagates on dependant packages:
  pip install OpenCV --upgrade --no-index --trusted-host=None --find-links=D:\my_wheels
  pip install Mayavi--upgrade --no-index --trusted-host=None --find-links=D:\my_wheels

- if a "D:\my_wheels\my_re.txt" file exist with in it
       Mayavi
       OpenCV
  then you can try:
  pip install -r D:\my_wheels\my_re.txt --upgrade --no-index --trusted-host=None --find-links=D:\my_wheels

- I hope that the "Warehouse" successor of "pypi" will allow "cgohlke" and "scipy team" to upload his binaries on a sub-directories of pypi (like binstar), and the complexity will go away interely
  (and maybe WinPython need also)

==> Is it what you wanted to know ?

[stonebig]

oups, I meant in first solution:
  pip uninstall OpenCV
  pip install OpenCV   --no-index --trusted-host=None --find-links=D:\my_wheels
  pip uninstall Mayavi
  pip install Mayavi  --no-index --trusted-host=None --find-links=D:\my_wheels

(no--upgrade option)

[stonebig]

also the big trick to upgrade pip itself and keeping WinPython "movability"

click on "WinPython Command Prompt.exe", then type

%WINPYDIR%\python.exe -m pip install --upgrade --force-reinstall  pip
%WINPYDIR%\python.exe -c "from winpython import wppm;dist=wppm.Distribution(r'%WINPYDIR%');dist.patch_standard_packages('pip')

[Gökhan Sever]

Hello,

I was thinking more on the line of installing packages to a separate directory other than Python's site-packages, and have each WinPython version to use this location for additionally installed packages. Eventually once a new WinPython version comes out and after a few tests, I can simply delete the older WinPython version, so simplify my workflow. Do you think this would be feasible? Thanks.

[stonebig]

If the shared location is a location of .whl packages, yes. if you mean a location of installed package, I guess no.

[v+py...@g.nevcal.com]

Thank you for making WinPython Zero.