Max size of Python source code and compiled equivalent

Malcolm Greene

unread,

Jul 21, 2016, 12:51:22 PM7/21/16

to

We're writing a DSL parser that generates Python code. While the size of
our generated code will be small (< 32K), I wanted to re-assure the rest
of our team that there are no reasonable code size boundaries that we
need to be concerned about. I've searched for Python documentation that
covers max Python source (*.py) and compiled file (*.pyc) sizes without
success. Any tips on where to look for this information?

Background: Python 3.5.1 on Linux.

Thank you,
Malcolm

Chris Angelico

unread,

Jul 21, 2016, 1:08:39 PM7/21/16

to

On Fri, Jul 22, 2016 at 2:51 AM, Malcolm Greene <pyt...@bdurham.com> wrote:
> We're writing a DSL parser that generates Python code. While the size of
> our generated code will be small (< 32K), I wanted to re-assure the rest
> of our team that there are no reasonable code size boundaries that we
> need to be concerned about. I've searched for Python documentation that
> covers max Python source (*.py) and compiled file (*.pyc) sizes without
> success. Any tips on where to look for this information?

Heh, great question, and I'm curious too! But one place to get a bit
more info is the standard library.

rosuav@sikorsky:~/cpython/Lib$ find -name \*.py|xargs ls -lS|head
-rw-r--r-- 1 rosuav rosuav 624122 Jul 17 17:38 ./pydoc_data/topics.py
-rw-r--r-- 1 rosuav rosuav 229348 Jun 15 23:20 ./_pydecimal.py
-rw-r--r-- 1 rosuav rosuav 210900 Jul 21 22:41 ./test/test_decimal.py
-rw-r--r-- 1 rosuav rosuav 205399 Jun 15 23:20 ./test/test_email/test_email.py
-rw-r--r-- 1 rosuav rosuav 203297 Jun 23 13:22 ./test/test_socket.py
-rw-r--r-- 1 rosuav rosuav 183195 Jul 17 17:38 ./test/test_descr.py
-rw-r--r-- 1 rosuav rosuav 166684 Jun 29 16:26 ./tkinter/__init__.py
-rw-r--r-- 1 rosuav rosuav 164583 Jun 15 23:20 ./test/test_argparse.py
-rw-r--r-- 1 rosuav rosuav 162603 Jun 15 23:20 ./test/test_buffer.py
-rw-r--r-- 1 rosuav rosuav 159528 Jun 15 23:20 ./test/datetimetester.py

rosuav@sikorsky:~/cpython/Lib$ find -name \*.pyc|xargs ls -lS|head
-rw-r--r-- 1 rosuav rosuav 387893 Jul 22 00:03
./pydoc_data/__pycache__/topics.cpython-36.pyc
-rw-r--r-- 1 rosuav rosuav 231742 Jul 18 12:45
./test/__pycache__/test_descr.cpython-36.pyc
-rw-r--r-- 1 rosuav rosuav 202168 Jul 11 12:40
./test/test_email/__pycache__/test_email.cpython-36.pyc
-rw-r--r-- 1 rosuav rosuav 179521 Jul 11 12:40
./tkinter/__pycache__/__init__.cpython-36.pyc
-rw-r--r-- 1 rosuav rosuav 178331 Jul 11 12:40
./test/__pycache__/test_socket.cpython-36.pyc
-rw-r--r-- 1 rosuav rosuav 166424 Jul 11 12:40
./test/__pycache__/test_argparse.cpython-36.pyc
-rw-r--r-- 1 rosuav rosuav 162697 Jul 11 12:40
./__pycache__/_pydecimal.cpython-36.pyc
-rw-r--r-- 1 rosuav rosuav 160564 Jul 22 00:04
./test/__pycache__/test_decimal.cpython-36.pyc
-rw-r--r-- 1 rosuav rosuav 157038 Jul 11 12:40
./test/__pycache__/test_inspect.cpython-36.pyc
-rw-r--r-- 1 rosuav rosuav 149682 Jul 11 12:41
./lib2to3/tests/__pycache__/test_fixers.cpython-36.pyc

So you can have 600KB of source code or 350KB of pyc without any
trouble whatsoever. (You won't actually be using .pyc files if you use
exec as recommended in the other thread, but it's still a reasonable
way of estimating compiled size.) If there are limits, they're
certainly no lower than this.

ChrisA

Malcolm Greene

unread,

Jul 21, 2016, 1:28:17 PM7/21/16

to

> Heh, great question, and I'm curious too! But one place to get a bit more info is the standard library.
>
> rosuav@sikorsky:~/cpython/Lib$ find -name \*.py|xargs ls -lS|head
> -rw-r--r-- 1 rosuav rosuav 624122 Jul 17 17:38 ./pydoc_data/topics.py

Brilliant! :)

Thanks Chris!
Malcolm

Peter Otten

unread,

Jul 21, 2016, 2:20:18 PM7/21/16

to

Malcolm Greene wrote:

> We're writing a DSL parser that generates Python code. While the size of
> our generated code will be small (< 32K), I wanted to re-assure the rest
> of our team that there are no reasonable code size boundaries that we
> need to be concerned about. I've searched for Python documentation that
> covers max Python source (*.py) and compiled file (*.pyc) sizes without
> success. Any tips on where to look for this information?
>

> Background: Python 3.5.1 on Linux.

I don't know if/where this is documented, but there are structural limits:

>>> def nested(N):
... return "".join(" " * i + "if 1:\n" for i in range(N)) + " " * N +
"print('hi')"
...
>>> print(nested(3))
if 1:
if 1:
if 1:
print('hi')
>>> exec(nested(99))
hi
>>> exec(nested(100))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 101
print('hi')
^
IndentationError: too many levels of indentation