Loading modules faster

[Nicolas Pinault]

Hi,

Still want to accelerate the loading of Brython modules ?
Why not use zipapp ?
https://docs.python.org/3.5/library/zipapp.html#module-zipapp

Nicolas

[Olemis Lang]

AFAICS , zipapp is mainly about three aspects

1. Bundling i.e. multiple files in a single archive
2. (De)compression , with little impact upon performance
3. Command-line execution

All these are fine in the desktop environment . I have the following
answers regarding browser-oriented implementations like Brython :

1.1 Do we need anything beyond packaging and loading .vfs.js files ?
2.1 Is it a good idea to deal with data compression explicitly ?
2.2 Isn't it better to rely upon built-in HTTP data compression [1]_ ?
3.1 Use cases for "command-line" execution (or an equivalent
abstraction) when it comes to web browsers rendering web pages ?

.. [1] https://en.wikipedia.org/wiki/HTTP_compression

--
Regards,

Olemis - @olemislc

Apache™ Bloodhound contributor
http://issues.apache.org/bloodhound
http://blood-hound.net

Brython committer
http://brython.info
http://github.com/brython-dev/brython

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:

[Nicolas Pinault]

Le 13/10/2015 20:32, Olemis Lang a écrit :
> On 10/12/15, Nicolas Pinault <d...@famillepinault.fr> wrote:
>> Hi,
>>
>> Still want to accelerate the loading of Brython modules ?
>> Why not use zipapp ?
>> https://docs.python.org/3.5/library/zipapp.html#module-zipapp
>>
> AFAICS , zipapp is mainly about three aspects
>
> 1. Bundling i.e. multiple files in a single archive
> 2. (De)compression , with little impact upon performance
> 3. Command-line execution
>
> All these are fine in the desktop environment . I have the following
> answers regarding browser-oriented implementations like Brython :
>
> 1.1 Do we need anything beyond packaging and loading .vfs.js files ?

zipapp is a Python standard way of packing applications. .vfs.js files
is Brython specific.

> 2.1 Is it a good idea to deal with data compression explicitly ?
> 2.2 Isn't it better to rely upon built-in HTTP data compression [1]_ ?

zipapp compresses more efficiently since all files in package are
compressed all together, not one at a time.

> 3.1 Use cases for "command-line" execution (or an equivalent
> abstraction) when it comes to web browsers rendering web pages ?

No need for a special syntax : <script type="text/python"
src="test.pyz"></script>
The full app is in the package which is loaded using one http request.
When importing a module, no more http request, files are already in the
browser.
No need for a special tool for creating .vfs files.
No need for a special Brython distribution.
This can also serve as some sort of code obfuscation some people request.
>
> .. [1] https://en.wikipedia.org/wiki/HTTP_compression
>
However, this is only a proposition...

Nicolas

[Pierre Quentel]

Slow imports is one of the major problems in Brython. Several proposals have been made to improve speed : Virtual File System (a JS object mapping module names to Python source code), precompile modules to Javascript, now zipapp. I may forgot other suggestions. Of those, only the VFS is implemented and works well.

For precompilation, as suggested by Glyph, Olemis and I started something in files tests/precompile.html and cgi-bin/precompile.py (see the thread in https://github.com/brython-dev/brython/issues/222). The issue is still pending, and we don't know to what extend it would improve the situation : the generated Javascript is much bigger than the original Python source code, so it's not sure that the extra time to load bigger modules (especially on mobile networks) doesn't exceed the time taken to load and convert the Python source.

Distributing a zipapp reduces the number of Ajax requests, like VFS, but it requires an extra operation on the client side : unzipping the file. How long would this take ? with which program ?

The only way to answer these questions would be to implement at least a minimal version of the proposed solutions and run a few test cases. I currently don't have time for that - I dedicate the little time I have for Brython to making the built-in types more compliant, using the test suite in Lib/test. Contributors are welcome.
 

[Nicolas Pinault]

If I had the necessary knowledge, I would certainly work on a test case.

 

--
You received this message because you are subscribed to the Google Groups "brython" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brython+u...@googlegroups.com.
To post to this group, send email to bry...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/brython/73a60e70-f42f-47ef-94ce-e25b8de1dab0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Stu.Axon]

It looks like chrome can throttle bandwidth - which could be good for testing locally

http://stackoverflow.com/questions/10328568/simulate-limited-bandwidth-from-within-chrome

that might make it easier to rule in or out approaches.

another approach could be preloading - there's a POC preloader for images using webworkers here

https://gist.github.com/mseeley/9321422

but benchmarking is really needed otherwise there's no way of knowing how much difference anything makes.

[Olemis Lang]

On 10/13/15, Nicolas Pinault <d...@famillepinault.fr> wrote:
> Le 13/10/2015 20:32, Olemis Lang a écrit :
>> On 10/12/15, Nicolas Pinault <d...@famillepinault.fr> wrote:

[...]

>> AFAICS , zipapp is mainly about three aspects
>>
>> 1. Bundling i.e. multiple files in a single archive
>> 2. (De)compression , with little impact upon performance
>> 3. Command-line execution
>>
>> All these are fine in the desktop environment . I have the following
>> answers

oops ! «questions» :P

>> regarding browser-oriented implementations like Brython :
>>
>> 1.1 Do we need anything beyond packaging and loading .vfs.js files ?
> zipapp is a Python standard way of packing applications. .vfs.js files
> is Brython specific.

... but , unless I get some strong reasons, *right now* *I* do not
consider it suitable for the particular use-case of Brython loading
modules over HTTP . The only benefit I see is related to
interoperability concerns , which IMO should not be a priority atm .

>> 2.1 Is it a good idea to deal with data compression explicitly ?

since there is no answer to this topic , I'll clarify what I mean :
Considering performance , is it a good idea for the browser to run JS
code to explicitly execute data compression in the client side ?

>> 2.2 Isn't it better to rely upon built-in HTTP data compression [1]_ ?
> zipapp compresses more efficiently since all files in package are
> compressed all together, not one at a time.

This is exactly what happens if a .vfs.js file containing multiple
packages is downloaded over compressed HTTP streams . I still do not
think zipapp will be quite better .

>> 3.1 Use cases for "command-line" execution (or an equivalent
>> abstraction) when it comes to web browsers rendering web pages ?
> No need for a special syntax : <script type="text/python"
> src="test.pyz"></script>

I do not understand this aspect . How would you achieve this using
zipapp *only* ? I mean why not just zipimport [2]_ , which a more
"curated" solution available since early 2.x ?

> The full app is in the package which is loaded using one http request.
> When importing a module, no more http request, files are already in the
> browser.

This is exactly what happens in aforementioned scenario .vfs.js + HTTP
compression

> No need for a special tool for creating .vfs files.

Agreed , but IMHO all other performance issues are far more important
. The tooling may be improved with time .

> No need for a special Brython distribution.

There will always be a transformed copy of CPython stdlib , if that's
what you mean , since there are significant differences with respect
to Brython e.g. no file-system, SOP, ...

> This can also serve as some sort of code obfuscation some people request.

Good point

>>
>> .. [1] https://en.wikipedia.org/wiki/HTTP_compression
>>
> However, this is only a proposition...
>

Yes , yes , it's cool . I just do not think it's worth the effort ,
even if it will work in the end . Besides the command-line executable
capabilities of zipapp , which is what makes the difference with
respect to e.g. zipimport, I honestly do not see how they fit for
adoption by Brython .

.. [2] https://docs.python.org/2/library/zipimport.html

[...]

[Olemis Lang]

On 10/13/15, Pierre Quentel <pierre....@gmail.com> wrote:
> Le lundi 12 octobre 2015 21:11:59 UTC+2, nico38 a écrit :

[...]

>
> Distributing a zipapp reduces the number of Ajax requests, like VFS, but it
> requires an extra operation on the client side : unzipping the file.

The key issue IMHO is not about unzipping/decompressing stream
contents in the client.side , since this is also what HTTP compression
working in client's browsers is all about . The point is that it is
necessary to execute JS code for this purpose ...

> How
> long would this take ?

... and I do not think running on top of the JS engine will be faster
than optimized built-in browser compression .

> with which program ?
>

... even if a zip decompression JS lib is involved .

> The only way to answer these questions would be to implement at least a
> minimal version of the proposed solutions and run a few test cases.

[...]

+1

[Nicolas Pinault]

>>> regarding browser-oriented implementations like Brython :
>>>
>>> 1.1 Do we need anything beyond packaging and loading .vfs.js files ?
>> zipapp is a Python standard way of packing applications. .vfs.js files
>> is Brython specific.
> ... but , unless I get some strong reasons, *right now* *I* do not
> consider it suitable for the particular use-case of Brython loading
> modules over HTTP . The only benefit I see is related to
> interoperability concerns , which IMO should not be a priority atm .

Brython claims to be as close as possible from Python3.4.
I know, zipapp is not part of the language. But when you speak about
Python3.4, the library is of concern.

>
>>> 2.1 Is it a good idea to deal with data compression explicitly ?
> since there is no answer to this topic , I'll clarify what I mean :
> Considering performance , is it a good idea for the browser to run JS
> code to explicitly execute data compression in the client side ?

Maybe a zippapp application can be more compressed than what http
compression do ?

>
>>> 2.2 Isn't it better to rely upon built-in HTTP data compression [1]_ ?
>> zipapp compresses more efficiently since all files in package are
>> compressed all together, not one at a time.
> This is exactly what happens if a .vfs.js file containing multiple
> packages is downloaded over compressed HTTP streams . I still do not
> think zipapp will be quite better .

Agreed.

>
>>> 3.1 Use cases for "command-line" execution (or an equivalent
>>> abstraction) when it comes to web browsers rendering web pages ?
>> No need for a special syntax : <script type="text/python"
>> src="test.pyz"></script>
> I do not understand this aspect . How would you achieve this using
> zipapp *only* ? I mean why not just zipimport [2]_ , which a more
> "curated" solution available since early 2.x ?

When you run a zipapp application with Python, you do not use a special
interpreter.
Just run "python myapp.pyz".
It could be the same with brython "<script type="text/python"
src="myapp.pyz"></script>"

>> The full app is in the package which is loaded using one http request.
>> When importing a module, no more http request, files are already in the
>> browser.
> This is exactly what happens in aforementioned scenario .vfs.js + HTTP
> compression

Agreed

>> No need for a special Brython distribution.
> There will always be a transformed copy of CPython stdlib , if that's
> what you mean , since there are significant differences with respect
> to Brython e.g. no file-system, SOP, ...
>

Correct me if I am wrong but with .vfs.js packages, everything is in the
package. The application and brython core and libraries also.
With zipapp, you keep the standard brython distribution outside the
zipapp package.
You can use many zipapp packages in one application.

[Olemis Lang]

On 10/20/15, Nicolas Pinault <d...@famillepinault.fr> wrote:
>
>>>> regarding browser-oriented implementations like Brython :
>>>>
>>>> 1.1 Do we need anything beyond packaging and loading .vfs.js files ?
>>> zipapp is a Python standard way of packing applications. .vfs.js files
>>> is Brython specific.
>> ... but , unless I get some strong reasons, *right now* *I* do not
>> consider it suitable for the particular use-case of Brython loading
>> modules over HTTP . The only benefit I see is related to
>> interoperability concerns , which IMO should not be a priority atm .
>
> Brython claims to be as close as possible from Python3.4.
> I know, zipapp is not part of the language. But when you speak about
> Python3.4, the library is of concern.

That would be the only reason (I see now) for doing something about it
, but , considering there is no evident use case , that still would be
low priority .

>>
>>>> 2.1 Is it a good idea to deal with data compression explicitly ?
>> since there is no answer to this topic , I'll clarify what I mean :
>> Considering performance , is it a good idea for the browser to run JS
>> code to explicitly execute data compression in the client side ?
> Maybe a zippapp application can be more compressed than what http
> compression do ?

If this made a substantial difference I'm quite sure browser and
Internet players would have pushed towards implementing it .
Nonetheless , I might change my mind if enough data is gathered and
presented as evidence .

[...]

>>>> 3.1 Use cases for "command-line" execution (or an equivalent
>>>> abstraction) when it comes to web browsers rendering web pages ?
>>> No need for a special syntax : <script type="text/python"
>>> src="test.pyz"></script>
>> I do not understand this aspect . How would you achieve this using
>> zipapp *only* ? I mean why not just zipimport [2]_ , which a more
>> "curated" solution available since early 2.x ?
> When you run a zipapp application with Python, you do not use a special
> interpreter.
> Just run "python myapp.pyz".
> It could be the same with brython "<script type="text/python"
> src="myapp.pyz"></script>"

I do not get it , the initial python in "python myapp.pyz" is your
interpreter , even if it is not "interactive" .

[...]

>>> No need for a special Brython distribution.
>> There will always be a transformed copy of CPython stdlib , if that's
>> what you mean , since there are significant differences with respect
>> to Brython e.g. no file-system, SOP, ...
>>
> Correct me if I am wrong but with .vfs.js packages, everything is in the
> package. The application and brython core and libraries also.

No , multiple vfs may coexist in sys.path .

> With zipapp, you keep the standard brython distribution outside the
> zipapp package.
> You can use many zipapp packages in one application.
>

So it is possible to do all this with .vfs.js as well .

[Nicolas Pinault]

>>>>> 2.1 Is it a good idea to deal with data compression explicitly ?
>>> since there is no answer to this topic , I'll clarify what I mean :
>>> Considering performance , is it a good idea for the browser to run JS
>>> code to explicitly execute data compression in the client side ?
>> Maybe a zippapp application can be more compressed than what http
>> compression do ?
> If this made a substantial difference I'm quite sure browser and
> Internet players would have pushed towards implementing it .
> Nonetheless , I might change my mind if enough data is gathered and
> presented as evidence .

As I already said, this was just a proposition.