Speeding up initial load of pages....
140 views
Skip to first unread message
Jérémie Lefrancois
Mar 19, 2021, 3:16:04 AM3/19/21
to bry...@googlegroups.com
Hello.
I made this wonderful site in brython (I quoted brython on the technical corder page... :-) )
My "customers" - they are not customers in fact - complain of course they say it is too slow.
I have already done following optimizations :
- removed import configparser (using json instead) for huge gain
- removed import typing (goodbye full type checking) for 40% approx gain
My python imports are now :
json, datetime, time, math, enum, abc
Every time I add/remove an import I do :
brython-cli --modules
and the following files are deployed :
brython.js brython_stdlib.js brython_modules.js
index.html has :
<body onload="brython(1)">
<!-- just calling python --><script type="text/python" src="index.py"></script>
<script type="text/python" src="home.py"></script>
etc...
Any ideas of how to optimize ?
regards
Kiko
Mar 19, 2021, 3:55:13 AM3/19/21
to bry...@googlegroups.com
2021-03-19 8:15 GMT+01:00, Jérémie Lefrancois <jeremie.l...@gmail.com>:
> Hello.
>
> I made this wonderful site in brython (I quoted brython on the technical
> corder page... :-) )
> https://diplomania-gen.fr/
> My "customers" - they are not customers in fact - complain of course they
> say it is too slow.
<WARNING rant>
> Hello.
>
> I made this wonderful site in brython (I quoted brython on the technical
> corder page... :-) )
> https://diplomania-gen.fr/
> My "customers" - they are not customers in fact - complain of course they
> say it is too slow.
It is not so slow (when compared to web pages in 2021). Yes, it is sad
the state of the (surveillance) internet :-(
<END WARNING>
>
> I have already done following optimizations :
> - removed import configparser (using json instead) for huge gain
> - removed import typing (goodbye full type checking) for 40% approx gain
>
> My python imports are now :
> json, datetime, time, math, enum, abc
>
> Every time I add/remove an import I do :
> and the following files are deployed :
> brython.js brython_stdlib.js brython_modules.js
>
> index.html has :
>
> *<body onload="brython(1)">*
> brython.js brython_stdlib.js brython_modules.js
>
> index.html has :
>
>
>
> *<!-- just calling python --><script type="text/python"
> src="index.py"></script>*
> *<script type="text/python" src="home.py"></script>*
>
> *etc...*
>
> Any ideas of how to optimize ?
I see you are using the optimisation section here:
> Any ideas of how to optimize ?
https://brython.info/static_doc/en/import.html
But it seems you are not using it as expected. When you create your
brython_modules.js you shouldn't use brython_stdlib.js and you
shouldn't add
<script type="text/python" src="index.py"></script>
<script type="text/python" src="home.py"></script>
...
<script type="text/python" src="home.py"></script>
As index.py, home.py, etc, should be already in the brython_modules.js file.
So, when you use 'brython-cli --modules' you should just use
brython.js and brython_modules.js (with brython_modules.js including
the needed packages from brython_stdlib.js as well as the rest of the
py files you are using).
This have two effects:
- Potentially reduce the size of the downloaded files.
- Avoid the use of ajax calls.
Kind regards.
>
> regards
>
> --
> You received this message because you are subscribed to the Google Groups
> "brython" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to brython+u...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/brython/CAAuNSdEsQcdrvUaGZLUGLUkMu1mZccWTD%3Dv3GaNtsnjYDNOYKQ%40mail.gmail.com.
>
Jérémie Lefrancois
Mar 19, 2021, 11:57:11 AM3/19/21
to brython
Thanks a lot, this gains another 50 % (very approx).
Changing body onload="brython(1)" to brython(0)" does not seem to make much difference.
Let's see what the customers say...
Ian Sparks
Mar 19, 2021, 4:08:22 PM3/19/21
to brython
An idea.. What if the initial page showed some kind of animated loading icon against a background color. Build the interface inside a container with display:none and then once loading is done clear the background and loading icon and make the UI visible.
It won't actually load any faster but instead of staring at a blank screen for 1-2 seconds your user will feel like something is happening.
Since diplomacy is kind of a game (I recall it being a good way to start a fistfight among my teenage friends) you could also consider fading in some appropriate quotes (Machiavelli perhaps?) with a random quote chosen every time you load the page - might make the "wait" something that the user actually looks forward to?
You'd need to have this boostrapping done by a small amount of JS and/or CSS for speed but it would help?
Ray Luo
Apr 3, 2021, 3:28:03 AM4/3/21
to brython
Thanks, Ian, for sharing that interesting article. I've already used a loading animation in my projects, but that article got me adding some loading messages.
To give back to the community, I wrap my work into a generic template. See it in action here: https://rayluo.github.io/brython-project-template/
Feel free to fork or clone or download it from github https://github.com/rayluo/brython-project-template
Regards,
Ray Luo
Ian Sparks
Apr 3, 2021, 8:45:19 AM4/3/21
to brython
Thanks for trying it the idea Ray. Something is a bit odd about the way it works though. The first time I visit it, it stays on "Loading main program" for a really long time which is odd because your main program is very short.
I made a video of this.
I tried on Chrome on Mac and Firefox on Ubuntu with the same effect.
On subsequent loads it is almost instant but still stays longer than you would expect on "Loading main program".
This happens for me each time I clear the indexdb cache for the page.
When I run locally with <body id="body" onload="brython({indexedDB:false})" (i.e. no indexdb) the loading icon hardly moves (loading standard libraries message shows) and its all loaded in about 2 seconds or less.
And If I change the end of your page to:
<script type="text/javascript">
console.log("Loading main python program...")
document.getElementById("psychology").text = "Loading Main Program..."
</script>
<!-- Your main logic should go into here -->
<script type="text/python" src="main.py"></script>
<script type="text/javascript">
console.log("Ready to use");
document.getElementById("loading").style.display = "none";
document.getElementById("main").style.display = "";
document.getElementById("body").style.cursor = "default";
</script>
console.log("Loading main python program...")
document.getElementById("psychology").text = "Loading Main Program..."
</script>
<!-- Your main logic should go into here -->
<script type="text/python" src="main.py"></script>
<script type="text/javascript">
console.log("Ready to use");
document.getElementById("loading").style.display = "none";
document.getElementById("main").style.display = "";
document.getElementById("body").style.cursor = "default";
</script>
i.e. I use javascript to set the "Loading main.." and then "Ready to use" after main.py has loaded it is perceptibly faster still.
Not sure what is going on.
Andy Lewis
Apr 3, 2021, 11:35:31 AM4/3/21
to brython
Hi Ian
If you change `<body onload="brython()">` to `<body onload="brython(2)">`to switch on detailed messages, you can see in the console what Brython is doing if the indexedDB is empty. Basically it is compiling all the modules in the standard library and storing them in indexedDB. Once it has finished, the animation is replaced by the "Echo" page. Of course, this is a one-off task (unless the indexedDB is subsequently cleared) - on subsequent visits to the page the modules are just loaded from indexedDB, which is much faster.
Hope that helps
Andy
Ray Luo
Apr 3, 2021, 3:16:00 PM4/3/21
to brython
Thanks for testing it, Ian! Also thank you, Andy, for providing more info on this.
I can not fully answer Ian's question, but this is my different perspective.
First of all, you can also enable the console of your browser, to see some more developer-oriented messages which would also mention the "Brython" or "Python" keywords accordingly. You can even enable your browser to display the timestamp of each console message (at least Firefox allows you to do that).
I do not know enough internals about whether Brython runs Javascript and Python script concurrently, but that is exactly why I choose to use Javascript in the first half of the template to show "Loading brython..." and then use Python at the second half of the template to show those "Loading main python program..." and "Ready to use". Those 2 messages being printed by Python, are meant to be a signal of "the PYTHON part of the this webpage is loading ... and now ready". I do have same observation as Ian, that the "Loading main program" stays on screen for a little longer than what Ian would expect, even on the subsequent page reload. I do not know what Brython does behind the scene during the printing of "Loading main program" and the final "Ready to use" (only Pierre would know?), but it is what it is, from the perspective of this template.
Ian's experiment of using Javascript to show the some more messages in between, might be a way to generate more perceivable progress for the end user, although at that point I don't know how those messages would correspond to the actual progress, so we might as well use some irrelevant humours in those messages. :-)
Lastly, I also observe that, especially during the initial loading of the page (when the indexedDB is unavailable), the loading animation hardly moves. I guess that is a single-threaded limitation. Unless Brython can do the initialization in a background thread, there is nothing this template can do. For what it's worth, I did add a "waiting" mouse cursor effect to the page body tag to mitigate that. The mouse animation seems functional even when the page is unresponsive. :-) But such mouse animation is only noticeable if the end user hovers the mouse on the body area, which unfortunately does not equal to the entire browser visible page area. :-(
Regards,
Ray Luo
Ian Sparks
Apr 3, 2021, 3:44:25 PM4/3/21
to brython
Hi Andy, that was helpful to track the order of things. For example the output of this surprised me:
</script>
<!-- Your main logic should go into here -->
<script type="text/python" src="main.py"></script>
<!DOCTYPE html>
<html lang="en">
<head>
</head>
<body onload="brython()">
<script type="text/javascript">
console.log("1 - Javascript : Before Loading Brython")
</script>
<script src="https://raw.githack.com/brython-dev/brython/master/www/src/brython.js"></script>
<script type="text/javascript">
console.log("2 - Javascript : After Loading Brython")
</script>
<script type="text/python">
print("3 - Python : After Loading Brython")
</script>
<script type="text/javascript">
console.log("4 - Javascript : Before loading brython stdlib")
</script>
<script src="https://raw.githack.com/brython-dev/brython/master/www/src/brython_stdlib.js"></script>
<script type="text/python">
print("5 - Python : After Loading std lib")
</script>
<script type="text/javascript">
console.log("6 - Javascript : After loading brython stdlib")
</script>
</body>
</html>
<html lang="en">
<head>
</head>
<body onload="brython()">
<script type="text/javascript">
console.log("1 - Javascript : Before Loading Brython")
</script>
<script src="https://raw.githack.com/brython-dev/brython/master/www/src/brython.js"></script>
<script type="text/javascript">
console.log("2 - Javascript : After Loading Brython")
</script>
<script type="text/python">
print("3 - Python : After Loading Brython")
</script>
<script type="text/javascript">
console.log("4 - Javascript : Before loading brython stdlib")
</script>
<script src="https://raw.githack.com/brython-dev/brython/master/www/src/brython_stdlib.js"></script>
<script type="text/python">
print("5 - Python : After Loading std lib")
</script>
<script type="text/javascript">
console.log("6 - Javascript : After loading brython stdlib")
</script>
</body>
</html>
I'd forgotten that what the brython() on load task does is go through and process all the text/python scripts so the output might surprise people who expect each script in turn to process.
The output is:
1 - Javascript : Before Loading Brython
2 - Javascript : After Loading Brython
4 - Javascript : Before loading brython stdlib
6 - Javascript : After loading brython stdlib
3 - Python : After Loading Brython
5 - Python : After Loading std lib
2 - Javascript : After Loading Brython
4 - Javascript : Before loading brython stdlib
6 - Javascript : After loading brython stdlib
3 - Python : After Loading Brython
5 - Python : After Loading std lib
All javascript runs in order, followed by all python scripts in order. This explains some things about Rays original code. He has this:
<script type="text/javascript">console.log("Loading brython...")</script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/brython/3.9.1/brython.min.js"></script>
<script type="text/javascript">document.getElementById("psychology").innerHTML = "Loading Standard Libraries..."</script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/brython/3.9.1/brython_stdlib.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/brython/3.9.1/brython.min.js"></script>
<script type="text/javascript">document.getElementById("psychology").innerHTML = "Loading Standard Libraries..."</script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/brython/3.9.1/brython_stdlib.min.js"></script>
"Loading Brython" will only be on the screen for as long as it takes to load the brython.min.js but, and this is critical, NO BUILDING OF INDEXDB etc happens at this point so it will be a flash and not a lot more than that. Same for the next step of "Loading Standard Libraries".
In Rays program we then get to:
<script type="text/python">
print("Loading main python program...")
from browser import document
document["psychology"].text = "Loading Main Program..."
print("Loading main python program...")
from browser import document
document["psychology"].text = "Loading Main Program..."
</script>
<!-- Your main logic should go into here -->
<script type="text/python" src="main.py"></script>
These are python scripts so I expect they get read but nothing happens immediately because the body brython() has yet to run to initialize them. Then brython() runs and it puts "Loading Main Program" onto the screen. Then it runs the code of main.py and its at that point that the importing and transpilation starts to happen This is why "Loading Main Program" stays on the screen so long - it's not just "loading" the main program it's doing a lot of other work. The reason it's doing a lot of work is that this program looks innocuous:
# This script is developed based on Brython https://brython.info
from browser import document, bind
from string import capwords as say
@bind("#say", "click")
def echo(event):
message = document["message"].value
document["output"].text = say(message)
from browser import document, bind
from string import capwords as say
@bind("#say", "click")
def echo(event):
message = document["message"].value
document["output"].text = say(message)
--- but the string module is poison for performance. It's actually a very good example of what a largish Brython codebase might take to process. Look at what it ends up pulling in:
load string from precompiled
brython.js:9614 load re from precompiled
brython.js:9614 load enum from precompiled
brython.js:9614 load sys from precompiled
brython.js:9614 load types from precompiled
brython.js:9614 load sre_compile from precompiled
brython.js:9614 load _sre from precompiled
brython.js:9614 load operator from precompiled
brython.js:9614 load _operator from precompiled
brython.js:9614 load sre_constants from precompiled
brython.js:9614 load sre_parse from precompiled
brython.js:9614 load functools from precompiled
brython.js:9614 load abc from precompiled
brython.js:9614 load _py_abc from precompiled
brython.js:9614 load _weakrefset from precompiled
brython.js:9614 load _weakref from precompiled
brython.js:9614 load collections from precompiled
brython.js:9614 load _collections_abc from precompiled
brython.js:9614 load heapq from precompiled
brython.js:9614 load itertools from precompiled
brython.js:9614 load keyword from precompiled
brython.js:9614 load reprlib from precompiled
brython.js:9614 load _thread from precompiled
brython.js:9614 load _collections from precompiled
brython.js:9614 load _functools from precompiled
Change the code to:
# This script is developed based on Brython https://brython.info
from browser import document, bind
@bind("#say", "click")
def echo(event):
message = document["message"].value
document["output"].text = message.title() # Does same as capwords
from browser import document, bind
@bind("#say", "click")
def echo(event):
message = document["message"].value
document["output"].text = message.title() # Does same as capwords
And it runs instantly and doesn't import all that other stuff from the standard library.
There's something else going on with the animation related to the storagedb saving happening on the main thread and blocking it so that the svg animation is blocked. We can get around that with css animations which are not so blocked but I'll ping Ray on his repo for that.
Ray Luo
Apr 3, 2021, 6:37:39 PM4/3/21
to brython
1. Thanks, Ian, for the investigation. That is a good (re)learning for us that your Brython program would run faster if you can minimize its "import" need, and that is also true even when the program is "just" importing some of the Python standard libraries. I've updated the template again with this finding. The do-nothing template now runs faster, indeed. On the other hand, I think a real application which needs some "import xyz" would still see the relatively long waiting time with "Loading Main Program...".
2. Do you think it is possible for Brython to avoid "the storagedb saving happening on the main thread and blocking UI"? If so, please start an issue in Brython's github repo and mention Pierre there. Here in forum, my understanding is you can not @somebody, therefore, Pierre might not even notice this entire discussion.
3. Your css animation improvement would be highly appreciated! Send me a PR in the template repo please. :-)
Regards,
Ray Luo
Pierre Quentel
Apr 4, 2021, 4:16:07 AM4/4/21
to brython
Le dimanche 4 avril 2021 à 00:37:39 UTC+2, raylu...@gmail.com a écrit :
1. Thanks, Ian, for the investigation. That is a good (re)learning for us that your Brython program would run faster if you can minimize its "import" need, and that is also true even when the program is "just" importing some of the Python standard libraries. I've updated the template again with this finding. The do-nothing template now runs faster, indeed. On the other hand, I think a real application which needs some "import xyz" would still see the relatively long waiting time with "Loading Main Program...".2. Do you think it is possible for Brython to avoid "the storagedb saving happening on the main thread and blocking UI"? If so, please start an issue in Brython's github repo and mention Pierre there. Here in forum, my understanding is you can not @somebody, therefore, Pierre might not even notice this entire discussion.3. Your css animation improvement would be highly appreciated! Send me a PR in the template repo please. :-)Regards,Ray Luo
Sorry, I did not follow the discussion day by day. Imports are one of the main performance issues, and I haven't found a better solution than the indexedDB cache so far, but I would be glad if someone did !
A few comments:
- as Ian says, Javascript programs are executed during page load, and Brython programs are ignored at this stage. It's only when the page is loaded that function brython() is called and the Brython scripts are executed
- Ray, you should not have to use Javascript to show the "Loading main program..." message : just start the Brython program with
from browser import document
document["psychology"].text = "Loading main program..."
and put the other imports after that.
- I'm not sure if this is what Andy meants, but it's not all the scripts in brython_stdlib.js that are translated and stored in the cache, only those that are needed by the Brython script currently executing. What wight give this impression is that many modules in the stdlib require importing dozens of other modules...
- one the modules that is often imported and currently expensive to import is re. This will hopefully change in the next version, I have spent the last months working on a Javascript version of the Python regular expression engine. My first tests with the configparser module (Ray's issue #1592) show a dramatic improvement with this version.
A few comments:
- as Ian says, Javascript programs are executed during page load, and Brython programs are ignored at this stage. It's only when the page is loaded that function brython() is called and the Brython scripts are executed
- Ray, you should not have to use Javascript to show the "Loading main program..." message : just start the Brython program with
from browser import document
document["psychology"].text = "Loading main program..."
and put the other imports after that.
- I'm not sure if this is what Andy meants, but it's not all the scripts in brython_stdlib.js that are translated and stored in the cache, only those that are needed by the Brython script currently executing. What wight give this impression is that many modules in the stdlib require importing dozens of other modules...
- one the modules that is often imported and currently expensive to import is re. This will hopefully change in the next version, I have spent the last months working on a Javascript version of the Python regular expression engine. My first tests with the configparser module (Ray's issue #1592) show a dramatic improvement with this version.
Ian Sparks
Apr 4, 2021, 2:01:07 PM4/4/21
to brython
Sent Ray a PR implementing the following system (see video):
I believe that the saving to indexedb blocks the main thread which will cause any javscript or svg animation running there to stall. Instead this attempts to use CSS animations which run on the GPU and so are not blocked by the main thread. That limits the kinds of things you can do because you have to avoid causing a repaint/reflow of the DOM.
My suggestion is a proof of concept, I knew nothing about CSS animations before this weekend and I've only tested in Chrome but I think it's a useful technique to keep users engaged while (long loading) brython programs start up.
That said, if you look at Andy's website: http://mathsanswers.org.uk/investigations/polyominoes/index.html?size=4 he doesn't go through all this shenanagins. There's a delay while the brython content loads but something is immediately loaded and you're not staring at a blank screen so you feel something is happening.
Ray Luo
Apr 4, 2021, 3:32:47 PM4/4/21
to brython
1. Thanks, Ian. I reviewed that PR. We can continue our interaction there. (PS: I use github daily.)
2. I also appreciate Ian to go extra mile and provide the video. For what it's worth, my template repo has a github workflow/action to enable the github pages. Not sure whether that also works in a forked repo, but you may try to enable github pages for your repo, and then see whether the website will be automatically hosted online. If so, you do not need to bother recording video anymore, just share audience with a link to your website *in action*! How cool is that. :-)
3. It is true that all the html, css and small javascript stuff are immediately available when loading the page. So, if a website happens to have quite some visual content at the beginning, you do not need loading messages (a.k.a. shenanigans, LOL). The better example would the a "4-Fours" website shared in this forum the other day. It came with fully screen of instructions to engage users. (However, a text-heavy homepage is not necessarily user-friendly, but that is a different topic anyway.)
4. Pierre, glad to hear that you made good progress on the re. Keep us updated. :-)
5. By the way, #4 is a perfect example of how multiple update messages during the process can give audience a sense of progress. Ian, I discuss that point in our PR, too. :-)
Regards,
Ray Luo
Ian Sparks
Apr 4, 2021, 4:48:57 PM4/4/21
to brython
Sorry to post in two places, discussion is going on in Rays repo also.
Ray made the point that the animations are ADDING time to the loading because of my eye-candy transition taking at least 1 second. So I remove it. Another video:
I will look at enabling hosting on my own copy of the repo.
As I say to Ray, for me, in this example faster feels worse, "jumpy" but your mileage may vary! It's an interesting exercise.
Ray Luo
Apr 4, 2021, 9:42:02 PM4/4/21
to brython
It is a difficult choice. I do agree the smooth transition looks better especially for end users, but we as developers who do "modify source code and refresh browser" all the time, I know I'd have mixed feeling of that tangible 1-extra-second animation every time. Wish that animation would not block the UI, then we'd have the best of both worlds. Until then, now, that PR is merged without ending animation. See it in action here. By the way, normally I'd do a squash. But this time I keep all the commits there, just in case that in the future we'd like to switch back to the longer wait time version. :-)
Thanks again, Ian!
Andy Lewis
Apr 5, 2021, 1:08:29 PM4/5/21
to brython
On Sunday, April 4, 2021 at 11:01:07 AM UTC-7 ianjs...@gmail.com wrote:
>That said, if you look at Andy's website: http://mathsanswers.org.uk/investigations/polyominoes/index.html?size=4
he doesn't go through all this shenanagins. >There's a delay while the
brython content loads but something is immediately loaded and you're not
staring at a blank screen so you feel something is happening.
On the other hand for some of my pages, such as http://mathsanswers.org.uk/investigations/tessmaker , I do have a "splash screen". It is static rather than animated, but in this case it shows an example of the type of tessellation that can be created with TessMaker, so hopefully it gives something of interest to look at while the page loads.
Another thing I do is avoid importing modules from the standard library so far as possible, especially (as Pierre says) the ones which end up pulling in a large number of dependencies. Sometimes I write my own class to emulate the thing I would otherwise import. But of course for some things that is not practicable.
Pierre has done amazing work to speed up Brython where possible, and with that coupled with increases in processor speeds and download speeds, I would says that over the past few years delays have become less noticeable, but I agree that it is important to get something happening on the screen as soon as possible so that end-users don't think a site isn't going to load. Your project is starting to look really useful in that respect.
Andy
Reply all
Reply to author
Forward
0 new messages