(Suggested) changes to PWA (sw.js and other files)

[Milan Zimmermann]

Gilad,

Based on my observation that both the PWA client and the browser version do not update, for the last several nights I was digging into why and how to fix it. Now I have fixed and localhost served tested code that works well.

I plan to commit the code to my fork of newspeak; I suppose the best way to communicate this would be via a pull request from my fork of the newspeak github repo, let me know if otherwise. I suppose I will send the pull request anyway, you can always reject it or ask me for changes.

I am not sure where to start describing it, so maybe I just paste here part of the text I plan to place to the commit that should end up in the pull request. It is long, and you will see it in the pull request, but here it is ahead of time, if there is something I need to clarify:

TL;DR:
  - This commit fixes the failure of the Newspeak service worker (sw.js) to
    update its caches. This issue caused the Newspeak PWA client app on the user's
    device to never pick up server updates, notably the changed HopscotchWebIDE.vfuel.
  - The changed files are sw.js, index.html, build.sh; there is a new script 'increase-pwa-version.sh'
    called from 'build.sh'.
  - Apart from fixing the end-user experience, this change also touches Newspeak development
    (without any changes in the developer's workflow), as running '. ./build.sh' now changes
    the variable pwaVersion in 'sw.js'
  - How to test this change:
    - Server / build
      - Get the latest version and make sure webIDE/index.html is copied to the webIDE directory
        where Newspeak is served locally for https://localhost:8080/webIDE,
        or for https://newspeaklanguage.org/webIDE.
        I am serving from the git version of webIDE, so I do not have to copy anything.
      - Run . ./build.sh as normally.
      - Make sure the updated webIDE/sw.js is copied into the webIDE directory where Newspeak is served.
        I am serving from the git version of webIDE, so I do not have to copy anything.

    - PWA Client / browser
      - You do NOT have to do anything, the versioned sw.js should now be installed,
        and use its versioned cache. But if you want to check:
        - In Browser, navigate to  https://localhost:8080/webIDE,
          or https://newspeaklanguage.org/webIDE.
        - Open the PWA
        - In both browser and PWA, if you press F12 to debug (Chrome) and look at:
            Application -> Storage -> Cache Storage.
          you should see a versioned cache number, such as 'newspeak-ide-cache-version-13'.
      - Every time a new server version is build and the new build deployed,
        you should see a new cache version on the client PWA or browser,
        such as 'newspeak-ide-cache-version-14'.


Motivation for change:

  - 'Before this change': After a new Newspeak version deployment on
    https://newspeaklanguage.org/webIDE, the PWA client app on the user's device
    did not update even after multiple launches, unless a manual cache clean was performed.
    I had the same experience (the PWA client not updating) with the webIDE server
    running on localhost, and the PWA client created from it.
    In fact, even the in-browser running webIDE did not update after server deployment.

  - Having looked into how service workers function, it seems clear that
    the mechanism that causes the service worker update on the user's device
    (changing some bits) was never triggered.  As a result, the service worker caches are never automatically
    updated, causing, among other things, the same (old) HopscotchWebIDE.vfuel to be used after server update.

Service worker lifecycle - summary

  - From https://developer.mozilla.org/en-US/docs/Web/Progressive_web_apps/Tutorials/CycleTracker/Service_workers:
    - Once the PWA is installed on the user's machine, THE ONLY WAY to inform
      the browser that there are updated files to be retrieved is to change
      the service worker (this file, sw.js) on the server.

  - Corollary: The only way for a PWA website to force an update of the service worker
      is to make a bit-change on the server's service worker (the sw.js).

      Such enforced update of the service worker, causes the browser to call a one-time 'install' and 'activate' on the
      service worker, which the service worker can use to delete old caches and cache the new assets.

Changed files in more detail:

  - index.html
    - moved caching of assets (cache.addAll) to the 'install' event in sw.js
    - added catching errors during serviceWorker.register
  - sw.js
    - almost complete rewrite, although keeping the basics of using cache-first strategy
      (assuming used to lighten load on the server)
    - added a version number which should be changed on every deployment,
      to support the reinstall of the service worker on the client PWA browser.
      This number is increased automatically, during running each '. ./build.sh'
      by calling the added 'increase-pwa-version.sh'
      - 'const pwaVersion = 12;'
    - added a versioned cache name
      - 'const pwaCacheName = 'newspeak-ide-cache-version-' + pwaVersion;'
    - added a full list of assets
      - 'const pwaAppResources = [ .. see code .. ];'
    - added processing in the 'install' event:
      - all assets listed in 'pwaAppResources' are added to the NEW cache version 'pwaCacheName'
    - added processing in the 'activate' event:
      - OLD caches are deleted
    - modified processing in the 'fetch' event
      - In principle, this remains the cache-first fetching, but error checking was added etc
      - Assets not in 'pwaAppResources' would still cache here from network
  - build.sh
    - added call to 'increase-pwa-version.sh' which increases version number in sw.js by 1, example
      - 'const pwaVersion = 12;' => 'const pwaVersion = 13;'
  - increase-pwa-version.sh
    - a simple sed script that edits sw.js and increases version number by 1.

[Gilad Bracha]

Hi Milan,

This sounds good. Please file a pull request. Thanks!

--
You received this message because you are subscribed to the Google Groups "Newspeak Programming Language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to newspeaklangua...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/newspeaklanguage/CAEc2VK22%3D17fi6Z5_pFvvBBU2YUjwwCWJ_e2LswLCfufGZhM-g%40mail.gmail.com.

--

Cheers, Gilad

[Gilad Bracha]

Thanks Milan. I've integrated your changes and both the soucre repo and the Newspeak PWA site are now up to date.

[Gilad Bracha]

Oh, and you'll need to update the Psoup repo as otherwise, things will break when you rebuild. That's because I made a small change to mirrors.

[Milan Zimmermann]

Great, thanks for merging the changes.

I have pulled both the latest of primordialsoup and newspeak repos, the build process worked, and I see the service worker update in and working for the PWA at https://newspeaklanguage.org/webIDE/. 

Also, early this morning, before updates, I disconnected my network, and tried to run PWA pointing to https://newspeaklanguage.org/webIDE/, it was not showing at all. After the update, then disconnecting my network, the same PWA (with the new versioned cache etc) works well.

While so far it looks good to me, let me know if you run into any issues. 

Milan

PS : One thing that may be worth looking into for me, is a 'development workflow' for this: 

The pwaVersion in sw.js is being updated for anyone who runs build.sh locally - but it should really be only pushed by you, otherwise there may be commit conflicts. I am not sure how many people are pushing to the newspeak master. Long term, using something like a globally unique ID instead of the linearly increasing pwaVersion by 1 would be better - the simple number + 1 was deceitfully attractive, and .. simple

To view this discussion on the web visit https://groups.google.com/d/msgid/newspeaklanguage/4b400e11-71db-48d3-a0da-c3991708f53cn%40googlegroups.com.

[Gilad Bracha]

Hi Milan,

Yes, you're right, the version number must only be increased in a coordinated fashion, so it increases monotonically.  I think the only sustainable version is to manually ensure that the version number is max(local_version_number, repo_version_number+1) when commiting changes. This could/should be automated as well.  

To view this discussion on the web visit https://groups.google.com/d/msgid/newspeaklanguage/CAEc2VK204guppodyteuezVv-YEc6JrmypyFqG%3D9aEeGPFvB2Mg%40mail.gmail.com.

--

Cheers, Gilad

[Milan Zimmermann]

Gilad,

I would be happy to implement changes you are suggesting for the pwaVersion increase by looking both at the local and repo version number (although I'd ask about what version to use the local git repo or remote repo, staging, or worktree, or committed etc).

Having said that, I feel the core of the problem is that the pwaVersion serves both the role of 'release version' on  newspeaklanguage.org as well as 'user version' when running in development mode on localhost. So the pwaVersion serves BOTH as 'user data' AND 'release identifier'.  The fact that the 'pwaVersion as user data' is actually stored in git is the reason for potential merge conflicts - users using the build.sh locally (even if they do not commit and push their 'sw.js'), would get merge conflicts after pull.

I can only think of one way to solve this correctly described in 'Longer term: The only correct solution I can think of'. But there may be other solutions I fail to think of.

So let me suggest the following:

'Near term': In the near term (days to a few weeks): We can avoid the problem, by 

- you being the only person who *deploys* to newspeaklanguage.org/webIDE as is the case anyway, 

- AND NO ONE commits or pushes the (constantly changing pwaVersion in) the 'sw.js'. 

That way: 

a) your local sw.js in your 'git worktree' increases the pwaVersion on your every build and 

b) your local sw.js  is the keeper of the 'globally last' pwaVersion for newspeaklanguage.org/webIDE, and 

c) your deployments from the 'git worktree' to the newspeaklanguage.org (which is in your git worktree or is a copy of it)  always changes the pwaVersion from the previous version, which makes the PWA to renew caches, as intended (both on your localhost and newspeaklanguage.org). 

d) Localhost testing for everyone developing and running their localhost works on the same principle: their uncommitted sw.js works as the keeper of the pwaVersion on their localhost.

e) No merge conflicts on sw.js

'Longer term: The only correct solution I can think of':  I feel we need to get away from keeping the pwaVersion in git, for the reason described in the second paragraph, while still ensuring any new deployment to localhost, newspeaklanguage.org, or any other site hosting the PWA gets its version changed for all clients who already installed their PWA. How to achieve that? This is the way I can think of:

1.   The git version of 'sw.js' will always have this line: 'const pwaVersion = 1;'
2.   We use a separate directory for every deployment from where our git 'working tree' is located. If you deploy both localhost and newspreaklanguage.org, there would be 3 directories a) the working tree b) directory where localhost is served c) directory where newspeaklanguage is served
3.   Everything will be scripted - see below
4.   The changing of the pwaVersion happens during deployment - the process of copying from the working tree to the serving directory
5.   We use a GUID generator (or at least a random number) for each changed pwaVersion
6.   We separate build from deploy, so we have 3 scripts:

• build.sh # Same as the current build, EXCEPT editing the pwaVersion. The built files are copied to 'out' directory
• deploy.sh   server_directory # COPIES/SYNCs all files from the git 'working tree' to the server deployment directory AND replaces  'const pwaVersion = 1;' with  'const pwaVersion = a_new_GUID;'
• build_and_deploy.sh   server_directory # a combination of the above

  

So the developer would typically run

build.sh

or

deploy.sh  localhost_server_dir

build_and_deploy.sh  localhost_server_dir

or 

deploy.sh  newspeaklanguage_server_dir

build_and_deploy.sh  newspeaklanguage_server_dir

I realize implementing the above would require changes on how you (and others) do things during your development (actually that is the reason I was not suggesting it initially, I do not like to disturb or suggest to others how to do their work and hobby :) ). 

So if the above 'workflow' process would be working for you and others, I would work on implementing it. If you would prefer some other (perhaps simpler) process around the updating pwaVersion, I would work on implementing it as well, let me know,

Milan

PS:

Purely from the perspective of 'achieving correct behavior' we need that:

  - A site serving a PWA publicly (newspeaklanguage.org/webIDE) should ensure two versions in sw.js on the server never repeat, as there may be many clients with different versions already installed, and a change must kick in for all. A monotonically increasing number works for this purpose, but that brings the question of who is the keeper of the 'last version used for the site' and mechanism of keeping it. That is why I feel a GUID or high random number is better.

  - Private local development (localhost:8080/webIDE), that has a single user, all we need to ensure a new deployment changes the version from previous deployment.

To view this discussion on the web visit https://groups.google.com/d/msgid/newspeaklanguage/CAOq-cNy1kD58mhawFoDohK3HBe3Qifh6U%3D%3DCXH3P3zyhCwm_xQ%40mail.gmail.com.

[Gilad Bracha]

Hi Milan,

I agree that the release & user versions should be distinct.  However, I think it would be good to have a release version on git. So here is a counter proposal:

(1) The release version is recorded in a file that is committed to the central repo at newspeaklanguage.org.

(2) The user version is recorded in a file that is never committed.

(3) The build script increments the user version in the file, and copies it into sw.js.

(4) the deploy script pulls the release version from the repo, increments it, and copies it into sw.js.

(5) Each developer chooses where they are serving the local version from, and has a script that copies sw.js, the psoup files and the vfuel from the the main repo's out directory to their chosen directory. 

Thoughts?

To view this discussion on the web visit https://groups.google.com/d/msgid/newspeaklanguage/CAEc2VK0ACpLaWsAr-84qaCBNJen8EGUZ%2BNk0fUHz47rPeuUyyg%40mail.gmail.com.

--

Cheers, Gilad

[Milan Zimmermann]

Gilad,

Sounds great; I think I understand your (1) - (5), the gist of my extrapolated understanding is that:

a) We store, AND keep in git, a 'release version file' which is effectively for newspeaklanguage.org. We also store but do NOT keep in git a 'user version file' for localhost.
b) Each developer has a user-named HTTP deployment directory (OUTSIDE OF the git directory 'newspeak') for localhost; additional user-named HTTP deployment directories are possible.
c) (3) suggests that the 'user version file' is managed by the 'build' step, and (4) suggests that the 'release version file' is managed by the 'deploy' step. I feel that both the 'user version' and the 'release version' (and even *other* version) should be managed by the 'deploy' step only, never the 'build' step - my reason is that it seems more symmetrical. All steps are equivalent, whether the deployment is to a private (localhost) or public (newspeaklanguage.org, myBlog.org) site. The steps differ only by arguments passed to 'deploy.sh'.  [But I am fine to apply the (3) (4) instead of c)].


Short TL;DR:
----
- Development and deployment lifecycle after this change:
  - One-time: Developer initializes deployment directory/ies. I plan to provide a script for it
  - One-time: Only if developer wants to build and deploy some non-localhost, Newspeak based HTTP such as https://giladsBlog.org, developer adds a file *newspeak/deployed-versions/giladsBlog.org.ver*
  - Developer makes a change in a Newspeak file
  - Developer runs 'build.sh'
  - Developer runs *deploy.sh version-file-for-url deploy-dir-running-http-url* described in D).
    - Examples:
      - *deploy.sh localhost   /gilad/deploy/localhost-http*
      - *deploy.sh newspeaklanguage.org /gilad/deploy/newspeaklanguage-http*
    - Note: If the http directory is on a remote system, we can eventually support something like this, if ssh/scp is available on the remote system:
      - *deploy.sh newspeaklanguage.org gi...@newspeaklanguage.org:/servers/newspeaklanguage-http*

Notes:
----
  - 'deploy' or 'deployment', is implemented by the script 'deploy.sh' in step D) below. It is the process that copies files built by 'build.sh' to the 'deploy directory' (the directory where the HTTP server runs - the '_DEPLOY_DIR' in C) below).
  - 'build.sh' is not updating versions. Version updates are performed only in 'deploy.sh' described in C) and D) below.

Here is a Long TLDR, in capital letter items A) - F). Not sure anyone has the stamina to read it though :) so I hope everything above is somewhat understandable. Although at least going over A) may set the stage.

Long TL;DR:
----
A) Deployed versions tracking: We add a 'deployed-versions' directory, containing 'version files', ending with '.ver' suffix. Each '.ver' file holds the last version number in the corresponding deployed http server. The file names suggest but do not enforce the URLs. It may look like this:
   + deployed-versions <== added directory
     - newspeaklanguage.org.ver <== git committed, version deployed on newspeaklanguage.org
     - localhost.ver            <== git ignored, version deployed on localhost
     - vvv optional vvv
     - giladsBlog.org.ver       <== git committed or ignored, version deployed on giladsBlog.org
     - etc ..
       
       
B) .gitignore changes - see the PS section

C) Deployment directories: Developer already has or adds one deployment directory for each HTTP server the developer runs; each such directory contains a deep copy of the newspeak/platform/webIDE directory. To update a HTTP server (on a local or public facing URL) a developer runs the 'build.sh', command, followed by the command *deploy.sh version-file-for-url deploy-dir-running-http-url*. Item D) has examples of use of 'deploy.sh'.

Deploy directories examples (again, *all directories are free, and developer-named*):

- LOCALHOST DEPLOY DIR examples:/milan/deploy/localhost-http, /gilad/deploy/localhost-http

- NEWSPEAKLANGUAGE DEPLOY DIR examples: /gilad/deploy/newspeaklanguage-http, gi...@newspeaklanguage.org:/servers/newspeaklanguage-http

- MYBLOG DEPLOY DIR examples: /milan/deploy/newspeak/myblog-http, mi...@mysite.org:/deploy/myblog-http, gi...@giladsblog.org:/some-dir/deploy/blog-http


D) Deployment script: We add a script deploy.sh, which takes 2 arguments, as follows:
  - *deploy.sh version-file-for-url deploy-dir-running-http-url*
  - Performs in Summary:
    - increases the version, in the version file named in arg1
    - copies the sw.js with the increased version, AND the results of build.sh (primordialsoup.wasm, primordialsoup.js, *.vfuel) to the deploy directory in arg2 (the directory where the HTTP server runs).
  - Examples:
    - *deploy.sh localhost   /gilad/deploy/localhost-http*
      - copies the git version of sw.js (the newspeak/platforms/webIDE/sw.js) to the deployment directory /gilad/deploy/localhost-http/webIDE
      - increases the git version of localhost.ver (the newspeak/deployed-versions/localhost.ver), and places the increased version to the sw.js in the deploy directory (the /gilad/deploy/localhost-http/webIDE/sw.js). Git version of sw.js is unchanged.
      - copies the result of 'build.sh', from out/{primordialsoup.wasm, primordialsoup.js, *.vfuel} to /gilad/deploy/localhost-http/webIDE/public/assets/lib/
    - *deploy.sh newspeaklanguage.org /gilad/deploy/newspeaklanguage-http*
      - equivalent to above, substitute 'localhost' -> 'newspeaklanguage'

E) Further parametrization (not part of this implementation): The above implementation assumes a user/developer only uses one 'build from' directory (this is the newspeak git dir) but multiple 'deploy to' directories (where HTTP serve(s) run). The 'deploy to' is parametrized, the 'build from' is not. Perhaps this should change?

F) Deploy directory is remote (support later) : Ability for 'deploy.sh' to handle deployments running on remote HTTP servers in one step. Initially if a HTTP server is running on a remote system, developer has to do the remote copy after 'deploy.sh' runs and copies to local deploy directory. This is not hard to do, but assumes there is a ssh-like connection etc. We can eventually support something like this, if ssh/scp is available on the remote system:
      - *deploy.sh newspeaklanguage.org gi...@newspeaklanguage.org:/servers/newspeaklanguage-http*

Milan

PS: gitignore changes

.gitignore: we add the following entries. This will ignore all versions except newspeaklanguage.org.ver:
!deployed-versions
deployed-versions/*
!deployed-versions/newspeaklanguage.org.ver

To view this discussion on the web visit https://groups.google.com/d/msgid/newspeaklanguage/CAOq-cNw%3D4qTydARDSEs-r3qH1axJdMYubsbTOkSrAuHYo8S6NQ%40mail.gmail.com.

[Gilad Bracha]

Hi Milan,

Sounds good (I read everything). Go for it! Thanks.

To view this discussion on the web visit https://groups.google.com/d/msgid/newspeaklanguage/CAEc2VK1_5sFtmRJ%2BHfXUiHV-zeLDbz8cw%2BK4%3D-gWF-cMYBTxjw%40mail.gmail.com.

--

Cheers, Gilad

[Milan Zimmermann]

Gilad, great. Thanks for reading it all :). I will start on it.  Will update here or on Github,

Milan

To view this discussion on the web visit https://groups.google.com/d/msgid/newspeaklanguage/CAOq-cNw2UFdzdn-J3HVv6kBZwHGnSC0UAbgV-kAS7qxM_zUs0A%40mail.gmail.com.