Trying to configure a localhost server for the PWA WebIDE Newspeak

[Milan Zimmermann]

Gilad,

I was not able to configure a localhost server serving the PWA (well, I was, but only a hand-made flat version that included editing files, as a proof of concept).

Would it be possible to share the directory structure (perhaps just a result of ls -lR) of the http server that runs the https://newspeaklanguage.org/webIDE/ ? I think that would help me to figure it out.

Thanks,

Milan

PS : Context only (no need to read this context - just in case it reveals something around the multiple versions of primordialsoup.js described in it)

======

For a while, I have been able to run '. ./build.sh' in Newspeak to build the  'primodialsoup.wasm', 'primodialsoup.js' and  'HopscotchWebIDE.vfuel' and use the 'out' directory versions.

After you released the PWA version, I assumed that creating a localhost server would need the following:

1. Copy to the top level of the http server, the webIDE directory from https://github.com/newspeaklanguage/newspeak/tree/master/platforms/webIDE .
• Notes:
• - This directory contains 'primodialsoup.js' in https://github.com/newspeaklanguage/newspeak/blob/master/platforms/webIDE/public/assets/lib/primordialsoup.js
• - This directory does not contain 'primodialsoup.wasm'
• - This directory does not contain 'HopscotchWebIDE.vfuel'
2. Place the  'newspeak/out' results of build.sh of 'primodialsoup.wasm' and  'HopscotchWebIDE.vfuel' into places indicated by index.html:

      const otherAssets = {
        codemirror_css: '/webIDE/public/assets/lib/codemirror.css',
        primordialsoup_wasm: '/webIDE/public/assets/lib/primordialsoup.wasm',
        hopscotch_vfuel: '/webIDE/public/assets/lib/HopscotchWebIDE.vfuel',
      };

But the above did not produce a working PWA version.

I have actually done a massive amount of permutations, stemming from my observation that there are multiple versions of 'primordialsoup.js', every single one with a different size. But none of it lead to success :(

size=114508 Oct  8 01:27  ./newspeak/platforms/webIDE/public/assets/lib/primordialsoup.js (GIT)
size=157458 Sep 25 13:58  ./newspeak/primordialsoup.js
size=86790  Nov 22 00:40  ./newspeak/out/primordialsoup.js
size=332036 Nov 22 00:40  ./newspeak/out/primordialsoup.wasm

I also wonder why those js files are made so different, but that is secondary 

Thanks,

Milan

[Gilad Bracha]

Hi Milan,

a. The correct primordialsoup.js is  https://github.com/newspeaklanguage/newspeak/blob/master/platforms/webIDE/public/assets/lib/primordialsoup.js, the one with size 114508. The structure of the webIDE directory on the newspeak web site is identical to the one at  https://github.com/newspeaklanguage/newspeak/blob/master/platforms/webIDE/. I believe they are identical (I need to re-verify that). 

b. I usually run the local web server from the out directory, because that way it is easy to do development on the IDE itself. 

c. Running that way, I don't get a true PWA; I have the IDE running in a web page, but it is not recognized as a PWA so one cannot install it from that page. You can see that the little icon allowing for installing/updating the PWA does not appear when running from localhost. This is to be expected, because I am not using the webIDE structure etc.

d. If you run the server inside the webIDE directory, as you say, I'd expect it to work. it might work. Or perhaps it needs to be run in the directory above it.

Let me know how it goes.

--
You received this message because you are subscribed to the Google Groups "Newspeak Programming Language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to newspeaklangua...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/newspeaklanguage/CAEc2VK3QcbneqSfkKEf%2BJHnPDvEJZA82ifDCcHiAwj4HYjzLBQ%40mail.gmail.com.

--

Cheers, Gilad

[Milan Zimmermann]

Thanks Gilad for responding.

I tried a few things again, always placing on the top of the webserver:

  -  the GIT version of the 'webIDE',  including the primordialsoup.js size 114508

  - with the primordialsoup.wasm and the HopscotchWebIDE.vfuel (results of the build.sh) copied from the out directory to the lib directory

But I still cannot get the server to work, there is a browser console error related to 'wasm streaming compile': primordialsoup-setup.js:40 wasm streaming compile failed: LinkError: WebAssembly.instantiate(): Import #21 module="env" function="emscripten_memcpy_js": function import requires a callable.  I pasted the full error in PS.

I hate to bother you with this; I would hope if I see the result of 'ls -lR' from the server that runs the webIDE in https://newspeaklanguage.org/webIDE/ , I may understand more about the issue.

======

Feel free to ignore everything below if it does not make sense or is too long (hmm well, it is long for sure)

======

But - let me try from another direction:  It seems to me there is something 'asymmetrical' (or potential file duplications) between the files in GIT and the result of build.sh. But I am looking at it as a black box, and may well be wrong. Anyway, let me describe what I mean:

1. The important files in running the PWA are (not specifying full paths here)

- index.html (defines the paths of all files: The js files, the vfuel file, the wasm file)

- sw.js

- primordialsoup-setup.js (defines the path of the vfuel file AGAIN, but the path matches the index.html, so that should be fine)

- primordialsoup.js

- primordialsoup.wasm

- HopscotchWebIDE.vfuel

2. index,html, pw.js and primordialsoup-setup.js: The only versions exist in GIT. So there is no danger of duplicates

3. primordialsoup.wasm and HopscotchWebIDE.vfuel: The only versions are a result of build.sh. So there is no danger of duplicates

4. Conflict/duplication: primordialsoup.js: there are versions BOTH in GIT (lib under webIDE), AND a version in the 'out' directory resulting from build.sh.

Because the GIT version of 'primordialsoup.js' and all built versions of  'primordialsoup.js' are different, I have a hard time making sense of what should be built and what should be present in GIT. If the built 'primordialsoup.js' is wrong (different from the GIT version), where did the GIT version originate from? Also, if the 'primordialsoup.js' is in GIT, should the 'primordialsoup.wasm' and HopscotchWebIDE.vfuel also be in GIT? (that would not answer my inability to build, but it would provide a basis for a working server)

Thanks

Milan

PS:  The errors I am getting in the browser console :

primordialsoup-setup.js:40 wasm streaming compile failed: LinkError: WebAssembly.instantiate(): Import #21 module="env" function="emscripten_memcpy_js": function import requires a callable
printErr @ primordialsoup-setup.js:40
primordialsoup-setup.js:40 falling back to ArrayBuffer instantiation
printErr @ primordialsoup-setup.js:40
primordialsoup-setup.js:40 failed to asynchronously prepare wasm: LinkError: WebAssembly.instantiate(): Import #21 module="env" function="emscripten_memcpy_js": function import requires a callable
printErr @ primordialsoup-setup.js:40
primordialsoup-setup.js:40 Aborted(LinkError: WebAssembly.instantiate(): Import #21 module="env" function="emscripten_memcpy_js": function import requires a callable)
printErr @ primordialsoup-setup.js:40
primordialsoup.js:1361 Uncaught (in promise) RuntimeError: Aborted(LinkError: WebAssembly.instantiate(): Import #21 module="env" function="emscripten_memcpy_js": function import requires a callable)
    at abort (primordialsoup.js:1361:11)
    at primordialsoup.js:1538:7
webIDE/:1 Uncaught (in promise) Error: A listener indicated an asynchronous response by returning true, but the message channel closed before a response was received
sw.js:16 [Service Worker] Fetching resource: http://localhost:8081/webIDE/
sw.js:20 [Service Worker] Caching new resource: http://localhost:8081/webIDE/

To view this discussion on the web visit https://groups.google.com/d/msgid/newspeaklanguage/CAOq-cNyzZ4%3DMM31yxgot1w6c0HL0WAKKS_nx-J%3D3VCt8X7XEvA%40mail.gmail.com.

[Gilad Bracha]

Hi Milan,

Again, aplogies, these days I find it hard to respond in a timely manner.

Have you looked at

https://github.com/newspeaklanguage/newspeaklanguage.github.io

That is where the PWA is hosted. 

[Milan Zimmermann]

Gilad,

I absolutely understand, stormy and difficult times.  

Thanks for the link. I failed to realize what I was asking for, is on Github already. I will look into it and get back here (whether I find the reason or not). 

Thanks

Milan

PS

The primordialsoup.js my 'emsdk' is building (no matter what version I tried) is quite different from the version in git, so I am starting to suspect something to do with my failures - although my non-PWA version works with my builds. But anyway, I will get back here.

To view this discussion on the web visit https://groups.google.com/d/msgid/newspeaklanguage/5d6f2c1d-2e5d-48a4-9641-33a0df86ccf1n%40googlegroups.com.

[Milan Zimmermann]

Gilad,

Thanks for the link to the github repo https://github.com/newspeaklanguage/newspeaklanguage.github.io that is used to run the PWA in https://newspeaklanguage.org/webIDE/. The link helped me to finally understand how the PWA is served, and how to duplicate it locally (with caveats described below)

Let me summarize my findings, items 5 and 6 are relevant (although maybe not important) to the directory structure behind  https://newspeaklanguage.org/webIDE/  is serving the IDE.

I will try to be reasonably brief (although my path was bumpy :) )

1. My initial goal was to be able to run the full emsdk + Newspeak build.sh that would produce a fully working local IDE using the https://github.com/newspeaklanguage/newspeak/tree/master/platforms/webIDE as the directory structure serving the IDE.
2. I was able to build an "almost fully working" localhost served versions (e.g. http://localhost:8081/webIDE/) using both
•  The version 2.0.0 of emsdk (which does not require any changes to SConstruct)
•  The 'latest' emsdk (which does require changes to SConstruct, namely adding "_free" and "_malloc" to the build: '-s', 'EXPORTED_FUNCTIONS=["_load_snapshot", "_handle_message", "_handle_signal", "_free", "_malloc"]',  '-s', 'EXPORTED_RUNTIME_METHODS=["stringToUTF8"]',
3. By "almost" in "almost fully working" I mean that no images are showing in the IDE in the browser (see PS), everything else works.
4. It turns out the reason for the images not showing, is that the images are requested from the 'top level' of the serving directory - from the same level as the 'webIDE'. So the lack of images can be fixed by adding the images on the top level (beside webIDE) although I feel this is not expected or ideal.
5. I was able to duplicate the same issue with the version in  https://github.com/newspeaklanguage/newspeaklanguage.github.io - if the webserver runs from that directory, the presence of the png images on the "top level" (the same level as the "webIDE") is crucial for the images to show. Removing the images from the top level and cleaning up caches ends up with the IDE without images in the browser.
6. As a consequence of 5, simply taking the https://github.com/newspeaklanguage/newspeak/tree/master/platforms/webIDE  structure and serving from there, results in an IDE without images. Also the (duplicate) images in https://github.com/newspeaklanguage/newspeak/tree/master/platforms/webIDE/public/assets/lib are currently not used at all afaict. Although I assume that is the intended location from where the images should be ideally loaded, I could be wrong.

I am curious if 5. and 6 are known or intended. If not, and if there is a value of looking into it, let me know, I could (look into it further).

Thanks,

Milan

PS 1: I would be curious to know the version of emsdk (e.g. 'source $EMSDK_HOME/emsdk_env.sh; emsdk list; em++ --version') you are using. I suspect it is not much higher than 2.0.0 as after that some of the SConstruct changes would be needed.

PS 2: This is what I mean "almost fully working" version of IDE with no images:

 

[Gilad Bracha]

Hi Milan,

I think the problem is that the UI system uses the image data to determine a URI, which is always related to the top level domain being served. Since this needs to work on the Newspeak web site, it can't work at the same level as the nested webIDE directory. I think I discovered the problem when I set things up, and realized that the images in the asset directory would not work. I think fixing this would entail changing the way the Vfuel is built, and/or the way images are handled in the UI. It just isn't worth dealing with - life is too short.

You received this message because you are subscribed to a topic in the Google Groups "Newspeak Programming Language" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/newspeaklanguage/cGzw65b35vM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to newspeaklangua...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/newspeaklanguage/CAEc2VK0YKtvDeu%3DPk9F6WqBNTidaFaiyajgaJ-RWWUTorcbd-Q%40mail.gmail.com.

--

Cheers, Gilad

[Milan Zimmermann]

Hi Gilad,

Thanks for the update. My main reason for digging into this was that my build process consistently resulted in .vfuel + (primordialfuel) .wasm + .js that created a non-fully-working system due to the missing images. I thought before it was due to my build process, version of emsdk, or in that area. It became clear from the debugger the system always requests the images from the top level. Now you confirmed this cannot be fixed easily, I am fine leaving it behind (although the debugging led me to places that woke up more interest in the way Aliens-images are handled).

Anyway it all sounds good (well apart from the shortness of life :) but that is an irrefutable part of it I suspect), and I'll move on to other things Newspeak... likely resulting in more questions in the future.

Thanks for your help and appreciate your time taken,

Milan

To view this discussion on the web visit https://groups.google.com/d/msgid/newspeaklanguage/CAOq-cNx%3D2_iwFwnbCP39HsrpgzfmtOS-v%2BpK9VKSNKgfs8Dvtg%40mail.gmail.com.

[Gilad Bracha]

Hi Milan,

I would suggest you simply install a WebIDE subdirectory in the out directory and serve localhost from out/, using http://localhost:8080/webIDE/. That works well for me.  If you also make sure to copy the results of the build to webIDE/public/assets/lib/ you should be able update the environment whenever you rebuild it.  You can run this in the browser, but also use it to install a version of the PWA which can subsequently be run independently of the local web server. Running Chrome, this produces a distinct PWA from the one installed via newspeaklanguage.org (i.e., one can have a different PWA per source URI). I think one could keep different subdirectories of out/ and serve and install versions of the PWA that way, though I haven't verifed that.

To view this discussion on the web visit https://groups.google.com/d/msgid/newspeaklanguage/CAEc2VK3wwj%2BSaWrsk%3DGbk11GDaEiukWhhkYcUgG10hh%3DO%2BbYNA%40mail.gmail.com.

--

Cheers, Gilad

[Milan Zimmermann]

Thanks Gilad for the information and suggestions. A few notes inline

On Fri, Dec 1, 2023 at 7:56 PM Gilad Bracha <gi...@bracha.org> wrote:

Hi Milan,

I would suggest you simply install a WebIDE subdirectory in the out directory and serve localhost from out/, using http://localhost:8080/webIDE/.

Yes, serving from the 'out' directory with the webIDE subdirectory added is a good idea since 'out' already contains all the images. I do have scripts that clean it by removing all contents though, so I run my http server from a separate location 'newspeak-server'.

 

That works well for me.  If you also make sure to copy the results of the build to webIDE/public/assets/lib/ you should be able update the environment whenever you rebuild it. 

Yes, I have a local wrapper around build.sh which copies the built IDE vfuel, primordialsoup.wasm and js from 'out' to newspeak-server/webIDE/public/assets/lib

You can run this in the browser, but also use it to install a version of the PWA which can subsequently be run independently of the local web server.

For my development experiments, I generally run the IDE from the browser pointing to my local web server.  I often use the PWA created from a local URL in a side-by-side window with the browser where I do development, and use the PWA for searches, or side-by-side browsing of other classes. 

This actually brings me to a note of what I miss the most in the Newspeak IDE in browser: split panels where I can see multiple things side by side (like emacs split-window-horizontally/vertically). Even thinking about trying to add a split window but there is more education before that :)

Running Chrome, this produces a distinct PWA from the one installed via newspeaklanguage.org (i.e., one can have a different PWA per source URI).

Yes, that works, I have two desktop icons like this and they show nicely the URL of the backing server.

 

I think one could keep different subdirectories of out/ and serve and install versions of the PWA that way, though I haven't verifed that.

Do you mean a single local Newspeak server, serving from e.g. webIDE and webIDE2 accessed as localhost:8080/webIDE and  localhost:8080/webIDE2? I agree that should work (with edited index.html in webIDE2). Would be useful to run for example the version '2' that is built using the latest 'emsdk'.  I may do that as I have sunk quite a bit of time into porting to 'emsdk latest' which requires a different SConstruct, so I should keep it alive for testing while doing my development with 'emsdk 2.0.0' that is probably close to what you use (would be curious what you use)

Thanks,

Milan

PS: My next step is to finish my SimpleIDEBrowser (simplified IDE Browser from scratch with a bit of borrowing from Browsing and HopscotchWebIDE). This is purely as a discovery exercise of Hopscotch on a deeper level that I interrupted 18+ months ago.

To view this discussion on the web visit https://groups.google.com/d/msgid/newspeaklanguage/CAOq-cNy5-_dd9MTiaDffgMfjekXNS-59xqXt7sMth%3D9SK-QFkA%40mail.gmail.com.