How to use Node modules in browser?

[bimlas]

Dear all,

In a Node.js Javascript project, I want to use TiddlyWiki to display data.

To run Node methods, I need to create modules with "module-type: utils-node" (if I understand it well), thus I can reach them through $tw.utils. These will only be loaded by TiddlyWiki if $:/info/node is "yes" (so the value of $tw.node is not null). I have several Node wiki, checked in each (in the browser), but it seems that nowhere does it detects that the wiki is running under Node, because viewing $:/info/node in the browser says "no".

I'm confused, because if I check in the code, it detects that it is running under Node.

$tw.boot.argv = ['myproject', '--listen'];
$tw.boot.boot();


console.log($tw.node);
// {}
console.log($tw.wiki.getTiddlerText('$:/info/node'));
// yes

If I create a widget that uses this Node module, it doesn't work in the browser, instead of the widget, RSOD appears: $tw.utils.myNodeFunction is not a function

My question is how can I use the Node module in the browser?

[bimlas]

To be clear: I need Node because I want to use NPM modules.

[Jeremy Ruston]

Hi bimlas

$tw.boot.boot() is an async function, and takes a callback. TW won’t be fully initialised until it gets to the callback. So try something like:

var $tw = require("./boot/boot.js").TiddlyWiki();

$tw.boot.argv = ['myproject', '--listen'];

$tw.boot.boot(function() {

console.log($tw.node);

console.log($tw.wiki.getTiddlerText('$:/info/node'));

});

Best wishes

Jeremy

-- 
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.
To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywikidev.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/3e0d6bc9-86c3-443c-8146-81acc7c98809%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jeremy Ruston]

Hi bimlas

Apologies, I should have pointed out that $tw.node having the value {} does evaluate to true, as expected.

Best wishes

Jeremy

[bimlas]

Jeremy,

Thank you for your reply, but I still don't understand that although it can recognize Node in the code, $:/info/node is always "no" in the browser, so it does not load utils-node modules. How do I use them, for example, in widgets? For example, if I want to use the $tw.utils.getEditionInfo() method in my browser, how do I do it?

[bimlas]

Maybe it's basically a bad way of thinking, so I am reformatting the question: how can I use NPM modules in a widget?

[Jeremy Ruston]

Hi bimlas

Maybe it's basically a bad way of thinking, so I am reformatting the question: how can I use NPM modules in a widget?

Ah, OK, you’re asking if it is possible from the browser to invoke a function that is running on the server. That would be some kind of RPC mechanism, and we don’t have that, I’m afraid.

If you have something running in the browser that wants to talk to something on the server the simplest way to do it is to build on the sync mechanism: have the browser create a tiddler with some flags that gets synced to the server. The serverside code picks up the tiddler (by searching for the same flags), and then performs some operation, putting the results back in the same tiddler.

Alternatively, you can use websockets or serversentevents to communicate directly from code running in the browser to code running in the server.

Best wishes

Jeremy.

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.
To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywikidev.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/72c2fe2f-b746-4a2b-9d50-224b6e18ff4f%40googlegroups.com.

[bimlas]

The sync mechanism seems to be usable, I go around the topic, thank you for the idea!

PS.: According to your comment, if you would like GitLab saver to 5.1.20, please merge the request, otherwise mark it with "post-v5.1.20" just to give me some feedback. :) 

[Brian Theado]

Bimlas,

If the npm modules you want are compatible with the browser, then you can bundle them into a tiddler and load them directly in the browser. It can be tricky to get it working.  I've had the most success using browserify --standalone to get it bundled in a way that it will work with TW's require. If the library provides a umd bundle, I think that usually that works as well. I've also used webpack before to bundle the code.

But if the module needs to execute functionality only available in node, then Jeremy's suggestion is the way to go.

Brian

On Thu, Jul 11, 2019 at 6:55 AM bimlas <bimba....@gmail.com> wrote:

The sync mechanism seems to be usable, I go around the topic, thank you for the idea!

PS.: According to your comment, if you would like GitLab saver to 5.1.20, please merge the request, otherwise mark it with "post-v5.1.20" just to give me some feedback. :) 

--

You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.
To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywikidev.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/0a206645-1cfa-4de9-9563-f4b91dc43bf7%40googlegroups.com.

[bimlas]

Jeremy,

... have the browser create a tiddler with some flags that gets synced to the server. The serverside code picks up the tiddler (by searching for the same flags), and then performs some operation, putting the results back in the same tiddler.

 So far it works so that the server receives the query and executes the change, but how can I "upload" the changes? On the file system I can see that the tiddler has changed, but how can I update it in the browser? There I still see the query, not the response.

Example code:

* Create a new directory and cd into it

* `npm init && npm install --save tiddlywiki`

* Create `index.js` with this content:

// Start TiddlyWiki
var $tw = require('tiddlywiki').TiddlyWiki();
$tw.boot.argv = ['gui', '--listen'];
$tw.boot.boot(function() {});

$tw.wiki.addEventListener("change", function (changedTiddlers) {
  if (!"$:/communicate-with-server" in changedTiddlers) return;
  var tiddler = $tw.wiki.getTiddler("$:/communicate-with-server");
  if (!tiddler || (tiddler.fields["sender"] === "server")) return;
  console.log("Query got from browser");

  var query = tiddler.fields.text;
  var response = $tw.wiki.getModificationFields();
  response["sender"] = "server";
 
  // Parse query...
  console.log("Original query:" + query);
  response.text = query.toUpperCase();
 
  $tw.wiki.addTiddler(new $tw.Tiddler(tiddler,response));
});

* `node index.js`

* Create "$:/communicate-with-server" and write something in it

When you save it, it notifies you in the terminal (where Node is running) that it has received a request from the browser and executes the change (capitalizing the text), saves the file, but the browser knows nothing about it: it still does shows the version in memory. (If you want to send a request again, make sure the "sender" field is cleared)

How do I get the browser to read the file system change? And how do I tell the browser when the change was made (so I don't have to use a timer after I send the request)?

[bimlas]

I think I found the solution: I have to use this (after the tiddler is modified) in the browser (by an ActionWidget for example):

  $tw.utils.nextTick(function() {
    $tw.syncer.enqueueSyncTask({type: "load", title: "$:/communicate-with-server"})
  });

[Jeremy Ruston]

Hi Bimlas

To make the browser to pick up changes from the server then you can trigger polling with the tm-server-refresh message. In my own experiments I often just change the default polling interval from 60s down to 3-5s (via $:/config/SyncPollingInterval).

You may also be interested in this pending PR which attempts to address some of these issues by introducing a new framework for invoking external tasks on the server:

https://github.com/Jermolene/TiddlyWiki5/pull/3461

Best wishes

Jeremy

--

Jeremy Ruston

jer...@jermolene.com

https://jermolene.com

On 27 Aug 2019, at 13:32, bimlas <bimba....@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/030d5158-3cac-472c-8044-5c48284b40f1%40googlegroups.com.

[bimlas]

Jeremy,

To make the browser to pick up changes from the server then you can trigger polling with the tm-server-refresh message. In my own experiments I often just change the default polling interval from 60s down to 3-5s (via $:/config/SyncPollingInterval).

 

I'm afraid something is wrong: I created a new wiki ("tiddlywiki --init server"), made a tiddler, changed the text on the file system and issued the $tw.rootWidget.dispatchEvent({type: "tm-server-refresh"}) command in the browser console, but the tiddler has not been updated.

I also set the configuration value as you wrote it, restarted the Node server, and saw in the console that it searches for changes at specified intervals ("syncer-browser-tiddlyweb: retrieving skinny tiddler list"), but the tiddler did not take over the file system changes made.

Called $tw.syncer.syncFromServer() directly, same results.

The only solution that really updated it is the $tw.syncer.enqueueSyncTask({type: "load", title: "New Tiddler"}) command issued in the browser.

$ tiddlywiki --version

5.1.20

$ node --version

v8.10.0

Is it possible that Node 10 is needed for proper operation?

You may also be interested in this pending PR which attempts to address some of these issues by introducing a new framework for invoking external tasks on the server:

https://github.com/Jermolene/TiddlyWiki5/pull/3461

It looks promising, but I'm not digging into it yet if I'm so close to the goal. In fact, maybe this may be closer to my needs if I really understand what it is: https://github.com/Jermolene/TiddlyWiki5/pull/4123 

PS.: $:/config/SyncPollingInterval could be documented at https://tiddlywiki.com/#Hidden%20Settings if you agree.

[bimlas]

I also tried Node 12.9.1 installed through NVM, which is the result.

Need to create a skinny tiddler and it will be synced? As far as I know it doesn't work with text files:

$ mkdir files
$ echo "Some text" > ./files/foo.txt
$ tiddlywiki --listen

• Check that http://localhost:8080/files/foo.txt works by opening it in browser
• Create a new tiddler with _canonical_uri: ./files/foo.txt
• It says "Trying to load external content from ./files/foo.tid", file is not found
• Set type: text/plain
• The text is empty, so the file is not found

To sum up my thoughts:

I want to solve the communication between browser and server by modifying a tiddler, which I see on both sides, and if I change it, the other party updates itself. Server side modification works, only browser side update is missing. This requires the syncadaptor to load the tiddlers from the file system at intervals or for a specific event. This does not seem to work, it updates only skinny tiddlers, but they do not work with text files.

At the moment, I see the only workable solution is $tw.syncer.enqueueSyncTask({type: "load", title: "New Tiddler"}).

Am I right? If not, what do I do wrong?

[Jeremy Ruston]

Hi Bimlas

To make the browser to pick up changes from the server then you can trigger polling with the tm-server-refresh message. In my own experiments I often just change the default polling interval from 60s down to 3-5s (via $:/config/SyncPollingInterval).

 

I'm afraid something is wrong: I created a new wiki ("tiddlywiki --init server"), made a tiddler, changed the text on the file system and issued the $tw.rootWidget.dispatchEvent({type: "tm-server-refresh"}) command in the browser console, but the tiddler has not been updated.

TW5 doesn't support external changes to the filesystem. You should use the HTTP API to create/update tiddlers.

PS.: $:/config/SyncPollingInterval could be documented at https://tiddlywiki.com/#Hidden%20Settings if you agree.

Yes, it should be documented there.

Best wishes

Jeremy

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/0cd9d225-d507-4b24-abdd-98db0c0e9e4c%40googlegroups.com.

[bimlas]

Jeremy

I found why my files didn't load: the code above (https://groups.google.com/d/msg/tiddlywikidev/wKqzwaMfBJw/YwJVbBjGBAAJ) works, but only if I am not using a system tiddler, as they are not loaded by Syncadapter.

The description of Syncadaptor is ambiguous (https://tiddlywiki.com/dev/#Syncadaptor): "provides functionality ... to load, save and delete single tiddlers ... honours a special system tiddler $:/config/SyncFilter containing a filter string. Tiddlers matching this filter string are not synced with a syncadapter." The reality is that it uses the $:/config/SyncFilter filter when saving from browser to server, but ignores all system tiddlers when syncing from server to browser (by using /recipes/default/tiddlers.json).

(Not a real UML, but I hope it helps understanding.)

As far as I can track back in Git history, filtering system tiddlers is intentional (and logical), but not mentioned in the documentation.

[bimlas]

Opened https://github.com/Jermolene/TiddlyWiki5/pull/4211 and https://github.com/Jermolene/TiddlyWiki5/pull/4212

[bimlas]

 

Yes, it should be documented there.

Opened https://github.com/Jermolene/TiddlyWiki5/pull/4213 

[okido]

Hi Bimlas,

I use TWc in a straight forward nw.js install.

You can include every npm packet as simple as using something like:

let UglifyJS = require("uglify-es")

in your scripts or inline scripts.

Saving requires a plugin that can be found here: https://github.com/nwOkido/nwTWcSaver.

Access to the filesystem runs with:

 let fs = require("fs")

Have a nice day, Okido