“Seamless interaction with NPM” not so seamless

[Krzysztof Jurewicz]

Hello,

I want to write a simple proof-of-concept (which may eventually evolve into something more serious) GUI wallet for a cryptocurrency that I’ve been writing. As desktop libraries/languages are either not very functional or not very portable, ClojureScript plus local storage in browser as a database looks like a good choice. To make it work, I need support for Ed25519 (a public-key signature system).

Fortunately, there are implementations of Ed25519 compiled to JavaScript. js-nacl is a “pure-Javascript High-level API to Emscripten-compiled libsodium routines”, available also as a NPM package, and post at https://clojurescript.org/news/2017-07-12-clojurescript-is-not-an-island-integrating-node-modules advertises “Seamless interaction with NPM dependencies”. I tried to use it, roughly in the following way:

⒈ lein new reagent-frontend ercoin-wallet
⒉ Added :npm-deps {:js-nacl "1.2.2"} to :cljsbuild ⇨ :builds ⇨ app ⇨ :compiler in project.clj.
⒊ Added [js-nacl :as nacl] to :require in core.cljs

However :npm-deps seemed to be ignored. Fixed this by downgrading ClojureScript from 1.9.908 to 1.9.671 (BTW, these versions are not tagged in the Git repository), but then there are warnings:

WARNING: JSC_JS_MODULE_LOAD_WARNING. Failed to load module […] at […]

, where modules are “fs”, “path” and “crypto”.

spinningtopsofdo on IRC said that “From a quick skim for js-nacl it looks like it's using emscripten and doing some unique module loading that Google Closure isn't aware of (https://github.com/tonyg/js-nacl/blob/master/lib/nacl_factory.js#L30-L39).” and “I think it's more the way js-nacl is using ASM / Emscripten and creating JavaScript Modules. :npm-deps covers the common JavaScript module patterns (e.g. CommonJs, Node, UMD) but there is still many edge cases out there”.

I’ve tried also tweetnacl-js, but similarly there is a warning:

“WARNING: JSC_JS_MODULE_LOAD_WARNING. Failed to load module "crypto" at […]”.

What is the status of NPM interoperability then? Is it supposed to work as plug & play or does it require hacky knowledge in some cases?

[António Monteiro]

You also need to pass the `:install-deps true` compiler option from 1.9.854 onwards.

Some libraries are not consumable by Google Closure if they use constructs like dynamic exporting and such.

In any case, it looks like your problem is even another one: you're trying to use a Node.js library in the browser. Obviously, `fs`, `path` and `crypto` (which are Node.js builtins) are not going to be available in any browser.

[Thomas Heller]

“Seamless interaction with NPM” not so seamless

This has been exactly my experience as well which is why I wrote an entirely new implementation for JS dependencies.

The detailed article can be found here

https://code.thheller.com/blog/shadow-cljs/2017/09/15/js-dependencies-going-forward.html

In short the implementation is much less aggressive and sacrifices some potential byte savings for greater compatibility with the JS ecosystem.

I added js-nacl to my demo here:

https://github.com/thheller/shadow-cljs-examples/blob/bb29828917080b143030ff557590c7c94b806ed7/browser/src/demo/app.cljs#L25-L29

The relevant bits from the config are

https://github.com/thheller/shadow-cljs-examples/blob/bb29828917080b143030ff557590c7c94b806ed7/browser/shadow-cljs.edn#L26-L28

which basically just tell the compiler to use the window.crypto property when require("crypto") is used.

I don't know if everything works since I know nothing about js-nacl. The Basic example from their README seems to work.

This is still in pre-release mode but I could some more testers with "troublesome" JS dependencies.

[Hendrik Poernama]

I want to add my experience with this when trying to fit react-apollo into a cljs project. Maybe it is useful to others going in this direction. Overall, I find it works better than I expected for an alpha feature, though not yet seamless.

Unfortunately I did not keep good enough notes while hacking through the project, so some bits below are from my memory. Also, I'm no JS expert.

I started a project with latest CLJS, reagent 0.8 (cljsjs deps excluded), react 15.6.2 from npm, apollo-client, and react-apollo both from npm.

This is in order of discovery:

1. NPM module that does not export using ES6/CommonJS did not generate a namespace.

The first thing that broke was 'whatwg-fetch' polyfill. Because it did not 'export' anything, module_deps did not generate any 'provide'.

But somewhere in apollo-client, there is a 'require', so dependency resolution failed. I forked whatwg-fetch and added these lines just to fool module_deps:

           const whatwg_fetch = 1; 

           export { whatwg_fetch };

 

2.  Closure compiler assumes a file is either CommonJS or ES6. If a file has both types of import/export, it will ignore the other.

react-apollo contains source with both 'require' and 'import' in the same file. Closure ignored one, did not rewrite module names into

the 'provide' string ("full$path$to$module"), leading to undefined vars. I forked the library and manually converted the imports to ES6. 

 

3. Closure compiler does not support circular dependency. https://github.com/google/closure-compiler/issues/1883

apollo-client contains some circular deps. Forked the library and manually merged the files... New version of apollo-client

was released with more circular deps, I will not be upgrading for a while. 

 

At this point, I got myself a working although fragile dev environment. I could (:require [apollo-client]) like a cljs library, and interop was seamless.

Then it was time for advanced optimization....

4. Libraries not written with closure compiler in mind..

Well.. mangled names everywhere. These libraries have some dynamic codes. Not much to say here, just use the extern shotgun until

no more errors. I'm not even sure if something like tsickle (typescript to closure) can even help here.

5. Passing complex objects back and forth between js and cljs

The choice here is either being really careful when using js->clj, clj->js, and :keywordize-keys or extern shotgun. I did some experiments

to understand how to do this safely, but got frustrated and used the extern shotgun.

After all that, I have a working app at about 700k uncompressed compared to 3MB+ simple optimization. 

I don't see an easy way to run arbitrary node module into closure's advanced optimization yet.. Maybe a pre-processor can do the above tasks

automatically?