A new take on ClojureScript and npm

[Thomas Heller]

Hello,

I wrote a few lines of code that should hopefully make it easier to get started with CLJS for people coming from the JS world.

I created a demo here:
https://github.com/thheller/npm-module-example

The gist is that you gain: var foo = require("shadow-npm/demo.foo");

Where "shadow-npm" is a pseudo node-module generated by the tool and "demo.foo" is just the CLJS namespace you want. The require will return the NS so foo.bar() just works. The CLJS sources can use any npm module and local sources as well.

See: https://github.com/thheller/npm-module-example/blob/master/src/main/demo/foo.cljs

Given that the generated code is NPM compliant it should work with any npm JS tools. The demo has a basic setup for webpack, no special configuration for CLJS is needed.

I barely know anything about any NPM tools so I might be doing something completely wrong, please create issues on the shadow-devtools [1] project if you run into issues. I only tested node itself and webpack.

The sacrifice is that you can't use the Closure Compiler but you gain everything the npm world has to offer with hopefully seemless interop. Do not underestimate the impact of this sacrifice though, I cannot recommend using this for production builds.

The example still uses leiningen by I may drop that and create something standalone.

Need to work on a lot of things still but the proof of concept works, maybe it is useful to someone.

Cheers,
/thomas

[1] https://github.com/thheller/shadow-devtools

PS: This is not related to :npm-deps at all.

[Thomas Heller]

Just added another example that shows how you'd use CLJS in a create-react-app setup.

https://github.com/thheller/npm-module-example/tree/master/examples/create-react-app

[Jiyin Yiyong]

Is that project compiling with something like lein-cljs(in shadow-devtools) inside and then emit JavaScript that can be processed by Webpack?

[Thomas Heller]

Yes, currently it is just shadow-devtools via lein. Could also run it through boot, no problem.

I want to package it all up so you can "npm install shadow-npm".

Need to figure out how to best do that first though. Should probably also chose a better name.

[Shaun LeBron]

> The example still uses leiningen by I may drop that and create something standalone.

I'd like to show you around my "cljs/tool" some time for your standalone thing. I'm a year into building it to wrap a good UX around cljs on npm things as they become available, and I think this experimental feature could be delivered pretty easily through it.

[Thomas Heller]

On Monday, May 15, 2017 at 8:30:26 AM UTC+2, Shaun LeBron wrote:
> > The example still uses leiningen by I may drop that and create something standalone.
>
> I'd like to show you around my "cljs/tool" some time for your standalone thing. I'm a year into building it to wrap a good UX around cljs on npm things as they become available, and I think this experimental feature could be delivered pretty easily through it.

Hey Shaun,

I've been following your work on github.com/cljs pretty much since the beginning. It would safe me a ton of work if you want to cover the npm package side of things.

My plan so far was:

- create an uberjar for shadow-devtools (which would contain clojure, clojurescript, core.async and could add more)
- create a npm package with a shadow-cljs bin that uses java or node-jre
- shadow-cljs cli opts --watch --repl etc
- zero conf CLJS compilation mode that compiles to the shadow-npm module to quickstart things

When the user is ready to go into full-blown CLJS first mode it should be as simple as creating a build config and then running the full featured mode which includes Google Closure.

The hard part for that transition would be that the require("shadow-npm/cljs.core") would no longer work since it is pretty much impossible to make that work with :advanced.

The results of "webpack -p" on the shadow-npm thing are ok but nowhere close to what Closure would give you. I guess it is good enough if you are JS first with a bit of CLJS sprinkled on top.

I have no attachments to the shadow-cljs or shadow-npm names, I just used them because there already are so many cljs+npm, clojurescript+npm variations out there that I just didn't want to conflict with.

If you want to use this for your cljs/tool I'd be happy to make any changes you'd need. Although your current mix of lumo and figwheel really doesn't translate too well to the "create a pseudo node module" (ie. "./node_modules/shadow-npm/cljs.core.js" to enable require("shadow-npm/cljs.core"). Also the configs for shadow-devtools are substantially different from lumo/figwheel/cljsbuild, hopefully simpler though.

Currently it is all driven by this little CLI ns:
https://github.com/thheller/shadow-devtools/blob/master/src/main/shadow/npm/cli.clj

I just use it through leiningen since I didn't want to work through the dependency management, but you already have that covered. lein is not needed at all beyond that.

Since I'm not a JS dev and have been purely CLJS for 3+ years I really don't feel qualified to make decisions on what would be ideal for a JS dev. Current solutions all felt unnecessary complex though.

Happy to consider all feedback.

[Thomas Heller]

I just did a quick test for React Native.

https://github.com/thheller/npm-module-example/tree/master/examples/react-native-example

Since it is the first time I started react-native I really have no idea what is going. It works but is extremely slow to start. I don't know why, just wanted to see if it can work at all.

If someone has suggestions on how to make it start faster let me know, also need to look into their HMR which doesn't seem to work.

[Thomas Heller]

Also removed all traces of lein, the examples now just use an uberjar of shadow-devtools.

Certainly not the final solution I have in mind but getting closer.

[Rohit Aggarwal]

On Monday, 15 May 2017 12:48:38 UTC+1, Thomas Heller wrote:
> Also removed all traces of lein, the examples now just use an uberjar of shadow-devtools.
>
> Certainly not the final solution I have in mind but getting closer.

I think it should be possible to use closure-compiler-js [1] with this project to do DCE if needed.

[1] https://github.com/google/closure-compiler-js

[Thomas Heller]

>
> I think it should be possible to use closure-compiler-js [1] with this project to do DCE if needed.
>

In a way you can and you can't. The same way :npm-deps has issues.

The Closure Compiler (JS or JVM version, doesn't matter) is not yet capable of compiling everything in the npm world. Some things just don't work.

So instead of making something that "maybe" works I opted to drop the Closure Compiler entirely.

The intent here is to enable JS devs to start using CLJS seamlessly without giving up their tools. They don't have the Closure compiler to begin with so they won't miss it.

Enabling the Closure Compiler is my top priority though. I'm definitely keeping my eye on this and will provide a proper solution if it ever becomes solid.

[Thomas Heller]

Too tired to make a proper announcement but I just pushed the first preview version of the shadow-cljs npm package.

https://www.npmjs.com/package/shadow-cljs

npm install -g shadow-cljs
shadow-cljs --once
node
var x = require("shadow-cljs/cljs.core");
x.enable_console_print_BANG_(); // oops
x.prn(x.assoc(null, "foo", "bar"));

More tomorrow ...

[Thomas Heller]

I wrote a short introduction here:
https://github.com/thheller/shadow-devtools/wiki/ClojureScript-for-JS-Devs

Looking for any kind of feedback.

[Shaun LeBron]

The quick start is great! I have three questions:

1. Should we call this "cljs-require"? With the understanding that it is a bit like a portal to cljs's ":require" form:

require('cljs-require/foo.core')

2. Is there a way to set the ClojureScript version, or is it tied to shadow's version?

3. Thinking back on Jiyin's question about using cljs in webpack—have you put any thought into the possibility of this workflow as a webpack loader? Maybe that's a next step?

Awesome job turning this webpack discussion into a real tool we can explore! 🎉

[Thomas Heller]

> 1. Should we call this "cljs-require"? With the understanding that it is a bit like a portal to cljs's ":require" form:
>
> require('cljs-require/foo.core')

The package name must be something you add to your package.json, otherwise yarn will nuke the directory every time you run "yarn install". "npm" isn't as strict but still. Using the name of the tool seemed like the obvious choice. We can however create an empty cljs-require npm package that just serves as a placeholder.

I'm totally for this if other tools want to adopt this idiom. I definitely do not want this to be something you can only do with shadow-cljs.

> 2. Is there a way to set the ClojureScript version, or is it tied to shadow's version?

You can specify dependencies in package.json, basically like you would in lein.

"shadow-cljs: {
"version":"1.0.20170516",
"dependencies": [
["org.clojure/clojurescript", "1.9.542"]
]
}

where "version" is the version of shadow-devtools, any other dependencies are resolved using the fantastic pomegrenate lib. (thanks btw for writing the dep-resolver, I didn't think it would be this simple to do the dependency loading).

I'm not sure how the conflict resolution works but you should be able to use any cljs version you want. The only things written in stone are the deps of the shadow-cljs uberjar, can't change the clojure version after the fact.

See:
https://github.com/thheller/shadow-devtools/blob/master/npm-package/project.clj

>
> 3. Thinking back on Jiyin's question about using cljs in webpack—have you put any thought into the possibility of this workflow as a webpack loader? Maybe that's a next step?

Don't need a loader. It could be a plugin that just calls "shadow-cljs --once"
before proceeding with the rest of the JS compilation.

I actually started by writing a plugin but to be honest I do not think it is super useful. You can achieve the same via npm scripts & co. You'll need the CLI tool for more advanced tasks (eg. REPL) anyways.

I want this to be generally useful, not just webpack. No specific webpack configuration is needed, you just somehow need to compile before calling webpack. "webpack -w" and "shadow-cljs --dev" side-by-side actually just works.

> Awesome job turning this webpack discussion into a real tool we can explore! 🎉

Thanks.

One issue that needs to be addressed is the widespread use of CLJSJS packages in popular libs like reagent&co. Can't have cljsjs.react react.js if you are supposed to use npm react.

I can easily skip over all foreign-libs but then (:require [cljsjs.react]) wouldn't have the effect of ensuring that "React" is available on global. Not yet sure how to best handle that yet.

Still have a few ideas I want to explore, haven't totally given up on the Closure Compiler yet but it making life harder than it should be.

[jiyinyiyong]

Fascinating.. it's using `node_modules/shadow-cljs` to do the trick. I can see ClojureScript files are compiled into CommonJS compatible form...

Do you think it's possible that we put all js built from `cjs.core`(also with `goog`) into a single npm package. And then for each ClojureScript file, it is mapped to a JavaScript file by `shadow-cljs` tools, which use `require("./cljs.core.js");` to access it's cljs dependencies(like in your demo), and use `require('package_name/foo/bar')` to require sibling files. For other packages, maybe we can do the some trick, like precompile them with `shadow-cljs` tools and then release them to npm. Seeing from a CoffeeScript user, I think somehow in this way ClojureScript can also fit into npm's ecosystem.

Here's an example, we may create a new project in such file structure, where:

`src/` is for source code,

`target` is for compiled code,

`package.json` is copied from `assets/package.json` to `target/package.json`,

```

.

├── assets

│   └── package.json

├── project.clj

├── src

│   └── demo

│       └── main.cljs

└── target

    ├── demo

    │   └── main.js

    └── package.json

```

if it's in this way, `target/` can be released on npm as a module and others surely may use `require('package_name/demo/main.js')` to load the package built from ClojureScript.

--
Note that posts from new members are moderated - please be patient with your first post.
---
You received this message because you are subscribed to a topic in the Google Groups "ClojureScript" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojurescript/AGXku7Ous0Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojurescrip...@googlegroups.com.
To post to this group, send email to clojur...@googlegroups.com.
Visit this group at https://groups.google.com/group/clojurescript.

[jiyinyiyong]

Some typo. Since the package name in my previous post is demo, it is actually written like `require('demo/main.js')`, which means the structure of the folder should be(no `target/demo/`):

```

.

├── assets

│   └── package.json

├── project.clj

├── src

│   └── demo

│       └── main.cljs

└── target

    ├── main.js

    └── package.json

```

[Thomas Heller]

If I understand correctly the short answer is: no.

I chose this setup specifically to avoid anyone ever putting compiled CLJS code in npm packages. The problem with this would be that everyone would bring their own cljs.core version again.

That is an issue currently when you write a npm package in CLJS and publish to npm. Each package contains its own conflicting version of cljs.core (and maybe others) even if you compile with :advanced.

All CLJS sources must be compiled with the same CLJS compiler, you cannot use a precompiled "some-package/cljs.core.js" as there is no way to guarantee that everyone was using the same version and so on. The way npm resolves version conflicts is by creating nested node_modules folders.

./node_modules/A
./node_modules/shadow-cljs (v1.0)
./node_modules/B
./node_modules/B/node_modules/shadow-cljs (v1.1)
and so on.

There is no way to ensure that B uses the same shadow-cljs version as A. By not having any compiled CLJS code in any npm package we can sort of get away with it though.

See:
https://docs.npmjs.com/how-npm-works/npm3-nondet
https://docs.npmjs.com/how-npm-works/npm3-dupe

yarn fixes some of these issues but not all.

I do however think that it would be relatively straightforward to put uncompiled CLJS code into npm packages and have them picked up by shadow-cljs.

So if you have

./fancy-package/src-cljs/fancy/foo.cljs

You could just require("shadow-cljs/fancy.foo") and (:require [fancy.foo]) to use that. While you can still require("fancy-package") normally if there also was JS code but never require("fancy-package/fancy.foo").

It's a bit confusing but I repeat: there must never be compiled CLJS code in a npm package, only uncompiled .cljs files just like we do in CLJS .jar files published to clojars (or maven). For now I do recommend using the normal CLJS methods to publish CLJS libs. Publishing to npm would be something only currently supported by shadow-cljs and that would not be a good idea (for now). Things need to be ironed out first.


I cannot explain this very well, as I am still very confused about it myself.

Sorry if you wanted something completely different.

[Thomas Heller]

On Tuesday, May 16, 2017 at 8:27:44 PM UTC+2, Thomas Heller wrote:
> If I understand correctly the short answer is: no.
>

I should add: You can use shadow-cljs in your coffeescript project if you just want to use cljs.core or some other CLJS package. You do not need to write CLJS yourself to make use of shadow-cljs.

var x = require("shadow-cljs/cljs.core");

x.assoc(null, "foo", "bar");

is basically the same as

var x = require("mori")
x.assoc(null, "foo", "bar");

But you would need to run shadow-cljs in your project.

[Thomas Heller]

I just added another demo using react-native+haul.

https://github.com/thheller/npm-module-example/tree/master/examples/AwesomeProject
https://github.com/callstack-io/haul

The default react-native packager tries to run cljs.core through a large number of babel transforms which we don't need. The code works fine but the development experience is extremely terrible with about 4min of packager bundle time.

haul seems to be a drop-in replacement for this packager and works without modification since it just uses webpack.

I don't do any react-native development but the proof of concept works well.

I'll work on adding live-reloading and a REPL for everything tomorrow or so.

[jiyinyiyong]

I don't like the way of writing `x.assoc(null, 'foo', 'bar')` by myself. Also I don't people would accept that.

I see the problem now. ClojureScript compilers may have tricky behaviors that make it different from CoffeeScript's compiling processes.

Putting `x.cljs` files in npm is fine, it's just slower. To me compiling ClojureScript is always slower, I already accepted that.

I think here are may main concerns:

* in js projects, we regard code in `node_modules` are modules instead of source code. For web projects using Webpack, I would prefer using a 'compiled/' folder to hold the code and then config `resolve.alias` to make sure I import the code with `require('cljs/foo.bar.core')`. https://webpack.js.org/configuration/resolve/

*  For nodejs projects, somehow I can accept the solution that we use `require('../../compiled/foo.bar.core')`. But I guess it may bring problems, which makes your `shadow-cljs` solution a better choice...

* does shadow-devtools compiled cljs files incrementally? If it does, Webpack may use it to hot replace module.

[Thomas Heller]

On Wednesday, May 17, 2017 at 3:51:14 AM UTC+2, Jiyin Yiyong wrote:
> I don't like the way of writing `x.assoc(null, 'foo', 'bar')` by myself. Also I don't people would accept that.
>

I wouldn't as well, but that was an example to show that you can use CLJS code directly without modifying it in any way.

import { assoc } from "shadow-cljs/cljs.core";
assoc(null, "foo", "bar");

also works. assoc is an ugly example because of the leading null but it is the shortest form to replace the CLJS {} empty map literal. {} would mean empty JS object which doesn't work with assoc.

> I see the problem now. ClojureScript compilers may have tricky behaviors that make it different from CoffeeScript's compiling processes.
>

The compiler is pretty straightforward actually, the tricky parts are the version conflicts. If there was a way to ensure that everything always used the exact some compiler configuration/version this would be possible, but there is not AFAICT.

> Putting `x.cljs` files in npm is fine, it's just slower. To me compiling ClojureScript is always slower, I already accepted that.
>

[:script] Build completed. (23 files, 1 compiled, 0 warnings, 0.12s)

I think 0.12s is pretty fast. That is running in --dev mode recompiling one changed file with hot-reload enabled.

Of course it is slower if you always run "shadow-cljs --once" which includes starting the JVM each time. "shadow-cljs --dev" will be substantially faster when re-compiling.

>
> I think here are may main concerns:
>
>
> * in js projects, we regard code in `node_modules` are modules instead of source code. For web projects using Webpack, I would prefer using a 'compiled/' folder to hold the code and then config `resolve.alias` to make sure I import the code with `require('cljs/foo.bar.core')`. https://webpack.js.org/configuration/resolve/
>

Not sure what you mean. Where is the difference in putting code into node_modules or compiled then using webpack? The behavior is identical, the node_modules version just doesn't need the resolve.alias? Both versions require that you have your code compiled before running webpack, which directory you load them from should be automatic?

> *  For nodejs projects, somehow I can accept the solution that we use `require('../../compiled/foo.bar.core')`. But I guess it may bring problems, which makes your `shadow-cljs` solution a better choice...
>

Yes, I hate relative paths. Actually the output folder where things end up is a config option. So if you really wanted to have your compiled folder that can be done. I just don't see an upside to it.

>
> * does shadow-devtools compiled cljs files incrementally? If it does, Webpack may use it to hot replace module.
>

Yes, incrementally. Don't know anything about HMR but shadow-devtools already has hot-reloading (ala figwheel) built-in. I just didn't enable it yet because other things were more important.

Can't tell if Webpack HMR would work since I have never used it.

Thanks for the feedback, I hope I lessened your concerns.

Keep it coming.

Remember: this is evolving as we discuss it, I have no idea what JS devs want so without feedback I'll be building what I want. That may or may not align.

[jiyinyiyong]

> Where is the difference in putting code into node_modules or compiled then using webpack?

I'm not sure either. But here's what I see:

* in `node_modules/` there are always tens or even thousands of folders, which I'm never happy to open it.

* for hot module replacement issues, sometimes we want to ignore the whole `node_modules/` because of the amount of the files, https://webpack.js.org/configuration/watch/#watchoptions-ignored . I don't think all of us will ignore that folder, but we might do and we consider code in `node_modules/` remaining the same.

As a result, using `resolve.alias` appears more acceptable to me.

[Thomas Heller]

> * in `node_modules/` there are always tens or even thousands of folders, which I'm never happy to open it.
>

That is indeed scary, but you'd never need to open it. Same way you never look at any other node_modules folder probably.

>
> * for hot module replacement issues, sometimes we want to ignore the whole `node_modules/` because of the amount of the files, https://webpack.js.org/configuration/watch/#watchoptions-ignored . I don't think all of us will ignore that folder, but we might do and we consider code in `node_modules/` remaining the same.
>

Given that you can specify a regexp you can just use

ignored: /node_modules\/(?!shadow-cljs)/

where it still ignores any node_modules folder except shadow-cljs. Watching this folder however is currently not the best idea because I was lazy and didn't optimize for it yet. That means that the compiler will emit all files on compile, not just the files that were actually recompiled/changed.

I will fix that eventually.
https://github.com/thheller/shadow-cljs/issues/9

[jiyinyiyong]

Just realized that ClojureScript is different from CoffeeScript because of the macro system. If a macro changed, it may cause changes of multiple files. So that it's hard to just detect which file changed and compiled it alone. The only window left is to read and compare file content before writing, if we are trying to get rid of the unnecessary change events...

[Thomas Heller]

The shadow-cljs compiler already does the correct thing when it comes to macros and recompiles all affected CLJS files.

It does however know nothing about JS files that may be affected as well so it would not reload those. I don't know if the webpack HMR will reload dependent files?

Does it reload ./index.js if that does require("shadow.cljs/some.foo") and some.foo was recompiled?

I have never even seen a HMR config, can you share one so I can try a few things?

[jiyinyiyong]

Yes Webpack will replace the dependent files. Webpack made a concept called "module tree" based on dependency tree. File changes will propagate along the tree, and hit the root at last. But if a change propagates to root, the whole page reloads. That's not what we wanted.

I don't have an example, but you may follow the docs: https://webpack.js.org/guides/hmr-react/#app-code

roughly 3 steps(I did that before but not sure about every detail)

1) add configs in `webpack.config.js`

  entry: [
    'react-hot-loader/patch',
    // activate HMR for React

    'webpack-dev-server/client?http://localhost:8080', <---- add this
    // bundle the client for webpack-dev-server
    // and connect to the provided endpoint

    'webpack/hot/only-dev-server', <---- add this
    // bundle the client for hot reloading
    // only- means to only hot reload for successful updates

    './index.js'
    // the entry point of our app
  ],

  plugins: [
    new webpack.HotModuleReplacementPlugin(),  <---- add this
    // enable HMR globally

    new webpack.NamedModulesPlugin(),
    // prints more readable module names in the browser console on HMR updates
  ],

2) add `module.hot.accept('./demo', cb)` to handle the changes from './demo', so it will not reload the page:

// Hot Module Replacement API
if (module.hot) {
  module.hot.accept('./components/App', () => {
    render(App)
  });
}

3) replace `webpack` to `webpack-dev-server` to start bunding:

webpack-dev-server

Webpack would inject code that starts a WebSocket and responds to file changes. Since it contains many steps, it's highly possible you run into problems setting up Webpack HMR.

[Thomas Heller]

I made a quick video showing how you'd use what we have talked about here with "create-react-app".

https://www.youtube.com/watch?v=BLDX5Twt2zk

[Kurt Harriger]

Thomas,

First thank you for starting this project!

I recently joined a team at Atlassian that uses ClojureScript to support collaborative editing.  I don't think I need to sell the community here on the advantages of ClojureScript, but I think packaging still needs work.

If you use ClojureScript for your entire UI you probably don't have any issues, but in larger organizations we need to "Play as a Team." It does not seem feasible or reasonable to expect that other teams switch to using the ClosureScript compiler and make their js libraries play nice with closure because the end result will be marginally better, especially given the work involved in ensuring they still work with advanced optimizations may be non-trivial.

The only practical alternative I'm currently aware of is to precompile the ClojureScript with optimizations and have them include this as a script tag in their page.  The disadvantages of this are many:

* separate download - not bundled with other js libraries

* not commonjs / cannot require / pollutes global namespace 

* install/upgrade/versioning process is different / change script tag rather than npm/yarn

* each precompiled ClosureScript library will include closure and cljs again

As we add support for multiple editors I want to split the library into multiple artifacts that can be released separately, however that last bullet point is undesirable.  I have had some success in adding a cljsbuild along side a webpack build, but CommonJS has solved a lot of pain points for JS devs and most modern js devs now expect an npm artifact. 

Although shadow-cljs doesn't solve having different install/version process it does appear to solve the other issues and is better than using a script tag.

I did have a general question though. I understand that if you compile multiple ClosureScript libraries independently each will end up with duplicate bits of closure and cljs core. Shadow-cljs, as I understand it, appears to solve this by first compiling all the ClosureScript code together and exposing it as a commonjs module. But thinking out-loud here...isn't sharing dependencies the problem npm is trying to solve? could we leverage that instead of reinvent it?  

What if google closure libraries and cljs.core were published as their own commonjs npm libraries - precompiled but not minified yet? Couldn't cljs libraries then just 'require' them without resulting in another copy or requiring the closurescirpt compiler (and jvm)?  In my ideal world, I think I would be able to specify :npm-module as the clojurescript compiler target and it would generate js that can be packaged in npm that 'requires' rather than bundles cljs and closure.  Minification can be done downstream with *all* the javascript, not just the cljs code. I also found a closure plugin for webpack https://www.npmjs.com/package/webpack-closure-compiler (I never used it though), so maybe we can still get just as good if not better results doing it later? 

I'm not sure what would be involved in rewriting cljs output to require "require" closure and cljs.core as their own libraries, but maybe the partially solved already with https://github.com/clojure/clojurescript/wiki/JavaScript-Module-Support-(Alpha)?

Thanks!

[Dustin Getz]

Closure is also doing dead code elimination which is only possible if you code in a restricted dialect of javascript, which cljs compiler does emit. Dead code elimination is important because, for example, the standard library and data structures are not implemented naively and bundled with the js vm, the standard lib has to be shipped on the wire and parsed.

[Kurt Harriger]

I understand it is important since only a small portion of the relatively large closure code base is used in any given app... but isn't this part of what the uglify plugin and/or webpack-closure-compiler does? Are you saying that dead code elimination will not work as well after the code has been commonjs-ified? Perhaps it will still work well enough?

On Sun, Jul 9, 2017 at 1:34 PM Dustin Getz <dusti...@gmail.com> wrote:

Closure is also doing dead code elimination which is only possible if you code in a restricted dialect of javascript, which cljs compiler does emit. Dead code elimination is important because, for example, the standard library and data structures are not implemented naively and bundled with the js vm, the standard lib has to be shipped on the wire and parsed.

--

[Thomas Heller]

Hey,

I understand what you want to achieve and I definitely want to make the whole thing easier but some issues don't have an easy solution.

TL;DR: It's complicated.

First we need to split compilation and bundling.

CLJS->JS compilation just takes one .cljs file and compiles it to .js. To do that it needs analyzer data of all CLJS dependencies. If you were to publish compiled CLJS to npm you'd need to also ship the analyzer data somehow. The other problem is that certain compiler options affect the generated JS which means that for every .cljs file there are many possible .js results. Another thing is that you also need to account for the compiler version since that may also affect the output or the analyzer data which then may affect other files.

The second part is bundling things together, which means reducing the number of individual files down to a more reasonable number. shadow-cljs enables JS tools to do that for us (eg. webpack, create-react-app or any other), since those tools use "require" to discover dependencies.

The best option for this however is to use the Closure Compiler but to get the whole benefit of it you need to optimize your whole program, which includes EVERY .js file. There is promising progress in Closure to support optimizing common js and ES6 code but it is not quite there yet and can't compile everything. It will be coming to shadow-cljs soon though, it is already in CLJS but I didn't hook it up yet since I wanted to solve some issues first. 

:npm-module tries to bridge the gap a little but as soon as you don't optimize your whole program you need externs to teach Closure about the things it can't see. Which means you can still use :advanced compilation for the CLJS/Closure parts but the rest of the code remains untouched. It also means that you need a second bundler (eg. webpack) to then combine the Closure output with the remaining unoptimized parts.

Given all these constraints I don't think it is feasible to publish compiled (but unbundled) CLJS code as a library to NPM. It would be relatively easy to consume uncompiled .cljs code from NPM packages but unless everybody agrees to do that it would just make things a whole lot more complicated for everyone. Maven is also a whole lot better than NPM IMHO, npm or yarn just have better UX (until something breaks).

:npm-deps is a feature in CLJS that lets you declare which npm packages your CLJS code requires to work. Technically this would need a complement in package.json to allow declaring which CLJS packages the JS code needs. I don't have a solution for that and I was and still am opposed to whole idea of :npm-deps. It does however solve the issue from the CLJS perspective, just not from the "Play as a Team" JS perspective.

I do want integration to be painless and I do think shadow-cljs already makes life easier for some. I am very interested to hear about any other pain points people may have with CLJS<->JS interop. I tested webpack, create-react-app and create-react-native-app. They are all work reasonably well but I haven't gone beyond basic examples.

I really don't have a good solution for the problem since part of the solution must come from the JS world and I have basically no idea what people are doing over there.

The whole problem basically boils down to which bundler you want to use.

I'm personally only considering the Closure Compiler for that and shadow-cljs is trying to make that as easy as possible with support for common JS / ES6 coming soon.

If you want to use webpack or so that also works reasonably well already but CLJS really is built with the Closure Compiler in mind so the end result of this won't be optimal.

If you want both that is always going to require more manual work for :externs and come with certain caveats.

I hope that made some sort of sense, I have been thinking about this a lot but haven't gotten very far.

/thomas

[Kurt Harriger]

On Sunday, July 9, 2017 at 8:06:00 PM UTC+10, Thomas Heller wrote:

Hey,

I understand what you want to achieve and I definitely want to make the whole thing easier but some issues don't have an easy solution.

TL;DR: It's complicated.

Yeah! :)

 

First we need to split compilation and bundling.

Yes.  Currently optimization is something we are doing when compiling clojurescript, but perhaps *ideally* optimization could be done after bundling when all the javascript is available.   However, I realize this may not actually work very well in practice since most modern js applications include dozens or hundreds of npm libraries that are not closure ready and would break in unexpected ways.  Perhaps this is why UglyifyJS is more common for webpack users?  UglyifyJS does dead code elimination as well I believe, is it as good? probably not? is it good enough? not sure.

 

CLJS->JS compilation just takes one .cljs file and compiles it to .js. To do that it needs analyzer data of all CLJS dependencies. If you were to publish compiled CLJS to npm you'd need to also ship the analyzer data somehow. The other problem is that certain compiler options affect the generated JS which means that for every .cljs file there are many possible .js results. Another thing is that you also need to account for the compiler version since that may also affect the output or the analyzer data which then may affect other files.

Do we still need analyzer data after its compiled? Can't we just use the javascript as a lib? I assume that google closure isn't written in cljs but cljs has no problem using or optimizing it... perhaps the packages are distributed as unoptimized google closure compatible js and the webpack plugin turns google closure modules into commonjs modules? not sure. just ideas. 

 

The second part is bundling things together, which means reducing the number of individual files down to a more reasonable number. shadow-cljs enables JS tools to do that for us (eg. webpack, create-react-app or any other), since those tools use "require" to discover dependencies.

The best option for this however is to use the Closure Compiler but to get the whole benefit of it you need to optimize your whole program, which includes EVERY .js file. There is promising progress in Closure to support optimizing common js and ES6 code but it is not quite there yet and can't compile everything. It will be coming to shadow-cljs soon though, it is already in CLJS but I didn't hook it up yet since I wanted to solve some issues first. 

Best optimized perhaps, but possibly not the best js dev ux.  Many js developers don't want to run java as part of their build process and might prefer to use uglifyjs even if the results are not as good.  

:npm-module tries to bridge the gap a little but as soon as you don't optimize your whole program you need externs to teach Closure about the things it can't see. Which means you can still use :advanced compilation for the CLJS/Closure parts but the rest of the code remains untouched. It also means that you need a second bundler (eg. webpack) to then combine the Closure output with the remaining unoptimized parts.

But many js devs do optimize and the best results would probably be to put the unoptimized closure together with the unoptimized javascript and then optimize everything together. But my guess is that wont work in practice.  UglifyJS I think is safer than closure, as good? probably not, good enough? not sure.  Your solution probably produces best optimization, but its worth asking if thats what devs care about the most?  Slower build process with more complexity vs a faster website.  There will be devs on both sides I think.  

 

Given all these constraints I don't think it is feasible to publish compiled (but unbundled) CLJS code as a library to NPM. It would be relatively easy to consume uncompiled .cljs code from NPM packages but unless everybody agrees to do that it would just make things a whole lot more complicated for everyone. Maven is also a whole lot better than NPM IMHO, npm or yarn just have better UX (until something breaks).

I don't necessarily agree or disagree on mvn but npm is what modernjs devs use and they are familiar with its quirks, maven has its own set of tools to troubleshoot dependency conflicts and js devs unfamiliar with java may have more difficulty configuring repo authentication and troubleshooting the errors.  I would rather deploy to npm, but if compile must happen down stream then I think your solution is a pretty good one. 

 

:npm-deps is a feature in CLJS that lets you declare which npm packages your CLJS code requires to work. Technically this would need a complement in package.json to allow declaring which CLJS packages the JS code needs. I don't have a solution for that and I was and still am opposed to whole idea of :npm-deps. It does however solve the issue from the CLJS perspective, just not from the "Play as a Team" JS perspective.

I do want integration to be painless and I do think shadow-cljs already makes life easier for some. I am very interested to hear about any other pain points people may have with CLJS<->JS interop. I tested webpack, create-react-app and create-react-native-app. They are all work reasonably well but I haven't gone beyond basic examples.

I really don't have a good solution for the problem since part of the solution must come from the JS world and I have basically no idea what people are doing over there.

The whole problem basically boils down to which bundler you want to use. 

True a more general tool is probably better than a webpack plugin.

 

I'm personally only considering the Closure Compiler for that and shadow-cljs is trying to make that as easy as possible with support for common JS / ES6 coming soon.

If you want to use webpack or so that also works reasonably well already but CLJS really is built with the Closure Compiler in mind so the end result of this won't be optimal.

If you want both that is always going to require more manual work for :externs and come with certain caveats.

I hope that made some sort of sense, I have been thinking about this a lot but haven't gotten very far.

Yeah. Hopefully my rambling is useful feedback, your goals are probably a bit different than mine and thats okay - its your tool :)  

[Thomas Heller]

Yes.  Currently optimization is something we are doing when compiling clojurescript, but perhaps *ideally* optimization could be done after bundling when all the javascript is available.   However, I realize this may not actually work very well in practice since most modern js applications include dozens or hundreds of npm libraries that are not closure ready and would break in unexpected ways.  Perhaps this is why UglyifyJS is more common for webpack users?  UglyifyJS does dead code elimination as well I believe, is it as good? probably not? is it good enough? not sure.

UglifyJS is not good enough in my opinion, not even close. The JS world has "accepted" that you need to make lots of many small files to make it acceptable. We don't since we have Closure.

 

Do we still need analyzer data after its compiled? Can't we just use the javascript as a lib? I assume that google closure isn't written in cljs but cljs has no problem using or optimizing it... perhaps the packages are distributed as unoptimized google closure compatible js and the webpack plugin turns google closure modules into commonjs modules? not sure. just ideas. 

You need to analyzer data if you want to compile any .cljs. It is not needed at runtime (which is where the closure library comes in).

 

Best optimized perhaps, but possibly not the best js dev ux.  Many js developers don't want to run java as part of their build process and might prefer to use uglifyjs even if the results are not as good.  

But many js devs do optimize and the best results would probably be to put the unoptimized closure together with the unoptimized javascript and then optimize everything together. But my guess is that wont work in practice.  UglifyJS I think is safer than closure, as good? probably not, good enough? not sure.  Your solution probably produces best optimization, but its worth asking if thats what devs care about the most?  Slower build process with more complexity vs a faster website.  There will be devs on both sides I think. 

This is one place where it is far more important to optimize for production users not the developer UX. I care very little about the fact that my production build may take a while longer if the resulting .js my users have to download is substantially smaller with much better runtime performance. The fear of Java is unreasonable and mostly FUD. If I run builds with create-react-app or create-react-native-app they are frequently much slower than a :advanced compiled CLJS build.

There are still lots of places where we can optimize the dev UX but production concerns will always come first.

JS integration is still a work in progress and will get a whole lot better soon. Stay tuned. ;)

/thomas