Standard way to provide externs for library authors

[Nikita Prokopov]

Hi!

Right now lein-cljsbuild allows for externs to be specified on user side. E.g. I’m building app with react-cljs, I add dependency, but there’s also a second step: I need to put :externs ["react/externs/react.js"] into _my_ project.clj.

This is tedious and easy to forget, leading to a lot of confusion (I developed for 2 month and everything was fine, then I turn on advanced build and nothing works).

On the other hand, conceptually, externs are part of a library, not my user code. So, if library author thinks his library needs externs, he specifies it and packs externs along with library code, and they’re automatically used when I add dependency. So as a user I don’t manage external deps’ externs at all. As Thomas Heller puts it, it might be [a quality of life change](https://github.com/tonsky/datascript/issues/30#issuecomment-62253696).

Any thought on it? Maybe there’s already a way to do that?

Nikita.

[Julien Eluard]

I agree this would be a really great feature.

There used to be a similar facility part of lein-cljsbuild allowing to package both the externs and eventual JS dependencies (see [1]) but it does not work anymore.

Maybe something equivalent could be part of the clojurescript compiler? There already are some projects packaging externs and JS files (see [2]) so it would be a matter of defining a convention. It could also be an opportunity to remove the need to rely on :preamble for non closure dependencies.

This would allow the creation of self-contained libraries and would greatly lower the barrier of using clojurescript libraries with dependencies.

[1] https://github.com/jeluard/cljs-ol3js/tree/master/resources/closure-js

[2] https://github.com/swannodette/react-cljs/tree/master/src/react

Nikita.

--
Note that posts from new members are moderated - please be patient with your first post.
---
You received this message because you are subscribed to the Google Groups "ClojureScript" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojurescrip...@googlegroups.com.
To post to this group, send email to clojur...@googlegroups.com.
Visit this group at http://groups.google.com/group/clojurescript.

[Thomas Heller]

Hey,

the feature already exists like I mentioned on github. I'm just not sure about its state since it is "marked" experimental [1]. I'm not sure how it came into existence or what the backstory was but if I understand the code correctly you are supposed to include a deps.cljs in your .jar which contains something like

{:externs ["my/externs.js"]} where my/externs.js is also included in your .jar so everything is available via the classpath.

Not sure how well this solution works since you might run into conflicts if 2 libraries provide similar externs. Not sure how Closure handles this sort of conflict.

The feature was introduce in February 2012, maybe the original authors can chime in here.

Cheers,
/thomas

[1] https://github.com/clojure/clojurescript/blob/9fd6bf5bd55421c3d5becacc5230ed661d6fb3c3/src/clj/cljs/closure.clj#L838

PS: the feature probably won't work with shadow-build currently, need to look into that.

[Paul Bostrom]

"remove the need to rely on :preamble for non closure dependencies."

I took a stab at solving this problem with a lein plugin:
https://github.com/pbostrom/lein-cljsasset
It would be simple to expand to support an :externs field. Then the workflow would be that libraries declare the externs path in their project.clj, and users declare the target externs file in their environment, which they would then specify in their cljsbuild config. I'll admit it sounds a bit cumbersome still, and the whole workflow should probably be managed by a single tool. Just thinking out loud, the ideal tool would:
1) Resolve JS/CSS dependencies from classpath and concat (and minify?) to a single file for each
2) Resolve externs dependencies from classpath
3) Advanced compile all CLJS/Closure dependencies and CLJS application code with externs file(s) from step 2
4) Generate initialization/loading JS to load the JS/CSS files from step 1
5) Output a single JS file that initializes and loads your application

There would probably need to be another step in there which resolves image dependencies.

[Micha Niskin]

The boot cljs compiler task solves these problems:
https://github.com/adzerk/boot-cljs#preamble-externs-and-lib-files

[Julien Eluard]

Looks like both boot cljs and lein-cljsasset have their own way to define externs/lib. This makes the task for library writer harder.

By defining a common standard directly supported by the ClojureScript compiler we would have a generic solution all tools / IDE could benefit.

Probably the best option would be to rely on a directory layout / filename convention so that non leiningen based solution are not penalized.

Is that something we want to include in the compiler?

[Nikita Prokopov]

Agree that should be on compiler level. Relying to third-party solution does not improve situation: without them, I have a possibility to forget to include externs, with third-party solutions I have a possibility to forget to include them (also library author will have to choose whose convention to support).

My proposal:

Cljs-compiler should just include every 'externs.js' file it can found on the classpath on top level.

Library authors: just include externs.js to the jar at top level

Library consumers: just add dependency (as you normally do) and it’ll put externs.js to the classpath where it could be seen by compiler.

Any thoughts? Who can make such a decision? I can work on an implementation.

[Micha Niskin]

Hi Nikita,

How would you find these externs in JAR files? Would you scan all entries for them? This takes a really long time. 

This is actually what boot v1 does, and it's not really workable for large projects with a lot of dependencies (this can be hundreds of thousands of JAR entries to sift through). 

In boot v2 we namespaced them so we can very efficiently prune the JARs that don't have JS includes or externs, and the JARs that do typically won't have a lot of entries.

--

Micha Niskin

--
Note that posts from new members are moderated - please be patient with your first post.
---

You received this message because you are subscribed to a topic in the Google Groups "ClojureScript" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojurescript/LtFMDxc5D00/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojurescrip...@googlegroups.com.

[Nikita Prokopov]

Micha,

I don’t understand what kind of optimization we’re talking about.

If I understand correctly, boot v2 scans every jar for the mask `*.jar!/hoplon/include/**/*.ext.js`.
My proposal is to scan each jar for a single possible path `*.jar!/externs.js`.
Current `:externs` directive in lein-cljsbuild _does_ scan all the jars for the path you’ve specified for it (e.g. if you put :externs ["react/externs/react.js"] it will scan entire classpath for this path). So at least we’re not making things worse.

Also, as the content of the jar (non-snapshot at least) is immutable, we can cache this information heavily.

[Micha Niskin]

Nikita,

Ah okay, I misunderstood; I'm tracking now. I do think that namespacing would be a good idea because of possible conflicts with jars containing entries with that name, but there are of course no performance issues with what you propose.

--

Micha Niskin

[David Nolen]

As Thomas Heller mentioned this already supported via deps.cljs.
Someone needs to enumerate what is missing from the existing
functionality and how it may be improved.

David

On Sat, Nov 15, 2014 at 11:14 AM, Nikita Prokopov <prok...@gmail.com> wrote:

[Nikita Prokopov]

deps.cljs more or less works, at least for my use case (one extern file).

1) I found that path to externs file should be unique across all dependencies. I think it’s counter-intuitive, and may lead to serious problems in the future. E.g. two different lib authors decide to use name `externs.js` for their libs, and everything is fine until somebody decides to use both of them libs. From technical point of view, we can distinguish these externs (they come from different libs), but current compiler implementation fails, and not in a very gracious way. I created repo reproducing the issue: https://github.com/tonsky/clojurescript_deps_test. Should I create a ticket/submit a patch somewhere?

2) I believe deps.cljs should be promoted to official convention, so other build tools may start relying on it. It would help if most popular CLJS projects will adopt this (e.g. swannodette/react-cljs could use it as an example. DataScript already uses it)

3) I can’t anything about feature completness of :libs and :foreign-libs in deps.cljs (don’t know the use-cases)

Nikita.

[Martin Klepsch]

Another potential issue with `deps.cljs` is that it might be present in projects already, serving a different purpose. To make sure this does not cause conflicts `deps.cljs` should reside in a location where it can be safely assumed that it's intended for the Clojurescript compiler.

Also maybe it's more sensible to name the file `deps.edn` as it's not really a source file that contains a namespace `deps`?

[David Nolen]

On Sun, Nov 16, 2014 at 1:49 PM, Nikita Prokopov <prok...@gmail.com> wrote:
> deps.cljs more or less works, at least for my use case (one extern file).
>
> 1) I found that path to externs file should be unique across all dependencies. I think it’s counter-intuitive, and may lead to serious problems in the future. E.g. two different lib authors decide to use name `externs.js` for their libs, and everything is fine until somebody decides to use both of them libs. From technical point of view, we can distinguish these externs (they come from different libs), but current compiler implementation fails, and not in a very gracious way. I created repo reproducing the issue: https://github.com/tonsky/clojurescript_deps_test. Should I create a ticket/submit a patch somewhere?

Ticket & patch welcome.

> 2) I believe deps.cljs should be promoted to official convention, so other build tools may start relying on it. It would help if most popular CLJS projects will adopt this (e.g. swannodette/react-cljs could use it as an example. DataScript already uses it)

Agreed.

David

[Thomas Heller]

Hey,

had a bit of time thinking about things.

"deps.cljs" seems like a band aid to be honest, it might be better than without but I imagine we run into all sorts of issues in the future.

In my projects I use CodeMirror quite frequently with a handcrafted (sort of incomplete) externs.js. To get everything working on the page I need to include the externs.js into my build and then load the codemirror.js (+ all modes and css) manually in the HTML, before my optimized CLJS. What happens if another CLJS library uses a different CodeMirror version than mine? Say v3 vs v4? The externs are not compatible (well almost), but I basically cannot use that library without running into issues.

I thought we might exploit the maven/leiningen ecosystem. Say someone released a [someone/codemirror-4.7.1], another library could still depend on 4.1.1 but maven will resolve that for us. The artifact itself could include a library.edn that describes it to the compiler (externs, preamble, provides, requires, variants, ...). We'd just need a way to tell the cljs compiler to include "library" codemirror with variants html-mode, css-mode and others so the can emit the correct preamble and include the externs in the optimization. Not sure how the library.edn would look but we should be able to figure this out.

Of course that would require that someone takes to time to update and release javascript packages as a jar. But David is already doing that with [com.facebook/react "0.11.2"], maybe github+clojars is already a good enough platform to handle this.

Just my 2 cents,
/thomas

[Martin Klepsch]

With what Thomas Heller described here it might be worth pointing out again how this is handled within boot-cljs. boot-cljs looks for a specific path in jars and uses files in that location as #{preamble, extern, gclosure-lib} depending on their file endings.

For example, a file ending in `.ext.js` will be supplied as extern to the Clojurescript compiler.

This would provide a clear pattern to provide externs and via regular Maven dependencies and some of the problems described by Thomas would be solved. Also creating a file like `library.edn` wouldn't be necessary.

When I first saw this approach I also wasn't sure if I like it but after some more discussions
with the Hoplon Team it seems very sane.

[Martin Klepsch]

Sorry, forgot the link: https://github.com/adzerk/boot-cljs#preamble-externs-and-lib-files

[Thomas Heller]

I'm not a big fan of automatic file discovery and inclusion. When evaluating new libraries I have to "scan" all files manually to get a glimpse of what they are up to instead of just looking at one file. I do the same now when browsing clojure projects, first look at project.clj to get an overview of what dependencies they carry and the like. But that might just be me ...

Also one BIG problem that is not addressed is that some javascript libraries may include "optional" resources. I mentioned variants in my post, let me use CodeMirror [1] as an example.

CodeMirror ships with alot of optional "modes", basically extensions. Since you usually don't need all modes you only load the ones you need. I only use htmlmixed, css and vim mode for example. These ship in seperate files which you include in the HTML. In an ideal world these would be added to the preamble after codemirror itself automatically. How we describe that in a library.edn I don't know, but it should be possible to express this somehow. [someone/codemirror-htmlmode] and others seems like overkill.

But this does not address the CSS issues, I'm not even sure I want to get into that at all.

Also most libraries tend to ship .min.js files which IMHO is not required, we just need to teach compiler to minify (not optimize) the preamble (something closure already is capable off).

Maybe it is best left to the user to determine the build config as this stuff all seems very fragile at best. The more I think about this stuff the more hesitant I get. I'm very particular with my CLJS builds since they are shipped to thousands of people every day and I don't want to waste bytes especially when approaching higher mobile percentages. The whole reason I created shadow-build [2] in the first place was to save bandwidth for the user.

/thomas

[1] http://codemirror.net/
[2] https://github.com/thheller/shadow-build

[Paul Bostrom]

My $0.02, I think it would be nice to leverage the extensive list of libraries already packaged at http://www.webjars.org as well as use that as a model for packaging new libraries on clojars. This requires using something like a library.edn file or just the project.clj file to specify what should get pulled in as part of the library. For example, here is the project.clj file I used in my proof of concept code-mirror wrapper to include support for Clojure mode and paren matching:
https://github.com/pbostrom/om-codemirror/blob/master/project.clj

In any case, any potential solution will probably have to ride the coattails of a popular library before it gains any traction.

[Micha Niskin]

Thomas,

Hoplon handles the whole shebang. We've been using it successfully for about a year now in production. The system encompasses JS preamble, externs, and resources. All of these are processed in correct dependency order according to the dependency graph. Versions of our JARs are pinned to the versions of the 3rd party libraries they contain.

An example: https://github.com/tailrecursion/hoplon-demos/tree/master/jquery-date-picker. This demo has dependencies on two Hoplon "contrib" libraries, which are CLJS wrappers for 3rd party JS things (in this case Twitter Bootstrap and the jQuery Date Range Picker plugin). These contrib dependencies themselves depend on "vendor" libraries which contain the 3rd party JS and corresponding resources.

Hoplon understands how to integrate these into your project, and it works.

Here is the source:

* contrib libraries: https://github.com/tailrecursion/hoplon/tree/master/contrib

* vendor libraries: https://github.com/tailrecursion/hoplon/tree/master/vendor

The README files describe the methodology used to create new ones. (Note that this is for Boot v1, which is a little different from how we will be doing things with Boot v2. Still in the process of transitioning.)

Here are the associated artifacts on Clojars:

* contrib libraries: https://clojars.org/groups/io.hoplon

* vendor libraries: https://clojars.org/groups/io.hoplon.vendor

Our system addresses a number of problems:

* Correctly resolve transitive dependencies (eg. the jQuery date picker has transitive dependencies on jQuery and MomentJS, which need to be loaded in the correct order in the client).

* Prevent inclusion of multiple instances of 3rd party JS (eg. including jQuery more than once is a no-go).

* Automate and encapsulate resources in the dependency jars (eg. the date picker ships with CSS which is automatically loaded correctly in the app).

* Ensure that externs are provided and used transparently to the user, with the correct version used with the associated JS.

Things that are not implemented currently and are waiting for boot v2:

* Properly namespace resources to avoid collisions with resources in other JARs and facilitate efficient search from the classpath.

* Use dependency classifiers to select between minified (the default) and unminified JS, source maps, etc.

* Better tasks for visualizing JS dependencies during the build.

This is a complete solution that has had a good deal of thought put into it and has been used successfully for some time now. The improvements listed above are the result of this.

--

Micha Niskin

--
Note that posts from new members are moderated - please be patient with your first post.
---

You received this message because you are subscribed to a topic in the Google Groups "ClojureScript" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojurescript/LtFMDxc5D00/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojurescrip...@googlegroups.com.

[Nikita Prokopov]

I feel that managing CSS inclusion and third-party JS lib minification is kind of out of scope for CLJS compiler. Yes, these problems exists, but they are very broad, with a lot of solutions. I’m not sure we should try to solve them in CLJS compiler. I’m not even sure “preamble” feature should belong to CLJS compiler.

The goal of CLJS is get .cljs sources in and produce executable .js code out. How this .js will be used depends on your app: you might include it as is, you might concatenate it with libs (preabmle), etc.

:externs are important because without them libraries would not compile into executable .js source, therefore should be addressed somehow in compiler. Managing JS dependencies, minification, concatenation etc has, in my opinion, nothing to do with compilation.

[Thomas Heller]

Well yes, the compiler only needs to produce .js from .cljs.

The "second phase" is "packaging" everything together, this is where Closure kicks in. That part needs externs, preamble and the like. I would agree that does not need to be in ClojureScript itself, the whole Closure part ist not actually needed for anything CLJS compilation related.

Tools like shadow-build or lein-cljsbuild could handle that part on their own (shadow-build already does) and provide extended "features". Minification of the preamble is actually just Closure :simple optimizations of that code, I'll eventually add that.

CSS is a whole other issue though, I would not attempt to address that but the sad part is that many JS libraries require them to display properly. Anyways, that has no place in ClojureScript.

It might be worthwhile to remove the CloSure parts from ClojureScript, given the tooling required to make "proper" builds. It would simplify CLJS alot and since a big majority use external tools already, probably no one would notice.

But that went a bit off-topic, since we'd still need a way to package external dependencies properly. ;)

/thomas

[Micha Niskin]

I agree that these are not things I'd want to see incorporated into the CLJS compiler, for sure. I think that it's the role of build tooling to do this stuff. Boot does just fine driving the CLJS compiler, telling it where externs are and so forth, and this is the way I'd most prefer, in a perfect world. Even with Hoplon it's separated into a number of different composable tasks, because asset pipeline stuff can't be made monolithic and general. We need to leverage composition of simple components there and let the end user decide how they need to be composed to build their unique snowflake of a project.

The point I was making was that it's a problem that can and has been solved elegantly with tooling alone, and doesn't need CLJS compiler integration. I see a trend emerging where frustration with tooling is resulting in more and more fascination with loading down core things with build concerns, which I think would be a shame. The CLJS compiler does the job admirably as-is (I also really have high hopes for shadow-build, but I'm not very familiar with it yet; still learning). 

--

Micha Niskin