Easily setup Intellij-friendly cross-ScalaJS/JVM projects, in seconds

[Ben Hutchison]

Haoyi's earlier work around the JsCross build and an Intellij-friendly layout has made developing cross-ScalaJS/JVM projects much better. [https://github.com/lihaoyi/utest#jscrossbuild]

But Ive found that its still surprisingly fiddly to setup new projects: creating the directories, Build.scala, plugin.sbt, js and jvm directories, symlinking etc...

And if done in the wrong order, it won't work right in Intellij. You must do the symlink *before* initial import into Intellij, or else the IDE metadata will be wrong and stay wrong.

If you live demo this, the audience will think "Scala.js is a big hassle and not mature".

So after yet more time waste today, I wrote a script that creates projects that import into Intellij and "Just Work", first time:

https://gist.github.com/benhutchison/5e8e3a8d29eace84006f

(Keep in mind this is aimed at codebases that work in both JS and JVM, not a pure JS project.)

-Ben

[Justin du coeur]

This reminds me of a question for those who have been playing with this stuff.  As I understand it, this approach is symlink-based.  Does it work on Windows?  For better or worse, that's where my dev environment still lives...

--
You received this message because you are subscribed to the Google Groups "Scala.js" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-js+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scala-js/CAGJkGJ9R38i5br2kCBCNLDPD0c-BZsO2PiqHfUK4iFnRsxvnnw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Ben Hutchison]

Window's does have its own form of symlink [http://en.wikipedia.org/wiki/NTFS_symbolic_link], I once got them to work. But dev on Windows was such an uphill (and increasingly, lonely) battle, so glad Im not fighting that one anymore.

https://www.google.com/search?q=sad+cats&tbm=isch

[Egon Nijns]

FWIW, I recently cloned https://github.com/lihaoyi/workbench-example-app/tree/autowire to play with this setup on windows. (after seeing Haoyi's wonderful presentation: https://www.youtube.com/watch?v=Ksoi6AG9nbA) 

This example has a server module (using spray), a shared module, and a client module.

After using sbt-idea to generate the idea project intellij imported the project without any problem. I did some small experiments with this setup and everything just worked as expected / hoped :)

I have no idea if this is using the windows symlinks or some other trick to make it work, but am also interested to know more about how this works.

[Ben Hutchison]

On Sat, Oct 11, 2014 at 5:08 AM, Egon Nijns <egon....@gmail.com> wrote:

FWIW, I recently cloned https://github.com/lihaoyi/workbench-example-app/tree/autowire to play with this setup on windows. (after seeing Haoyi's wonderful presentation: https://www.youtube.com/watch?v=Ksoi6AG9nbA) 

This example has a server module (using spray), a shared module, and a client module.

After using sbt-idea to generate the idea project intellij imported the project without any problem. I did some small experiments with this setup and everything just worked as expected / hoped :)

For curiosity, I just checked this out and imported, but it didn't work 100% correct for me: 

The shared code is a separate peer module rather than being embedded within the client and server. AFAIK that doesn't work well in Intellij, the non-shared code doesnt "see" the shared code.

For example, if you go into Server.scala L32 and hyperlink on "Api", does it go through to the shared trait? I get "Cannot find declaration.."

Also, when I build in SBT, Im seeing: "unresolved dependency: org.jsawn#jawn-parser_2.11;0.5.4"

-Ben

[Egon Nijns]

I had this setup on my desktop at the office, so can't check right now, but I'm pretty sure code navigation worked. I'll double-check this as soon as I get the chance.

Did you run gen-idea to generate the intellij project before importing? (I don't think so since you had trouble resolving some artefact)

Op 11-okt.-2014 01:24 schreef "Ben Hutchison" <brhut...@gmail.com>:

[Egon Nijns]

I just tested this and code navigation + renames of methods in the API class definitely work.

I don't think this project is using symlinks anyway, both the client and the server project just depend on 'shared':

As expected only the client and shared are using the scala-js compiler plugin. (Note that I'm not using the sbt support from IntelliJ)

Thinking more about this, I have no idea why you would even need symlinks for this to work?

egon.

On Saturday, October 11, 2014 8:22:38 AM UTC+2, Egon Nijns wrote:

I had this setup on my desktop at the office, so can't check right now, but I'm pretty sure code navigation worked. I'll double-check this as soon as I get the chance.

Did you run gen-idea to generate the intellij project before importing? (I don't think so since you had trouble resolving some artefact)

Op 11-okt.-2014 01:24 schreef "Ben Hutchison" :

[Alexandru Nedelcu]

There’s something I’m missing in that build.sbt file.

If shared is a ScalaJS project using scalaJSSettings, how can the server depend on it?

​

On Sat, Oct 11, 2014 at 4:31 PM, Egon Nijns <egon....@gmail.com> wrote:

As expected only the client and shared are using the scala-js compiler plugin. (Note that I'm not using the sbt support from IntelliJ)

Thinking more about this, I have no idea why you would even need symlinks for this to work?

--
Alexandru Nedelcu
www.bionicspirit.com

PGP Public Key:
https://bionicspirit.com/key.aexpk

[Ben Hutchison]

I'm going to offer my best guess at what's going on. But it would be good to get Haoyi's input, since he pioneered alot of this stuff [https://groups.google.com/forum/#!msg/scala-js/snmOMbI5hhk/cNnYOyeSVCoJ]:

- There are 2 approaches to sharing code in mixed Scala/Scala.js projects that "work"

 (a) A SBT subproject that is shared between client and server, as demonstrated [https://github.com/lihaoyi/workbench-example-app/blob/autowire/build.sbt] and [https://github.com/lihaoyi/upickle/tree/cake]

 (b) A symlink that embeds the same shared src folder into both client and server projects, as documented, and employed, here [https://github.com/lihaoyi/utest#scalajs]

- If you read the above-linked thread carefully, you'll notice that Haoyi starts on Aug 13 talking about option (a), but 5 days later in the thread he had switched over to using option (b). Although he doesn't explicitly give the reason he migrated from approach (a) to (b).

I guess the motivation is the difference is that (a) only support dependencies from non-shared onto shared code, hence the need to "cake" (ie abstract over) any refs from shared code to platform specific code. 

Whereas (b) seems more forgiving. The shared code is just peer source folder in both client and server, so cross dependencies are fine without abstracting over, as long as they are fulfilled in both environments.

-Ben

[Sébastien Doeraene]

Hi,

Actually option (a) is just plain wrong, as it reuses the same sbt subproject for the JVM version and the JS version of the artifacts. You should never do that! There should be a sharedJVM and a sharedJS.

Sébastien

--
You received this message because you are subscribed to the Google Groups "Scala.js" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-js+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/scala-js/CAGJkGJ8%3DqGOfFynUnpuHa5_dqT-u9OcP%2Bme8x%3DKfvKD%3D1z%2BOWw%40mail.gmail.com.

[Justin du coeur]

I don't follow -- why not?  If the shared directory is just code, and isn't being *compiled* as such, I don't see anything wrong with it.  This is the way I'm operating, based on the Play skeleton project; the shared "project" is basically just a resting place for code that is separately compiled in the JVM and JS subprojects...

[Alistair Johnson]

Hi

My two cents can be found at sbt-scalajs . But please see/read the issues that I'm currently working on, so some things will change (soon).

The documentation will be improved to make it clearer that the real purpose of the plugin is more to get a bunch of ideas related to sbt, play, etc and make them easily accessible for people to use and try via code - rather than reading FAQ's, forum's, forking this'n'that, etc. It's certainly not meant to be just "my way". 

So if anyone wants to add /change anything - please just ask, add issue or PR. 

Alistair 

[Matthew Pocock]

Where are we up to on this? I've got a project derived from the play + scalajs template on github and I need to run a main class in the debugger in intellij to work out what's going on (there's a pathological performance case, but without debugging it's difficult to work out why it's running away), but the project is a patchwork of red, and idea can't even build it.

Thanks,

Matthew

To view this discussion on the web visit https://groups.google.com/d/msgid/scala-js/e84abf98-3424-4612-a35e-72afeab0f32d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Dr Matthew Pocock

Turing ate my hamster LTD

mailto: turingate...@gmail.com

Integrative Bioinformatics Group, School of Computing Science, Newcastle University

mailto: matthew...@ncl.ac.uk

gchat: turingate...@gmail.com

msn: matthew...@yahoo.co.uk

irc.freenode.net: drdozer

skype: matthew.pocock

tel: (0191) 2566550

mob: +447535664143

[Alistair Johnson]

Hi,

Yesterday I applied for an IntelliJ license from their Open Source Program as only their commercial version has Play support.

I was turned down. 

Therefore I cannot do anything further to help, unless you consider some other IDE solution that works, such as bash + vi.

Regards

Alistair

[Matthew Pocock]

Hi,

I'm at that time again when I'm about to start another cross-compiled library for scala 2.11.4 but jvm+js that I need to be able to edit in intellij. Is there a reliable template I can work from as of today? I'm continually baffled by this stuff ;/

Thanks,

Matthew

On 14 October 2014 14:36, Alistair Johnson <alistair...@johnsonusm.com> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/scala-js/e84abf98-3424-4612-a35e-72afeab0f32d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Alistair Johnson]

Hi Matthew

What we have at banana-rdf is stable and works well.

If you can't/don't want to copy that, I could get a template together for you tomorrow, bit busy today. You can then compare the alternatives and pick the one you like best. It would be interesting to see how they compare!

If you would like this, perhaps you could give a link to an existing, similar project that you might have and/or detail the new lib so I/we can help you get started.

Cheers

Alistair

[Haoyi Li]

I've put together a simple utest-example-module which has a basic demonstration-module set up. Obviously this one is utest-specific (hence the name) but apart from utest being my own thing, I've been using it in almost a dozen projects right now and it works great. This should import into IntelliJ no problem, although you may have to remember to refresh the symlinked files  (Cmd/Ctrl-Shift-Y) if you're jumping back and forth from editing them in js/share/ and jvm/shared/

To view this discussion on the web visit https://groups.google.com/d/msgid/scala-js/8d88335c-e479-4e24-ab77-f3cde4d29c14%40googlegroups.com.

[EECOLOR]

My 2 cents: It might be a smart idea to create an activator template for Scala.js mixed project setups.

To view this discussion on the web visit https://groups.google.com/d/msgid/scala-js/CALruUQ%2BDe%3DJH8wjQnQmeZm7unYBUn4%2BU-hoO0G30B1vQOa%2Bm9g%40mail.gmail.com.

[Matthew Pocock]

Hi Alistair,

A template would be really helpful. I'm trying to work from the script Ben posted at the moment, but I don't even seem to be able to get a dependency to be picked up :( It would be really useful to have a template project with all the directories already there and place-holders for dependencies in the various projects were I just need to fill in the missing bits. I've really honestly tried, but I don't seem to be able to grasp how SBT works. I'd go back to maven, but I can't work out how to get scalajs to work in a maven build.

Matthew

To view this discussion on the web visit https://groups.google.com/d/msgid/scala-js/8d88335c-e479-4e24-ab77-f3cde4d29c14%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Alistair Johnson]

Hi Matthew,

I'll do my best to get something ready for tomorrow morning.

Also, I've heard that others have had some issues with IntelliJ 14 - what version do you use? Rather interestingly, I updated to 14 and found that it works fine *without* using the gen-idea plugin.

Cheers

Alistair 

[Matthew Pocock]

Hi Alistair,

I'm using IntelliJ 14, scala 2.11.2 and sbt 1.13.5 (not upgraded to 2.11.4 and 1.13.6 yet - will for this next project). 

Matthew

To view this discussion on the web visit https://groups.google.com/d/msgid/scala-js/985f5383-532e-4221-8b71-6cd7e7b8e853%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Alistair Johnson]

Hi

I have added  a template to https://github.com/InTheNow/sbt-scalajs/tree/master/example .

There are no tests , please can you check this first ? Please use gen-idea.

Regards

[Dan Di Spaltro]

Basically Intellij 14 made the gen-idea command feel deprecated (using an older project model).

And with Alistair's plugin, importing an sbt project doesn't work because of shared roots.

Ideally they can relax the shared root issue and then we can ditch the gen-idea command.

-Dan

To unsubscribe from this group and stop receiving emails from it, send an email to scala-js+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scala-js/e05b3551-8ef9-44e7-8914-525f0eba484d%40googlegroups.com.

[Anton Kulaga]

>Ideally they can relax the shared root issue and then we can ditch the gen-idea command.

Opened and issue on that https://youtrack.jetbrains.com/issue/SCL-7859

[Jakub Liska]

Thanks Haoyi for this : 

js/shared 

jvm/shared    shared -> ../js/shared

Finally something that works !

[Haoyi Li]

After talking with people at ScalaeXchange, I'd like to ressurrect this thread again with an old idea.

Given the crazy contortions we're doing to do this multiple-projects-shared-source thing in SBT, and trying to get the IntelliJ guys to play along, and trying to make it work with windows (no symlinks), etc,

Is there a possibility to falling back to an earlier idea: compile the module once, and share the module (with both .class files + .sjsir files) for both Scala.js and Scala-JVM?

@sjrd has said that that is wrong in the current set-up because the Scala.js compiler may generate different class files from the Scala-JVM compiler. However, that leaves one more option open: perhaps we could have a flag to mark a subproject as being compiled twice: once with Scala.js, once with Scala-JVM, and having the Scala-JVM .class files stomp over the Scala.js .class files.

This may be a bit awkward to set up, but I suspect it'll relieve a lot more awkwardness later. In particular, looking at 

- The crazy things alistair is doing to try and make his cross-builds works in all IDEs on all OSs, 

- The work that sjrd/tobias have done to try and get cross builds (which still doesn't work with IntelliJ =P by default) and 

- The amount we've pestered the IntelliJ guys to make fundamental changes to their project mode to support our use case,

I can't help thinking that maybe doing the double-compile-and-merge gymnastics may be the path-of-least resistance to getting this working. If we actually do have a "real" module that can be shared on both -js and -jvm, everyone else (IntelliJ, Eclipse, SBT, windows, blah) all fall in place automatically.

[Sébastien Doeraene]

Hi,

Sorry, Haoyi, but this will simply kill sbt's incremental compilation: *everything* will be recompiled *every time*.

Sébastien

--
You received this message because you are subscribed to the Google Groups "Scala.js" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-js+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/scala-js/CALruUQ%2BVjE0SzXDAYM1O%2BsuiW0vLwy4USS%2B2eA9to_jM%3DXEbGQ%40mail.gmail.com.

[Haoyi Li]

What if we had two subprojects for JS and JVM, and used a single "merged" project which did nothing but aggregate the .class files from JVM and .sjsir files from JS? That should make incremental compilation work right? And then other modules could depend on merged and automatically get what they want, and IntelliJ/Eclipse could (maybe?) see the .class files that merged has and do the right thing.

[Sébastien Doeraene]

Le 10 déc. 2014 08:08, "Haoyi Li" <haoy...@gmail.com> a écrit :
>
> What if we had two subprojects for JS and JVM

Then you're back to the problem that IntelliJ cannot handle two separate projects with a shared folder.

Sébastien

> and used a single "merged" project which did nothing but aggregate the .class files from JVM and .sjsir files from JS? That should make incremental compilation work right? And then other modules could depend on merged and automatically get what they want, and IntelliJ/Eclipse could (maybe?) see the .class files that merged has and do the right thing.

That would be very risky at best, and would most certainly destroy any non-base features of sbt (packaging and publishing, for example), not to mention other sbt plugins.
Metadata would also be very difficult to manage.

Sébastien

[Haoyi Li]

Why would it destroy packaging and publishing? Wouldn't it just slurp up whatever files are in the target folder? In general, I was under the impression that apart from the tasks which actually compile the source code, the other tasks just take the resultant output and do stuff with it. If we just override compile to grab the compiled files from JS and JVM, packaging and publishing shouldn't even notice that the compiled files didn't come from sources in that project.

You're right, maybe IntelliJ still might not be happy though.

[Anton Kulaga]

To be honest even after Sebasien's explanations I still do not undestand one thing. When I do crosscompilations to different ScalaJVM versions (like 2.10 and 2.11) I get several subfolders in target folder and IDEA works fine. I wonder why cannot we treat ScalaJS as just another scala version? Yes, the project will be compiled twice but for me it is not a big issue, most of popular libs already compile for several versions of scala.

--
You received this message because you are subscribed to a topic in the Google Groups "Scala.js" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/scala-js/Ws8iBtygUXY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to scala-js+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/scala-js/CALruUQ%2BPVkAFAVjH_f6UBO%2Bw05e9N%3DMkCSJObxQ9%3DcmRMci5zw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Best regards,
Anton Kulaga

[Sébastien Doeraene]

Hi all,

OK, I'll try to make it clear once and for all.

1. There is a reason the .class files are different

The Scala.js .class files contain more information than JVM class files, which we need for exports. Not only in the same compilation run, but also in separate runs when you compile other source files against them. If you bundle up the JVM .class files in the .jars and use them for Scala.js classes, things will break.

It is therefore not an option to put jars out there which serve both for Scala.js and Scala/JVM. You need separate artifacts.

2. There is a reason sbt wants separate projects

Since the artifacts need to be different, it is normal for sbt to want two different projects. It is not only binaries that need adapting according to JS or JVM. Library dependencies can be different: you may want to write a mostly cross-compiling Scala wrapper around a Java library on the JVM and a JS library on JS.

3. The cross-ScalaJVM handling cannot be used because it is not good enough

The way Scala "handles" compiling against different major versions of Scala is a hack, which does not compose. In particular, have you ever tried to have, in the same build, a cross 2.10/2.11 library L, that you depend on by two separate projects, A for 2.10, and B for 2.11? In that setup, you cannot build all your projects at the same time. In particular, you need to use ++ every time you switch between compiling A to compiling B and vice versa.

It is not so much of an issue for cross-ScalaJVM compilation, because that setup does not happen very often (unless your build contains one subproject which is an sbt plugin, like ours). But in the context of Scala.js, this happens most of the time: you typically do want to write your server and client code in parallel, with a shared cross-compiling library, which means that you would have to constantly switch manually the target with ++. You're not going to like it.

4. IDEA wants A, every one else wants B

All tools except IDEA are very happy with two separate projects, with different metadatas, different base locations, and a stupid shared source folder. They like it very much because it is the natural thing to do, which respects all the concepts behind what a project, in the sense of artifact, is.

IDEA wants something else because it's not able to cope with a shared source folder.

Therefore, if we "fix" the structure to make IDEA happy, we have to introduce workarounds for all other tools (sbt included)! If we leave as is, we simply use the symlink workaround if we want to use IDEA.

=> The equation is simple: do we want 1 workaround in IDEA, or do we want n-1 workarounds in all other tools? The answer is pretty obvious to me, especially since the workaround is stupid simple: just create 2 symlinks once when you clone a project.

I hope this will end the endless stream of proposals for alternative models. The current model is the right one: all tools but IDEA agree on this. For IDEA, there is a simple workaround.

Sébastien

[Haoyi Li]

Thanks for the write-up! That clears things up considerably

[Benjamin Jackman]

Has anyone tried what I am doing in cgta/open? It's been working fine for us for several months.

We don't use symlinks.

This is a bit out of date compared to what we are doing internally but here is an example: 

https://github.com/cgta/open/blob/master/project/SharedProjects.scala

It works well enough for us.

You basically end up with 5 projects 1 is the shared project

1 jvm project

1 sjs project

1 jvm-test project that dependsOn the jvm project

1 sjs-test project that dependsOn the sjs project

The shared project is named oscala It is configured in sbt to be an aggregating project that builds both oscala-jvm and oscala-sjs

so command run in there run in both, if i just want to run commands on one or the other i will go into oscala-jvm or oscala-sjs and

run the commands there

the test projects are just dummy projects configured to do nothing in sbt, but they do give intellij what it needs to build the project

Then I use bin/gen-idea to make the source files.

[Michael Werk]

I have restructured the scalajs-cross-compile-example projects SBT configuration in a way that seems to work well with IntelliJ. 

https://github.com/scala-js/scalajs-cross-compile-example/pull/3

We might not want to pull this configuration into the main cross build example, as it is an IDEA-only workaround. But it works well for us and could potentially help other IDEA users getting started. 

Background: I just started using Scala.js 2 days ago. My actual problem was not cross building but just to share Scala code between a client and a server and have it all working nicely in IntelliJ. Just like in this video https://www.youtube.com/watch?v=Ksoi6AG9nbA where Haoyi Li demos his autowire library. But this IDEA problem seems to be a little secret when reading the documentation and testing the various example projects.