Yarn findings

[Cliff Meyers]

I have been evaluating Yarn and the updated version of maven-frontend-plugin. I really hoped this e-mail would begin with TL;DR "yarn is awesome, let's go!" but I definitely can't say that. Sorry for the length of this e-mail but the devil's in the details.

Maven-y Things:

maven-frontend-plugin 1.4 supports yarn, so that makes it pretty easy to use yarn. An older version of MFP also supports caching the downloaded node, npm and yarn binaries inside ~/.m2/repository, so we can also eliminate some custom maven-download-plugin code we had to download it and cache it ourselves. Here is a change that removes that config and relies on MFP to do it. Seems to work great:

https://github.com/cliffmeyers/plugin-pom/commit/87f0aefde074721517fdc96eeccb0a23d0919480

For enabling yarn builds, the relevant code changes are here:

https://github.com/cliffmeyers/plugin-pom/commit/0a56287ffbdffa60a50a100c6128396515a928ed

https://github.com/cliffmeyers/plugin-pom/commit/959283c4d5e10840f6eb088affafe14acecfb8ec

As far as Yarn in general, our main Maven build seems to work ok and not cause any regressions in JS build behavior. Porting blueocean-plugin and child modules to use yarn, the "mvnbuild" and "mvntest" targets both worked, seemed to produce valid bundles that load okay in the browser, and that UI build passed ATH.

Yarn Pros:

- It uses lockfiles by default, so we'll get consistent builds by default. We should also see fewer bugs than we did with npm, as npm's shrinkwrap has always been a bit of an afterthought. Yarn also suggests always committing a lockfile irrespective of whether the project is an "app" or "library"  (https://yarnpkg.com/blog/2016/11/24/lockfiles-for-all/) whereas npm tries to be more nuanced.

- Yarn's lockfile format is much more diff-friendly than shrinkwrap since it's not a JSON file.

- Yarn seems to handle inconsistencies between package.json/yarn.lock and the local file system better (it warns that an integrity check failed and it seems to handle it better than npm). I haven't tested it super thoroughly though.

- The "--har" option is super useful to see the network traffic generated by yarn. Unfortunately the file is not generated if yarn fails part way through.

- Its API distinguishes between "install my deps"  (yarn or yarn install) and "install a new dep" (yarn add) so it's a little easier for newbies to grok.

- It is anecdotally slightly faster than npm with shrink. It is anecdotally much faster than npm without shrink. In practice, this is not a big deal for us.

Yarn Cons:

- Concurrency: It does not handle concurrent installs in a resilient manner by default. When running mvn with the "-T 1C" option - especially when skipping tests - it's easy to see "yarn install" fail either for dashboard or personalization because two yarns are writing and reading from the yarn cache at the same time. The workaround is to use the "--mutex network" option which pauses that yarn if another yarn with that option is also running. There doesn't seem to be a way to set the mutex globally per machine via .yarnrc which is a real drag.

- Local Development: link and unlink appear to suffer from the the same limitations as npm's equivalent commands, which led to the creation of slink. Furthermore, yarn's equivalent of "npm install ../path/to/project" is "yarn add file:../path/to/project" and doesn't play nicely w/ our browserify / js-builder set up at all. I haven't taken the time yet to understand why but when doing it from blueocean-web -> blueocean-core-js it results in a totally broken app that can't resolve the "init.jsx" that loads Blue Ocean. It's possible we could work around these issues if we just modified slink a bit so it doesn't require a local install of the package before use. Have to defer to Tom for some input there.

Registry Issues - note that these issues probably affect npm and yarn equally

- Registries and Lockfiles: yarn lockfiles are generated with the fully-qualified path to the artifact, and a checksum. These URL's will trump any subsequent registry settings. I was able to get yarn working with the Artifactory mirror, after I set the registry, wiped the lockfiles, and then re-ran the build to generate new lockfiles. Note this is the same behavior as npm with shrinkwrap. Also note that our mirror is slower for me here in Virginia USA than the yarn registry.

- Registries and Publishing: with the Artifactory registry set, I cannot publish modules to npm - I get an authentication error. This isn't a huge surprise since I'm using my npm registry credentials and presumably Artifactory would expect different credentials. Maybe there's a way to create an Artifactory account that can associate an npm registry auth token; I haven't investigated - seems doubtful. The workaround is to clear your registry settings (yarn config delete registry), publish the artifact, then bring the registry back (yarn config set registry https://myrepo/path/to/mirror)

I have some opinions on next steps but I think I'll reserve them until others can chime in with their thoughts.

Thanks,

Cliff

[Joshua McDonald]

Nice work Cliff, cheers! How much of this (the registry / contention issues) could be addressed perhaps with a registry service on localhost?

--
You received this message because you are subscribed to the Google Groups "Jenkins User Experience" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-ux+unsubscribe@googlegroups.com.
To post to this group, send email to jenkin...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-ux/CAFm%3D0hd2vaKcYGSs9fuVYPbnZFJ7V2cR83POmrqyR%2BaPTh5E8Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Cliff Meyers]

I am not sure I follow. Are you proposing mounting a prepopulated yarn cache in CI?

[Joshua McDonald]

Actually my firs thought was to just run a smarter repo on localhost and turn off cacheing. But it's probably easier to just fix Yarn, since it's OSS :)

[Cliff Meyers]

It appears that you *can* specify arbitrary arguments passed to yarn via the .yarnrc file. It's a bit of a hack through. This wasn't documented but I debugged through the yarn code and figured it out:

https://github.com/yarnpkg/yarn/blob/v0.23.2/src/rc.js#L41

This code converts a "--argname argvalue" entry in .yarnrc into an argument that's passed on to yarn as though it were a normal command line arg. If you don't preprend the double-dash, it gets ignored.

So adding this to yarnrc gives the more resilient behavior we'd like:

--mutex network

Interestingly, it seems that this needs to added by hand, since "yarn config set --mutex network" complains with "invalid subcommand" due to how the commander lib is parsing the arguments.

Probably worth opening an issue and possibly contributing a PR to resolve, since .yarnrc has giant "DO NOT EDIT THIS FILE DIRECTLY" warnings at the top so hand-editing might not be supported in future versions.

-Cliff

[Michael Neale]

so yarn sounds like a bit of an improvement.. what is best way to get people trying this out? 

[Cliff Meyers]

This PR should be purely additive to enable use of yarn, and still support npm without any breaking changes:

https://github.com/jenkinsci/plugin-pom/pull/57

Working on getting that through, and opening another PR shortly that shows the changes to use yarn in blueocean-plugin.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-ux/2e92c570-da60-4942-b3c4-bd30363e86e3%40googlegroups.com.