Frontend upgrade design choices

[Stefano Maggiolo]

Hi!

We've got recently a couple of ideas around modernising our frontend development. This thread is intended to make a decision on what to do about it. So, here I'll present the alternatives, state some guiding principles coming either from me or from opinions shared by other people on gitter or github, and summarise how they are satisfied by the solutions.

Alternatives

1. Stay where we are. This means having our serving libraries (jquery and some plugin, raphael, syntaxhighlighter, bootstrap and a few more) hardcoded in the repo, with no update scripts.
2.  wil93@'s "Use npm to manage frontend dependencies". In short, it deletes the hardcoded copies of the libraries, and handle their download (and possibly update) via npm; there's a new prerequisite to fetch the libraries, that are then included as they are now in setuptools' package.
3. my "yarn + webpack + babel + typescript ( + react)". This not only does everything wil93's PR does (just with yarn instead of npm), but also introduces a bunch of development practices supported by the JS ecosystem (minification and bundling of TS/JS/CSS, and transpiling of TS into JS, mainly).

Note that the difference between yarn and npm is not really important - we can do 3 with npm and 2 with yarn, it's an independent decision. Apparently new versions of npm have more or less feature-parity with yarn.

Guiding principles

• Repo
  
• it's good to have an easy way to update the frontend libraries (to avoid getting stuck with old version)
• it's bad to have external code included in our repo, especially if we want to update it often (spurious diffs, commits, ...)
• Package managers
• it's bad to ask users to install a JS package manager to use CMS
• it's ok to ask developers to install a JS package manager, but it should preferably be an official package of most distributions
• Frontend development and serving
• it's bad not to have minification for external libraries
• it's nice to have minification, bundling, dead code elimination
• it's nice to have a good frontend language (typescript)
• it's good to have ES5 compatibility by default
• it's bad to have the repo in a half-and-half situation (some servers using one solution, some using another)
• Security and convenience
• it's bad to have old libraries
• it's bad to have thousands of dependencies (relevant old news)
• it's bad to have to trust small, poorly maintained packages (relevant news and a thought experiment)

Assessment

• Repo: this is easy: solution 1 is bad, 2 and 3 are both good.
• Package managers: 1 is good, 2 is not bad, 3 is a bit worse.
• We can include the required libraries (for 2) or the minified bundles (for 3) in the targz we release, so users are similarly happy.
  
• For developers, yarn requires a PPA, npm is in Ubuntu but with a very old version, that lacks some important features such as the lock file. 2 only uses serving packages, so there shouldn't be any non-explicit dependency and a lock file might not be necessary. For 3 it's important to have a lock file.
  
• Frontend development and serving: 1 and 2 are similar, 3 has pros and cons.
• All three have minification for external libraries.
• Only solution 3 offers typescript and other advanced features (minification, bundling, out-of-the-box es5 compatibility...).
• Solutions 3 would also require longer transitioning time - RWS is relatively easy to port, but for AWS and CWS one would have to extract all the JS in all templates first, making sure there was no Jinja stuff in them; also moving to idiomatic typescript is not trivial.
• Security and convenience: 2 wins here, followed by 1 than 3.
• Solution 1 would give zero external dependencies but will get us stuck with old versions (as we are now) which might be a liability for the users.
• Solution 2 would give new versions with minimal dependencies.
• Solution 3 would also give new versions, but with thousands of minuscule dependencies.
  

Conclusion

These are my opinions now. I really want to solve the issues of 1 in the repo sections (want to have up to date dependencies and remove them from our repo). So 1 is not an option for me. 

I worked on 3 so I'm a little biased, but I really would like to have TS and all the fancy stuff, but TS doesn't come for free - we would need to actively port the JS codebase, since it's not that it's very active. RWS is not too difficult, but AWS and CWS would be way more. I'm also very scared by the security implications of having 4k dependencies or so.

2 seems relatively painless, with fewer advantages but also fewer potential risks.

So I see either we do everything with 2, or we decide to do RWS now with 3 and wait for the next time I will have a lot of free time to transition the other servers.

[Konstantin Tretyakov]

A vote for 2 or 3, but with a pip-installable package where all the JS is prebuilt (+ a couple of setup.py commands which do the environment preparation and JS bulding).
As 2 is an intermediate step on the way to 3 anyway (webpack can be added later), and if there's already a PR, I'd vote for merging the PR.

On a side note - why is CMS not "officially" available from PyPI via pip install? (And the probably unofficial contestms package posted to PyPI by wil93 is outdated).

If it were the case, propositions like "users are asked to install a JS package manager" wouldn't even come up, as building everything would be the standard process anyway.

K.

--
You received this message because you are subscribed to the Google Groups "Contest Management System (discussion)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to contestms-disc...@googlegroups.com.
To post to this group, send email to contestm...@googlegroups.com.
Visit this group at https://groups.google.com/group/contestms-discuss.
To view this discussion on the web visit https://groups.google.com/d/msgid/contestms-discuss/CA%2B%3DdVjoxh92BcY4_rgu7nzBAX-iV_Pmp_PEXLYnAbSTHcrwkMw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Cristian Consonni]

On 13/12/18 23:12, Stefano Maggiolo wrote:
> We've got recently a couple of ideas around modernising our frontend
> development. This thread is intended to make a decision on what to do
> about it. So, here I'll present the alternatives, state some guiding
> principles coming either from me or from opinions shared by other people
> on gitter or github, and summarise how they are satisfied by the solutions.
>

> *Alternatives*
>
> 1. Stay where we are. This means having our serving libraries (jquery

> and some plugin, raphael, syntaxhighlighter, bootstrap and a few
> more) hardcoded in the repo, with no update scripts.

> 2.  wil93@'s "Use npm to manage frontend dependencies
> <https://github.com/cms-dev/cms/pull/459>". In short, it deletes the

> hardcoded copies of the libraries, and handle their download (and
> possibly update) via npm; there's a new prerequisite to fetch the
> libraries, that are then included as they are now in setuptools'
> package.

> 3. my "yarn + webpack + babel + typescript ( + react)
> <https://github.com/stefano-maggiolo/cms/commits/yarnrws_for_wil93>". This

> not only does everything wil93's PR does (just with yarn instead of
> npm), but also introduces a bunch of development practices supported
> by the JS ecosystem (minification and bundling of TS/JS/CSS, and
> transpiling of TS into JS, mainly).
>
> Note that the difference between yarn and npm is not really important -
> we can do 3 with npm and 2 with yarn, it's an independent decision.
> Apparently new versions of npm have more or less feature-parity with yarn.

I'd vote either for 2 or 3, with a slight preference for 2 (but that's
because I already know how to use npm, and the workflow looks simpler to
me).

C
--
Cristian Consonni
PhD student, DISI - University of Trento

[Andrey Vihrov]

Hi,

I think that this can be separated to three individual choices.

A. Whether to ask regular users to install a JS package manager

B. Which JS package manager to use

C. Actual JS toolchain steps

I can't comment on C as I'm not familiar with modern web development.

About A, to me it doesn't look that bad to use a package manager, unless you need extra steps to install it. After all, users are already expected to work with pip, and its operation is similar. From a security point of view, as I understand, both pip and a JS package manager can execute arbitrary code during install. Shipping already prepared JS dependencies inside the package is possible too, but it will probably make release preparation more complex.

About B, both npm and yarn will probably work fine, but in my opinion npm is a better candidate because it is included in Ubuntu standard repositories. Assuming Ubuntu's npm version is good enough (I see above that it doesn't have some features), it should be better to include it as just another Ubuntu package amongst many to install, than to ask the user to add PPAs or even do a manual install.

Currently the CMS installation process [1] relies solely on official Ubuntu packages, which is great. In my opinion it would be unfortunate if this changed. In such a case the non-official dependency should be required only for developers/contributors to install, leaving out regular users. This would also mean a "no" answer to point A.

However, if Ubuntu's npm lacks important features that developers need, then it doesn't leave much of a choice. Maybe the configuration can be written in a way so that Ubuntu's npm can still understand it, but skip unsupported features.

Thoughts?

[1] https://cms.readthedocs.io/en/latest/Installation.html#dependencies-and-available-compilers