Moving away from Yeoman (slowly)

[Deepu K Sasidharan]

 Hey guys,

So some thoughts for our future, I have been toying with this idea for a while and let me list out the reasons for that first

Yeoman is great, really great, for code generators, but lately, as you may have noticed we have outgrown what a code generator does and we are now more of a platform than a simple code generator. I have looked at the code of many popular Yeoman generators and by far JHipster is the most complex one hands down, especially due to our module system and now the blueprint system. For me, we have outgrown what Yeoman offers and hence currently we are kind of held back by limitations in the yeoman system. To list down some major issues

1. Yeoman doesn't have a proper extension mechanism other than composability, which doesn't cater exactly to our needs and is very tricky to understand, use and debug. I have spent so much time working around the composability limitations for our module/blueprint systems, that today its more of a hack than a neat feature.
2. We are stuck with JDL v2 ability to generate multiple projects due to limitations, Yeoman 3 might have some good news, but I won't be so sure unless we try it.

There are other minor things but these 2 are the noteworthy ones that I consider as slowing us down in terms of what more we can do.

I believe that JHipster can become much more powerful, and simpler in terms of internal code if we move away from Yeoman as much as possible. That doesn't mean I want to move to a new fancy language or systems like Rug or something, I believe in the power of JS and being a Javascript system makes it extremely easy for contributors and for distribution via NPM. Being written in JS is one of the reasons why we have so many contributors and moving to something fancy and complex would be death sentence IMO.

So here is what I would like to propose.

I would like to slowly phase out our dependency on Yeoman

Step 1: 

• Dual publish to NPM on both `generator-jhipster` and `jhipster`
  
• Document to install `npm install jhipster` | `yarn add jhipster` (its also much cooler)
• Change docs everywhere to remove any use of `yo <command>` and replace with `jhipster <command>`
• Deprecate support for `yo <command>` and advise users over the depreciation period to use jhipster CLI directly, in docs and in the generator itself

Step 2: 

1. Use prompts and memory-fs directly instead of using the bundled yeoman ones(they are exactly the same)
2. Replace utilities from yeoman with their direct dependent libs

Step 3: (In a major release, JH 6 preferably)

1. Run sub-generators from the CLI instead of using Yeoman (This shouldn't be difficult)
2. Use plain JS classes and inheritance for modules and blueprints instead of yeoman composability
   
3. Remove all non-essential Yeoman dependencies
4. Remove support for  `yo <command>`
5. Stop publishing `generator-jhipster` if possible and redirect to `jhipster`

So what do you guys think? If everyone agrees I'll do step 1 and step 2 over the next few months

Thanks & Regards,

Deepu

[Pierre BESSON]

+1 for everything

Totally agree on the Idea of keeping javascript for the generator. I also think that Yeoman is showing its limitations for our use case, especially with the JDL app generation but also with prompts. It would be much better to define options in a central declarative definition to keep them in sync accross prompts, the JDL and JHipster online.

I have found a great project to create CLIs in JS https://oclif.io/. It is from Heroku and looks very interesting for what we want to do. Maybe we should investigate.

Best regards,

Pierre

--
You received this message because you are subscribed to the Google Groups "JHipster dev team" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jhipster-dev...@googlegroups.com.
To post to this group, send email to jhipst...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jhipster-dev/CAFXry544RVa7Y%2BNUnRutr4dfjJ00__F_jpFN%2Bs3hxk6yuVP6-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Matt Raible]

How does this affect modules that work with “yo module-name”? I’m particularly concerned about the Ionic Module that runs outside of a JHipster project.

--

[Deepu K Sasidharan]

​​​

Hi Matt,

For independent modules which don't use anything from jhipster, it shouldn't matter. But for JHipster modules, they also would have to stop relying on Yeoman and start using something like `jhipster --module <module name>` similar to blueprints. 

I would imagine that the module developers would have to do a small migration as well, but this is normal for major releases anyway and IMO its a small price to pay for more flexibility and power for the platform

Thanks & Regards,

Deepu

To view this discussion on the web visit https://groups.google.com/d/msgid/jhipster-dev/61F0A2A0-B611-44FD-AED6-D300D5302E14%40raibledesigns.com.

[Deepu K Sasidharan]

Thanks, Pierre, Oclif looks excellent, exactly what we would need if we are migrating away from Yeoman. I love the plugin feature, it means modules can also register them as plugins to our CLI 

Thanks & Regards,

Deepu

[Pascal GRIMAUD]

I'm following you Deepu on this.

The only main problem I can see, is to keep compatibility to all OS. If we can test on Windows, MacOS and Linux and be sure it will work on all platform, I'm fine.

Clearly, the target is JHipster 6 for that.

Cheers,

Pascal

To view this discussion on the web visit https://groups.google.com/d/msgid/jhipster-dev/CAFXry55CtKzBDC4nh7A6kVDjdofFR4_DJq_%3DDgmRJ0asyqC-LQ%40mail.gmail.com.

[Deepu K Sasidharan]

Hi Pascal,

I'm definitely not suggesting to reinvent the wheel, the idea is that use all the building blocks used by Yeoman, like Inquirer, MemFs, and so on directly and may be even some of the utils from Yeoman core and remove the limitations we have. It means we will be using tools that are cross os compatible. I really like oclif suggested by Pierre if we are moving in this direction, and it seems to be os compatible.

Thanks & Regards,

Deepu

To view this discussion on the web visit https://groups.google.com/d/msgid/jhipster-dev/CALUG8VvXa43JARjn%2Ba0x9i11U7LNAizvNFXJ9EZcdqvDzu0%3DTg%40mail.gmail.com.

[Sendil Kumar N]

Totally agree to this change, we are at a point where we have to choose some other framework that is extendible and suitably matches our use case. 

oclif is attractive but we need to be more careful in choosing things. Since we very well know our use cases now.

Recreating Yeoman is not so easy, It is definitely doable but it requires a lot of work. May be we can fork yeoman and modify the fork and use that for our application generation.

Thanks

Sendil Kumar

To view this discussion on the web visit https://groups.google.com/d/msgid/jhipster-dev/CAFXry55h%3D7YN6MK9SstQ3at49NX%2BVwqXP6G5-k1DqVzhQSStRQ%40mail.gmail.com.

[William MARQUES]

+1, didn't know about oclif, looks good for what we want to build :)

To view this discussion on the web visit https://groups.google.com/d/msgid/jhipster-dev/CAPH4DvRMUgukwgzfyA2pmbWQ8UGziHPvL2JQTe_BpAxajMEFcA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

William Marques

[Deepu K Sasidharan]

Sendil, as I told before, my idea is not to drop everything from Yeoman. The most important parts of Yeoman are already seperate lib (inquirer, memfs etc) so we would reuse as much as possible and only change the shell to something else or our own. 

Yes, oclif needs to be studied first to see if it fits well. If everyone agrees here, then I guess we could do a Poc with oclif to test it out first. 

I'm still eager to hear from Julien and others

To view this discussion on the web visit https://groups.google.com/d/msgid/jhipster-dev/CAJJQRhfYgLGj19qVz26VLJNGdZJnKGJEayK%3Do3wmdU%2B8roPUuQ%40mail.gmail.com.

[Pierre BESSON]

Agree, to me Yeoman is providing a number of facilities to create a generator but at the same time it contrains its user on how they can do it. This is very good to create a new generator quickly from scratch but not to create and build on a complex platform. Maybe this is why ng-cli, react-cli and vue-cli chose not to use yeoman. In the end what we do is not very hard to write from scratch at all using some of the same building blocks as yeoman.

To view this discussion on the web visit https://groups.google.com/d/msgid/jhipster-dev/CAFXry56bB8chGLB_Kbm%2BEk9_YhWcBv%2B82JkJd9Hi1%2B8tF_gWZA%40mail.gmail.com.

[Julien Dubois]

Hi everyone,

Yes we have already discussed this in the past, and I have mixed feelings.

Indeed, Yeoman is too limited for our needs. Deepu knows it better than I do, and I'll just take one example as I'm currently working on JHipster Online: you can't script the Yeoman prompts. One of the original reasons I chose Yeoman was that it was a CLI tool, and my idea was to use it in scripts - and that fails as soon as you have prompts. That's the reason why JHipster Online only works with sub-generators that can work without asking any questions.... and that's really bad.

I also had very high hopes on having GUI tools on top of Yeoman. That was a very interesting part of the Yeoman documentation (you still have this: http://yeoman.io/authoring/integrating-yeoman.html ), and the idea was that third-party tools would provide GUIs for free on top of JHipster. As we all know, none of those tools worked. There were also IDE integration with VS Code and Intellij IDEA, but they all fail because they don't understand correctly the way we use validators in the prompts - once again, we're going way farther than the Yeoman team created.

This is linked to the previous point, but we seriously lack to have a good "decision tree" that can be re-used. So we could use it on JHipster Online (I'm coding the generator rules there a second time and I always mistakes), and there is also the research team that wants to generate all possible combinations of JHipster, and they just can't do it.

Last but not least, both the Angular team and the React team chose competing solutions. That's not Yeoman's fault, and in fact I blame Angular to have a huge NIH syndrome. Instead of reinventing the wheel, they should have let people like us do CLIs, and spend more time doing a decent developer experience or coding lazy-loading and routing correctly (yes, that's a rant). I know less React but I have the same feeling.

Anyway, the 4 points above make me think about going away from Yeoman.

Then, we have a also a huge success thanks to Yeoman. It is very fast and the prompts work very well in all environments. Their team is very friendly. In terms of marketing, they have helped us a lot to get JHipster well-known.

We also have all the modules that rely on Yeoman - whatever breaking change we make, we will lose many integrations here, and that would be really bad. There are also many people doing modules and blueprints inside their companies, we can't let them down.

So that all makes me very reluctant to move away from Yeoman.

Deepu, and also Mathieu, you are our best experts on this field, so of course I'm trusting you on this. But couldn't we work more closely with the Yeoman team, and push the features we need into Yeoman itself? As you know, I've always tried to make people contribute and not compete - that's why we have so many options in JHipster. Couldn't we do the same on the core generator and Yeoman?

Cheers,

Julien

To view this discussion on the web visit https://groups.google.com/d/msgid/jhipster-dev/CAFXry56bB8chGLB_Kbm%2BEk9_YhWcBv%2B82JkJd9Hi1%2B8tF_gWZA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Julien Dubois

Twitter: @juliendubois

[Deepu K Sasidharan]

Hi Julien,

Let me answer some of your concerns

Indeed, Yeoman is too limited for our needs. Deepu knows it better than I do, and I'll just take one example as I'm currently working on JHipster Online: you can't script the Yeoman prompts. One of the original reasons I chose Yeoman was that it was a CLI tool, and my idea was to use it in scripts - and that fails as soon as you have prompts. That's the reason why JHipster Online only works with sub-generators that can work without asking any questions.... and that's really bad.

I started working on this some time ago but didn't progress due to my lack of time. This can be improved with or without yeoman the only issue is it is time-consuming as it would need refactoring of all prompts and carefully testing them

I also had very high hopes on having GUI tools on top of Yeoman. That was a very interesting part of the Yeoman documentation (you still have this: http://yeoman.io/authoring/integrating-yeoman.html ), and the idea was that third-party tools would provide GUIs for free on top of JHipster. As we all know, none of those tools worked. There were also IDE integration with VS Code and Intellij IDEA, but they all fail because they don't understand correctly the way we use validators in the prompts - once again, we're going way farther than the Yeoman team created.

I think generic GUI tools will never work for us, with or without Yeoman and that's why I think its best that we do this ourself, I know how this can be done, see my earlier point, I just lack the time 

This is linked to the previous point, but we seriously lack to have a good "decision tree" that can be re-used. So we could use it on JHipster Online (I'm coding the generator rules there a second time and I always mistakes), and there is also the research team that wants to generate all possible combinations of JHipster, and they just can't do it.

Same as my previous answer, this can be done with or without Yeoman, but just need time

Last but not least, both the Angular team and the React team chose competing solutions. That's not Yeoman's fault, and in fact I blame Angular to have a huge NIH syndrome. Instead of reinventing the wheel, they should have let people like us do CLIs, and spend more time doing a decent developer experience or coding lazy-loading and routing correctly (yes, that's a rant). I know less React but I have the same feeling.

Agree both React and Angular team have the NIH syndrome, I think coz of the competing companies may be

Anyway, the 4 points above make me think about going away from Yeoman.
Then, we have a also a huge success thanks to Yeoman. It is very fast and the prompts work very well in all environments. Their team is very friendly. In terms of marketing, they have helped us a lot to get JHipster well-known.

I think probably I should have put the title differently. When I say move away from Yeoman, I'm not talking about ditching it completely. I love Yeoman and I'm grateful for their support. Fortunately, Yeoman is not a monolith it's extremely modular and we would need most of the building blocks regardless of if we move to another shell or not. Most of the issues I face are only in the Yeoman generator shell, not with underlying blocks like the prompts, mem-fs and so on. IMO the shell is restrictive as it wants to be generic. For our needs, we don't need that much of generic solution and we can craft something simple and efficient with plain JS class inheritance and a simple run loop. That's why I suggested to move away from the Yeoman generator shell and become an independent CLI. But we will still be using a lot of Yeoman components underneath.

We also have all the modules that rely on Yeoman - whatever breaking change we make, we will lose many integrations here, and that would be really bad. There are also many people doing modules and blueprints inside their companies, we can't let them down.

Yes there will be some breaking changes and like any of our Major releases, module/blueprint writers would have to make some changes to be compatible with newer version, this is true even today. I dont expect a heavy rewrite if we do this, but rather some minor changes as we can also make sure the changes are mostly backward compatible.

So that all makes me very reluctant to move away from Yeoman.
Deepu, and also Mathieu, you are our best experts on this field, so of course I'm trusting you on this. But couldn't we work more closely with the Yeoman team, and push the features we need into Yeoman itself? As you know, I've always tried to make people contribute and not compete - that's why we have so many options in JHipster. Couldn't we do the same on the core generator and Yeoman?

Yes, that would also be a great option. I have in the past tried to work with the Yeoman team for the composability part but it didn't get anywhere. I'm definitely not talking about competing or reinventing the wheel, I just want us to be less dependent on yeoman and if the Yeoman team is willing to collaborate with us to sort out the issues we face I'll be happy to do so. I saw there was one response from a Yeoman contributor for my tweet but nothing more. I would also understand if the Yeoman team thinks that our needs are too specific and not generic enough for them to make changes in Yeoman.

Thanks & Regards,

Deepu

To view this discussion on the web visit https://groups.google.com/d/msgid/jhipster-dev/CADNXADG4JUY0f4tm%2BUafiDGm60iX%2BcBi5FUxEwbD-r%2By57bcnQ%40mail.gmail.com.

[Julien Dubois]

As Deepu talked about Atomist / Rug and several of you were interested by it in the past, I just had a look and it seems that it has been killed: https://github.com/atomist-attic?tab=repositories

All the Rug stuff is "deprecated" as far as I can see.... I was following their Twitter account, etc, but I totally missed that.

They did several conferences about it not that long ago, including the keynote of Jax London, so that's surprising.

Anyway, for me that proves the 2 points we already discussed in the past, to have a successful generator:

- You need to have a huge community, contributors are the most important part of the project 

- You need to have something very simple: all projects that tried to do something "smart" failed (think of JBoss Forge also, but I have lots of other examples)

Those 2 points are linked, of course: if the generator is too complex, or coded in a "rare" language, then you will have fewer contributors, and hence fail.

So once again, let's first try to work with the Yeoman team. I have a few contacts there, let me do an introduction next week.

Julien 

[Deepu K Sasidharan]

Yes, Lets try with yeoman team first.

Btw it seems Atomist is going full closed source. Anyway, I completely agree a fancy language is the easiest way to kill an OSS project. TBH I hate languages which are fancy just for the sake of being cool. I'll always bet on JavaScript and I'm pretty sure languages like Java and JS will be there long after the death of all these fancy languages

Thanks & Regards,

Deepu

To view this discussion on the web visit https://groups.google.com/d/msgid/jhipster-dev/CADNXADEm--K0d_hdkiQe2ZufKrnEZyWV0W0sENPDJexFrpkK-w%40mail.gmail.com.