Xtend performance problem during gradle build

[Daniele Antonini]

Hi,

I have a quite large java codebase:

 ~ 10500 java files

~ 1050 xtend files

~ 100 gradle submodules

recently I migrated from xtend 2.8.3 to xtend 2.12.

I also use jenkins to do CI/CD stuff with a quite complex pipeline.

After the migration I had to raise the abort timout on almost every step.

Before the migration the checkout phase (git clone + gradle testClasses + other stuff) took around 8 minutes, 

after the upgrading the same phase takes around 19 minutes.

Also on local machine the time build is higher, and these performances are becoming a problem.

On my machine a full build takes around 10 minutes

I attach a gradle report of a full build , as you can see xtend takes a relevant part of the build.

Most of projects are mixed, java & xtend files refer each other.

I show you also a big overview of the xtend related projects:

                 engineBuilder       java:932  xtend-src:114  xtend-test:74

                   coderCommon       java:753   xtend-src:32   xtend-test:3

      cloudlet/engineFramework      java:1346   xtend-src:36   xtend-test:1

                     livetable       java:432   xtend-src:30  xtend-test:17

              expressionParser       java:495    xtend-src:2   xtend-test:1

                queryGenerator       java:198   xtend-src:27  xtend-test:33

   cloudlet/workgroupFramework       java:492   xtend-src:14   xtend-test:8

                      utilsSql        java:96    xtend-src:5   xtend-test:3

                        common       java:253    xtend-src:1   xtend-test:0

                   reportCoder       java:179    xtend-src:1   xtend-test:0

                 livetable-otf       java:111   xtend-src:54  xtend-test:34

           hostingServant/core       java:171    xtend-src:0   xtend-test:4

                     restCoder        java:36    xtend-src:4   xtend-test:0

            engineBuilder-demo        java:21    xtend-src:4   xtend-test:0

runtimeQueryGenerator/services        java:18    xtend-src:6   xtend-test:2

                         utils       java:130    xtend-src:0   xtend-test:0

                   dbunit-plus        java:17    xtend-src:1   xtend-test:0

              connectionPooler        java:28    xtend-src:2   xtend-test:0

         livetable-annotations         java:4    xtend-src:2   xtend-test:0

     runtimeQueryGenerator/api        java:12    xtend-src:5   xtend-test:0

Do you know are in which xtend requires more time during traspiling process?

What I can tell you is that EngineBuilder/coderCommon use a lot of xtend template, so maybe process template instructions is time consuming activity.

As soon as possible I send you also the time with the previous xtend version.

Regards

Daniele

[Daniele Antonini]

Hi,

you can find as attachment the profiled build using xtend 2.8.3

The overall time are:

- xtend 2.8.3 the build takes 3m 13s

- xtend 2.12 the build takes 12m17.17s

So xtend 2.12 is 4x slower than 2.8.3 ..

Do you confirm this times or something is wrong with my build?

Regards

--

Daniele Antonini

fhoster.com, CTO

(Skype) daniele.antonini_fhoster

daniele....@fhoster.com

Tel. ufficio: +39 06 39389666

[Christian Dietrich]

could you use a profiler to see "what" makes it slow in your case.
stepping from 2.8 to 2.12 is a huge gap. so i cannot nail down any specific changes but there where a lot, e.g. in the 2.9 release.
thus it would be interesting to have more specific data "what" is slow.

and yes there are some kinds of writing xtend code what makes it slow,
but that again would need a (java)profiler run of the task execution

[Christian Dietrich]

another question: which plugin in which version do you use?

do you know the xtend gradle plugin is obsolete and now part of the xtext gradle plugin
thus the tasks should read

- generateXtext
- generateTestXtext

[Daniele Antonini]

Hi Chris,

Yes I know that xtend plugin is obsolete,

In the first attachement (xtend 2.12) you can see that the build invoke 

generateTestXtext

generateXtext

while the second (xtend 2.8.3) use 

compileXtend

compileTestXtend

I'm using the following gradle configuration:

buildscript {

  repositories {

    maven {

      url "https://plugins.gradle.org/m2/"

    }

  }

  dependencies {

    classpath "org.xtext:xtext-gradle-plugin:1.0.17"

  }

}

//see https://discuss.gradle.org/t/how-do-i-include-buildscript-block-from-external-gradle-script/7016 

//apply plugin: "org.xtext.xtend"

apply plugin: org.xtext.gradle.XtendLanguagePlugin

dependencies {

  compile org.eclipse.xtend:org.eclipse.xtend.lib:2.12.+

}

I notice is now available org.xtext:xtext-gradle-plugin:1.0.18 so I'll give it a chance.

Ok I can use a profiler to inspect the build.

I see this gradle plugin that can help (I think) with profiling activity: https://github.com/gradle/gradle-profiler

Can you confirm this is the right tool? 

I'm not an expert on profiling activity can you sugge me some useful link about it?

Can you suggest me any other tool or profiler I can use to inspect the problem?

Regards

[Christian Dietrich]

no i meant a java profiler like yourkit or jvisualvm etc.i dont know what the gradle profiler is capable of but it seems to be a tool integrating these.
there is no relevant difference between the 1.0.17 and 1.0.18 (the latter was released yesterday and brings gradle 3.5 support to a rarely used usecase)

one issue might be that the plugins scans all the classpath for model files, which will be very pure in performance.
https://github.com/xtext/xtext-gradle-plugin/issues/57

but without any data that is pure guessing....

inside xtext there some wired stuff done but i have no idea what the purpose it
but they use a different xtextClasspath

https://github.com/eclipse/xtext-core/blob/ba911b4790738b6432e4de2da4ea39b754a138a4/gradle/bootstrap-setup.gradle
https://github.com/eclipse/xtext-core/blob/c65b348e92152b4719b9e9f2e6e05cda10695852/gradle/xtend-compiler-settings.gradle