Compilation time survey

[Jean-François Côté]

Hi everyone,

We are using Play Framework in my company for a medium-sized application. It has grown from small to medium in the latest months because of a big effort of development of my team. Unfortunately, the "compilation" time has been growing out of control.

I would like to ask you what are your compilation times to see if what we are experiencing is normal or not. When I talk about compilation time, I'm talking about a compilation from scratch, having deleted the "target" folder and "project/project" and "project/target" folder too.

When you calculate your compilation time, do not include the time used for the resolving of dependencies.

So, for us, here is the information:

151 scala files

1224 java files

It takes 14 minutes in total. We use Play 2.5.15 and we compile using a very recent computer with a SSD. Is this normal? 

The compilation occurs like that:

First for 30 to 45 seconds

[info] Compiling 151 Scala sources and 1224 Java sources to C:\dev\CS\trunk\target\scala-2.11\classes...

Then it prints this

[info] C:\dev\CS\trunk\app\services\xxxxx.java: Some input files use or override a deprecated API.
[info] C:\dev\CS\trunk\app\services\xxxxxxx.java: Recompile with -Xlint:deprecation for details.
[info] C:\dev\CS\trunk\app\com\EnumValueMarshaller.java: C:\dev\CS\trunk\app\com\EnumValueMarshaller.java uses unchecked or unsafe operations.
[info] C:\dev\CS\trunk\app\com\EnumValueMarshaller.java: Recompile with -Xlint:unchecked for details.

It stay stuck there for the next 13 minutes, without any information in the console. If at least I could see a progression or some indication where it is, but just nothing...

Then it finally compile LESS files

[info] LESS compiling on 1 source(s)
INFO  - Creating Pool for datasource 'default'
INFO  - Database [default] connected at jdbc:mysql://localhost/database?characterEncoding=UTF-8
DEBUG - Starting application default Akka system: actorWorld
log4j:WARN No appenders could be found for logger (org.jboss.logging).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
INFO  - Application started (Dev)

And the application is started!

I can't wait to read about your compilation times in the comment below so I can compare and see if my situation is normal or not.

Thanks!

[Justin du coeur]

Data point: no -- my compilation isn't lightning-fast (let's see -- 83 seconds for a clean recompile), but it's nothing like that slow.  I have about 600 Scala files, but no Java files at all.

--
You received this message because you are subscribed to the Google Groups "Play Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to play-framework+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/e1a0ae9e-256c-4122-83c6-b8901eee4b55%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jean-François Côté]

Thanks for your input. Quick questions to sort it out:

-Do you have a single module?

-Do you have a single route file or did you split it in multiple route files?

-Do you have any views (or is it a simple REST api)

Thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to play-framewor...@googlegroups.com.

[Justin du coeur]

On Wed, Oct 4, 2017 at 11:13 AM, Jean-François Côté <jfcote....@gmail.com> wrote:

Thanks for your input. Quick questions to sort it out:

-Do you have a single module?

In the Guice sense, you mean?  IIRC, yes.  This application is biggish, but is mostly Akka and Scala.js code -- the Play part per se is fairly slim.

 

-Do you have a single route file or did you split it in multiple route files?

Single, with a couple dozen routes.

 

-Do you have any views (or is it a simple REST api)

A modest number of views -- originally more, but my UI gradually went from being Twirl-based to Scala.js, so it's mostly API.  (And mostly *not* REST -- there's one primary entry point, handling RPC calls that get routed into the Akka cluster.)

[Byron Weber Becker]

A clean recompile takes 92sec on a recent MacBook Pro.  That includes time to update/resolve/etc.  

I have several modules that depend on each other:

• xplatform  (scala.js, 61 files)
  
• server    (scala, 45 files)
• wapp     (scala, 67 files)
• wsvc     (scala, 35 files)
• browser   (scala.js, 83 files)

wapp delivers the application

wsvc is web services

browser is stuff that runs on the browser

I have a routes file that delegates to two sub-routes files.

My pain point is the amount of time it takes to make a change to the scala.js code and view the results in the browser.

[Jülide Meyer]

Yes, our compilation is slow, too. Up to 20 minutes for a clean recompile. We have three submodules, mainly java files.

1612 java files

573 scala files

92 routes files

We try to split our project in more submodules and to simplify our code

[Manabu Tokunaga]

This has been the biggest pain point in using the play framework. I really like Scala, I really like that this is similar to .NET MVC, since I move back and forth. But a large size MVC project builds in just a few seconds, but with a single line of change, especially in our DAO object, it takes up several minutes for me to get the web page up and running. A modest size node.js project utilizing Typescript is also a breeze to work with.

Sorry I am ranting and venting here a bit, but given it takes two minutes and I do comple-build-run at least 20 times a day. That's 40 minutes a day, that' comes out to be 5 of 24-hour days of lost time, or realistically 15 regular working days out of this, that's a vacation! And I work pretty much every day of the year to keep my startup running. 

 

I understand we cannot compare like apple and oranges w/ node or C# but, the only thing I am hoping that this will get fast some day soon and for someone who is just an application level developer without a desire to dive deep into the architecture of Scala, SBT, Play etc. It is supposed to make it easier for people like us.

[info] Loading project definition from C:\WinguMD\BoostServer\project

*** SBT has not good cross platform support for npm, cd to public and do npm install ***

[info] Set current project to portal (in build file:/C:/WinguMD/BoostServer/)

[info] Updating {file:/C:/WinguMD/BoostServer/}portal...

[info] Resolving org.fusesource.jansi#jansi;1.4 ...

[info] Done updating.

[info] Compiling 81 Scala sources and 287 Java sources to C:\WinguMD\BoostServer\target\s

[warn] Note: Some input files use unchecked or unsafe operations.

[warn] Note: Recompile with -Xlint:unchecked for details.

[success] Total time: 135 s, completed Oct 6, 2017 11:23:08 AM

[Justin du coeur]

On Fri, Oct 6, 2017 at 2:36 PM, Manabu Tokunaga <man...@wingumd.com> wrote:

This has been the biggest pain point in using the play framework. I really like Scala, I really like that this is similar to .NET MVC, since I move back and forth. But a large size MVC project builds in just a few seconds, but with a single line of change, especially in our DAO object, it takes up several minutes for me to get the web page up and running.

This usually suggests that some refactoring might be helpful -- if you are *frequently* making single-line changes that cause the whole world to recompile (which sounds like what you're seeing), you can often drop that to near-zero by pulling out an abstract trait/interface and using that in preference to concrete types.

I don't know your particular situation, and this approach doesn't fix *every* problem -- in particular, if the issue is that you really are changing the shape of your primary data structures constantly, that's simply a tough situation.  But in my experience, when this is happening frequently, refactoring dependencies fixes it 95% of the time.  Certainly this problem is *not* typical in Play applications: I haven't yet hit a case that I couldn't refactor away.

All of this is true of essentially any strongly-typed language.  The same problems are true in C++, C#, Java, etc, and broadly speaking the same solutions are necessary -- I'm still applying the same lessons I learned from C++ 20 years ago.  I suspect that your .NET programs aren't structured quite the same ways as your Play ones, and that that explains most of the recompile-time difference; last I checked, C# wasn't immune to these problems.

(There may well be an *educational* problem here: it wouldn't surprise me if .NET MVC is encouraging code patterns that mitigate this problem, and Play isn't yet doing enough to do so.  And to be fair, we all acknowledge that Scala's compile time is still somewhat slow, although improving gradually.  But in practice, this usually isn't a huge problem for change-compile-test cycles unless you have dependency issues.)

But note that we're talking apples and oranges here.  Your problem is that your system is recompiling too *much* every time you make a change -- that's a classic dependency-hell problem, and is pretty well-understood.  The original poster's problem is quite different: clean compiles are taking a mysteriously long time.  That's a very different issue...

[Manabu Tokunaga]

Hello Justin,

 

Thank you for taking time and providing some great insight into this. For Java/Scala, play makes it very easy to whip up an web app and the possibility of scaling up big-time, and so I will continue to be a part of the community.

 

At any rate, I think part of the compilation (and my complain) time issue for us ties very much in dependencies as you point out. Likely we (over) use Spring DI, and I think if we change any Component it results in long compilation time.  In our C# MVC, we are archaic (in healthcare as you can guess)  and don’t do any D-I.

 

What has been puzzling me for years is that “activator ~run” will launch right away.

 

Then after we fire http://localhost:9000 it does several re-compile cycles, often not just once. It looks as though  the first set outputs more dynamic sources and that in turn emits more sources. Often resulting in hundreds of sources over a one line change.

 

Now we have a bit better ideas.

 

Thanks again for the insights and also my apology for diverting this topic.

 

[Justin du coeur]

On Sat, Oct 7, 2017 at 11:10 AM, Manabu Tokunaga <man...@wingumd.com> wrote:

At any rate, I think part of the compilation (and my complain) time issue for us ties very much in dependencies as you point out. Likely we (over) use Spring DI, and I think if we change any Component it results in long compilation time.  In our C# MVC, we are archaic (in healthcare as you can guess)  and don’t do any D-I.

Hmm.  I've never used Spring per se, but I use Guice a lot in Play nowadays.  Question: is your DI providing an *interface*, or a concrete class?  If you have concrete classes in your DI, that can easily cause the sort of problem you are describing.  I am absolutely religious that the types exposed by DI should always be interfaces, for exactly this reason.

 

Then after we fire http://localhost:9000 it does several re-compile cycles, often not just once. It looks as though  the first set outputs more dynamic sources and that in turn emits more sources. Often resulting in hundreds of sources over a one line change.

Yeah, that can sometimes happen -- one change causes something else to recompile, which forces other dependencies to recompile.  Again, it tends to be due to too many dependencies on concrete classes instead of traits/interfaces.

 

Thanks again for the insights and also my apology for diverting this topic.

No problem, and happy to help.  Hope you can track this down and improve it...

[Jean-François Côté]

People here might be interested to know that I have found what take so much time in my project.

It's the Play Enhancer. The compilation itself is pretty quick for a full build. Then, like for 13-14 minutes, it is the Play Enhancer modifying the bytecode behind the scene. This is unacceptable. So what I did is disabled this completely by adding this in the build.sbt

playEnhancerEnabled := false

Then, the trade-off is that all your Ebean model needs to have private members and getters/setters. Setters are only needed if you do modification to the object.

After adding all getters/setters and changing the calls in you app, you will see a crazy compilation speed.

For us, the compilation that was around 18-19  minute now takes 1m21 for a FULL recompile.

Hot recompile now takes only 1 or 2 seconds compared to 30 to 60 seconds before.

Try it and let me know.

When I will be completely confident in this solution, I will probably go in stack overflow to share the good news and also do an article. I'll post a link here.

Thanks !

[Manabu Tokunaga]

That's insanely faster. We also do have the enhancer on so I ought to try this. We will give this a shot. Time is $.

I've also read elsewhere that people who switched to Graddle had much less issue with compile time. From I recall in the post, SBT performs a lot more dependency scanning or checking and that eats up a lot of time. I have neither confirmed nor tried it, so it could be a fake news.

[Marcos Pereira]

Thanks for reporting that, Jean-François! Curious to see a more detailed article about it.

Best.

On Tue, Jan 9, 2018 at 3:04 PM, Manabu Tokunaga <man...@wingumd.com> wrote:

That's insanely faster. We also do have the enhancer on so I ought to try this. We will give this a shot. Time is $.

I've also read elsewhere that people who switched to Graddle had much less issue with compile time. From I recall in the post, SBT performs a lot more dependency scanning or checking and that eats up a lot of time. I have neither confirmed nor tried it, so it could be a fake news.

--
You received this message because you are subscribed to the Google Groups "Play Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to play-framework+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/CANVzukRZQithnVM%2BkQNa%3D8kq6xXVR%2B97hwCbB8q%2B0DauxAHyrw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcos Pereira

Software Engineer, Lightbend.com

[Jean-François Côté]

Hello again!

Here is my post about the solution: https://medium.com/@jfcote/the-ultimate-solution-to-play-framework-slow-compilation-53f4fd499df4

Enjoy!

Le mardi 16 janvier 2018 08:18:22 UTC-5, Marcos Pereira a écrit :

Thanks for reporting that, Jean-François! Curious to see a more detailed article about it.

Best.

On Tue, Jan 9, 2018 at 3:04 PM, Manabu Tokunaga <man...@wingumd.com> wrote:

That's insanely faster. We also do have the enhancer on so I ought to try this. We will give this a shot. Time is $.

I've also read elsewhere that people who switched to Graddle had much less issue with compile time. From I recall in the post, SBT performs a lot more dependency scanning or checking and that eats up a lot of time. I have neither confirmed nor tried it, so it could be a fake news.

--
You received this message because you are subscribed to the Google Groups "Play Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to play-framewor...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/CANVzukRZQithnVM%2BkQNa%3D8kq6xXVR%2B97hwCbB8q%2B0DauxAHyrw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.