Skip to first unread message
Ezh
Jun 11, 2012, 4:29:49 AM6/11/12
to simple-b...@googlegroups.com
I have small Scala project that compile around minute or two. It is very annoying that CPU usage not more than 40% at i7-2600K. I thought about buying new, more powerful workstation, but after little investigation I am in frustration.
Every build take minute or two... I have thousands builds per release...
Usual timing looks like
real 1m10.248s
user 2m39.056s
sys 0m3.744s
While compilation in progress I got
Tasks: 282 total, 1 running, 281 sleeping, 0 stopped, 0 zombie
Cpu0 : 52.7%us, 2.3%sy, 0.0%ni, 44.3%id, 0.0%wa, 0.7%hi, 0.0%si, 0.0%st
Cpu1 : 60.7%us, 2.0%sy, 0.0%ni, 36.9%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu2 : 90.8%us, 1.3%sy, 0.0%ni, 7.6%id, 0.0%wa, 0.3%hi, 0.0%si, 0.0%st
Cpu3 : 18.0%us, 2.0%sy, 0.0%ni, 79.3%id, 0.0%wa, 0.3%hi, 0.3%si, 0.0%st
Cpu4 : 5.7%us, 0.3%sy, 0.0%ni, 93.6%id, 0.0%wa, 0.3%hi, 0.0%si, 0.0%st
Cpu5 : 16.6%us, 1.3%sy, 0.0%ni, 82.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 8.9%us, 1.0%sy, 0.0%ni, 89.8%id, 0.0%wa, 0.3%hi, 0.0%si, 0.0%st
Cpu7 : 50.5%us, 1.7%sy, 0.0%ni, 47.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
or Cpu(s): 36.6%us, 1.4%sy, 0.0%ni, !!!!61.8%id!!!, 0.0%wa, 0.2%hi, 0.1%si, 0.0%st
Is it possible to fix compilation performance with sbt? Maybe some magic option or key?
parallelExecution in Global := true useless
King regards,
Alexey
Every build take minute or two... I have thousands builds per release...
Usual timing looks like
real 1m10.248s
user 2m39.056s
sys 0m3.744s
While compilation in progress I got
Tasks: 282 total, 1 running, 281 sleeping, 0 stopped, 0 zombie
Cpu0 : 52.7%us, 2.3%sy, 0.0%ni, 44.3%id, 0.0%wa, 0.7%hi, 0.0%si, 0.0%st
Cpu1 : 60.7%us, 2.0%sy, 0.0%ni, 36.9%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu2 : 90.8%us, 1.3%sy, 0.0%ni, 7.6%id, 0.0%wa, 0.3%hi, 0.0%si, 0.0%st
Cpu3 : 18.0%us, 2.0%sy, 0.0%ni, 79.3%id, 0.0%wa, 0.3%hi, 0.3%si, 0.0%st
Cpu4 : 5.7%us, 0.3%sy, 0.0%ni, 93.6%id, 0.0%wa, 0.3%hi, 0.0%si, 0.0%st
Cpu5 : 16.6%us, 1.3%sy, 0.0%ni, 82.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 8.9%us, 1.0%sy, 0.0%ni, 89.8%id, 0.0%wa, 0.3%hi, 0.0%si, 0.0%st
Cpu7 : 50.5%us, 1.7%sy, 0.0%ni, 47.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
or Cpu(s): 36.6%us, 1.4%sy, 0.0%ni, !!!!61.8%id!!!, 0.0%wa, 0.2%hi, 0.1%si, 0.0%st
Is it possible to fix compilation performance with sbt? Maybe some magic option or key?
parallelExecution in Global := true useless
King regards,
Alexey
Eugene Vigdorchik
Jun 11, 2012, 4:47:38 AM6/11/12
to simple-b...@googlegroups.com
run of scala compiler, which is single-threaded. To do more job in
parallel you should have multiple projects, in which case sbt will
schedule multiple compilations at once.
HTH,
Eugene.
>
> parallelExecution in Global := true useless
>
> King regards,
> Alexey
>
> --
> parallelExecution in Global := true useless
>
> King regards,
> Alexey
>
> You received this message because you are subscribed to the Google Groups
> "simple-build-tool" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/simple-build-tool/-/9aZ_344rbakJ.
> To post to this group, send email to simple-b...@googlegroups.com.
> To unsubscribe from this group, send email to
> simple-build-t...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/simple-build-tool?hl=en.
Ezh
Jun 11, 2012, 4:56:18 AM6/11/12
to simple-b...@googlegroups.com
Great idea, thank you very much! I am always split my projects only because of logic. Now I will split them for speed boost ;-) Thank you for sharing!
Eugene Vigdorchik
Jun 11, 2012, 6:41:42 AM6/11/12
to simple-b...@googlegroups.com
On Mon, Jun 11, 2012 at 12:56 PM, Ezh <alexe...@gmail.com> wrote:
> Great idea, thank you very much! I am always split my projects only because
> of logic. Now I will split them for speed boost ;-) Thank you for sharing!
You didn't share the details why you see the same time of the next
> Great idea, thank you very much! I am always split my projects only because
> of logic. Now I will split them for speed boost ;-) Thank you for sharing!
build, and I didn't ask:)
I suppose it's incremental, right?
Eugene.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "simple-build-tool" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/simple-build-tool/-/3Nalb2Mfe-8J.
>
> --
> You received this message because you are subscribed to the Google Groups
> "simple-build-tool" group.
> To view this discussion on the web visit
Ezh
Jun 11, 2012, 6:59:11 AM6/11/12
to simple-b...@googlegroups.com
Unfortunately, this is android project. Scala have very poor android framework support. So I do post compilation bytecode correction that prevent dalvik from blowing up. I use code weaving too. So byte code mess like crazy ;-) Incremental compilation broken most of the time. But a least it is working.
Actually I thought about compilation farm/cluster, something like 'distcc'. But split project by hands is solution too.
понедельник, 11 июня 2012 г., 14:41:42 UTC+4 пользователь Eugene Vigdorchik написал:
Actually I thought about compilation farm/cluster, something like 'distcc'. But split project by hands is solution too.
понедельник, 11 июня 2012 г., 14:41:42 UTC+4 пользователь Eugene Vigdorchik написал:
On Mon, Jun 11, 2012 at 12:56 PM, Ezh <alexe...@gmail.com> wrote:
> Great idea, thank you very much! I am always split my projects only because
> of logic. Now I will split them for speed boost ;-) Thank you for sharing!
You didn't share the details why you see the same time of the next
build, and I didn't ask:)
I suppose it's incremental, right?
Eugene.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "simple-build-tool" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/simple-build-tool/-/3Nalb2Mfe-8J.
>
> To post to this group, send email to simple-build-tool@googlegroups.com.
> To unsubscribe from this group, send email to
> simple-build-tool+unsubscribe@googlegroups.com.
Eugene Vigdorchik
Jun 17, 2012, 8:11:58 AM6/17/12
to simple-b...@googlegroups.com
On Mon, Jun 11, 2012 at 2:59 PM, Ezh <alexe...@gmail.com> wrote:
> Unfortunately, this is android project. Scala have very poor android
> framework support. So I do post compilation bytecode correction that prevent
> dalvik from blowing up. I use code weaving too. So byte code mess like crazy
> ;-) Incremental compilation broken most of the time. But a least it is
> working.
I'd love to know more about the instrumentation you are doing. If it
> Unfortunately, this is android project. Scala have very poor android
> framework support. So I do post compilation bytecode correction that prevent
> dalvik from blowing up. I use code weaving too. So byte code mess like crazy
> ;-) Incremental compilation broken most of the time. But a least it is
> working.
uses only the information local to the class itself, then it should be
possible to do it post-compile and preserve incremental compilation.
Otherwise, if you know the set of classes you instrumentation affects
given the changes for the next compilation, then you should be able to
provide a selective clean, that should still be faster than full
rebuild.
Eugene.
>
> Actually I thought about compilation farm/cluster, something like 'distcc'.
> But split project by hands is solution too.
>
> понедельник, 11 июня 2012 г., 14:41:42 UTC+4 пользователь Eugene Vigdorchik
> написал:
>>
>> On Mon, Jun 11, 2012 at 12:56 PM, Ezh <alexe...@gmail.com> wrote:
>> > Great idea, thank you very much! I am always split my projects only
>> > because
>> > of logic. Now I will split them for speed boost ;-) Thank you for
>> > sharing!
>> You didn't share the details why you see the same time of the next
>> build, and I didn't ask:)
>> I suppose it's incremental, right?
>>
>> Eugene.
>> >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "simple-build-tool" group.
>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msg/simple-build-tool/-/3Nalb2Mfe-8J.
>> >
>> > To unsubscribe from this group, send email to
>> > simple-build-t...@googlegroups.com.
>> > For more options, visit this group at
>> > http://groups.google.com/group/simple-build-tool?hl=en.
>
>> > http://groups.google.com/group/simple-build-tool?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "simple-build-tool" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/simple-build-tool/-/1te1NgAZ6AAJ.
> You received this message because you are subscribed to the Google Groups
> "simple-build-tool" group.
> To view this discussion on the web visit
>
> To post to this group, send email to simple-b...@googlegroups.com.
> To unsubscribe from this group, send email to
> simple-build-t...@googlegroups.com.
Alexey Aksenov
Jun 17, 2012, 9:45:38 AM6/17/12
to simple-b...@googlegroups.com
At first stage I use aspectj weaving.
At second stage I use org.objectweb.asm
I change not only class files but nested jars too. Class modification follow internal patterns. I don't know how many classes modified. A lot.
Also I use a pack of sbt plugins (for example com.typesafe.sbtmultijvm to fork jvm for some types of tests).
It give me unusual sporadic side effects all together. Especially while testing.
I have no time to deal with it. I must concentrate on my primary task.
IMHO If write some sbt unit tests, discover pluging chain, the problem was gone. But I have no time. My build system more complex than code that I write :-(
If you interested you may find "DigiControl" at android market for fun (install, deinstall). It is in early alpha stage, but it is working!
Anyway thank you for idea. It is much faster than spend time for setup compilation cluster.
At second stage I use org.objectweb.asm
I change not only class files but nested jars too. Class modification follow internal patterns. I don't know how many classes modified. A lot.
Also I use a pack of sbt plugins (for example com.typesafe.sbtmultijvm to fork jvm for some types of tests).
It give me unusual sporadic side effects all together. Especially while testing.
I have no time to deal with it. I must concentrate on my primary task.
IMHO If write some sbt unit tests, discover pluging chain, the problem was gone. But I have no time. My build system more complex than code that I write :-(
If you interested you may find "DigiControl" at android market for fun (install, deinstall). It is in early alpha stage, but it is working!
Anyway thank you for idea. It is much faster than spend time for setup compilation cluster.
2012/6/17 Eugene Vigdorchik <eugene.v...@gmail.com>
Reply all
Reply to author
Forward
0 new messages