I don't have lot of memory (4Go), but it's enough to daily code in java with several projects,... But it's hell when I work on scala projects (including or.scala-ide.sdt.core), to be able to spawn/debug eclipse plugin I limit my eclipse IDE to 1024m. But I raise limit quick limit, I've got lot freeze,... so I started to "profile" and search : * from mat (MemoryAnalyzer) the 2 main leak are ScalaPresentationCompiler and EclipseBuildManager * with visualvm rapid analyse, I saw that freeze are due to lot of GC activity - can 40% - 80% CPU.
I try to investigate into code. I replace the ScalaPresenrtationCompiler cache (mutable.HashMap[ScalaCompilationUnit, CachedCompilerResult]) (in some analyze there is 128 entries) by I simplist tuple, and now my IDE play yo-yo with memory (with high amplitude I win ~ 100 Mo) but currently I no longer have high GC activity.
I don't know if it's the right solution, but I would like to add a PreferencePage "plugin tuning" where we could store configuration about some strategies, experimental solutions. In my case, I'll start with something like a check box for "cache AST" that allow to use HashMap or my simple store (code bellow) after restart. I also think it will be the good place for other/future customization (like statistics generation, ...).
WDYT ?
/davidB
-----8<-------------------------------------------------------------------- --- private val results = new Object{ private var _kv : (ScalaCompilationUnit,CachedCompilerResult) = null def apply(k : ScalaCompilationUnit) = { val kv = _kv if (kv == null || kv._1 != k) { val v = new CachedCompilerResult(k) _kv = (k, v) v } else { kv._2 } }
I like the idea of improving the performance and reducing the memory
footprint. I believe, your solution reduces performance, although it
also uses less memory. A good tradeoff would be to use a map with soft
keys (SoftReference), so objects could be released when free memory
gets low. Unfortunately, the JDK only contains a WeakHashMap and not a
SoftHashMap. So, you may have to implement it yourself. Just for
testing purpose you could use Apache Commons Collections' ReferenceMap
and see if it really would help.
Michel
On Jul 13, 4:07 pm, David Bernard <david.bernard...@gmail.com> wrote:
> I don't have lot of memory (4Go), but it's enough to daily code in
> java with several projects,... But it's hell when I work on scala
> projects (including or.scala-ide.sdt.core), to be able to spawn/debug
> eclipse plugin I limit my eclipse IDE to 1024m.
> But I raise limit quick limit, I've got lot freeze,... so I started to
> "profile" and search :
> * from mat (MemoryAnalyzer) the 2 main leak are
> ScalaPresentationCompiler and EclipseBuildManager
> * with visualvm rapid analyse, I saw that freeze are due to lot of GC
> activity - can 40% - 80% CPU.
> I try to investigate into code. I replace the
> ScalaPresenrtationCompiler cache
> (mutable.HashMap[ScalaCompilationUnit, CachedCompilerResult]) (in some
> analyze there is 128 entries) by I simplist tuple, and now my IDE play
> yo-yo with memory (with high amplitude I win ~ 100 Mo) but currently I
> no longer have high GC activity.
> I don't know if it's the right solution, but I would like to add a
> PreferencePage "plugin tuning" where we could store configuration
> about some strategies, experimental solutions. In my case, I'll start
> with something like a check box for "cache AST" that allow to use
> HashMap or my simple store (code bellow) after restart. I also think
> it will be the good place for other/future customization (like
> statistics generation, ...).
> WDYT ?
> /davidB
> -----8<-------------------------------------------------------------------- ---
> private val results = new Object{
> private var _kv : (ScalaCompilationUnit,CachedCompilerResult) = null
> def apply(k : ScalaCompilationUnit) = {
> val kv = _kv
> if (kv == null || kv._1 != k) {
> val v = new CachedCompilerResult(k)
> _kv = (k, v)
> v
> } else {
> kv._2
> }
> }
We are currently using Scala in Eclipse for a large OSGi project. This
means we have approximately 50 projects in our workspace of which 35
are Scala projects. We have 8GB in one machine and 4GB in another
machine.
Even when we set -Xmx=2048M it starts out ok, but after a while gc
starts using all CPU power. Eventually you will have to restart
Eclipse, and if you wait too long, it is not even possible to quit/
restart Eclipse as it stops responding.
Without having analyzed anything, it definitely looks like something
in the Scala IDE is leaking memory.
We have a similar sized OSGi project with Java only which runs fine on
the same machines and which doesn't leak memory.
Currently , from my user experience point of view, performance are better (less freeze), but it's probably because I have less and shorter GC activity. I have done a SoftReferenceCacheMap (SoftReference for value) in scala 2 years ago. I'll try to combine it with the cache for only the latest entry (should not be removable, to avoid eviction and recomputation,...) In fact I would like to use something like ehcache to replace all those cache (and adjust strategy , size,... of thoses cache).
@Martin, Michel
Thanks for feedbacks.
/davidB
PS: As I'm a newbee in Eclipse developement (and Scala-ide), have a soft desktop ;-) (I bought a SSD HD to improve, not yet installed). I need lot of time to investigate, try, experiment so more time to have public effect.
> I like the idea of improving the performance and reducing the memory > footprint. I believe, your solution reduces performance, although it > also uses less memory. A good tradeoff would be to use a map with soft > keys (SoftReference), so objects could be released when free memory > gets low. Unfortunately, the JDK only contains a WeakHashMap and not a > SoftHashMap. So, you may have to implement it yourself. Just for > testing purpose you could use Apache Commons Collections' ReferenceMap > and see if it really would help.
> Michel
> On Jul 13, 4:07 pm, David Bernard <david.bernard...@gmail.com> wrote: >> Hi,
>> I don't have lot of memory (4Go), but it's enough to daily code in >> java with several projects,... But it's hell when I work on scala >> projects (including or.scala-ide.sdt.core), to be able to spawn/debug >> eclipse plugin I limit my eclipse IDE to 1024m. >> But I raise limit quick limit, I've got lot freeze,... so I started to >> "profile" and search : >> * from mat (MemoryAnalyzer) the 2 main leak are >> ScalaPresentationCompiler and EclipseBuildManager >> * with visualvm rapid analyse, I saw that freeze are due to lot of GC >> activity - can 40% - 80% CPU.
>> I try to investigate into code. I replace the >> ScalaPresenrtationCompiler cache >> (mutable.HashMap[ScalaCompilationUnit, CachedCompilerResult]) (in some >> analyze there is 128 entries) by I simplist tuple, and now my IDE play >> yo-yo with memory (with high amplitude I win ~ 100 Mo) but currently I >> no longer have high GC activity.
>> I don't know if it's the right solution, but I would like to add a >> PreferencePage "plugin tuning" where we could store configuration >> about some strategies, experimental solutions. In my case, I'll start >> with something like a check box for "cache AST" that allow to use >> HashMap or my simple store (code bellow) after restart. I also think >> it will be the good place for other/future customization (like >> statistics generation, ...).
>> WDYT ?
>> /davidB
>> -----8<-------------------------------------------------------------------- --- >> private val results = new Object{ >> private var _kv : (ScalaCompilationUnit,CachedCompilerResult) = null >> def apply(k : ScalaCompilationUnit) = { >> val kv = _kv >> if (kv == null || kv._1 != k) { >> val v = new CachedCompilerResult(k) >> _kv = (k, v) >> v >> } else { >> kv._2 >> } >> }
I welcome every attempt at improving performance; I've now worked several months with the Scala IDE and before that many years with Eclipse - and I never had to restart Eclipse so much as now with the Scala plugin..
On Tuesday 13 July 2010 16:07:53 David Bernard wrote:
> -----8<-------------------------------------------------------------------- > --- private val results = new Object{
What type does results have now? Won't this lead to the same problems -- that a structural type is created -- mentioned here: http://www.scala- notes.org/2010/06/avoid-structural-types-when-pimping-libraries/ ? I'm not sure how much it actually matters, but when we're already optimizing...
Eg when my eclipse start to freeze 1. I start a visualvm (could be download alone or use the one provide in jdk 1.6.0_21) 2. I open the monitor of the eclipse process 3. I wait few second, take a screenshot (see attachment)
for gc and memory issue I also do a Heap dump (and save it for postmordem analyze with VisualVM or Mat).
@Mirko
In my local wip_tuning branch I replace new Object{...} by named classe. the link you provide seems to be invalid.
On Wed, Jul 14, 2010 at 10:51, Mirko Stocker <m...@misto.ch> wrote: > Hi David,
> I welcome every attempt at improving performance; I've now worked several > months with the Scala IDE and before that many years with Eclipse - and I > never had to restart Eclipse so much as now with the Scala plugin..
> On Tuesday 13 July 2010 16:07:53 David Bernard wrote: >> -----8<-------------------------------------------------------------------- >> --- private val results = new Object{
> What type does results have now? Won't this lead to the same problems -- that > a structural type is created -- mentioned here: http://www.scala- > notes.org/2010/06/avoid-structural-types-when-pimping-libraries/ ? I'm not > sure how much it actually matters, but when we're already optimizing...
In case, someone who want to give a try to my wip_tuning (note that it is based on tycho-reorg branch): update site (include full scala-ide plugin) : http://alchim31.net/scala-ide/update-wip_tuning
On Thu, Jul 15, 2010 at 00:30, David Bernard <david.bernard...@gmail.com> wrote: > I'm interesting by some data.
> Eg when my eclipse start to freeze > 1. I start a visualvm (could be download alone or use the one provide > in jdk 1.6.0_21) > 2. I open the monitor of the eclipse process > 3. I wait few second, take a screenshot (see attachment)
> for gc and memory issue I also do a Heap dump (and save it for > postmordem analyze with VisualVM or Mat).
> @Mirko
> In my local wip_tuning branch I replace new Object{...} by named classe. > the link you provide seems to be invalid.
> /davidB
> On Wed, Jul 14, 2010 at 10:51, Mirko Stocker <m...@misto.ch> wrote: >> Hi David,
>> I welcome every attempt at improving performance; I've now worked several >> months with the Scala IDE and before that many years with Eclipse - and I >> never had to restart Eclipse so much as now with the Scala plugin..
>> On Tuesday 13 July 2010 16:07:53 David Bernard wrote: >>> -----8<-------------------------------------------------------------------- >>> --- private val results = new Object{
>> What type does results have now? Won't this lead to the same problems -- that >> a structural type is created -- mentioned here: http://www.scala- >> notes.org/2010/06/avoid-structural-types-when-pimping-libraries/ ? I'm not >> sure how much it actually matters, but when we're already optimizing...
> In case, someone who want to give a try to my wip_tuning (note that it
> is based on tycho-reorg branch):
> update site (include full scala-ide plugin) :http://alchim31.net/scala-ide/update-wip_tuning
> On Thu, Jul 15, 2010 at 00:30, David Bernard <david.bernard...@gmail.com> wrote:
> > I'm interesting by some data.
> > Eg when my eclipse start to freeze
> > 1. I start a visualvm (could be download alone or use the one provide
> > in jdk 1.6.0_21)
> > 2. I open the monitor of the eclipse process
> > 3. I wait few second, take a screenshot (see attachment)
> > for gc and memory issue I also do a Heap dump (and save it for
> > postmordem analyze with VisualVM or Mat).
> > @Mirko
> > In my local wip_tuning branch I replace new Object{...} by named classe.
> > the link you provide seems to be invalid.
> > /davidB
> > On Wed, Jul 14, 2010 at 10:51, Mirko Stocker <m...@misto.ch> wrote:
> >> Hi David,
> >> I welcome every attempt at improving performance; I've now worked several
> >> months with the Scala IDE and before that many years with Eclipse - and I
> >> never had to restart Eclipse so much as now with the Scala plugin..
> >> On Tuesday 13 July 2010 16:07:53 David Bernard wrote:
> >>> -----8<--------------------------------------------------------------------
> >>> --- private val results = new Object{
> >> What type does results have now? Won't this lead to the same problems -- that
> >> a structural type is created -- mentioned here:http://www.scala- > >> notes.org/2010/06/avoid-structural-types-when-pimping-libraries/ ? I'm not
> >> sure how much it actually matters, but when we're already optimizing...
updates includes : * re-sync with last works on tycho-reorg branch * replace home made SoftHashMap by a WeakSoftHashMap build with google-collections MapMaker to cache/memoize result of Presentation compilation * (start) integrate StopWatch (form alex Boisvert) to count/mesure some compilations (from builder and presentation) by exemple on test project, writing : def foo() = "hhhh"[enter] generate 10 compilation (I tried to call to code completion), with the default configuration (If I understand right, JDT call completion engine every 200ms if there is typing)
In this version, stopwatch are always enabled, to disable or to see data collected open your browser (or eclipse internal browser) to http://localhost:9999/
Note that as I require 2 additional plugins (stopwatch who is provide by same update-site as scala-ide and google-collections provide by orbit update (I hope you don't need to declare it)). tell me if you have issue to install this version.
> On Jul 15, 1:19 am, David Bernard <david.bernard...@gmail.com> wrote: >> In case, someone who want to give a try to my wip_tuning (note that it >> is based on tycho-reorg branch): >> update site (include full scala-ide plugin) :http://alchim31.net/scala-ide/update-wip_tuning
>> On Thu, Jul 15, 2010 at 00:30, David Bernard <david.bernard...@gmail.com> wrote: >> > I'm interesting by some data.
>> > Eg when my eclipse start to freeze >> > 1. I start a visualvm (could be download alone or use the one provide >> > in jdk 1.6.0_21) >> > 2. I open the monitor of the eclipse process >> > 3. I wait few second, take a screenshot (see attachment)
>> > for gc and memory issue I also do a Heap dump (and save it for >> > postmordem analyze with VisualVM or Mat).
>> > @Mirko
>> > In my local wip_tuning branch I replace new Object{...} by named classe. >> > the link you provide seems to be invalid.
>> > /davidB
>> > On Wed, Jul 14, 2010 at 10:51, Mirko Stocker <m...@misto.ch> wrote: >> >> Hi David,
>> >> I welcome every attempt at improving performance; I've now worked several >> >> months with the Scala IDE and before that many years with Eclipse - and I >> >> never had to restart Eclipse so much as now with the Scala plugin..
>> >> On Tuesday 13 July 2010 16:07:53 David Bernard wrote: >> >>> -----8<-------------------------------------------------------------------- >> >>> --- private val results = new Object{
>> >> What type does results have now? Won't this lead to the same problems -- that >> >> a structural type is created -- mentioned here:http://www.scala- >> >> notes.org/2010/06/avoid-structural-types-when-pimping-libraries/ ? I'm not >> >> sure how much it actually matters, but when we're already optimizing...
Hi, David, interesting branch. as my recent profilings, the main performance problem is that the threads using the SynVar are not collaborative and the main UI thread is usually blocked shortly or longly. Martin's new ask method seem the way to rescue this.(In method is in the trunk, not the 2.8.0.final) For memory side, I have not seen the apparent full-gc activities when the Xmx be set to 1024m.
2010/7/20 David Bernard <david.bernard...@gmail.com>
> updates includes : > * re-sync with last works on tycho-reorg branch > * replace home made SoftHashMap by a WeakSoftHashMap build with > google-collections MapMaker to cache/memoize result of Presentation > compilation > * (start) integrate StopWatch (form alex Boisvert) to count/mesure > some compilations (from builder and presentation) > by exemple on test project, writing : def foo() = "hhhh"[enter] > generate 10 compilation (I tried to call to code completion), with the > default configuration (If I understand right, JDT call completion > engine every 200ms if there is typing)
> In this version, stopwatch are always enabled, to disable or to see > data collected open your browser (or eclipse internal browser) to > http://localhost:9999/
> Note that as I require 2 additional plugins (stopwatch who is provide > by same update-site as scala-ide and google-collections provide by > orbit update (I hope you don't need to declare it)). tell me if you > have issue to install this version.
> /davidB
> On Fri, Jul 16, 2010 at 11:29, David Bernard <david.bernard...@gmail.com> > wrote: > > Hello,
> > On Jul 15, 1:19 am, David Bernard <david.bernard...@gmail.com> wrote: > >> In case, someone who want to give a try to my wip_tuning (note that it > >> is based on tycho-reorg branch): > >> update site (include full scala-ide plugin) : > http://alchim31.net/scala-ide/update-wip_tuning
> >> or just the plugin jarhttp:// > alchim31.net/scala-ide/update-wip_tuning/plugins/org.scala-ide...
> >> /davidB
> >> On Thu, Jul 15, 2010 at 00:30, David Bernard < > david.bernard...@gmail.com> wrote: > >> > I'm interesting by some data.
> >> > Eg when my eclipse start to freeze > >> > 1. I start a visualvm (could be download alone or use the one provide > >> > in jdk 1.6.0_21) > >> > 2. I open the monitor of the eclipse process > >> > 3. I wait few second, take a screenshot (see attachment)
> >> > for gc and memory issue I also do a Heap dump (and save it for > >> > postmordem analyze with VisualVM or Mat).
> >> > @Mirko
> >> > In my local wip_tuning branch I replace new Object{...} by named > classe. > >> > the link you provide seems to be invalid.
> >> > /davidB
> >> > On Wed, Jul 14, 2010 at 10:51, Mirko Stocker <m...@misto.ch> wrote: > >> >> Hi David,
> >> >> I welcome every attempt at improving performance; I've now worked > several > >> >> months with the Scala IDE and before that many years with Eclipse - > and I > >> >> never had to restart Eclipse so much as now with the Scala plugin..
> >> >> On Tuesday 13 July 2010 16:07:53 David Bernard wrote:
> -----8<-------------------------------------------------------------------- > >> >>> --- private val results = new Object{
> >> >> What type does results have now? Won't this lead to the same problems > -- that > >> >> a structural type is created -- mentioned here:http://www.scala- > >> >> notes.org/2010/06/avoid-structural-types-when-pimping-libraries/ ? > I'm not > >> >> sure how much it actually matters, but when we're already > optimizing...
I have Xmx set to 1g (my full eclipse.ini below, may be I've got some misconfiguration). When I work on org.scala-ide.sdt.core my used memory (is ~ 750MB from The eclipse indicator (enabled via Prefences > General)). If i go into an other scala enabled project then when eclipse "build workspace" I've got more freeze And If I display visualvm I can see increase of GC Activy (see screenshot) until full freeze (like Martin Gamwell Dawids).
I also note some freeze due with usage of SyncVar, but as I explain, they could be reduced (in quantity) by reducing call to ScalaCompletionEngine (call too frequencly and generate unsuable AST for code completion suggestion, eg : (in sample pipe char is use to mark caret position) "bablbla".| <-- some code suggestion proposed "bablbla".to| <-- no code suggestion "bablbla".|to <-- no code suggestion (I expect to have as first case) )
On the WIP_Tuning page I start listing proposal, idea to improve,... I thinks we (developpers and users) could start a page on User Documentation part about some "eclipse tuning suggestion" about performance, usage,... (with preferences configuration or usefull complementary plugins to install)
DISCLAIMER : I'm not a optimization and profiling expert
On Tue, Jul 20, 2010 at 04:13, Jin Mingjian <jin....@gmail.com> wrote: > Hi, David, interesting branch. as my recent profilings, the main performance > problem is that the threads using the SynVar are not collaborative and the > main UI thread is usually blocked shortly or longly. Martin's new ask method > seem the way to rescue this.(In method is in the trunk, not the 2.8.0.final) > For memory side, I have not seen the apparent full-gc activities when the > Xmx be set to 1024m.
> 2010/7/20 David Bernard <david.bernard...@gmail.com>
>> updates includes : >> * re-sync with last works on tycho-reorg branch >> * replace home made SoftHashMap by a WeakSoftHashMap build with >> google-collections MapMaker to cache/memoize result of Presentation >> compilation >> * (start) integrate StopWatch (form alex Boisvert) to count/mesure >> some compilations (from builder and presentation) >> by exemple on test project, writing : def foo() = "hhhh"[enter] >> generate 10 compilation (I tried to call to code completion), with the >> default configuration (If I understand right, JDT call completion >> engine every 200ms if there is typing)
>> In this version, stopwatch are always enabled, to disable or to see >> data collected open your browser (or eclipse internal browser) to >> http://localhost:9999/
>> Note that as I require 2 additional plugins (stopwatch who is provide >> by same update-site as scala-ide and google-collections provide by >> orbit update (I hope you don't need to declare it)). tell me if you >> have issue to install this version.
>> /davidB
>> On Fri, Jul 16, 2010 at 11:29, David Bernard <david.bernard...@gmail.com> >> wrote: >> > Hello,
>> > On Jul 15, 1:19 am, David Bernard <david.bernard...@gmail.com> wrote: >> >> In case, someone who want to give a try to my wip_tuning (note that it >> >> is based on tycho-reorg branch): >> >> update site (include full scala-ide plugin) >> >> :http://alchim31.net/scala-ide/update-wip_tuning
>> >> On Thu, Jul 15, 2010 at 00:30, David Bernard >> >> <david.bernard...@gmail.com> wrote: >> >> > I'm interesting by some data.
>> >> > Eg when my eclipse start to freeze >> >> > 1. I start a visualvm (could be download alone or use the one provide >> >> > in jdk 1.6.0_21) >> >> > 2. I open the monitor of the eclipse process >> >> > 3. I wait few second, take a screenshot (see attachment)
>> >> > for gc and memory issue I also do a Heap dump (and save it for >> >> > postmordem analyze with VisualVM or Mat).
>> >> > @Mirko
>> >> > In my local wip_tuning branch I replace new Object{...} by named >> >> > classe. >> >> > the link you provide seems to be invalid.
>> >> > /davidB
>> >> > On Wed, Jul 14, 2010 at 10:51, Mirko Stocker <m...@misto.ch> wrote: >> >> >> Hi David,
>> >> >> I welcome every attempt at improving performance; I've now worked >> >> >> several >> >> >> months with the Scala IDE and before that many years with Eclipse - >> >> >> and I >> >> >> never had to restart Eclipse so much as now with the Scala plugin..
>> >> >> On Tuesday 13 July 2010 16:07:53 David Bernard wrote:
>> >> >>> -----8<-------------------------------------------------------------------- >> >> >>> --- private val results = new Object{
>> >> >> What type does results have now? Won't this lead to the same >> >> >> problems -- that >> >> >> a structural type is created -- mentioned here:http://www.scala- >> >> >> notes.org/2010/06/avoid-structural-types-when-pimping-libraries/ ? >> >> >> I'm not >> >> >> sure how much it actually matters, but when we're already >> >> >> optimizing...
> I have Xmx set to 1g (my full eclipse.ini below, may be I've got some > misconfiguration). When I work on org.scala-ide.sdt.core my used > memory (is ~ 750MB from The eclipse indicator (enabled via Prefences > > General)). > If i go into an other scala enabled project then when eclipse "build > workspace" I've got more freeze And If I display visualvm I can see > increase of GC Activy (see screenshot) until full freeze (like Martin > Gamwell Dawids).
> I also note some freeze due with usage of SyncVar, but as I explain, > they could be reduced (in quantity) by reducing call to > ScalaCompletionEngine (call too frequencly and generate unsuable AST > for code completion suggestion, eg : (in sample pipe char is use to > mark caret position) > "bablbla".| <-- some code suggestion proposed > "bablbla".to| <-- no code suggestion > "bablbla".|to <-- no code suggestion (I expect to have as first case) > )
> On the WIP_Tuning page I start listing proposal, idea to improve,... > I thinks we (developpers and users) could start a page on User > Documentation part about some "eclipse tuning suggestion" about > performance, usage,... (with preferences configuration or usefull > complementary plugins to install)
> DISCLAIMER : I'm not a optimization and profiling expert
> On Tue, Jul 20, 2010 at 04:13, Jin Mingjian <jin....@gmail.com> wrote: > > Hi, David, interesting branch. as my recent profilings, the main > performance > > problem is that the threads using the SynVar are not collaborative and > the > > main UI thread is usually blocked shortly or longly. Martin's new ask > method > > seem the way to rescue this.(In method is in the trunk, not the > 2.8.0.final) > > For memory side, I have not seen the apparent full-gc activities when the > > Xmx be set to 1024m.
> > 2010/7/20 David Bernard <david.bernard...@gmail.com>
> >> updates includes : > >> * re-sync with last works on tycho-reorg branch > >> * replace home made SoftHashMap by a WeakSoftHashMap build with > >> google-collections MapMaker to cache/memoize result of Presentation > >> compilation > >> * (start) integrate StopWatch (form alex Boisvert) to count/mesure > >> some compilations (from builder and presentation) > >> by exemple on test project, writing : def foo() = "hhhh"[enter] > >> generate 10 compilation (I tried to call to code completion), with the > >> default configuration (If I understand right, JDT call completion > >> engine every 200ms if there is typing)
> >> In this version, stopwatch are always enabled, to disable or to see > >> data collected open your browser (or eclipse internal browser) to > >> http://localhost:9999/
> >> Note that as I require 2 additional plugins (stopwatch who is provide > >> by same update-site as scala-ide and google-collections provide by > >> orbit update (I hope you don't need to declare it)). tell me if you > >> have issue to install this version.
> >> /davidB
> >> On Fri, Jul 16, 2010 at 11:29, David Bernard < > david.bernard...@gmail.com> > >> wrote: > >> > Hello,
> >> > On Jul 15, 1:19 am, David Bernard <david.bernard...@gmail.com> wrote: > >> >> In case, someone who want to give a try to my wip_tuning (note that > it > >> >> is based on tycho-reorg branch): > >> >> update site (include full scala-ide plugin) > >> >> :http://alchim31.net/scala-ide/update-wip_tuning
> >> >> or just the plugin > >> >> jarhttp:// > alchim31.net/scala-ide/update-wip_tuning/plugins/org.scala-ide...
> >> >> /davidB
> >> >> On Thu, Jul 15, 2010 at 00:30, David Bernard > >> >> <david.bernard...@gmail.com> wrote: > >> >> > I'm interesting by some data.
> >> >> > Eg when my eclipse start to freeze > >> >> > 1. I start a visualvm (could be download alone or use the one > provide > >> >> > in jdk 1.6.0_21) > >> >> > 2. I open the monitor of the eclipse process > >> >> > 3. I wait few second, take a screenshot (see attachment)
> >> >> > for gc and memory issue I also do a Heap dump (and save it for > >> >> > postmordem analyze with VisualVM or Mat).
> >> >> > @Mirko
> >> >> > In my local wip_tuning branch I replace new Object{...} by named > >> >> > classe. > >> >> > the link you provide seems to be invalid.
> >> >> > /davidB
> >> >> > On Wed, Jul 14, 2010 at 10:51, Mirko Stocker <m...@misto.ch> > wrote: > >> >> >> Hi David,
> >> >> >> I welcome every attempt at improving performance; I've now worked > >> >> >> several > >> >> >> months with the Scala IDE and before that many years with Eclipse > - > >> >> >> and I > >> >> >> never had to restart Eclipse so much as now with the Scala > plugin..
> >> >> >> On Tuesday 13 July 2010 16:07:53 David Bernard wrote:
> -----8<-------------------------------------------------------------------- > >> >> >>> --- private val results = new Object{
> >> >> >> What type does results have now? Won't this lead to the same > >> >> >> problems -- that > >> >> >> a structural type is created -- mentioned here:http://www.scala- > >> >> >> notes.org/2010/06/avoid-structural-types-when-pimping-libraries/? > >> >> >> I'm not > >> >> >> sure how much it actually matters, but when we're already > >> >> >> optimizing...
Hi, David. I find you are much right in the weekend. I do a heavy use on apsectj in this week. I find the I need to reboot the IDE after saving aspectj sources 30-50 times. The current ajdt(or ajdt+sdt?) seemly have serious memory leaking problem(there are many garbage can not be collected). Very terrible.
Jin
2010/7/20 David Bernard <david.bernard...@gmail.com>
> I have Xmx set to 1g (my full eclipse.ini below, may be I've got some > misconfiguration). When I work on org.scala-ide.sdt.core my used > memory (is ~ 750MB from The eclipse indicator (enabled via Prefences > > General)). > If i go into an other scala enabled project then when eclipse "build > workspace" I've got more freeze And If I display visualvm I can see > increase of GC Activy (see screenshot) until full freeze (like Martin > Gamwell Dawids).
> I also note some freeze due with usage of SyncVar, but as I explain, > they could be reduced (in quantity) by reducing call to > ScalaCompletionEngine (call too frequencly and generate unsuable AST > for code completion suggestion, eg : (in sample pipe char is use to > mark caret position) > "bablbla".| <-- some code suggestion proposed > "bablbla".to| <-- no code suggestion > "bablbla".|to <-- no code suggestion (I expect to have as first case) > )
> On the WIP_Tuning page I start listing proposal, idea to improve,... > I thinks we (developpers and users) could start a page on User > Documentation part about some "eclipse tuning suggestion" about > performance, usage,... (with preferences configuration or usefull > complementary plugins to install)
> DISCLAIMER : I'm not a optimization and profiling expert
> On Tue, Jul 20, 2010 at 04:13, Jin Mingjian <jin....@gmail.com> wrote: > > Hi, David, interesting branch. as my recent profilings, the main > performance > > problem is that the threads using the SynVar are not collaborative and > the > > main UI thread is usually blocked shortly or longly. Martin's new ask > method > > seem the way to rescue this.(In method is in the trunk, not the > 2.8.0.final) > > For memory side, I have not seen the apparent full-gc activities when the > > Xmx be set to 1024m.
> > 2010/7/20 David Bernard <david.bernard...@gmail.com>
> >> updates includes : > >> * re-sync with last works on tycho-reorg branch > >> * replace home made SoftHashMap by a WeakSoftHashMap build with > >> google-collections MapMaker to cache/memoize result of Presentation > >> compilation > >> * (start) integrate StopWatch (form alex Boisvert) to count/mesure > >> some compilations (from builder and presentation) > >> by exemple on test project, writing : def foo() = "hhhh"[enter] > >> generate 10 compilation (I tried to call to code completion), with the > >> default configuration (If I understand right, JDT call completion > >> engine every 200ms if there is typing)
> >> In this version, stopwatch are always enabled, to disable or to see > >> data collected open your browser (or eclipse internal browser) to > >> http://localhost:9999/
> >> Note that as I require 2 additional plugins (stopwatch who is provide > >> by same update-site as scala-ide and google-collections provide by > >> orbit update (I hope you don't need to declare it)). tell me if you > >> have issue to install this version.
> >> /davidB
> >> On Fri, Jul 16, 2010 at 11:29, David Bernard < > david.bernard...@gmail.com> > >> wrote: > >> > Hello,
> >> > On Jul 15, 1:19 am, David Bernard <david.bernard...@gmail.com> wrote: > >> >> In case, someone who want to give a try to my wip_tuning (note that > it > >> >> is based on tycho-reorg branch): > >> >> update site (include full scala-ide plugin) > >> >> :http://alchim31.net/scala-ide/update-wip_tuning
> >> >> or just the plugin > >> >> jarhttp:// > alchim31.net/scala-ide/update-wip_tuning/plugins/org.scala-ide...
> >> >> /davidB
> >> >> On Thu, Jul 15, 2010 at 00:30, David Bernard > >> >> <david.bernard...@gmail.com> wrote: > >> >> > I'm interesting by some data.
> >> >> > Eg when my eclipse start to freeze > >> >> > 1. I start a visualvm (could be download alone or use the one > provide > >> >> > in jdk 1.6.0_21) > >> >> > 2. I open the monitor of the eclipse process > >> >> > 3. I wait few second, take a screenshot (see attachment)
> >> >> > for gc and memory issue I also do a Heap dump (and save it for > >> >> > postmordem analyze with VisualVM or Mat).
> >> >> > @Mirko
> >> >> > In my local wip_tuning branch I replace new Object{...} by named > >> >> > classe. > >> >> > the link you provide seems to be invalid.
> >> >> > /davidB
> >> >> > On Wed, Jul 14, 2010 at 10:51, Mirko Stocker <m...@misto.ch> > wrote: > >> >> >> Hi David,
> >> >> >> I welcome every attempt at improving performance; I've now worked > >> >> >> several > >> >> >> months with the Scala IDE and before that many years with Eclipse > - > >> >> >> and I > >> >> >> never had to restart Eclipse so much as now with the Scala > plugin..
> >> >> >> On Tuesday 13 July 2010 16:07:53 David Bernard wrote:
> -----8<-------------------------------------------------------------------- > >> >> >>> --- private val results = new Object{
> >> >> >> What type does results have now? Won't this lead to the same > >> >> >> problems -- that > >> >> >> a structural type is created -- mentioned here:http://www.scala- > >> >> >> notes.org/2010/06/avoid-structural-types-when-pimping-libraries/? > >> >> >> I'm not > >> >> >> sure how much it actually matters, but when we're already > >> >> >> optimizing...
I think that the improvement of the speed of the plugin is essential
for its acceptance and thereby the efforts for plugin tuning will pay
off.
I also have a small contribution that optimizes the plugin (it
actually speeds up the building of the top-map for the project files)
Basically, my improvement is to replace a line from
ScalaProject.topLevelMap
I propose the line:
topLevelMap0.update(allSourceFiles)
In the original version (the first one) the methods TopLevelMap.update
and, in turn, TopLevelMap.parse are called once for each file. This is
highly inefficient!!!
Below is a complete patch (it comprises small modifications of the
files ScalaProject and TopLevelMap)
According to my measurements, the speed improvement in building the
"top-level map" is around 10%.
Cheers,
Daniel
------------------ patch start
diff --git a/org.scala-ide.sdt.core/src/scala/tools/eclipse/
ScalaProject.scala b/org.scala-ide.sdt.core/src/scala/tools/eclipse/
ScalaProject.scala
index 20ef850..e9036fe 100644
--- a/org.scala-ide.sdt.core/src/scala/tools/eclipse/
ScalaProject.scala
+++ b/org.scala-ide.sdt.core/src/scala/tools/eclipse/
ScalaProject.scala
@@ -333,7 +333,7 @@ class ScalaProject(val underlying : IProject) {
topLevelMap0 = new TopLevelMap
println("Building top-level map for: "+underlying.getName)
val start = System.currentTimeMillis
- allSourceFiles.map(topLevelMap0.update)
+ topLevelMap0.update(allSourceFiles)
val end = System.currentTimeMillis
println("Time: "+(end-start))
}
@@ -353,7 +353,7 @@ class ScalaProject(val underlying : IProject) {
}
On Sun, Jul 25, 2010 at 10:44 PM, Daniel Ratiu <rat...@googlemail.com> wrote: > Basically, my improvement is to replace a line from > ScalaProject.topLevelMap
> I propose the line: > topLevelMap0.update(allSourceFiles)
> In the original version (the first one) the methods TopLevelMap.update > and, in turn, TopLevelMap.parse are called once for each file. This is > highly inefficient!!! > Below is a complete patch (it comprises small modifications of the > files ScalaProject and TopLevelMap)
> According to my measurements, the speed improvement in building the > "top-level map" is around 10%.
This looks promising, but could you confirm that the map is completed for *all* the source file passed in, if there is an error in one of them?
If there aren't any problems, then please go ahead and push this to the tycho-reorg branch.
> This looks promising, but could you confirm that the map is completed
> for *all* the source file passed in, if there is an error in one of
> them?
I tested my solution on a scala project in which I injected an error
in one file. It seems to work -- by doing a "project clean" the top-
level map is built for all files.
By taking a look at the implementation of UnitParser I noticed that
the the method always returns a Tree object (that can be an empty
tree).
Do you know other cases that I should be aware of?
@all: I've noticed that the presentation compiler is destroyed and
completely reinitialized *every* time a scala source file is saved
(call chain below)
ScalaSourceFileEditor.doSave ->
ScalaProject.scheduleResetPresentationCompiler -> etc.
I think that this pessimistic treatment of the PresentationCompiler
(i.e. a complete reinitialization for each file save) is a possible
source for the performance bottleneck.
Furthermore, in this area might occur also some memory leaks (e.g.
objects of CompilerResultHolder that continue to hang around after the
presentation compiler is destroyed).
On Wed, Jul 28, 2010 at 11:10 PM, Daniel Ratiu <rat...@googlemail.com> wrote: >> This looks promising, but could you confirm that the map is completed >> for *all* the source file passed in, if there is an error in one of >> them? > I tested my solution on a scala project in which I injected an error > in one file. It seems to work -- by doing a "project clean" the top- > level map is built for all files.
Right, this is why the files are parsed individually currently ... we don't want to require continuous cleaning.
If you can come up with some mechanism for batching the files which accommodates failures (ie. retrying with the failing file excluded) and that mechanism still shows a significant speedup then that would be well worth having :-)
> @all: I've noticed that the presentation compiler is destroyed and > completely reinitialized *every* time a scala source file is saved > (call chain below) > ScalaSourceFileEditor.doSave -> > ScalaProject.scheduleResetPresentationCompiler -> etc.
> I think that this pessimistic treatment of the PresentationCompiler > (i.e. a complete reinitialization for each file save) is a possible > source for the performance bottleneck.
Unfortunately it's necessary as a workaround for bugs in the presentation compiler ... it's unable to cope with changes in the set of top-level types. See,
for example ... there are other related bugs which the presentation compiler reset addresses.
> Furthermore, in this area might occur also some memory leaks (e.g. > objects of CompilerResultHolder that continue to hang around after the > presentation compiler is destroyed).
Those should be invalidated fairly promptly ... can you point me at a case where that doesn't happen?
