What's the plan and timetable to fix Thunderbird performance at startup due to Lightning bugs and flaws

90 views
Skip to first unread message

Richard LEGER

unread,
Jul 18, 2019, 5:06:23 AM7/18/19
to tb-pl...@mozilla.org
Dear Thunderbird (TB) team,

Startup performance issues have been plaguing TB for years (7 years or more possibly) especially due to Lightning being enabled... and it became worth in past years...

Up to now the "false" assumption has been that all I/O happens in one thread (ui, email core features in addition of Lightning) causing too much processing... not responding issues...

While this may have been partially true, tremendous progress have been made so far in that regards within TB (e.g core improvements, use of workers and async features helping if I am not mistaking), so it had been more an aggravation of existing underlying issues in Lightning than a root cause.

Indeed I think such progress shadows the fact that Lightning still contains bugs and design flaws (no blame on developers here) that are mainly source of performance issues especially at startup that are still not being analysed and addressed the way they should (at least from end-users point of view) by implementing long term fixes/solutions into that code so it would performs "normally" ;-)

As an end-user, month ago, I have revealed (because end-users could not take it any more!) bug https://bugzilla.mozilla.org/show_bug.cgi?id=1502923 (patch issued but backed out without further support) and bug https://bugzilla.mozilla.org/show_bug.cgi?id=1543953 as the tip of the iceberg but received barely any support from TB/Lightning dev teams so far or feedback on helping resolving those long lasting and highlighted performance issues. It may not be a priority for developers (in their time and resources constraints) but it IS for many end-users...
[Update: there might be light at the end of the tunnel as per new bug https://bugzilla.mozilla.org/show_bug.cgi?id=1567055 created today]

From my impression, while it seems that lot of efforts (and communication) are oriented new UI revamping and new features... bugs fixing and especially those related to performance (quality, reliability, speed) seems relegated to a lower priority to some extent... while it should be in reverse order...

Currently are code reviews and new features subject to large scaling tests to assess performance of code with large data sets prior publishing it?
Are performance issues detected (or reported) fixed immediately or right after in next dev cycles?

For two years TB quality and reliability had decreased drastically, mainly due to lengthy discussion about governance that put TB development and maintenance to a stall, it was kept afloat by the remaining dev team and community (great thanks to them!).

As it is now past, shouldn't 2019 be the year where TB reliability and performance are brought back to life and take priority over new features? Wouldn't that be the best way to bring back confidence (and contributors) in the project? Wasn't it the priority for 2019 "Making Thunderbird fly faster" in the first place, as I recall?

Now that a new team is up and running, what is the plan to tackle reliability and performance issues in TB especially those linked to Lightning?

Would (and could) time and resources be dedicated to them in particular so they can finally be fixed once and for all in the next few months?

While Lightning is an add-on it provide a calendar features to TB and therefore shall be considered as an essential Core feature that need much care and attention ;-) I am not saying it is not already (because it certainly is)... just that it needs much more... especially performance wise...

As of today it is still not possible to have 4-5 calDav calendars (with thousands of items each) enabled and active at the same time, above two, it cripples TB especially at startup...

Maybe a plan and timetable is already in place if that the case, please advise and communicate on what can be expected...
Otherwise could a plan and timetable be established with clear goals and results to clear any performance issues in TB due to Lightning especially, as a priority?

It may be a wishful thinking but plan, support and fixes on the performance matters would be greatly appreciated by end-users...

Looking forward to seeing Thunderbird fly fast in 2019!

Regards,
Richard

Magnus Melin

unread,
Jul 18, 2019, 7:20:01 AM7/18/19
to tb-pl...@mozilla.org

Performance improvements are indeed high on the priority list. The first step will be to get some tests for UI performance so that we can measure what progress we make (and don't regress). We should have some of these later this year.

Like you noticed, there are many individual smaller things that can add up, and there's generally no magic bullet to just it all fixed at once.  OTOH, there are some larger topics like making proper use of multiple processes, process per tab like Firefox, that have potential to speed things up significantly. I don't have detailed timetable for that, though I'm hoping we can get there in 1-2 ESR releases.

 -Magnus

_______________________________________________
tb-planning mailing list
tb-pl...@mozilla.org
https://mail.mozilla.org/listinfo/tb-planning

Wayne Mery

unread,
Jul 18, 2019, 11:49:42 AM7/18/19
to Thunderbird planning (moderated), Magnus Melin
On 7/18/2019 7:05 AM, Magnus Melin wrote:

Performance improvements are indeed high on the priority list. The first step will be to get some tests for UI performance so that we can measure what progress we make (and don't regress). We should have some of these later this year.

Why should tests block fixing bugs, which in some cases prevent users from using Calendar, or keep users on an older version Thunderbird/Calendar?

And are you suggesting we need to make the switch to ical.js before making further tweaks?


Like you noticed, there are many individual smaller things that can add up, and there's generally no magic bullet to just it all fixed at once.  OTOH, there are some larger topics like making proper use of multiple processes, process per tab like Firefox, that have potential to speed things up significantly. I don't have detailed timetable for that, though I'm hoping we can get there in 1-2 ESR releases.

You're not suggesting not waiting 1-2 ESR to fix perf improvements that don't require multiprocess, correct?

Magnus Melin

unread,
Jul 18, 2019, 2:03:02 PM7/18/19
to Wayne Mery, Thunderbird planning (moderated)
On 18-07-2019 18:40, Wayne Mery wrote:
On 7/18/2019 7:05 AM, Magnus Melin wrote:

Performance improvements are indeed high on the priority list. The first step will be to get some tests for UI performance so that we can measure what progress we make (and don't regress). We should have some of these later this year.

Why should tests block fixing bugs, which in some cases prevent users from using Calendar, or keep users on an older version Thunderbird/Calendar?

They don't need to wait, I'm just saying that with tests it's much more easy to see if the changes have the desired effect.

And are you suggesting we need to make the switch to ical.js before making further tweaks?

Not really. We want to make ical.js performant enough to replace libical by 76 (preffed on by default) and drop libical shortly after. But to make sure it actually is performant enough, tests would be helpful.

Like you noticed, there are many individual smaller things that can add up, and there's generally no magic bullet to just it all fixed at once.  OTOH, there are some larger topics like making proper use of multiple processes, process per tab like Firefox, that have potential to speed things up significantly. I don't have detailed timetable for that, though I'm hoping we can get there in 1-2 ESR releases.

You're not suggesting not waiting 1-2 ESR to fix perf improvements that don't require multiprocess, correct?

All improvements are of course welcome to land as soon as someone comes up with them.

 -Magnus



ISHIKAWA,chiaki

unread,
Jul 18, 2019, 6:00:16 PM7/18/19
to Richard LEGER, Thunderbird planning (moderated)
In https://bugzilla.mozilla.org/show_bug.cgi?id=1502923#c33
you listed a few issues related to network download of data.

Is it possible that TB uses unbuffered I/O for these network I/O?

I have been trying to fix pop3 (and imap) to use buffered write.
Currently it does not.
(I am re-creating the patches to fit the modified source files over the
last serveral months.)
The delay may not be noticeable if all your data is on local disk.
However, once it is on a network file system, we lose badly due to the
large overhead of writing in a network file system.
We definitely should use buffered write for this.

Just my two cents worth.

Chiaki

Richard LEGER

unread,
Jul 19, 2019, 10:32:27 AM7/19/19
to tb-pl...@mozilla.org

Hi Magnus,

Thanks for your time and reply...

Date: Thu, 18 Jul 2019 14:05:18 +0300
From: Magnus Melin <mkmelin...@iki.fi>


Performance improvements are indeed high on the priority list.

It is good to hear about it, Thunderbird team should communicate more regularly in details about plans to tackle performance issues, maybe via a dedicated blog posts perhaps? Just a suggestion...

What the plan to improve performance in the next 3 to 6 months?

The first step will be to get some tests for UI performance so that we can measure what progress we make (and don't regress). We should have some of these later this year.

While this is certainly an essential step forward that is much welcomed, it should not be considered the first step (and only step) especially considering it would take another 6 months to put in place! Which mean another 6 months (roughly) from then to start seeing some results... so 1 year in total... Is it really not possible to improve performance before then?

As a possible plan in parallel of this "grand design", you may want to collect (as already started in https://bugzilla.mozilla.org/show_bug.cgi?id=487832 tb-startupperf  [meta] Thunderbird startup performance issues), analyse, and then distinguish between the various performance issues reported perhaps... not all may be related to the UI Performance itself but rather to underlying code bugs and flaws. No all perf bugs may need 6 months or more waiting to be fixed. You may want to battle on two fronts here...

Perhaps some of the performance issue could already be looked at and fixed today (as the 1st front of battle). Bug https://bugzilla.mozilla.org/show_bug.cgi?id=1502923 is only one example that may shows that changing a small peace of underlying code can have a HUGE performance improvement... but it require attention to perf bugs, time and resources... and further troubleshooting and testing... unfortunately patch for 67.x branch was backed out due to regression in 68.x where Lighting code changed very much already... so 67.x patch would not longer "fit"... now it could be a matter of tackling this regression to fix it possibly and to allow publication of a patch shortly... but again that may mean to put as a priority with resources and support to tackle such long lasting "major" perf issues that only require a "small" fix...

Like you noticed, there are many individual smaller things that can add
up, and there's generally no magic bullet to just it all fixed at once.?

While it is understandable perf issues cannot all be all tackled at once, they could be collected and analysed to see what could be already sorted and distinguished from what might require a grant re(design)... and those not (already fixable at next dev cycle iterations)... Then go step by step to fix perf issues one by one (where already possible of course) until greater goal is achieved... or getting closer to it... with a clear and transparent progression path...

This again may require time and resources, maybe a dedicated team temporarily could have a look at all perf issues... and see what could already be tackled and crack down... maybe by highlighting and narrowing down issues... maybe also helping raising awareness about perf issues within the dev teams responsible for the area of code affecting perf, so they look at it in their own code or part of their current ongoing code review... just a thought... from end-users point... but all this may already be in place...

OTOH, there are some larger topics like making proper use of multiple
processes, process per tab like Firefox, that have potential to speed
things up significantly.

Yes there are some larger topics that are already getting some attention that will fix part of the perf issues, and that is great long term plan, but as shown performance bugs may not be directly linked to those issues and may already be fixable to some extend if look after already... by fixing existing code...within a short term plan...
There may not be one case fit all situation here...

I don't have detailed timetable for that, though I'm hoping we can get there in 1-2 ESR releases.

Would it be a good time to get some details and timetable to evaluate progress?
And if not on the "grand design" plan, on small perf issues that could be already sorted?

How long 1-2 ESR represent... 6 months... 1 year?
Can Thunderbird wait another 6 months before its performance starts improving? Lighting related especially?

It would be expected to see at least slight performance improvements during the next 6 months of development... for the sake of end-users... workarounds such as lazy loading (e.g  https://bugzilla.mozilla.org/show_bug.cgi?id=1477958) may be welcomed as they may help hinder performance issues in short term (as part of the solution) but keeping in mind they would not fix underlying bugs and code flaws... it may only "hide" them deeper...

Not all has to be fixed at once, I don't even think it is even possible honestly :-) but are they really no ways to already bring some perf improvements into TB?
Any that would appear progressively at each next releases during the next 6 months? End-users may expect it as a "feature" and it should be considered as such, each release making the software perform better and better... in parallel of bug fixes and new features... that would be great news... at least to tide you over while more to come...

I am just trying to raise awareness here... that some perf issues may potentially be already addressable... without much waiting... it may just be a matter of looking into it...

Performance remain a critical issue for end-users and therefore a balance may need to be found between:
- a short term plan (1st front of battle) with immediate visible result/progress...
and
- a long term plan (2nd front of battle) to eliminate and prevent perf issues in the future...

One plan shall not prevent the other... or delay it...

Regards,

Richard LEGER

unread,
Oct 23, 2019, 6:47:56 AM10/23/19
to tb-pl...@mozilla.org

Hi,

On Fri, Jul 19, 2019 at 1:49 PM Richard LEGER <richar...@gmail.com> wrote:

Date: Thu, 18 Jul 2019 14:05:18 +0300
From: Magnus Melin <mkmelin...@iki.fi>

Performance improvements are indeed high on the priority list.

It is good to hear about it, Thunderbird team should communicate more regularly in details about plans to tackle performance issues, maybe via a dedicated blog posts perhaps? Just a suggestion...


A lot was heard about new core features of TB (or removed ones) but what about reliability and performance improvements?
Reliability and performance are what put people off Thunderbird in the first place, much much much more than UX theme/layout issues or missing features among others...

So any news about those?
What is happening on those fronts?
What has improved in the past 3 months? What will improve in the next 3 months?

At the meantime, I would like to praise the calendar dev team for bringing quite some improvements to the calendar feature...

In 70.x branch and later (results here https://bugzilla.mozilla.org/show_bug.cgi?id=1572823#c16 and here https://bugzilla.mozilla.org/show_bug.cgi?id=1572823#c19), it now only take 1mn+ with libical (or 2mn+ with ical.js) to load a CalDAV network calendar with ~4000 items... compared to the 100mn+ in 60.x branch that is quite an achievement!!! Bravo!

That said, efforts must continue and results must continue to improve in the course of next months, much before 76.x version foreseen in summer 2020... so far 72.x branch has not shown further improvements (yet) on that front compare to 70.x branch... and ical.js seems not performing as well or better than libical (yet)...

Would fixing bug https://bugzilla.mozilla.org/show_bug.cgi?id=1543953 be a way forward (or not) by reducing number of network requests... while waiting for ical.js and UI performance measurement and improvements? That remain to be seen...

I hope everyone would agree and understand that 1mn+ is still way too much of a delay to any end-user out there :-)
Especially considering that is preventing/delaying access to emails (via IMAP) for several minutes at startup (issue reported here https://bugzilla.mozilla.org/show_bug.cgi?id=1585259)

Work in progress I know... and thank you all for this... but please keep up and continue the good work...

End-users are very much looking already to such performance improvements...

Regards,
Richard

Richard LEGER

unread,
Jan 27, 2021, 7:28:32 AM1/27/21
to tb-pl...@mozilla.org

Date: Thu, 18 Jul 2019 14:05:18 +0300
From: Magnus Melin <mkmelin...@iki.fi>

Performance improvements are indeed high on the priority list.

Hi Magnus,

Since you made the statement above, would you be able to highlight what has been achieved in term of Thunderbird performance improvements, as well as what is in the pipeline to fix design flaws that still affect TB performance especially at start-up? 

What can we expect for the next stable version?

From my personal experience, from an end-users point of view so far, there still seems much to be done in term of visible results and I believe I may not be the only one to experience poor perf in TB... if I am please accept my sincere apology :-)

I know there is a plan to split processing in TB in different processes (as The Holly Grail toward a solution), which is coming up and that would be really nice to have, but that shall not be used to hide the performance bottleneck coming from the possible flaws in the TB code itself by design, and it is certainly not gonna be ready for the next stable(s)...

The same way while cache features have a side effect to improve performance, their main purpose by design is to allow access to the data offline and shall not be used as a permanent workaround to hide performance issues "under the carpet"... people shall be able to use TB in online mode without much more issues than in offline mode... even with large or very large data sets (the only limits shall be the physical ones network speed, storage space, etc...not TB itself) indeed not everyone may like data to be cached (copied/stored) on a local computer... or wait 2mn+ to be able to read first email at start-up!

Is there anyway results can be improved? How soon?
Is there anyway we can help? Better?

For now, I have recorded new performance profiles for various TB version in various setup that I can test with, and results can be found here:

I also put the links here for convenience...

Hopefully it would help the dev teams (and contributors) to identify bottlenecks and hopefully help fix some of the remaining of them... especially prior the next TB stable version...

In addition, I would like to encourage again all the dev teams (developers and contributors) to test features and code they design with very large data sets, that may help identify perf issues before they even land in a beta or stable version.

On another hand, if you could advise also how we can help from our end, how to best and better report performance issues encountered with steps and details for the common of mortal to follow, in a way that would be useful to developers and contributors, that may also help the overall project going in the right direction in term of performance... please do let us know.

Would there be already some resource available on the matter you could recommend?
Is recording a TB perf profil published and shared via https://profiler.firefox.com/ the best way?

Thanks to all developers and contributors who may understand my point and are already contributing towards better performance in TB. If you can advise from your experience how we can better help, let us know as well.

I am just dreaming of the time (that will come) where TB would be fast and reliable at all time... from start-up to exit (without crashing at all)!

Regards,
Richard



Reply all
Reply to author
Forward
0 new messages