What is Thunderbird doing on startup?
Nomis101
SSD. All applications now starting up very faster than before. The
onliest application which has the same startup time before and after,
is Thunderbird (maybe you can measure an improvement, but you don't
recognize it with the human eye). So, I'm wondering, what is
Thunderbird doing on startup which is so time-consuming?
Andrew Sutherland
The short story is we don't have a comprehensive understanding of what
is going on in terms of the distribution of CPU and I/O costs.
If you (or anyone!) would like to help improve start-up performance, the
two easiest to use set of tools on OS X would be:
1) The OS X profiler, Shark and/or Instruments. You will need a
Thunderbird build with debug symbols to get anything useful and
interesting out of it:
https://wiki.mozilla.org/Thunderbird:Backtraces_On_OS_X
You will also want to use Shark to trigger Thunderbird, rather than
trying to get a profile after Thunderbird starts by clicking very
quickly :). You will also want to make sure to stop the sampling once
you believe the Thunderbird UI has reached a usable state. If
Instruments is capable of breaking the data into multiple time ranges,
that might also help. (Namely, if it can profile 0-1sec, 1-2sec, 2-3sec
as discrete aggregated profiles rather than just lumping everything in
one big bucket.)
2) Use/build on the work by the firefox team that provides a startup
timeline for Firefox and much of the Platform. The bugs that covered
most of this work can be found here:
https://bugzilla.mozilla.org/show_bug.cgi?id=480735
https://bugzilla.mozilla.org/show_bug.cgi?id=560647
I would suggest starting with Shark. It's a really easy tool to use,
even if you aren't a hardcore developer. It can help give an
understanding of the broad strokes of what is happening, and might even
point out some smoking guns.
Also, if you are using Thunderbird 3.1.x, you may want to try our 3.3.x
alphas or nightlies because the transition to libxul results in much
friendlier I/O patterns in terms of seek costs (SSD or no) and benefits
from read-ahead.
Andrew
PS: If there are interested parties on linux / windows, there are other
tools available.
Nomis101
try that as soon as I find time. But I have two questions to shark:
1. Do I need to build the debug build with "--enable-shark" and
"MOZ_PKG_SPECIAL="shark""?
2. What should I than do with the datas shark gives me?
I use the current 3.3 nightlys.
Andrew Sutherland
> OK, I've tried Instruments a few times, but I never used shark. I will
> try that as soon as I find time. But I have two questions to shark:
> 1. Do I need to build the debug build with "--enable-shark" and
> "MOZ_PKG_SPECIAL="shark""?
No. What that does is make it possible for JS code to tell Shark to
start/stop profiling (and is broken in x64 builds). Since you want to
start profiling well before then, that support would not help you.
The thing you absolutely need for profiling is a binary whose internal
symbols have not been stripped. Our nightlies have this.
If you are building your own, or if the nightlies are providing
gibberish, you may also need/benefit from:
ac_add_options --enable-optimize="-O2 -fno-omit-frame-pointer"
That makes it easier to get backtraces without falling back to
heuristics or requiring debug info for everything. (It's possibly Shark
on OS X has sufficient heuristics that the lack of a frame pointer is
not a problem, but I get the impression the stock gdb is not clever
enough to do it.)
Alternatively, you could add:
ac_add_options --enable-debugger-info-modules=yes
Which will provide the debug info so that things like gdb (and maybe
shark) can find out how to walk the stack when there is no frame
pointer. (Although that may end up causing the profiling process to be
a bit more intrusive unless it is able to prefetch most of the debugging
info into RAM.)
> 2. What should I than do with the datas shark gives me?
Save the profiler output for any interesting runs you find to a new
bugzilla bug about startup performance on OS X and mention the bug here
or in tb-planning. You may also want to save a text-file dump of the
Shark output to the bug so people without Macs can play along and take a
look.
Andrew
Wayne Mery
> On 01/30/2011 09:55 AM, Nomis101 wrote:
>> A few weeks ago I've exchanged the harddrive in my MacBook with an
>> SSD. All applications now starting up very faster than before. The
>> onliest application which has the same startup time before and after,
>> is Thunderbird (maybe you can measure an improvement, but you don't
>> recognize it with the human eye). So, I'm wondering, what is
>> Thunderbird doing on startup which is so time-consuming?
>
> The short story is we don't have a comprehensive understanding of what
> is going on in terms of the distribution of CPU and I/O costs.
> Andrew
>
> PS: If there are interested parties on linux / windows, there are other
> tools available.
such as xperf. Taras has delved into it
http://blog.mozilla.com/tglek/2010/10/04/diagnosing-slow-startup/
--
QA/bugzilla advice ...
http://www.mibbit.com/chat/?server=irc.mozilla.org&channel=#tb-qa
evangelize ... http://www.spreadthunderbird.com/aff/165/
you can help with ... http://wiki.mozilla.org/Thunderbird:Testing
Wayne Mery
Please add any new bugs that affect startup performance to Bug 487832 -
[meta] Thunderbird startup performance issues
A few core meta bugs are educational
[Bug 627208] Preload startup cache
[Bug 572459] Tracking: Bad IO Patterns
[Bug 447581] startup performance tracking bug
Nomis101
> Save the profiler output for any interesting runs you find to a new
> bugzilla bug about startup performance on OS X and mention the bug here
> or in tb-planning. You may also want to save a text-file dump of the
> Shark output to the bug so people without Macs can play along and take a
> look.
>
> Andrew
I also have mshark files from both runs. But this files are too big
for bugzilla. :-(
Andrew Sutherland
> https://bugzilla.mozilla.org/show_bug.cgi?id=630645
>
> I also have mshark files from both runs. But this files are too big
> for bugzilla. :-(
Thank you very much for pursuing this!
For performance investigations, you don't want to use a DEBUG build.
DEBUG builds are much, much slower than normal builds. They cause a
whole bunch of extra logic to fire when reference counting, when
performing layout, etc. I assume you weren't using a debug build for
your original SSD test?
The (text) shark profile shows that most of the time was spend in
ImageLoaderMachOCompressed::findClosestSymbol which is confusing,
because I would not expect Shark to count its own symbol-finding efforts
in the profiling. Is it possible you ran Shark at the same time as
Instruments or something like that? (The all threads text thing just
seemed to dilute the activities of findClosestSymbol by averaging all of
the inactive threads).
I was unable to get Instruments to do anything with the .trace files...
do you know if I need your instruments project definition for that to
work or what not? Or more specifically, were you able to open or import
them without your project definition loaded?
The mshark files would probably be more useful. If they don't bzip2/zip
well, do you have any webspace where you could chuck them?
Andrew
Nomis101
> For performance investigations, you don't want to use a DEBUG build.
> DEBUG builds are much, much slower than normal builds.
Shark and/or Instruments. You will need a
you've told me to use a debug build. So, you mean its better to do the
same again with a non-debug build? An what do you than mean with
"debug symbols"?
On 2 Feb., 23:42, Andrew Sutherland <sombr...@alum.mit.edu> wrote:
> The (text) shark profile shows that most of the time was spend in
> ImageLoaderMachOCompressed::findClosestSymbol which is confusing,
> because I would not expect Shark to count its own symbol-finding efforts
> in the profiling. Is it possible you ran Shark at the same time as
> Instruments or something like that? (The all threads text thing just
> seemed to dilute the activities of findClosestSymbol by averaging all of
> the inactive threads).
I ran Safari and EyeTV at the same time with Shark. But I don't run
Shark and Instruments at the same time. I've set Shark to trigger the
startup from ShredderDebug.
On 2 Feb., 23:42, Andrew Sutherland <sombr...@alum.mit.edu> wrote:
> I was unable to get Instruments to do anything with the .trace files...
> do you know if I need your instruments project definition for that to
> work or what not? Or more specifically, were you able to open or import
> them without your project definition loaded?
>
> The mshark files would probably be more useful. If they don't bzip2/zip
> well, do you have any webspace where you could chuck them?
>
> Andrew
Yes, I've checked that before uploading on bugzilla. If I doubble
click the trace file, than it takes a few minutes and than it opens in
Instruments. I don't know if you need any project file, but I can
check if the file will open on my other Mac.
I've ziped the mshark file, but its not really smaller than. And it
contains personal information. Do you know how I can clean this file
from personal informations? I can upload it on a webspace for a
limited time.
Andrew Sutherland
> On 2 Feb., 23:42, Andrew Sutherland<sombr...@alum.mit.edu> wrote:
>> For performance investigations, you don't want to use a DEBUG build.
>> DEBUG builds are much, much slower than normal builds.
> Oh, but in your first response you've wrote "1) The OS X profiler,
> Shark and/or Instruments. You will need a
> Thunderbird build with debug symbols". I've interpreted this, that
> you've told me to use a debug build. So, you mean its better to do the
> same again with a non-debug build? An what do you than mean with
> "debug symbols"?
Yeah, that's a pretty ambiguous terminology and configuration area to
begin with, and my brain flubbed expressing what it really meant. (Or I
was hedging :)
Executables/libraries include various segments containing information
for different purposes.
Symbol table segments contain information about the locations of the
implementations of functions, locations of global variables, etc. In a
fully stripped production/release style build the only required symbols
are those that are publicly exposed for dynamic linking. In an
unstripped build (such as nightlies or builds we do ourself), we usually
keep the symbols around from earlier stages of the link process (where
all "non-static" symbols are exposed and even the "static" ones are
usually kept around).
Debug info segments (whose contents may also be known as debug symbols)
contain dwarf-2/other data that provides that information plus much more
extensive mappings of program code addresses to source lines, what is on
the stack when, structure definitions, etc. Debug info is much much
much more verbose. Debug info can be "stripped" (removed from the
binary) and uploaded to the symbol server so we can retrieve them later
if we need them. That is why you may see our various builds building
with debug info/symbols rather than strip them out later.
Shark interrupts the program when it is executing and is able to find
the current program counter (currently executing code location) and
potentially walk the stack which also tells it the program counters of
all the function calls on the stack. It can use the symbol information
to figure out what functions were executing, except for release builds
where there is so little information that most of its conclusions end up
very wrong. In fully optimized builds, the compiler may reuse a
processor register that simplifies the stack-walking process for other
purposes, making it harder to walk the stack. In that case, having the
dwarf-2 debugging info provides enough information to precisely walk the
stack, although it's also possible to walk the stack using heuristics.
It appears the stock gdb on mac is not smart enough to do this on our
3.3.x nightlies, for one. However, the stock Shark does seem
sufficiently clever.
Here's how the flags break down:
* --enable-strip gets rid of all non-global symbols from the symbol
tables and all debugging symbols from the debug segments.
* --enable-debug sets the DEBUG symbol during compilation which enables
a lot of expensive error-checking code. It also causes
MOZ_DEBUG_SYMBOLS to be defined, which is the same as the next flag.
* --enable-debugger-info-modules=yes gets us debug info, regardless of
optimization level. It has apparently been deprecated and its
replacement is "--enable-debug-symbols". This sets MOZ_DEBUG_SYMBOLS.
So you had this one well covered :)
* --enable-profiling cause MOZ_OPTIMIZE_FLAGS="-O3
-fno-omit-frame-pointer" instead of MOZ_OPTIMIZE_FLAGS="-O3
-fomit-frame-pointer"
All you definitely need is an unstripped symbol table that includes
local symbols. This happens by default. It may also be advisable to
add "--enable-profiling" if the tool you are using is having trouble
with the stacks. As I said, I don't think Shark has this problem.
Eliding the branding/update server stuff, this is what our OSX nightlies
use on trunk:
===
ac_add_options --enable-application=mail
ac_add_options --enable-optimize
ac_add_options --disable-debug
ac_add_options --enable-tests
ac_add_options --enable-codesighs
ac_add_options --disable-install-strip
export CFLAGS="-gdwarf-2"
export CXXFLAGS="-gdwarf-2"
# For NSS symbols
export MOZ_DEBUG_SYMBOLS=1
ac_add_options --enable-debug-symbols="-gdwarf-2"
===
As noted, all that debug stuff is for the symbol server, so if you
wanted something equivalent, you could do this.
===
ac_add_options --enable-application=mail
ac_add_options --enable-optimize
ac_add_options --disable-debug
===
I would suggest just using the nightlies, but either way works.
> I ran Safari and EyeTV at the same time with Shark. But I don't run
> Shark and Instruments at the same time. I've set Shark to trigger the
> startup from ShredderDebug.
Looking closer at your mozconfig you had also enabled --trace-malloc.
It (or some other debugging logic) was probably trying to do
stack-walking for allocation signature tracking or something. This
should go away. (Definitely do not build with trace-malloc!)
> On 2 Feb., 23:42, Andrew Sutherland<sombr...@alum.mit.edu> wrote:
>> The mshark files would probably be more useful. If they don't bzip2/zip
>> well, do you have any webspace where you could chuck them?
> Yes, I've checked that before uploading on bugzilla. If I doubble
> click the trace file, than it takes a few minutes and than it opens in
> Instruments. I don't know if you need any project file, but I can
> check if the file will open on my other Mac.
>
> I've ziped the mshark file, but its not really smaller than. And it
> contains personal information. Do you know how I can clean this file
> from personal informations? I can upload it on a webspace for a
> limited time.
To be honest, I'm not 100% sure what's in there. "Get info" suggests it
knows the username and host, but grepping doesn't show them, so
presumably there is compression and it's hard to really know. I could
also see it perhaps providing limited directory info about the path to
Thunderbird. I would rather expect the Instruments info to also have
that data too, though. If you saw shark providing explicit arguments
provided to function calls, we definitely don't want that, but I have no
idea how to turn that on, let alone off.
I think if you just put the mode to "Tree (Top-Down)" and do "Generate
Report" using that for just the main thread ("[NSApplication Run]"/the
first one) and once for "All", that should be a sufficient overview to
share with people. The tree mode provides much more useful information,
context-wise.
Andrew
Nomis101
Seems this subject area is more complex than I've thought.
I've done it now again. First I've downloaded a recent nightly, than
I've fetched the symbols for this nightly with this py script, than
I've run shark (with no app in the background) and saved the report in
4 different ways. This is now different from my first shark reports.
Hope this is now more usable than my first run.
Andrew Sutherland
The results are sadly still slightly confusing. I'm not sure why the
full tree hierarchy is not there? Maybe a build with --enable-profiling
is required after all?
In any event, the heaviness data shows no giant smoking guns, so unless
the tree can be made to root everything properly (which I presume it
can), the next best options are probably analysis of what file I/O is
happening and/or trying out the timeline stuff firefox did that I
mentioned before. It sounded like your instruments runs included some
file analysis. While it seems like maybe you can't share that data,
perhaps you could try your hand at investigating it locally and see if
you can draw any conclusions?
Thanks for your continued interest in improving startup performance;
sorry it's not more straightforward!
Andrew
Nomis101
> The results are sadly still slightly confusing. I'm not sure why the
> full tree hierarchy is not there? Maybe a build with --enable-profiling
> is required after all?
Than it looks like:
# Report 1 - Session 1 - Time Profile (All Thread States) of
thunderbird-bin
SharkProfileViewer
# Generated from the visible portion of the outline view
+ 74.7%, thread_start, libSystem.B.dylib
| + 74.7%, _pthread_start, libSystem.B.dylib
| | + 74.7%, _pt_root, libnspr4.dylib
| | | + 48.4%, nsThread::ThreadFunc(void*), XUL
| | | | + 48.4%, NS_ProcessNextEvent_P(nsIThread*, int), XUL
| | | | | + 48.4%, nsThread::ProcessNextEvent(int, int*), XUL
| | | | | | + 16.6%, nsEventQueue::GetEvent(int, nsIRunnable**), XUL
| | | | | | | + 16.6%, PR_Wait, libnspr4.dylib
| | | | | | | | + 16.6%, PR_WaitCondVar, libnspr4.dylib
| | | | | | | | | + 16.6%, _pthread_cond_wait, libSystem.B.dylib
| | | | | | | | | | 16.6%, __semwait_signal, libSystem.B.dylib
| | | | | | - 8.4%, nsCycleCollectorRunner::Run(), XUL
| | | | | | - 8.1%, nsSocketTransportService::Run(), XUL
| | | | | | - 7.8%, TimerThread::Run(), XUL
| | | | | | - 7.5%, nsThreadPool::Run(), XUL
| | | | | | - 0.0%, nsProxyObjectCallInfo::Run(), XUL
| | | - 8.1%, XPCJSRuntime::WatchdogMain(void*), XUL
| | | - 8.1%, js::GCHelperThread::threadLoop(JSRuntime*), XUL
| | | - 3.7%, nsSSLThread::Run(), XUL
| | | - 3.7%, nsCertVerificationThread::Run(), XUL
| | | - 2.8%, nsHostResolver::ThreadFunc(void*), XUL
- 16.8%, start_wqthread, libSystem.B.dylib
- 8.4%, start, thunderbird-bin
Is this better? I've tried to investigate the shark results like it is
explaned here: http://benjamin.smedbergs.us/blog/2008-09-04/profiling-dromaeo-testcases-with-shark/
But there is nothing obvious.
> In any event, the heaviness data shows no giant smoking guns, so unless
> the tree can be made to root everything properly (which I presume it
> can), the next best options are probably analysis of what file I/O is
> happening and/or trying out the timeline stuff firefox did that I
> mentioned before. It sounded like your instruments runs included some
> file analysis.
I've did some different Instrument runs, but it seems you can't save a
report as a text file from that. Hopefully I'll find a way to share
the datas.
Andrew Sutherland
Yes, that looks a lot better. It probably is more productive to just
look at the threads individually rather than aggregated together in "All
Thread States". Specifically, we expect most of the background threads
to spend most of their time just sitting waiting on asynchronous things
to happen. (I/O notifications, events being posted from other threads,
sleeps/timers to expire, etc.)
Whereas if you're just looking at the main thread we should expect that
its time can probably be broken down like so:
1) Actively doing things, some of which are perhaps things we should not
be doing at that point and should be deferring, and some of which are
things that are just running slow that could be optimized.
2) Waiting on things synchronously. For example, directly performing
I/O on the main thread, or causing the main thread to wait on something
another thread has to do.
3) Experiencing slow I/O costs related to loading the executable code
into memory from disk (which should be fairly cheap given that you have
an SSD). This can get smeared around everywhere and is one of the
things Taras Glek and Mike Hommey are working great magic on and which
we probably should not pursue ourselves. You can help take this kind of
thing out of the equation by doing warm profiling where you have already
run Thunderbird shortly before so the binary should ideally already be
cached into memory. (This does require that you have enough RAM, but I
would assume you would not use an SSD in a system with less than 4 gigs
of RAM.)
> I've did some different Instrument runs, but it seems you can't save a
> report as a text file from that. Hopefully I'll find a way to share
> the datas.
Any files jump out in the file stuff? Gloda's database and our session
restore should show up at least as expected blips on the main thread.
(Most gloda stuff happens on an asynchronous background thread, but we
open it on the main thread which will incur some synchronous I/O costs.)
Andrew
Mark Banner
> So, I'm wondering, what is
> Thunderbird doing on startup which is so time-consuming?
The instrumentation that you're working on is good. I've had a few other
thoughts recently as to things that might help to look at to assess what
we're doing better.
I think they all come down to:
- Are we starting off unnecessary network and other activities before
displaying the main window?
From the Firefox discussions about improving performance, it is
reasonably well known that displaying the main window as soon as
possible is better from the user experience/perception of start up time.
My suspicion is that we're actually kicking off a lot of activity a
little bit before we display the main window. You can see the fact that
something is happening before we actually display the main window if you
turn on the master password.
It would be good if we can make an assessment of what exactly is being
kicked off. Then we can think about if we can delay it until afterwards
or not.
One thing you could try is measuring the performance of Thunderbird
starting in offline mode versus online mode.
If you want slightly more consistent measures of startup time, then you
could look at https://bugzilla.mozilla.org/show_bug.cgi?id=462862 . It
is an extension and script that I wrote for measuring startup time. I
should have a 3.3 version around if you want that (I think the ones on
the bug are 3.1). There's also the about:startup items from Firefox that
may be worth investigating/porting.
Standard8
Nomis101
read from TB at startup. This is:
1. /Thunderbird.app/Contents/Info.plist
2. /Thunderbird.app/.../thunderbird-bin
3. /User/Library/Preferences/org.mozilla.thunderbird.plist
4. (same as 3.)
5. /User/Library/Preferences/ByHost/.GlobalPreferences.FF61....plist
6. (same as 5.)
7. /User/Library/Preferences/.GlobalPreferences.plist
8. (same as 7.)
9. /Library/Preferences/.GlobalPreferences.plist
10. (same as 9.)
11. /Thunderbird.app/.../Resources/de.lproj/InfoPlist.strings
12. /System/Library/CoreServices/SystemVersion.plist
13. /Thunderbird.app/Content/MacOS/chrome.manifest
14. /Thunderbird.app/.../components/interfaces.manifest
15. to 170. A lot of *.xpt files inside Thunderbird
171. to 192. A lot of *.js files inside Thunderbird
197. to 250. A lot of *.manifest files inside Thunderbird
After 250, read access to System files, after that a lot of CSS files
inside TB, than *properties files, than messenger.xul, than 346-359 is
localstore.rdf, than *.properties files, than 366-410 is global-
messages-db.sqlite, 417-899 is mailViews.dat, after that a lot of
different files.
I will see if I can save this as a text file.
On 11 Feb., 10:22, Mark Banner <bugzi...@invalid.standard8.plus.com>
wrote:
> One thing you could try is measuring the performance of Thunderbird
> starting in offline mode versus online mode.
>
> If you want slightly more consistent measures of startup time, then you
> could look athttps://bugzilla.mozilla.org/show_bug.cgi?id=462862. It
> is an extension and script that I wrote for measuring startup time. I
> should have a 3.3 version around if you want that (I think the ones on
> the bug are 3.1). There's also the about:startup items from Firefox that
> may be worth investigating/porting.
I've made shark profiles for online mode vs. offline mode. It seems to
me starting offline is a bit faster, but didn't measure it yet. Would
be nice if you could provide this extension for a 3.3 build. The one
from the bug is for 3.0b1pre only.
Philip Chee
> 13. /Thunderbird.app/Content/MacOS/chrome.manifest
> 14. /Thunderbird.app/.../components/interfaces.manifest
> 15. to 170. A lot of *.xpt files inside Thunderbird
> 171. to 192. A lot of *.js files inside Thunderbird
> 197. to 250. A lot of *.manifest files inside Thunderbird
>
> After 250, read access to System files, after that a lot of CSS files
> inside TB, than *properties files, than messenger.xul, than 346-359 is
> localstore.rdf, than *.properties files, than 366-410 is global-
> messages-db.sqlite, 417-899 is mailViews.dat, after that a lot of
> different files.
All the xpt, js, xul, properties, dtd etc files should be inside the
omnijar file. How are you building Thunderbird? What's your mozconfig like?
Phil
--
Philip Chee <phi...@aleytys.pc.my>, <phili...@gmail.com>
http://flashblock.mozdev.org/ http://xsidebar.mozdev.org
Guard us from the she-wolf and the wolf, and guard us from the thief,
oh Night, and so be good for us to pass.
Robert Kaiser
> 15. to 170. A lot of *.xpt files inside Thunderbird
> 171. to 192. A lot of *.js files inside Thunderbird
> 197. to 250. A lot of *.manifest files inside Thunderbird
You're not using something we release, then. If you are building
yourself, please do a "make package" and use either the package itself
or what ends up in dist/thunderbird.
Robert Kaiser
--
Note that any statements of mine - no matter how passionate - are never
meant to be offensive but very often as food for thought or possible
arguments that we as a community needs answers to. And most of the time,
I even appreciate irony and fun! :)
Nomis101
> On Sat, 12 Feb 2011 14:59:01 -0800 (PST), Nomis101 wrote:
> > 13. /Thunderbird.app/Content/MacOS/chrome.manifest
> > 14. /Thunderbird.app/.../components/interfaces.manifest
> > 15. to 170. A lot of *.xpt files inside Thunderbird
> > 171. to 192. A lot of *.js files inside Thunderbird
> > 197. to 250. A lot of *.manifest files inside Thunderbird
>
> > After 250, read access to System files, after that a lot of CSS files
> > inside TB, than *properties files, than messenger.xul, than 346-359 is
> > localstore.rdf, than *.properties files, than 366-410 is global-
> > messages-db.sqlite, 417-899 is mailViews.dat, after that a lot of
> > different files.
>
> All the xpt, js, xul, properties, dtd etc files should be inside the
> omnijar file. How are you building Thunderbird? What's your mozconfig like?
>
> Phil
>
> --
> Guard us from the she-wolf and the wolf, and guard us from the thief,
> oh Night, and so be good for us to pass.
Yes, you are right, this was not a packed build, this was a build
inside my OBJDIR. If I pack the build, than it looks like:
1. /Thunderbird.app/Contents/Info.plist
2. /Thunderbird.app/.../thunderbird-bin
3. /User/Library/Preferences/org.mozilla.thunderbird.plist
4. (same as 3.)
5. /User/Library/Preferences/ByHost/.GlobalPreferences.FF61....plist
6. (same as 5.)
7. /User/Library/Preferences/.GlobalPreferences.plist
8. (same as 7.)
9. /Library/Preferences/.GlobalPreferences.plist
10. (same as 9.)
11. /Thunderbird.app/.../Resources/de.lproj/InfoPlist.strings
12. /System/Library/CoreServices/SystemVersion.plist
13. /Thunderbird.app/.../pref/channel-prefs.js
14. (same as 13.)
15. /Thunderbird.app/Contents/MacOS/chrome.manifest
16. /Thunderbird.app/Contents/MacOS/components/binary.manifest
17. My prefs.js
18. (same as 17.)
19. My user.js
20. (same as 19.)
21. My userContent.css
22. write access from _SCNetworkReachabilityGetFlags
23. read access from _SCNetworkReachabilityGetFlags
24. to 70. global-messages-db.sqlite
71. to 78. mailViews.dat
79. to 83. /ColorSync/Profiles/Displays/ColorLCD-00610-..-0000.icc
The rest than is global-messages-db.sqlite