thunderbird and large Inbox
Rahul
articles warning against using large Inboxes with TB. Is this still an
issue in the newer TB versions? I haven't faced any issues or corruption so
far but I still wanted to double check. By large I mean around 4 GB. I use
IMAP.
In these day's of Terabyte disks and 8 core procs. is 4GB still a
unreasonably large size for a TB Inbox?
--
Rahul
Mike Easter
Any database is subject to corruption.
I read somewhere that somewhere along the way Tbird changed from one
database infrastructure to another which the writer of that remark
thought was an improvement. I am missing - not recalling - the details
of that information.
In my opinion, the features that make any database file more subject to
becoming corrupt are the factors of large size or large number of inner
pieces or lack of compacting (contributing to the previous two) or
accidents of compacting or accidents of antiviral agents deranging the data.
The databases of mail/news apps seem particularly subject to these
corruptions.
It doesn't seem to me to be 'smart' to conduct ones maintenance and
organization (or disorganization) in such a way as to increase the
factors that lead to a higher chance of corruption of the database.
So, that would mean organizing a big folder into smaller appropriate
folders and routinely compacting and trying to not let anything bad
happen during compacting or antiviral effects.
Large disks and multicore processors do nothing to reduce the likelihood
of corruption, as those features actually *invite* the kinds of factors
and behaviors that lead to corruption, rather than to assuage them.
That is, the more storage you have and the faster you can process your
(big fat overloaded) data, the less likely you are to be 'burdened' by
any need to help your mail/news client handle its (difficult) duties of
databasing by making its job any easier.
--
Mike Easter
Rahul
news:xPadnavxVcwg9v3R...@mozilla.org:
>> In these day's of Terabyte disks and 8 core procs. is 4GB still a
>> unreasonably large size for a TB Inbox?
>
> Any database is subject to corruption.
Agreed! I can't dispute that. The question I have though is how is the
scaling behavious of databases. With typical algorithms of the field is a
couple of GB's the size limit where the probability of data corruption
starts becoming significant?
> I read somewhere that somewhere along the way Tbird changed from one
> database infrastructure to another which the writer of that remark
> thought was an improvement. I am missing - not recalling - the details
> of that information.
Thanks for the pointer! I'll google that. I'd be curious to know. Is
there a certain size limit that the TB team reccomends not exceeding?
I've read that "Large Inboxes are evil" But not sure how large is
devilishly large.
> In my opinion, the features that make any database file more subject
> to becoming corrupt are the factors of large size or large number of
> inner pieces
But these are also the two factors that make the overlying database more
useful. A large database is more information rich (hopefully). The fact
that it is large shouldn't necessarily be termed bad. We just have to
find ways to deal with the size.
> So, that would mean organizing a big folder into smaller appropriate
> folders and routinely compacting and trying to not let anything bad
> happen during compacting or antiviral effects.
I like the gmail style approach of not splitting data into arbitrary
folders. With modern information retrival methods there's rarely a need
to do that. Unless it is a limitation of the clients mail handling
abilities.
> Large disks and multicore processors do nothing to reduce the
> likelihood of corruption, as those features actually *invite* the
> kinds of factors and behaviors that lead to corruption, rather than to
> assuage them.
>
> That is, the more storage you have and the faster you can process your
> (big fat overloaded) data, the less likely you are to be 'burdened' by
> any need to help your mail/news client handle its (difficult) duties
> of databasing by making its job any easier.
I don't think this is always true. An analogy is filesystems. ext2
started when 1 GB drives were termed large and now we have ext4 with
Terabyte drives. But I don't think performance or data corruption has
grossly degraded between the two eras. More isn't always evil. We just
need newer tools to handle it.
--
Rahul
Dave Warren
<nos...@nospam.invalid> was claimed to have wrote:
>Thanks for the pointer! I'll google that. I'd be curious to know. Is
>there a certain size limit that the TB team reccomends not exceeding?
One thing I will mention, in the early TB 2 days I used to corrupt a
mailbox on a fairly regular basis.
I haven't seen one get corrupted since 3.0 (or possibly earlier, I'm not
exactly sure when the problem went away)
Worst case, you can nuke the local ImapMail directory and TB rebuilds by
redownloading from the server without any data loss, so the risk isn't
huge.
>I've read that "Large Inboxes are evil" But not sure how large is
>devilishly large.
The numbers vary, but honestly, most of it depends on server
performance. Gmail is specifically designed to handle massive mailboxes
and it scales better than most.
That being said, TB's performance starts to suffer on my system beyond
5,000 messages or so.
It's not significant, but at least a little noticeable. Even with
20,000 messages it is still usable.
Rahul
news:51u0669n1j4s8c653...@4ax.com:
> In message <Xns9DCFA68B3383C66...@216.196.97.169> Rahul
> <nos...@nospam.invalid> was claimed to have wrote:
>
> One thing I will mention, in the early TB 2 days I used to corrupt a
> mailbox on a fairly regular basis.
>
> I haven't seen one get corrupted since 3.0 (or possibly earlier, I'm not
> exactly sure when the problem went away)
Thanks Dave! For me its hard to deconvolute the two factors of improved (?)
TB versions and my own steadily growing Inbox. I guess they might be
compensating to keep percieved performance the same.
> Worst case, you can nuke the local ImapMail directory and TB rebuilds by
> redownloading from the server without any data loss, so the risk isn't
> huge.
Thanks again! That's good to know. Again to confirm: So long as I have all
my mails on the IMAP server I can safely delete both files? [Somehow my
Inbox has started growing by itself since last night and I am hoping that
this will fix it. It is 4GB but now shows up as 8GB and rising.]
Inbox
&
Inbox.msf
>>I've read that "Large Inboxes are evil" But not sure how large is
>>devilishly large.
>
> The numbers vary, but honestly, most of it depends on server
> performance. Gmail is specifically designed to handle massive mailboxes
> and it scales better than most.
>
> That being said, TB's performance starts to suffer on my system beyond
> 5,000 messages or so.
>
> It's not significant, but at least a little noticeable. Even with
> 20,000 messages it is still usable.
I've around 35000 messages and 4GB Inbox.
--
Rahul
Rahul
> The numbers vary, but honestly, most of it depends on server
> performance. Gmail is specifically designed to handle massive mailboxes
> and it scales better than most.
That's interesting that the corruption at large sizes is a IMAP-server
issue. From what I had read so far I had thought that this was a IMAP-
client (TB) issue.
--
Rahul
Dave Warren
<nos...@nospam.invalid> was claimed to have wrote:
>Thanks again! That's good to know. Again to confirm: So long as I have all
>my mails on the IMAP server I can safely delete both files? [Somehow my
>Inbox has started growing by itself since last night and I am hoping that
>this will fix it. It is 4GB but now shows up as 8GB and rising.]
You should be able to delete the entire ImapMail directory, or an entire
server's worth of data at a time. It might be possible to delete in
smaller increments but I've not really experimented.
This is likely not something Mozilla formally supports, but it does work
nicely enough when issues do arise.
Part of the benefit of IMAP is that all of your data is stored on the
server, so the client stored client-side is disposable.
One other thing that I should mention, I have ThunderBird's global
indexer off. If you use it, ThunderBird is going to redownload
everything (not just the headers needed to display the message list), so
if you're on a slow connection expect a long rebuild time.
Dave Warren
<nos...@nospam.invalid> was claimed to have wrote:
It can be a server issue or a client issue really, both sides maintain
databases of some sort. The specific issues that you might encounter
will vary of course, but throw enough data at a design and performance
will suffer or resources consumed will increase (or both)
Modern hardware helps significantly, of course.
Ron K.
There are two factors that set file size limits. On is the hardware and he
other is the OS used. While upgrading to newer generation hardware will
improver performance, it will not resolve OS issues. For Windows, the Win32
OS's have a 4 GB file cap, it's a limitation of the OS's memory space. Use
of a 64 bit CPU with it's larger memory space results in some of that
capability being unused.
As long as the Mozilla factory builds produce binaries optimized for the P1
i586 CPU there will be limits imposed by the Opcodes of that obsolete CPU.
--
Ron K.
Who is General Failure, and why is he searching my HDD?
Kernel Restore reported Major Error used BSOD to msg the enemy!
Arivald
There are 2 files for each TB mail folder ("Inbox" is one folder).
One file, without extension, is a raw storage file, in mbox format. It
contain all mails bodied, one after another.
If mail is deleted, it is only marked as deleted, nothing is removed
physically.
If mail is added, it is always added at end of storage file.
If there is too much "deleted" space inside file, TB try to "compact
folder". It means all mail will be copied to beginning of storage file,
and unused space at end is truncated.
Because of all this methods, access and operations of storage file are
quite fast, and not very much affected by file size. Only compacting
folder takes longer on big file. It is why I set my TB to compact if
there is about 100 MB unused space.
Second file, *.msf, is index file. When TB try to show mail folder
contents, it reads this file only.
Size of index file affects speed of TB, large file may slow down
application.
If this file become corrupted, you may just delete it, and TB will
rebuild it by scanning mbox file.
Good way of improving performance and stability is to split folder to
sub-folders. For You, You may create 4 sub-folders in Inbox, name them
after last 4 years, and put messages You receive in any year in
appropriate sub-folder
--
Arivald
Rahul
news:646dnQAi3vLGuPzR...@mozilla.org:
> Good way of improving performance and stability is to split folder to
> sub-folders. For You, You may create 4 sub-folders in Inbox, name them
> after last 4 years, and put messages You receive in any year in
> appropriate sub-folder
>
The reason I hate splitting is that each time I now need to do a search
I'll have to do it four seperatee times. Or is there a smart way of using
dynamic searches or virtual folders to integrate these 4 sub-folders into
one virtual folder?
--
Rahul
Rahul
news:p4ydnRJtAuitLP3R...@mozilla.org:
> There are two factors that set file size limits. On is the hardware
> and he other is the OS used. While upgrading to newer generation
> hardware will improver performance, it will not resolve OS issues. For
> Windows, the Win32 OS's have a 4 GB file cap, it's a limitation of the
> OS's memory space.
I don't think that's the case. It is only a relic with fat32 file systems.
With NTFS I belive you can get files up to 16 Terabytes.
See here:
http://technet.microsoft.com/en-us/library/cc766145%28WS.10%29.aspx
So the arbitrary 4 GB limit seems purely a Thunderbird limitation. I don't
think the OS shares a blame.
>Use of a 64 bit CPU with it's larger memory space
> results in some of that capability being unused.
>
> As long as the Mozilla factory builds produce binaries optimized for
> the P1 i586 CPU there will be limits imposed by the Opcodes of that
> obsolete CPU.
Ah! Ok. maybe the trick is to compile my own build then for my
archetecture. Why doesn't Mozilla create a 32 bit and a 64 bit compile?
There must be plenty of people around these days using modern hardware!
--
Rahul
Wayne Mery
> Arivald<arivald_@AT_interia_DOT_pl> wrote in
> news:646dnQAi3vLGuPzR...@mozilla.org:
>
>> Good way of improving performance and stability is to split folder to
>> sub-folders. For You, You may create 4 sub-folders in Inbox, name them
>> after last 4 years, and put messages You receive in any year in
>> appropriate sub-folder
>>
>
> The reason I hate splitting is that each time I now need to do a search
> I'll have to do it four seperatee times.
this would be true only for quick search, or what is now in v3.1 called
quick filter
> Or is there a smart way of using
> dynamic searches or virtual folders to integrate these 4 sub-folders into
> one virtual folder?
1. you can do a virtual folder of course.
2. or construct the subfolders as previously suggested
the advantage of #1 is you can use quick filter on a virtual saved
search and hit all the folders. for #2 you can only search all the
folders if you use Edit | Search Messages aka advanced search - but at
least then you only need to do one search, because it will search
subfolders.
note - both searches/filters have disadvantages.
--
QA/bugzilla advice ...
http://www.mibbit.com/chat/?server=irc.mozilla.org&channel=#tb-qa
evangelize ... http://www.spreadthunderbird.com/aff/165/
you can help with ... http://wiki.mozilla.org/Thunderbird:Testing
Chris Ilias
When you delete a message, it doesn't really get deleted from the
folder. It gets marked as deleted. Thunderbird sees that marking, and
knows not to display it. Compacting a folder will command Thunderbird to
remove all messages marked as deleted, from that folder.
To compact a folder, right-click on it, and select Compact.
--
Chris Ilias <http://ilias.ca>
List-owner: support-firefox, support-thunderbird, test-multimedia
Rahul
dnYk0hvVvxvzRn...@mozilla.org:
> 1. you can do a virtual folder of course.
>
> 2. or construct the subfolders as previously suggested
>
> the advantage of #1 is you can use quick filter on a virtual saved
> search and hit all the folders. for #2 you can only search all the
> folders if you use Edit | Search Messages aka advanced search - but at
> least then you only need to do one search, because it will search
> subfolders.
>
>
Yes, #1 seems to be a winner clearly. If you look at your normal workflow
how often do you go to edit | Search versus just using the filters on the
main page? I'd say for every 30 times I use the filters I do one Edit
|Search.
So the ability to still think of all my Inbox as "one unit" is useful.
The sub-folders are only to pander to Thunderbirds inability to deal with
large single mbox files. It is an implimentation shortcoming not a design
requirement.
--
Rahul
Wayne Mery
> Wayne Mery<vsee...@lehigh.edu> wrote in news:K6-
> dnYk0hvVvxvzRn...@mozilla.org:
>
>> 1. you can do a virtual folder of course.
>>
>> 2. or construct the subfolders as previously suggested
>>
>> the advantage of #1 is you can use quick filter on a virtual saved
>> search and hit all the folders. for #2 you can only search all the
>> folders if you use Edit | Search Messages aka advanced search - but at
>> least then you only need to do one search, because it will search
>> subfolders.
>>
>>
>
> Yes, #1 seems to be a winner clearly. If you look at your normal workflow
> how often do you go to edit | Search versus just using the filters on the
> main page? I'd say for every 30 times I use the filters I do one Edit
> |Search.
just be aware that virtual folders is not yet perfect :)
bugs: http://bit.ly/djeRMJ
enh: http://bit.ly/bWBCGJ
so any assistance people can give to triaging unconfirmed bugs, or
re-triaging "too old" bugs will be welcome.
Rahul
> just be aware that virtual folders is not yet perfect :)
> bugs: http://bit.ly/djeRMJ
> enh: http://bit.ly/bWBCGJ
>
> so any assistance people can give to triaging unconfirmed bugs, or
> re-triaging "too old" bugs will be welcome.
>
Thanks for the heads-up. Looking at the number of bug reports this might
actually _increase_ my chances of corruption than decrease them. Maybe
stayin with a single large inbox is the lesser of the two evils for now....
--
Rahul
Rahul
news:fuWdnSMtlpFbwvzR...@mozilla.org:
>>
>> In these day's of Terabyte disks and 8 core procs. is 4GB still a
>> unreasonably large size for a TB Inbox?
>
> When you delete a message, it doesn't really get deleted from the
> folder. It gets marked as deleted. Thunderbird sees that marking, and
> knows not to display it. Compacting a folder will command Thunderbird to
> remove all messages marked as deleted, from that folder.
>
> To compact a folder, right-click on it, and select Compact.
>
I already have automatic compaction enabled. (I never did figure why this
isn't a default enabled option. If compaction is so critical to proper TB
operation why doesn't TB compact by default.)
--
Rahul
Annailis
and then realize they accidently deleted something important!?
--
Annailís
Wayne Mery
not at all.
you will note, that none of those bugs have the dataloss keyword.
besides that, not all bugs affect all users.
Rahul
HfzRnZ2dnU...@mozilla.org:
> Because people sometimes get carried away with deleting e-mails
> and then realize they accidently deleted something important!?
Oh no, I don't mean compact after every email. I meant "automatic
compaction" whenever TB realizes it's worth compacting either on grounds of
space, performance, corruption likelehood or all of the above. This is
exactly what the option currently does. But right now it has to be
explicitly enabled. So I bet, 90% of the users don't even bother.
--
Rahul
Dave Warren
<nos...@invalid.invalid> was claimed to have wrote:
>"Ron K." <kil...@gisco.net> wrote in
>news:p4ydnRJtAuitLP3R...@mozilla.org:
>
>> There are two factors that set file size limits. On is the hardware
>> and he other is the OS used. While upgrading to newer generation
>> hardware will improver performance, it will not resolve OS issues. For
>> Windows, the Win32 OS's have a 4 GB file cap, it's a limitation of the
>> OS's memory space.
>
>I don't think that's the case. It is only a relic with fat32 file systems.
>With NTFS I belive you can get files up to 16 Terabytes.
>
>See here:
>
>http://technet.microsoft.com/en-us/library/cc766145%28WS.10%29.aspx
>
>So the arbitrary 4 GB limit seems purely a Thunderbird limitation. I don't
>think the OS shares a blame.
Not the OS specifically (although OS APIs contribute), but the CPU
architecture itself dictates 32-bit pointers as being the easiest/most
natural, and as long as you rely on 32-bit pointers to seek within files
you end up with artificial 4GB limits.
This is especially significant with memory-mapped files (although I have
no idea if TB uses memory-mapped files), but it happens just as well
with other file access types.
Either way, 2GB/4GB limits are surprisingly common in applications that
were never really designed with larger-than-4GB files in mind.
Even changing to x64 won't automatically help, it makes using 64-bit
pointers somewhat more natural but the file formats might need to change
too if the files have internal pointers/counters/etc, this type of
change would be significant for Thunderbird since at the moment you can
take files from different TB installations and share them across
platforms.
>Ah! Ok. maybe the trick is to compile my own build then for my
>archetecture. Why doesn't Mozilla create a 32 bit and a 64 bit compile?
>There must be plenty of people around these days using modern hardware!
Lack of any real practical benefits for the vast majority, vs the cost
of building and testing on an additional architecture?
Don't get me wrong, I've been on x64 for many years and am a big fan,
but few applications really benefit just from recompiling to x64.
Bill Braun
Automatic compaction of all folders except Trash (left to manual
compaction) would resolve that.
Bill B
Chris Ilias
> Rahul wrote:
>> I already have automatic compaction enabled. (I never did figure why this
>> isn't a default enabled option. If compaction is so critical to proper TB
>> operation why doesn't TB compact by default.)
>
> Because people sometimes get carried away with deleting e-mails and then
> realize they accidently deleted something important!?
Are you assuming that, or do you have a reference to cite?
I would assume that that's not the reason, because:
* the trash folder is protection against accidentally deleted messages
* no-one expects users to know how to recover a lost message that way
Rahul
P4xDfzRnZ2d...@mozilla.org:
> I would assume that that's not the reason, because:
> * the trash folder is protection against accidentally deleted messages
> * no-one expects users to know how to recover a lost message that way
>
I suspect Annalis is confusing "undeleting" with "un-compacting" (I don't
even think Uncompacting is possible is it?)
--
Rahul
Rahul
>
> Either way, 2GB/4GB limits are surprisingly common in applications
> that were never really designed with larger-than-4GB files in mind.
True. I just hold TB and FF to much higher standards than a "typical"
application. :) Probably because I love them and have always been
impressed with the quality of Mozilla software.
> Even changing to x64 won't automatically help, it makes using 64-bit
> pointers somewhat more natural but the file formats might need to
> change too if the files have internal pointers/counters/etc, this type
> of change would be significant for Thunderbird since at the moment you
> can take files from different TB installations and share them across
> platforms.
But over time 32bit will disaapear anyways. It might make more sense to
provide a 32-64 bit conversion engine if the file formats need to change.
(a valid criticism though is that there's not enough interest in this. If
I want it so badly I should go ahead and code it. :) No contest. )
>
> Lack of any real practical benefits for the vast majority, vs the cost
> of building and testing on an additional architecture?
I'm not so sure that I am in the minority. Just google thunderbird /
performance / compaction / large mailboxes etc. There seems to be tons of
similar complaints. Now, one could argure this is just the stupid-user-
syndrome but I think there is more to this than that.
> Don't get me wrong, I've been on x64 for many years and am a big fan,
> but few applications really benefit just from recompiling to x64.
Absolutely. I agree that most apps get nothing out of moving to 64 bits.
But this might be one situation where 32 bits indeed presents a
roadblock. If what you say about the cause of the 4GB size limitation is
indeed 32bit buses. I'm not convinced yet. :)
--
Rahul
Mark Filipak
-snip-
> ... If what you say about the cause of the 4GB size limitation is
You have good reason to be skeptical.
Bus size has no bearing on virtual memory address size.
In the case of intel P5, the virtual memory limit is in the terrabytes.
Rahul
news:mailman.1720.1281467245....@lists.mozilla.org:
Thanks Mark! But then what is the real reason behind the so-called 4GB
limit? I keep reading this on blogs and a lot of places? Is this just a
myth? Or is there something in the implimentation that breaks apart at that
point?
--
Rahul
Mark Filipak
I suspect it may be disk addressing.
I don't have the latest ATAPI spec., but I believe it's there.
Try Googling "disk drive limits" and have a good read.
Also, try Googling "XP file size limits" (substitute your actual OS).
Rahul
> You should be able to delete the entire ImapMail directory, or an
> entire server's worth of data at a time. It might be possible to
> delete in smaller increments but I've not really experimented.
Just in case it helps anyone else here's how I resolved it. From the
impapail dir I deleted _all_ *.msf files. In addition I also deleted the
messed up INBOX file.
Then I started TB again. It reconstructed all the *.msf files and now
for the last one hour has been busily redownloading my INBOX from the
web imap server. The server did time out but it wasn't a problem since
the rebuild of INBOX resumed from the last spot.
Thu now after one hour I have a brand new INBOX file which is about
1/3rd the size of the corrupted bloated file TB was originally using.
Thanks for all the pointers, guys!
> This is likely not something Mozilla formally supports, but it does
> work nicely enough when issues do arise.
Sometimes the mozilla docs
> Part of the benefit of IMAP is that all of your data is stored on the
> server, so the client stored client-side is disposable.
Yes, exactly. I am so glad I retain all my mail on the IMAP server. :)
> One other thing that I should mention, I have ThunderBird's global
> indexer off. If you use it, ThunderBird is going to redownload
> everything (not just the headers needed to display the message list),
> so if you're on a slow connection expect a long rebuild time.
Where's the global indexer? I guess it will have to redownload
everything anyways coz' I have all my folders checked for local use.
--
Rahul
Rahul
> I suspect it may be disk addressing.
> I don't have the latest ATAPI spec., but I believe it's there.
> Try Googling "disk drive limits" and have a good read.
> Also, try Googling "XP file size limits" (substitute your actual OS).
>
I googled it up. I'm convinced it has nothing at all to do with 32bit, nor
WinXP (so long as you are using NTFS and not fat16/32). The OS nor
hardware have any blame.
If this 4Gig limit is indeed real, it has to a Thunderbird specific design
limitation. Maybe if I find some TB design docs. then I will find out.
--
Rahul
Wayne Mery
file, both opened and fixed - http://bit.ly/acbGOi
I have included bugs marked duplicate, which often have different
wording that the bugs they get duplicated to.