What storage format to choose?

[Cleverson Casarin Uliana]

Hello,

I'd like to know if there is currently a preferred storage format for
setting up Thunderbird to use, between MBox and Maildir. I've opted for
Maildir since last year, and now I notice my storage folder have near to
230,000 files. My system is a Windows 10 with the standard NTFS
filesystem. Because I'm not an expert in these things, I wonder for example:
Is the TB general performance and stability better when using a storage
format rather than another? Do regular backukps run better with one
rather than the other? Also, I've tried Linux some times, though never
migrated to it. Is any format better for the Linux filesystems e.g. ext4?

Thank you,
Cleverson

[Wolf K]

AFAICT, there's little difference. Maildir takes more space, since on
average a message will waste about 1/2 cluster. Mbox needs more ongoing
maintenance (compact folders). TB's Search works with both formats.

Main advantage of Maildir from my POV: you can rename associated/related
messages from the default file names that TB assigns.

So it's really a matter of taste, I think.

However:
I've found that about 1/2 the messages are obsolete as soon as I've
dealt with them (which usually takes at most a day or two), so I delete
them. The rest are keepers for operational/legal/sentimental/reference
reasons, and most of those don't need to be kept on the working-data
drive. So this is what I did/do:

a) create subfolders, using classifications expedient for me; and
b) delete any messages that were neither current nor keepers; and
c) back up the Profile every 6 months, then delete inactive subfolders
and items.

At the rate you accumulate messages, I'd suggest doing c) more often.


Best,
--
Wolf K
kirkwood40.blogspot.com
It's called an "opinion" because it's not a fact.

[Cleverson Casarin Uliana]

Hi, first many thanks for all of hints.

I see that TB has an option, under account settings, to sync or not the
messages to this computer, or sync only the latest messages. So perhaps
an alternative is to turn the syncing off or turn on syncing of latest
messages, then backup before it starts deleting the oldest ones,
something in this line...

Greetings,
Cleverson

[Mark Rousell]

On 29/08/2018 17:39, Cleverson Casarin Uliana wrote:

Hello,

I'd like to know if there is currently a preferred storage format for setting up Thunderbird to use, between MBox and Maildir. I've opted for Maildir since last year, and now I notice my storage folder have near to 230,000 files. My system is a Windows 10 with the standard NTFS filesystem. Because I'm not an expert in these things, I wonder for example:
Is the TB general performance and stability better when using a storage format rather than another? Do regular backukps run better with one rather than the other? Also, I've tried Linux some times, though never migrated to it. Is any format better for the Linux filesystems e.g. ext4?

I recommend against Maildir for now (on any OS platform). The reason for this is that there, I understand, still significant bugs in Thunderbird's Maildir handling. Sorry, I don't have a list of them but what I've read is enough to put me off using Maildir for now on any platform. Some users who use Maildir report no problems but I still wouldn't want to risk it; I suspect that whether or not you are hit by one of the bugs depends on what features of TB you use.

As and when the bugs are eventually ironed out (or if you really do want to risk it) then, if it were me, I'd certainly want to move to Maildir if I was on Linux or Mac.

However, I have very severe reservations about using Maildir on Windows (even with the TB maildir bugs fixed). This is because of (a) file path length limitations in the Win32 API and (b) file management issues in Windows with directories with very large numbers of small files.

For more on the issues in Windows, see the recent thread in the tb-enterprise list which begins at this message: https://mail.mozilla.org/pipermail/tb-enterprise/2018-August/001426.html

My contributions to the thread which outline the problem and attempt to indicate how much of an issue Maildir in TB might be on Windows are here:
https://mail.mozilla.org/pipermail/tb-enterprise/2018-August/001427.html
https://mail.mozilla.org/pipermail/tb-enterprise/2018-August/001428.html
https://mail.mozilla.org/pipermail/tb-enterprise/2018-August/001430.html
https://mail.mozilla.org/pipermail/tb-enterprise/2018-August/001431.html
https://mail.mozilla.org/pipermail/tb-enterprise/2018-August/001433.html
https://mail.mozilla.org/pipermail/tb-enterprise/2018-August/001434.html

The above messages also include references to a possible solution to file path length problem in up to date versions of Windows 10, but it's a solution that has not been well tested as yet.

-- 
Mark Rousell
 
 
 

[Wayne]

On 8/30/2018 4:28 AM, Mark Rousell wrote:
> On 29/08/2018 17:39, Cleverson Casarin Uliana wrote:
>> Hello,
>>
>> I'd like to know if there is currently a preferred storage format for
>> setting up Thunderbird to use, between MBox and Maildir. I've opted
>> for Maildir since last year, and now I notice my storage folder have
>> near to 230,000 files. My system is a Windows 10 with the standard
>> NTFS filesystem. Because I'm not an expert in these things, I wonder
>> for example:
>> Is the TB general performance and stability better when using a
>> storage format rather than another? Do regular backukps run better
>> with one rather than the other? Also, I've tried Linux some times,
>> though never migrated to it. Is any format better for the Linux
>> filesystems e.g. ext4?
>
> I recommend against Maildir for now (on any OS platform). The reason for
> this is that there, I understand, still significant bugs in
> Thunderbird's Maildir handling. Sorry, I don't have a list of them

The list is trivial via the dependency list of meta bug 845952 aka
"maildirblockers",
https://bugzilla.mozilla.org/showdependencytree.cgi?id=845952&hide_resolved=1

Some are quite severe. And there are surely more bugs to be discovered.

[Mark Rousell]

On 30/08/2018 11:21, Wayne wrote:

On 8/30/2018 4:28 AM, Mark Rousell wrote:

I recommend against Maildir for now (on any OS platform). The reason for this is that there are, I understand, still significant bugs in Thunderbird's Maildir handling. Sorry, I don't have a list of them

The list is trivial via the dependency list of meta bug 845952 aka "maildirblockers", https://bugzilla.mozilla.org/showdependencytree.cgi?id=845952&hide_resolved=1

Some are quite severe. And there are surely more bugs to be discovered.

Many thanks.

-- 
Mark Rousell
 
 
 

[opto]

I think Wayne posted somewhere that maildir has not yet all features of mboc

[Cleverson Casarin Uliana]

Hi, thank you all.

I don't need Windows Search integration, and I use to put the storage
folder hierarchy under another partition with a shorter name than
"c:\users\foobar\appdata\roaming". So, probably would never come across
the path length bugs.

As for the bugs where TB crashes while indexing, I think I've
experienced one, where TB crashes when it tries to display a folder
while the Internet connection is momentarily absent, then it starts
rebuilding all the messages' folder hierarchy, but doesn'd delete the
older hierarchy it won't use anymore.

In any case, can you say whether TB performance is better migrating to
MBox, or simply turning off the account synsing to my local computer? I
could for example let TB not to synchronize, then once in a while I
would manually sync and backup all the recent content.

Greetings,
Cleverson

[Mark Rousell]

On 30/08/2018 19:29, Cleverson Casarin Uliana wrote:

In any case, can you say whether TB performance is better migrating to MBox, or simply turning off the account synsing to my local computer?

I'm not sure whether mbox or maildir is faster in general. I can only suggest doing your own tests to see which works best for you.

I could for example let TB not to synchronize, then once in a while I would manually sync and backup all the recent content.

That sounds like a lot of annoyance to me. I would have thought that it would be better to sync regularly as normal, if you can.

-- 
Mark Rousell
 
 
 

[Cleverson Casarin Uliana]

Hello all, still on this topic, I'd like to do some experiments here,
but I'm asking some questions first as not to risk any data.

I don't want TB to continue syncing messages to my computer, and I want
the previous synced messages to be deleted from this computer, but not
from the imap server of course. In the account settings, under the sync
tab, I unchecked the option "preserve messages of this account on this
computer" (I don't know the exact wording in English, which isn't my
language...), then clicked OK. Now I guess that TB won't sync messages
any longer, but I notice it hasn't deleted the previous synced messages,
which constitute thousands of files. Is it safe for me to manually
delete them? My fear is that TB somehow detects I've deleted then, and
deletes them from the server.

Thanks,
Cleverson

[Mark Rousell]

On 01/09/2018 19:59, Cleverson Casarin Uliana wrote:

Hello all, still on this topic, I'd like to do some experiments here, but I'm asking some questions first as not to risk any data.

I don't want TB to continue syncing messages to my computer, and I want the previous synced messages to be deleted from this computer, but not from the imap server of course. In the account settings, under the sync tab, I unchecked the option "preserve messages of this account on this computer" (I don't know the exact wording in English, which isn't my language...), then clicked OK. Now I guess that TB won't sync messages any longer, but I notice it hasn't deleted the previous synced messages, which constitute thousands of files. Is it safe for me to manually delete them? My fear is that TB somehow detects I've deleted then, and deletes them from the server.

To be honest I am not completely certain. For my usage scenarios, I find POP3 to be more useful than IMAP and so I am not very familiar with the IMAP settings.

In your scenario, I think this would do what you need:

(a) Disable syncing for the IMAP account (already done as above).
(b) Move the messages you want to delete to a folder within Local Folders.
(c) Delete the messages from within Local Folders.

This is probably overkill but I think it should guarantee that you are only deleting your local copies without deleting from the server.

I should really get to know the exact meaning of all the IMAP-related settings.

Hope that helps but if anyone else can comment then it might be helpful.

-- 
Mark Rousell
 
 
 

[Wayne]

Deleted FILES locally does not delete from the server.
https://support.mozilla.org/en-US/questions/1085701#answer-787296

[opto]

In imap i think if they are Move that is like deleting or der delete Flagge.

So Move from imap to local = der delete Flagge in imap
Imap expunge ... they are gone
Local Folder delete ... they are gone there...
So in imap either gone or delete flag is set and they are invisible.

[Mark Rousell]

Ah, thank you.

If so, that's not helpful behaviour, although one would hope that the delete flag was not sent to the server if synchronisation is disabled.

I really must try out IMAP in TB much more thoroughly, it seems.

-- 
Mark Rousell
 
 
 

[Cleverson Casarin Uliana]

Hi, thank you all. The forum answer referred by Wayne is right. After
turning syncing off at the account settings yesterday, I now just closed
TB and deleted the previous content using a file manager, and
everything's alright.

Greetings,
Cleverson