wish for thunderbird: backup tool
Mark Aitchison
email (e.g. recognises mail added - or moved to a new folder - since the
last backup) supplied with thunderbird, and users encouraged to use it.
Backing up the individual mailbox files using conventional software has
three big problems:
a. a large file might only have the last few messages added since the
last backup, making backups much larger than they need to be;
b. knowing what files to back up, and how to extract individual
messages back into the system, is beyond the average user (and replacing
the current Inbox with a backed up version to restore an old message
risks wiping lots of new ones!).
c. periodic conventional backups are unlikely to help if mail is
constantly coming in and being deleted by users between backups.
There seems to be no useful backup add-on (my search showed only two
add-ons for "backup", neither was entirely appropriate).
Features needed:
1. works under linux, windows, mac etc.
2. uses some industry-standard archiver (preferably have the option of
.zip, .tgz, .zoo)
3. should be able to back up to local disk, area on LAN, with hooks to
allow external programs to use ftp upload, email, rsync, etc.
4. should allow user to select how to handle deleted messages, what
folders to back up and when (e.g. mail to some folders may be specified
for backup as soon as mail arrives, others on close or weekly backups, etc).
5. must have an easy-to-use "backup everything" option that ensures you
can recreate your mail setup (with messages, preferences, add-ons,
spelling additions even).
6. should have a tool to extract backed-up messages one-by-one or by
folder, to their original place or some other folder.
7. could also have a method of checking the backup against the current
folders/preferences and reporting differences.
8. document the storage details well enough for people to write tools
that can use the archives, e.g. to migrate to a new computer or report
mail statistics over a longer period than mail is held locally.
9. possibly allow archived mail to be searched (keep searchable details
in a fast database) and treated as another folder (large, and perhaps
slower to access - maybe even read-only).
10. it should be easy for system administrators (especially under
Unix-like systems) to implement a backup strategy for all users of a
large computer system.
Mark Aitchison, Christchurch
Joshua Cranmer
> I would very much like to see a backup tool especially designed for
> email (e.g. recognises mail added - or moved to a new folder - since the
> last backup) supplied with thunderbird, and users encouraged to use it.
Backups tend to need slightly more configuration than a simple email
client can provide.
1. Are you talking about backing up for enterprises, where
customizability is highly important? Or consumer usage, where
"point-and-click" is important?
2. What type of backups? Incremental backups for a week, followed by a
renewed snapshot? A snapshot every time you say "back up"? Automated
backups?
> Backing up the individual mailbox files using conventional software has
> three big problems:
>
> a. a large file might only have the last few messages added since the
> last backup, making backups much larger than they need to be;
1. Disk space is mostly cheap. 60GB+ hard drives are standard now, and,
even after OSs and programs are taken into account, most users (if they
don't have mammoth music or video libraries on their computers) are not
going to be using anywhere near that amount of space.
2. Have you tried incremental backups?
3. See below.
> b. knowing what files to back up, and how to extract individual
> messages back into the system, is beyond the average user (and replacing
> the current Inbox with a backed up version to restore an old message
> risks wiping lots of new ones!).
See below.
> c. periodic conventional backups are unlikely to help if mail is
> constantly coming in and being deleted by users between backups.
1. So what you want is a system that backs up mail as it comes on and
keeps everything on perma-archive until a backup is destroyed.
2. (This is the see below part) One of the current mid-term plans is to
implement a directory-style backup system. To what degree will your
problems still be valid when this is done?
> 1. works under linux, windows, mac etc.
> 2. uses some industry-standard archiver (preferably have the option of
> .zip, .tgz, .zoo)
.zoo? Haven't heard of that one.
> 3. should be able to back up to local disk, area on LAN, with hooks to
> allow external programs to use ftp upload, email, rsync, etc.
> 4. should allow user to select how to handle deleted messages, what
> folders to back up and when (e.g. mail to some folders may be specified
> for backup as soon as mail arrives, others on close or weekly backups,
> etc).
> 5. must have an easy-to-use "backup everything" option that ensures you
> can recreate your mail setup (with messages, preferences, add-ons,
> spelling additions even).
> 6. should have a tool to extract backed-up messages one-by-one or by
> folder, to their original place or some other folder.
> 7. could also have a method of checking the backup against the current
> folders/preferences and reporting differences.
I think you might need to make up your mind: a profile-level backup or a
specialized mail backup system. The first one is doable enough with
current tools, whereas the latter, at best, would be an extension.
> 8. document the storage details well enough for people to write tools
> that can use the archives, e.g. to migrate to a new computer or report
> mail statistics over a longer period than mail is held locally.
> 9. possibly allow archived mail to be searched (keep searchable details
> in a fast database) and treated as another folder (large, and perhaps
> slower to access - maybe even read-only).
And now you are beginning to sound to me like "Well, why not do Z in
addition to X and Y, since Z might be wanted by some people and sounds
sort of like X and Y?"
> 10. it should be easy for system administrators (especially under
> Unix-like systems) to implement a backup strategy for all users of a
> large computer system.
And here I ask you again: is this for simple end-users or enterprise
managers?
(P.S. Sorry if I'm sounding a bit agitated right now. I've been having
problematic code recently...)
Mark Aitchison
> Mark Aitchison wrote:
>> I would very much like to see a backup tool especially designed for
> Backups tend to need slightly more configuration than a simple email
> client can provide.
Yes, but email clients are in a position to know what/when to back up
that normal backup programs cannot; still, a complicated, all-singing,
all-dancing backup facility is more than a simple email client should
try to do - I do think that there is room for (say) thunderbird to do
some backup operations (which any well-run computer system, be it home
or office, *should* do) and provide hooks for external software to
provide more comprehensive options.
> 1. Are you talking about backing up for enterprises, where
> customizability is highly important? Or consumer usage, where
> "point-and-click" is important?
Both, really. I appreciate there are differences in the situations if
you want a whole-hog backup strategy, but there are important things in
common, which can be tackled. I am personally interested in both home
systems (where ease of use is important) and organisations (where a guru
should be able to set up all users at once to automatically back up to
somewhere sensible, perhaps off-site automatically after some encryption.
> 2. What type of backups? Incremental backups for a week, followed by a
> renewed snapshot? A snapshot every time you say "back up"? Automated
> backups?
Two priorities for me:
1. a tool that is easily found in the "Tools" pulldown menu, that
makes a complete backup copy of your mail system - messages and
configuration - so people can know that, if anything goes wrong with an
upgrade, etc, the mail will certainly be able to be restored completely.
Perhaps it would have a corresponding restore tool, or maybe "import"
could be adapted to use the backups. The only options needed would be:
a. where to make the backup (e.g. a USB drive, any local/network disk;
allow add-ons to augment the list with ftp servers or whatever).
b. do you want to stop any more mail activity, e.g. new messages
arriving, from the start of the backup until the program is closed
down (so you know the backup is completely up to date, e.g. before
an upgrade).
2. An automatic incremental backup, which is where regular backup
software has difficulties: a normal file-by-file incremental will
probably have to backup many old messages in the inbox; even if a mail
folder was used with a file per message, there is a problem missing many
emails in a nightly backup if people receive and then accidentally
delete a new message during the day. The built-in backup facility
should, at the least, allow:
a. new messages (in or out) to be appended to a backup archive as
they arrive, or before deletion, or nightly, or folder exceeding a
certain size, or not at all; the choice based on simple criteria such as
message size, subject, folder, sender etc.
b. the process of backing up would normally be something like gzip of
each message, appending a backup file whose name changes each month,
plus adding searchable information in a simple database.
c. some hooks to allow external programs to do the archiving in
larger organisations, that may include encryption or central logging or
distributed redundant databases.
d. a simple interface to backed-up messages, e.g. making them appear
as if they are in a folder tree of their own, so they can be copied back
if needed.
>> a. a large file might only have the last few messages added since the
>> last backup, making backups much larger than they need to be;
>
> 1. Disk space is mostly cheap.
But backup times can be annoyingly slow (e.g. if you want to do a backup
just before switching off the computer). Also, media such as CDroms,
rewritable DVDs and mag tapes are somewhat limited. And then how do you
search huge collections - e.g. you know a message came in last month,
scanning many large copies of folders made each day sequentially for the
message is inefficient... it is better in the hands of software that
understands the innards of the mail files.
And many organisations have a mixture of old and new computers, some of
which have very limited disk space/speed.
> 2. Have you tried incremental backups?
Yes, in two organisations in which I worked during the last decade the
problem was quite significant; emails were a quite common subject of
frantic requests for recovery yet it was slow looking in incremental
backups, and often the message wasn't there. Good practice should be to
have automated backups (as well as turning on the option of leaving
recent messages on the server), but also the fuss involved in making a
complete backup of the current mail before risky operations such as
upgrades is worse than it should be, so I suspect people don't do it
when they should.
> 1. So what you want is a system that backs up mail as it comes on and
> keeps everything on perma-archive until a backup is destroyed.
Backing up as it comes in (or is composed) should be an option, but
giving a choice would be ideal. Plus a "backup everything now" tool.
> 2. (This is the see below part) One of the current mid-term plans is to
> implement a directory-style backup system. To what degree will your
> problems still be valid when this is done?
Hmmm, probably helps with at least half the problem... where do I find
out more?
> ..zoo? Haven't heard of that one.
Zoo is an old archiver that runs under lots of systems, that has one
huge advantage for backups (not, perhaps, so much here, but I'd have to
think about it more): it allows *generations* of files in an archive,
e.g. if you set the generation limit to 3 you could have the latest
config file, the version before it, and the one before that. Basically,
though, I'm used to being able to say what program to call for a
particular activity, what parameters to give it, then giving it data on
its standrad input, which is a bit of a unixism, but it is exactly what
my thunderbird 2.0.0.9 does for printing under linux - it uses the
command "lpr -Php5550", and archivers like zip, zoo and so on will have
options that say "take the list of files to compress from stdin" or
"take the text to compress from stdin", so it shouldn't matter to
thunderbird what happens out there when it asks some external program to
do make an archive out of what it specifies. Making incremental
archives is also something many archivers can do, but with varying
complications, so this might need some thought - possibly a wrapping
batch file/shell script??
> I think you might need to make up your mind: a profile-level backup or a
> specialized mail backup system. The first one is doable enough with
> current tools, whereas the latter, at best, would be an extension.
I'm really wishing for something basic built into the mail client, but
with optional "fancy features" available as add-ons or external programs
that system administrators could set up.
>> 8. document the storage details well enough for people to write tools
>> that can use the archives, e.g. to migrate to a new computer or report
>> mail statistics over a longer period than mail is held locally.
>> 9. possibly allow archived mail to be searched (keep searchable
>> details in a fast database) and treated as another folder (large, and
>> perhaps slower to access - maybe even read-only).
>
> And now you are beginning to sound to me like "Well, why not do Z in
> addition to X and Y, since Z might be wanted by some people and sounds
> sort of like X and Y?"
Well, I'm thinking of the backup system in terms of how it might
actually be used - what people would need. It doesn't take much
implement the core of the backup activity - to make copies of a message
as it comes in, or of every folder and config file in a big backup - but
the work is in providing options that some people might want, as you
say; so if we think in advance what people may reasonably need in the
future the simple backup system can be designed correctly from the
ground up - not with all the fancy options built in but with an
understanding of hooks needed for add-ons that may come later.
> (P.S. Sorry if I'm sounding a bit agitated right now. I've been having
> problematic code recently...)
's ok.
Mark
ingemar
> I would very much like to see a backup tool especially designed for
I am using a setup that solves some of your problems.
Incoming mail is fetched by getmail from my ISP's
Mails are individually stored in a maildir structure,
ie. not merged into mailbox files. Backup by rsync.
Mails are served by Courier IMAP to Thunderbird clients
on linux and windows boxes.
I use an unattended linux box with cronscripts for my
mail server, The client boxes are on a LAN, I think it
would be possible to to use the maildir structure with
getmail, Courier IMAP and Thunderbird on one single linux
box.
I don't know if there is a similar windows solution.
/ingemar
Chandrabhan Gupta
know more:- https://www.data-recovery-solutions.com/thunderbird-mail-backup.php
Kiran Sharma
Know more- https://www.migrateemails.com/thunderbird-backup.php