Better Faster IMAP
David Ascher
very roughly and incorrectly calling "better offline IMAP", but which is
really about leveraging the offline mode capabilities to provide faster
workflow when online. See
http://wiki.mozilla.org/MailNews:Better_Faster_IMAP for more prose on that.
I think that pretty much all of that is exactly where I'd like to see
Thunderbird head in terms of increasing the usability, smoothing out
workflow for many users (viz. those with IMAP servers and persistent
network connections), leveraging the work that was put into the offline
infrastructure.
Emre and I reworked that page into a plan of attack, at
http://wiki.mozilla.org/MailNews:Better_Faster_IMAP_Plan
which is a version of the orginal that's a bit more broken down
step-by-step, and calls out UX decisions that need to be made before we
can release such a thing.
A couple of points:
* Emre is going to be point on this plan, but anyone who wants to
assist should get in touch with him.
* We'll see how much of this we can get into alpha 2.
--da
JoeS
Seems like a lot of those principals could also apply to a Better Faster NNTP as well.
Saving body content like a large graphics image always results in another content download from the server.
--
JoeS
Ron K.
Right, there have been NNTP bugs posted 3-5 years ago where David B.
commented He thought a fix would leverage future work in IMAP. We waste
a lot of time in NNTP where the cause is a re-fetch of another copy of
the posting, even though the first one was decode processed. Even a
local HDD temp cache. An alternative, can we setup another RAM cache
for decoded attachments. Basically, any strategy that gives a single
retrieval, view or save many efficiency, for both text/plain and
text/html news with binary attachments listed in the attachments bar.
--
Ron K.
Who is General Failure, and why is he searching my HDD?
Kernel Restore reported BSOD use by Major Error to msg the enemy!
Emre Birol
> Seems like a lot of those principals could also apply to a Better Faster
> NNTP as well.
I think everybody agree. There is already a decision about making the
implementation as generic/re-usable as possible -- as long as it is
doable in a reasonable timeline.
Frank Winkler
>Seems like a lot of those principals could also apply to a Better Faster
>NNTP as well.
I didn't see any apparent protocol probs for NNTP but the news reader UI is
_very_ poor.
Regards
fw
Ron K.
There is a possible NNTP protocol problem when the user is working with
a news group chartered for binary attachment postings. When one
accesses a post and processes the attachment, Tb wil not fetch the next
header with out first highlighting some other header then reselecting
the desired header.
--
Ron K.
Who is General Failure, and why is he searching my HDD?
Kernel Restore reported Major Error used BSOD to msg the enemy!
dalerw...@gmail.com
clamoring for my attention but now am back full time on Penelope/TB.
For several years I was the IMAP developer on the Classic Eudora team
and now spend most of my Penelope time implementing Classic Eudora
IMAP features in Penelope/TB.
Around the time of Eudora's demise we were going exactly the same
direction as described in the "Better Fast IMAP" document so I have a
lot of thoughts on this topic. I'll start with some quick background
on what we did and were planning to do.
We did not get around to the "download all messages for offline" idea
but that was next on our list. Regarding the comment:
"This will make it so whenever the user opens an imap folder, we
download the message bodies at the same time as we download the
message headers. If there are a lot of message bodies to download
(e.g., the new profile case), this will take a long time and not be a
great experience, but it's what a lot of IMAP clients do (Outlook
Express, Outlook, Mail.app)."
If a user was downloading full bodies then, yes, our plan was to just
download the full bodies all at once in the quickest way possible
(i.e., through a single IMAP command). However, in the case where the
user chose to download headers only we were going to download the
headers at first then in the background begin downloading the message
bodies one at a time. We felt that the improved user experience (not
having to wait while the app downloaded messages the user didn't
necessary want in the first place) more than offset the fact that it
might take significantly longer to download the entire set of message
bodies. So if a mailbox contained 5 new messages, we would download
the headers for all five, wait for an idle time then begin downloading
the body of the first new message. If the user clicked on the third
new message then at worst they would have to wait until the first
message download finished before the message they clicked on was
downloaded and displayed. As I said, we never got far enough to
implement this so we have no actual user experience data to report.
Regarding the following:
"Are there other imap operations that we would want to do "offline",
like renaming a folder, or should we stick with the most often used
commands, like reading a message, move, delete, etc?"
We did every possible command offline. We also took the extra step of
making all IMAP commands resulting from user actions, even those that
required immediate feedback from the server to happen (e.g., a mail
check or a mailbox list refresh), pass through the same code flow: do
any local action, place the command in a queue and then begin removing
items from the queue and performing them online. This required us to
implement the notion of priorities for events in the queue. Events
that required immediate data from the server were given a priority of
"now" meaning that they were popped of the queue pretty much
immediately after being placed on the queue. Even though this could
potentially slow down the operation our user experience suggested this
worked fine -- I am not aware of any users even being aware of this
extra activity.
Each mailbox maintained its own queue of actions to be performed and
used background threads to perform each action online. If the user
quit Eudora with tasks still waiting we stopped all processing and
resumed it the next time Eudora was started. By default, the user was
not made aware of what Eudora was doing in the background, however
Eudora did provide an optional "Task Status" pane in the window which
showed what tasks Eudora was performing at the time and these IMAP
tasks did show up in this pane.
I was probably the heaviest user of this feature because I used it
throughout the entire development process even before the first public
alpha and I have to say that I really liked the end result. To delete
a message, rename a mailbox, whatever, and see it happen immediately
was a joy. Of course, it helped that one of the servers I was using at
the time occasionally had latencies of 2 seconds or more, making the
apparently instantaneous response feel that much better.
I'll stop there for now but I am more than happy to discuss further
what we did (or planned to do) and why. I am obviously willing to
contribute to this effort in whatever way I can.
-Dale
Emre Birol
> Apologies for coming late to this discussion. I had other projects
> clamoring for my attention but now am back full time on Penelope/TB.
> For several years I was the IMAP developer on the Classic Eudora team
> and now spend most of my Penelope time implementing Classic Eudora
> IMAP features in Penelope/TB.
Welcome aboard :)
>
> We did every possible command offline. We also took the extra step of
> making all IMAP commands resulting from user actions, even those that
> required immediate feedback from the server to happen (e.g., a mail
> check or a mailbox list refresh), pass through the same code flow: do
> any local action, place the command in a queue and then begin removing
> items from the queue and performing them online. This required us to
> implement the notion of priorities for events in the queue. Events
> that required immediate data from the server were given a priority of
> "now" meaning that they were popped of the queue pretty much
> immediately after being placed on the queue. Even though this could
> potentially slow down the operation our user experience suggested this
> worked fine -- I am not aware of any users even being aware of this
> extra activity.
I think this is where we want to go eventually. Even possible to divide
the download task into sub tasks using partial fetch support in IMAP to
increase the granularity of the background downloads. As I understand
it, in the current implementation when the user selects another message
in the list before the previously selected one is completely downloaded,
TB interrupts the download and loses all downloaded parts. Partial
fetching combined with a prioritized queue like one you have mentioned
would improve the existing behavior drastically without sacrificing from
responsiveness.
>
> Each mailbox maintained its own queue of actions to be performed and
> used background threads to perform each action online.
How did you manage connections between Eudora and IMAP server? (1
Connection + 1 Thread) per folder or 1 manager thread and 1 connection
per folder?
If the user
> quit Eudora with tasks still waiting we stopped all processing and
> resumed it the next time Eudora was started. By default, the user was
> not made aware of what Eudora was doing in the background, however
> Eudora did provide an optional "Task Status" pane in the window which
> showed what tasks Eudora was performing at the time and these IMAP
> tasks did show up in this pane.
>
++
Have you seen Bryan's interactive status bar mockups yet:
http://clarkbw.net/designs/interactive-status-bar/interactive-status-bar-history%20(failure).png
dalerw...@gmail.com
> dalerwigg...@gmail.com wrote:
> > We did every possible command offline. We also took the extra step of
> > making all IMAP commands resulting from user actions, even those that
> > required immediate feedback from the server to happen (e.g., a mail
> > check or a mailbox list refresh), pass through the same code flow: do
> > any local action, place the command in a queue and then begin removing
> > items from the queue and performing them online. This required us to
> > implement the notion of priorities for events in the queue. Events
> > that required immediate data from the server were given a priority of
> > "now" meaning that they were popped of the queue pretty much
> > immediately after being placed on the queue. Even though this could
> > potentially slow down the operation our user experience suggested this
> > worked fine -- I am not aware of any users even being aware of this
> > extra activity.
>
> I think this is where we want to go eventually. Even possible to divide
> the download task into sub tasks using partial fetch support in IMAP to
> increase the granularity of the background downloads. As I understand
> it, in the current implementation when the user selects another message
> in the list before the previously selected one is completely downloaded,
> TB interrupts the download and loses all downloaded parts. Partial
> fetching combined with a prioritized queue like one you have mentioned
> would improve the existing behavior drastically without sacrificing from
> responsiveness.
Agreed, the finer the granularity the better. Partial message fetching
would certainly help.
> > Each mailbox maintained its own queue of actions to be performed and
> > used background threads to perform each action online.
>
> How did you manage connections between Eudora and IMAP server? (1
> Connection + 1 Thread) per folder or 1 manager thread and 1 connection
> per folder?
I was not the original author of the Eudora IMAP code. The original
author chose 1 thread + 1 connection + 1 mailbox and coded it in a way
that made it extremely hard to change so we stuck with that model. I
was not around for the original design discussion so I do not know
what pluses and minuses were discussed, but I don't recall our model
ever coming back to bite us.
> > If the user quit Eudora with tasks still waiting we stopped all processing and
> > resumed it the next time Eudora was started. By default, the user was
> > not made aware of what Eudora was doing in the background, however
> > Eudora did provide an optional "Task Status" pane in the window which
> > showed what tasks Eudora was performing at the time and these IMAP
> > tasks did show up in this pane.
>
> ++
> Have you seen Bryan's interactive status bar mockups yet:http://clarkbw.net/designs/interactive-status-bar/interactive-status-...
Not until now. That is quite similar to what we did in Eudora.
-Dale