nnimap eats all email when IMAP action fails

[Gleb Arshinov]

Now that Simon solved my nnimap performance problem I felt inspired to
investigate this other bug. I can consistently reproduce it at work
running against MS Exchange, and I've seen something similar running
against Linux IMAP servers as well.

The root cause seems that when UID SEARCH UID fails, nnimap ends up
deleting/expiring all articles involed in this UID SEARCH UID command.
UID SEARCH UID seems to fail on large lists of sequences. In Exchange
case "large" is pretty small :-), but I've seen similar problems on
very large "holey" mailboxes on uw-imapd.

More detailed report is below.

Gleb

Test case:

In 5.10.2 I need to (setq gnus-agent nil) to reproduce this. I also
notice that nnimap-nov-is-evil is set to true by default.

I have a large ~1200 message nnimap group. If all articles in the
group are marked !, there is no problems. If I go into the group and
and mark ~10 non-contiguous articles as E and then enter the group I get:

Retrieving newsgroup: nnimap+pivia.com:Mail/new/default...
nnimap: Updating info for nnimap+pivia.com:Mail/new/default...
nnimap: Updating info for nnimap+pivia.com:Mail/new/default...done
Fetching headers for nnimap+pivia.com:Mail/new/default...
nnimap: Retrieving headers... /
nnimap: Retrieving headers... -
...
nnimap: Retrieving headers... \
nnimap: Retrieving headers...done
Fetching headers for nnimap+pivia.com:Mail/new/default...done
No articles in the group
nnimap: Setting marks in Mail/new/default...
nnimap: Setting marks in Mail/new/default...done
No unread news

If I then C-u RET the group, I see that the artcles marked as E in the
last step are still marked as such, all the other articles (were
marked !) are now O.

Comparing *imap-log* files for no-bug with bug I see that nnimap does:

104 UID SEARCH FLAGGED

This returns large list of numbers. I see that articles I marked
as E are missing from the list, creating more "holes" in the
sequence.

118 UID SEARCH UID 1:15,17:27,29:105,... [... is mine]

It seems that nnimap compressed FLAGGED list into a list of
sequences and there more sequences involved when some articles are
marked as E.

118 NO Unspecified error.

Exchange fails when the list above is large. In the no-bug case
it succeds and returns the seemingly the same list as in step 104.
In no-bug case, this is the end of IMAP activity.

119 UID FETCH 1:15,17:27,29:105,... (UID RFC822.SIZE BODY BODY.PEEK[HEADER.FIELDS (Subject From Date Message-Id References In-Reply-To Xref To Newsgroups)])

In the bug case, this proceeds to fetch headers from all the
messages

120 UID STORE 1:15,17:27,29:105... -FLAGS (\Flagged)

* 1 FETCH (FLAGS (\Seen) UID 1)
* 2 FETCH (FLAGS (\Seen) UID 2)
* 3 FETCH (FLAGS (\Seen) UID 3)
....

We set the flags to equivalent of O

120 OK STORE completed.
121 STATUS "Mail/new/default" (unseen)
* STATUS Mail/new/default (UNSEEN 0)
121 OK STATUS completed.
122 EXAMINE "Mail/new/default"
* 1210 EXISTS
* 0 RECENT
* FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)
* OK [PERMANENTFLAGS ()] Permanent flags
* OK [UIDVALIDITY 34073] UIDVALIDITY value
122 OK [READ-ONLY] EXAMINE completed.
123 CLOSE
123 OK CLOSE completed.

The end