K9 IMAP bug or different IMAP version? (logs included)

44 views
Skip to first unread message

Joa H

unread,
Apr 21, 2016, 3:35:34 AM4/21/16
to K-9 Mail
When K9 connects to my IMAP server the most recent messages are not displayed. The reason can be seen in the below log (from the server).

This is my interpretation, but I'm no IMAP expert.
Note the four bold lines: In the first of these the server says that there are 112 messages. In the second it says that the next message is predicted to have UID 272, so the most recent UID should be 271. In the third one K9 asks which messages between UID 63 and 112 are not deleted. In the third line K9 asks for these messages. All messages in the UID range 113 to 271 are omitted, which is obviously wrong. K9 treats the number of existing messages as the greates UID.

Thunderbird on my laptop handles this correctly (see second log below) by asking for all messages to be fetched. Notable there is that the largest UID is 271.

Setting the "local folder size" (translation) of K9 to "all messages", as opposed to the default 25, does not remedy the problem.
I have a Sony Xperia Z5 running Android 6.0 and K9 5.010. I didn't have this problem before updating to Marshmallow.

Who's at fault here, K9 or my server (or me)?
Greatful for any help.
Joa


2016-04-21 08:21:39,429 -  - [     690] ************** New Connection from: x.x.x.x**************
2016-04-21 08:21:39,430 -  - [     690] S <-- * OK Synametrics IMAP4rev1 server ready 4/21/16 8:21 AM
2016-04-21 08:21:39,546 -  - [     690] C --> 1 CAPABILITY
2016-04-21 08:21:39,547 -  - [     690] S <-- * CAPABILITY IMAP4rev1 IDLE
2016-04-21 08:21:39,547 -  - [     690] S <-- 1 OK CAPABILITY completed
2016-04-21 08:21:39,550 -  - [     690] C --> 2 LOGIN "x@x.x" xxxxxx
2016-04-21 08:21:39,550 -  - [     690] S <-- 2 OK LOGIN completed
2016-04-21 08:21:39,554 -  - [     690] C --> 3 SELECT "INBOX"
2016-04-21 08:21:39,605 -  - [     690] S <-- * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
2016-04-21 08:21:39,605 -  - [     690] S <-- * 112 EXISTS
2016-04-21 08:21:39,605 -  - [     690] S <-- * OK [UIDVALIDITY 14] UID validity status
2016-04-21 08:21:39,605 -  - [     690] S <-- * OK [UIDNEXT 272] Predicted next UID
2016-04-21 08:21:39,605 -  - [     690] S <-- * 0 RECENT
2016-04-21 08:21:39,605 -  - [     690] S <-- * OK [PERMENENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft)]
2016-04-21 08:21:39,605 -  - [     690] S <-- 3 OK [READ-WRITE] SELECT completed
2016-04-21 08:21:39,647 -  - [     690] C --> 4 UID SEARCH 63:112 NOT DELETED
2016-04-21 08:21:39,649 -  - [     690] S <-- * SEARCH 64 65 66 67 69 81 85 92 93 94 95 96 97 98 101 102
2016-04-21 08:21:39,649 -  - [     690] S <-- 4 OK SEARCH completed
2016-04-21 08:21:39,718 -  - [     690] C --> 5 UID FETCH 102,101,98,97,96,95,94,93,92,85,81,69,67,66,65,64 (UID FLAGS)
2016-04-21 08:21:39,720 -  - [     690] S <-- * 102 FETCH (UID 102 FLAGS (\Seen))
2016-04-21 08:21:39,720 -  - [     690] S <-- * 101 FETCH (UID 101 FLAGS (\Seen))
2016-04-21 08:21:39,720 -  - [     690] S <-- * 98 FETCH (UID 98 FLAGS (\Seen))
2016-04-21 08:21:39,720 -  - [     690] S <-- * 97 FETCH (UID 97 FLAGS (\Seen \Answered))
2016-04-21 08:21:39,720 -  - [     690] S <-- * 96 FETCH (UID 96 FLAGS (\Seen))
2016-04-21 08:21:39,720 -  - [     690] S <-- * 95 FETCH (UID 95 FLAGS (\Seen))
2016-04-21 08:21:39,720 -  - [     690] S <-- * 94 FETCH (UID 94 FLAGS (\Seen))
2016-04-21 08:21:39,720 -  - [     690] S <-- * 93 FETCH (UID 93 FLAGS (\Seen))
2016-04-21 08:21:39,720 -  - [     690] S <-- * 92 FETCH (UID 92 FLAGS (\Seen))
2016-04-21 08:21:39,720 -  - [     690] S <-- * 85 FETCH (UID 85 FLAGS (\Seen))
2016-04-21 08:21:39,720 -  - [     690] S <-- * 81 FETCH (UID 81 FLAGS (\Seen \Answered))
2016-04-21 08:21:39,720 -  - [     690] S <-- * 69 FETCH (UID 69 FLAGS (\Seen))
2016-04-21 08:21:39,720 -  - [     690] S <-- * 67 FETCH (UID 67 FLAGS (\Seen))
2016-04-21 08:21:39,721 -  - [     690] S <-- * 66 FETCH (UID 66 FLAGS (\Seen))
2016-04-21 08:21:39,721 -  - [     690] S <-- * 65 FETCH (UID 65 FLAGS (\Seen))
2016-04-21 08:21:39,721 -  - [     690] S <-- * 64 FETCH (UID 64 FLAGS (\Seen))
2016-04-21 08:21:39,721 -  - [     690] S <-- 5 OK UID FETCH completed


Tbird log:
2016-04-21 08:20:40,431 -  - [     688] ************** New Connection from: x.x.x.x**************
2016-04-21 08:20:40,431 -  - [     688] S <-- * OK Synametrics IMAP4rev1 server ready 4/21/16 8:20 AM
2016-04-21 08:20:40,494 -  - [     688] C --> 1 capability
2016-04-21 08:20:40,494 -  - [     688] S <-- * CAPABILITY IMAP4rev1 IDLE
2016-04-21 08:20:40,494 -  - [     688] S <-- 1 OK CAPABILITY completed
2016-04-21 08:20:40,500 -  - [     688] C --> 3 login "x@x.x" xxxxxx
2016-04-21 08:20:40,500 -  - [     688] S <-- 3 OK LOGIN completed
2016-04-21 08:20:40,511 -  - [     688] C --> 4 select "INBOX"
2016-04-21 08:20:40,561 -  - [     688] S <-- * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
2016-04-21 08:20:40,561 -  - [     688] S <-- * 112 EXISTS
2016-04-21 08:20:40,561 -  - [     688] S <-- * OK [UIDVALIDITY 14] UID validity status
2016-04-21 08:20:40,561 -  - [     688] S <-- * OK [UIDNEXT 272] Predicted next UID
2016-04-21 08:20:40,561 -  - [     688] S <-- * 0 RECENT
2016-04-21 08:20:40,561 -  - [     688] S <-- * OK [PERMENENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft)]
2016-04-21 08:20:40,561 -  - [     688] S <-- 4 OK [READ-WRITE] SELECT completed
2016-04-21 08:20:40,621 -  - [     688] C --> 5 UID fetch 1:* (FLAGS)
2016-04-21 08:20:40,629 -  - [     688] S <-- * 1 FETCH (UID 1 FLAGS (\Seen))
2016-04-21 08:20:40,629 -  - [     688] S <-- * 2 FETCH (UID 2 FLAGS (\Seen))
2016-04-21 08:20:40,629 -  - [     688] S <-- * 3 FETCH (UID 3 FLAGS (\Seen))
2016-04-21 08:20:40,629 -  - [     688] S <-- * 4 FETCH (UID 4 FLAGS (\Seen))
2016-04-21 08:20:40,630 -  - [     688] S <-- * 5 FETCH (UID 5 FLAGS (\Seen))
2016-04-21 08:20:40,630 -  - [     688] S <-- * 6 FETCH (UID 6 FLAGS (\Seen))

more fetching...

2016-04-21 08:20:40,636 -  - [     688] S <-- * 109 FETCH (UID 268 FLAGS ())
2016-04-21 08:20:40,636 -  - [     688] S <-- * 110 FETCH (UID 269 FLAGS ())
2016-04-21 08:20:40,636 -  - [     688] S <-- * 111 FETCH (UID 270 FLAGS ())
2016-04-21 08:20:40,636 -  - [     688] S <-- * 112 FETCH (UID 271 FLAGS ())
2016-04-21 08:20:40,636 -  - [     688] S <-- 5 OK UID FETCH completed
2016-04-21 08:20:40,661 -  - [     688] C --> 6 IDLE
2016-04-21 08:20:40,661 -  - [     688] S <-- + idling

Joa H

unread,
Apr 21, 2016, 3:28:33 PM4/21/16
to K-9 Mail
Some more testing reveals that the Mailwise app behaves just like K9 (logs confirm the same problem), but the Gmail app works fine.

Joa H

unread,
Apr 22, 2016, 2:24:21 AM4/22/16
to K-9 Mail
A little too fast on the trigger there (the only time I think fast is when I jump to conclusion). Gmail does not display all messages in the inbox- When refreshing it does the same as the other apps for the same reason. When opening Settings->Accounts and manually synching, it first synchs messages up to UID = #existing mails (as usual) and then specifically fetches messages "SINCE 2019-04-21" (= yesterday). So I see the most recent mail, which is what deceived me.

Sorry about the fragmented narrative.

Questions:
Since the problem didn't arise until Android 6, and then for all apps, could it be that the apps use a common IMAP protocol stack (provided by Android 6)?
If so, is the problem with Android or my server?

BR,
Joa



Den torsdag 21 april 2016 kl. 09:35:34 UTC+2 skrev Joa H:

Philip Whitehouse

unread,
Apr 22, 2016, 3:35:25 AM4/22/16
to k-9-...@googlegroups.com, Joa H
Thunderbird doesn't really do the same thing here. Whether it's because it had never been synchronised before or because their IMAP implementation is poorly optimized I'm not sure. It just fetches all the emails (1:*)

We have received mails already so we start from 63. Clearly we are mistaking the email count for the next valid UID, which is terrible behaviour.

What amazes me is Gmail does the same.

It's possible that to implement IMAP with other mail servers they imported the IMAP code from the original AOSP app. But this should break everywhere, the number of emails will almost never be the next valid UID except if you never move or delete an email.

I will try and check the source code to see if this is what the code actually says we are doing and whether we have fixed it in master.

The only thing that I can think of right now is that both Gmail and we are expecting you to list some capability which you are not. But from the spec I can't see this as being the case.

Any thoughts cketti?
>> *2016-04-21 08:21:39,605 - - [ 690] S <-- * 112 EXISTS*
>> 2016-04-21 08:21:39,605 - - [ 690] S <-- * OK [UIDVALIDITY 14]
>UID
>> validity status
>> *2016-04-21 08:21:39,605 - - [ 690] S <-- * OK [UIDNEXT 272]
>> Predicted next UID*
>> 2016-04-21 08:21:39,605 - - [ 690] S <-- * 0 RECENT
>> 2016-04-21 08:21:39,605 - - [ 690] S <-- * OK [PERMENENTFLAGS
>> (\Answered \Flagged \Deleted \Seen \Draft)]
>> 2016-04-21 08:21:39,605 - - [ 690] S <-- 3 OK [READ-WRITE]
>SELECT
>> completed
>> *2016-04-21 08:21:39,647 - - [ 690] C --> 4 UID SEARCH 63:112
>NOT
>> DELETED*
>> 2016-04-21 08:21:39,649 - - [ 690] S <-- * SEARCH 64 65 66 67 69
>81
>> 85 92 93 94 95 96 97 98 101 102
>> 2016-04-21 08:21:39,649 - - [ 690] S <-- 4 OK SEARCH completed
>> *2016-04-21 08:21:39,718 - - [ 690] C --> 5 UID FETCH
>> 102,101,98,97,96,95,94,93,92,85,81,69,67,66,65,64 (UID FLAGS)*
>> *2016-04-21 08:20:40,561 - - [ 688] S <-- * 112 EXISTS*
>> 2016-04-21 08:20:40,561 - - [ 688] S <-- * OK [UIDVALIDITY 14]
>UID
>> validity status
>> *2016-04-21 08:20:40,561 - - [ 688] S <-- * OK [UIDNEXT 272]
>> Predicted next UID*
>> *2016-04-21 08:20:40,636 - - [ 688] S <-- * 112 FETCH (UID 271
>FLAGS
>> ())*
>> 2016-04-21 08:20:40,636 - - [ 688] S <-- 5 OK UID FETCH
>completed
>> 2016-04-21 08:20:40,661 - - [ 688] C --> 6 IDLE
>> 2016-04-21 08:20:40,661 - - [ 688] S <-- + idling
>>
>>
>
>--
>--
>You received this message because you are subscribed to the K-9 Mail
>Users List.
>To post to this group, send email to k-9-...@googlegroups.com
>To unsubscribe, email k-9-mail+u...@googlegroups.com
>To report an issue with K-9 Mail, visit
>http://code.google.com/p/k9mail/issues/list
>For more options, visit this group at
>http://groups.google.com/group/k-9-mail
>
>---
>You received this message because you are subscribed to the Google
>Groups "K-9 Mail" group.
>To unsubscribe from this group and stop receiving emails from it, send
>an email to k-9-mail+u...@googlegroups.com.
>For more options, visit https://groups.google.com/d/optout.

Best regards,

Philip Whitehouse

Philip Whitehouse

unread,
Apr 23, 2016, 9:13:01 AM4/23/16
to K-9 Mail
Ok right looked at this properly.

I misread the spec a bit:

The thing at fault is the server. For Thunderbird it happens to do the right thing:

UID SEARCH a:b

This says I want all the UIDs between sequence number a and sequence number b. 

SEARCH UID a:b

This says I want all the sequence numbers between UID a and UID b.

We are sending the first one. Your server is interpreting it as the second one.

Your server has a bug. It's not a bug in K-9 & Gmail. 

Thunderbird only gets the right response because it wants all the messages (from sequence number 1 onwards).

Either update or reach out to your email server provider. If this was broken, every IMAP user would have issues and that's just not the case.

- Philip
>To report an issue with K-9 Mail, visit
>http://code.google.com/p/k9mail/issues/list
>For more options, visit this group at
>http://groups.google.com/group/k-9-mail
>
>---
>You received this message because you are subscribed to the Google
>Groups "K-9 Mail" group.
>To unsubscribe from this group and stop receiving emails from it, send

Joakim Hirsch

unread,
Apr 23, 2016, 10:22:37 AM4/23/16
to k-9-...@googlegroups.com
I get it. Thank you for educating me Philip. You have been most patient with me.

BR,
Joa
To unsubscribe, email k-9-mail+u...@googlegroups.com

To report an issue with K-9 Mail, visit http://code.google.com/p/k9mail/issues/list
For more options, visit this group at http://groups.google.com/group/k-9-mail

---
You received this message because you are subscribed to a topic in the Google Groups "K-9 Mail" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/k-9-mail/JXVXSsZfRDQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to k-9-mail+u...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages