Issue 5707 in k9mail: IMAP PUSH not working in k9 mail 4.401

[k9m...@googlecode.com]

Status: New
Owner: ----
Labels: Type-Defect Priority-Medium Product-k9mail

New issue 5707 by jhnmlkv...@gmail.com: IMAP PUSH not working in k9 mail
4.401
http://code.google.com/p/k9mail/issues/detail?id=5707


What's going wrong?

Mail not fetched when using IMAP PUSH. I'm using the following settings on
my IDLE enabled IMAP account:

Refresh IDLE connection: currently at 6 mins. I've tried everything from 1
to 24 minutes.

Push Folders: All

For the same account on the same phone (Nexus 4), IMAP Push works
flawlessly with Aqua Mail. So I know its not the server.

On k9mail (4.401) however, there is absolutely no activity. I enabled
debugging and tried to record activity in aLogcat. With regular expression
k9|AndroidRuntime the logs are blank.

What steps will reproduce the problem?
1.
2.
3.

What do you expect to have happen?

I expect to see email arrive in my inbox

What do you see instead?

I see no new email in my inbox

What version of K-9 are you using?

4.401

What is your device and what version of Android are you using?

Nexus 4

Is your email account a POP account, Exchange Account or an IMAP account?

IMAP (IDLE enabled)

In order for us to have a chance at helping you with a complex issue...

I enabled debugging and tried to record activity in aLogcat. With regular
expression k9|AndroidRuntime the logs are blank.

What else should we know?

This works fine with Aqua Mail's (another email client for Android) IMAP
Push implementation.


--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[k9m...@googlecode.com]

Comment #1 on issue 5707 by cketti: IMAP PUSH not working in k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

K-9 Mail currently limits Push to 10 folders. If you set 'Push folders'
to "All" and have more than 10 folders it's unspecified which folders will
be checked with Push. Try setting 'Push folders' to "Only 1st Class folders"

[k9m...@googlecode.com]

Comment #2 on issue 5707 by jhnmlkv...@gmail.com: IMAP PUSH not working in

k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

I changed 'Push folders' to "Only 1st class folders". I checked to see I
have 3 first class folders. However, this has not made any difference.
There is absolutely no activity from k9 in the logs and I don't receive
mail until I manually hit the refresh button.

[k9m...@googlecode.com]

Comment #3 on issue 5707 by cketti: IMAP PUSH not working in k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

You have to use the adb method to record a debug log. Alternatively you can
use catlog if you have a rooted device.

[k9m...@googlecode.com]

Comment #4 on issue 5707 by jhnmlkv...@gmail.com: IMAP PUSH not working in

k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

ok. I've attached the log.

Attachments:
adblog.txt 115 KB

[k9m...@googlecode.com]

Comment #5 on issue 5707 by jhnmlkv...@gmail.com: IMAP PUSH not working in

k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

So, does the issue show up in the attached log. Is there anything else I
can try to make this work?

[k9m...@googlecode.com]

Comment #6 on issue 5707 by a...@scrumpy.me.uk: IMAP PUSH not working in k9
mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

fixed mine buy un-installing and re-installing

[k9m...@googlecode.com]

Comment #7 on issue 5707 by jhnmlkv...@gmail.com: IMAP PUSH not working in

k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

Thanks for the response but, uninstalling and reinstalling doesn't fix it
for me. The frustrating thing is I cant find any logs that hint at what the
problem could be. I've tried SSL (if available) and SSL (always) to see if
could be the certificate. No luck there either.

[k9m...@googlecode.com]

Comment #8 on issue 5707 by kevi...@kevincox.ca: IMAP PUSH not working in

k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

I'm having the same problem.

When I am looking at the folder list it says "(Push active)" does that mean
that push is actually active or that it is set to be active and it will if
it is one of the ten folders? Either way, I don'thave 10 folders set to
push.

I am connecting to gmail (although it is an apps account I don't think that
should make a difference).

[k9m...@googlecode.com]

Comment #9 on issue 5707 by michaelk...@gmail.com: IMAP PUSH not working in

k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

I have the same problem using version 4.409 of K9 on a Nexus 4!

Is there any solution for this issue available?

[k9m...@googlecode.com]

Comment #10 on issue 5707 by warb...@gmail.com: IMAP PUSH not working in k9
mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

Also having the same issue, K9 push email not working on a Galaxy S4.
Reinstalling hasn't helped... tried all the settings. Think I'll have to
try an alternative.

[k9m...@googlecode.com]

Comment #11 on issue 5707 by jhnmlkv...@gmail.com: IMAP PUSH not working in

k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

Yeah. No solution yet. I gave up on k9mail and started using AquaMail.

[k9m...@googlecode.com]

Comment #12 on issue 5707 by sc...@bytality.com: IMAP PUSH not working in

k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

Push won't work for me on my Galaxy S3 (CyanogenMod 10.1.3RC2-D2VZW)
running android version 4.2.2.

I control the IMAP server, and can see that the logs say the "Idle timeout
ended" though I don't know if that's because K9 is doing it, or the server
is ending IDLE session. I've got to think K9 has to be doing something in
order to establish the IDLE sessions, but it's not using them for mail
transfer.

I've got only 1st class set for push, and only my inbox is in the 1st
class. I've uninstalled/reinstalled to no resolution.

There's definitely something wrong with IMAP IDLE in K9. Please fix ASAP

[k9m...@googlecode.com]

Comment #13 on issue 5707 by cketti: IMAP PUSH not working in k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

https://github.com/k9mail/k-9/wiki/LoggingErrors

[k9m...@googlecode.com]

Comment #14 on issue 5707 by jhnmlkv...@gmail.com: IMAP PUSH not working in

k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

cketti, was the log I provided useful to narrow down the problem?

[k9m...@googlecode.com]

Comment #15 on issue 5707 by featherv...@gmail.com: IMAP PUSH not working

in k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

I'm having the same problem, it used to work, it no longer works. Even
forcing an update does not work.

[k9m...@googlecode.com]

Comment #16 on issue 5707 by shsw...@gmail.com: IMAP PUSH not working in k9
mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

I have the same problem.

Sony Xperia T,
Android 4.1.2.
K-9 version 4.409.

I have to pull down to refresh or hit the refresh button. I did have it set
to first class folders only, I set it to all and the next email I tried DID
come in pretty quickly. But after that it stopped again. I turned off "all"
folders and went back to first class only. THIS USED TO WORK, so it's not
because of too many folders. I have uninstalled and reinstalled. SOMETIMES,
and it is only sometimes, the message comes in ok after a few minutes
(still not as quick as it used to be). Sometimes it takes 30+ minutes.
Sometimes it's more than an hour (beyond that I don't know).

I have tried using the logging software on the phone but have read
elsewhere that this does not work beyond Android 4.1 so no wonder it had
nothing in the log! I have also tried the required software on my PC.
Following the instructions given a DOS type window opens up, flips through
a load of stuff then closes itself. There's nothing else I can do to
generate the required log so I'm stuck with the only answer being to turn
on polling, which I didn't want to have to do - I wanted push to work!

This is the same email account, same hardware and settings which used to
work perfectly. My host is the same, my account is the same. Only the
version of K-9 has changed.

Is there anywhere to get the old version from - the one that works?

[k9m...@googlecode.com]

Comment #17 on issue 5707 by byrox...@umn.edu: IMAP PUSH not working in k9
mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

Here are 4 log files from my phone, Let me know if there is more needed
from me for diagnosis/reproduction of the issue. I'll gladly provide any
additional info I can, or test new code/versions.

BTW, when you type "k9 mail push" into a google search, one of the top
suggested searches is "k9 mail push not working" so I think this is a
fairly widespread bug that isn't isolated to a few people. Please fix as
soon as you can :)

http://pastebin.com/icnUJmKC
--> FYI, This one is the only one where I could find the word "push" with a
text search

http://pastebin.com/tY9dT1uV
http://pastebin.com/05DNJNpt
http://pastebin.com/gz6f9A9f


K9 v4.409 on Samsung Galaxy S3 (CyanogenMod 10.1.3RC2-D2VZW) running
Android version 4.2.2

[k9m...@googlecode.com]

Comment #18 on issue 5707 by sc...@bytality.com: IMAP PUSH not working in

[k9m...@googlecode.com]

Comment #19 on issue 5707 by sc...@bytality.com: IMAP PUSH not working in

k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

Do you need anything more from me or other users encountering this push
mail issue to work on it?

[k9m...@googlecode.com]

Comment #20 on issue 5707 by sc...@bytality.com: IMAP PUSH not working in

k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

Please look into this ASAP.

[k9m...@googlecode.com]

Comment #21 on issue 5707 by jhnmlkv...@gmail.com: IMAP PUSH not working in

k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

It still does not work with 4.701. When I look at my push folders in the
folder view I see "Refreshed x minutes ago" in gray. Is this what I should
expect to see if the folder is connected to the server using IDLE?

Using Nexus 4 Android 4.3.

[k9m...@googlecode.com]

Comment #22 on issue 5707 by jhnmlkv...@gmail.com: IMAP PUSH not working in

k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

Quick update. Just upgraded to Android 4.4 on my Nexus 4 and installed k9
4.801. Still no dice.

[k9m...@googlecode.com]

Comment #23 on issue 5707 by lukas.fr...@gmail.com: IMAP PUSH not working

in k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

I have also problems with push - here my tests I did:

K9 4.801 on Nexus 5 with Cyanogenmod 11, Android 4.4.1:

-------------------------------------
K9 is configured for 3 EMail accounts:
imap.gmail.com, outlook.office365.com and the mail account of my university.

-------------------------------------
Push/Pull settings for all 3 accounts:

No Pull,
Push only 1st class folders (only the inboxes of the 3 accounts are set as
1st class)
Push refresh every 6 minutes (reduced it for the test - normally i have
24min)
No Pull on push refresh (for testing, to make sure I get only push mail).

-------------------------------------
Test Case:

K9 debug enabled.
Recording the logcat output of K9 for 1 hour with Android Studio - filtered
by "k9" (I saw the instructions at
https://github.com/k9mail/k-9/wiki/LoggingErrors after I was done - I
checked everything again without a filter and there were no AndroidRuntime
errors).

I have a script on a server running which sends a test email to all three
accounts every 15min.

The phone is connected over my home WLAN to the internet. (definitely no
connection interrupts durring the test)
The phone is configured to keep WIFI turned on durring sleep.

Durring the test, the phone was the only IMAP Client connected to these
accounts.

-------------------------------------
Results:

University email:
Note: My university is running a Dovecot server which sends [OK, Still
here] every 2min when idling.

Push works perfectly. I get every mail a few seconds after it is sent.
---
Office 365 email:
Push works sometimes. If the "push refresh rate" is lower it works more
often.
Even though "Pull on push" is disabled I get sometimes multiple emails at
once (I got three emails at the same time even tough they were sent in
15min intervals).
---
Gmail:
Works even less often than Office365.

---

The GMAil and Office 365 account produce this exception every 20-25min each
(pull refresh is still 6min):

12-09 12:11:00.560 4332-13494/com.fsck.k9 E/k9? Got exception while
idling for MAILBOX:INBOX/Thread-425
java.net.SocketTimeoutException: Read timed out
at com.android.org.conscrypt.NativeCrypto.SSL_read(Native
Method)
at
com.android.org.conscrypt.OpenSSLSocketImpl$SSLInputStream.read(OpenSSLSocketImpl.java:689)
at java.io.InputStream.read(InputStream.java:162)
at
java.io.BufferedInputStream.fillbuf(BufferedInputStream.java:142)
at
java.io.BufferedInputStream.read(BufferedInputStream.java:227)
at
com.fsck.k9.mail.filter.PeekableInputStream.peek(PeekableInputStream.java:33)
at
com.fsck.k9.mail.store.ImapResponseParser.readResponse(ImapResponseParser.java:42)
at
com.fsck.k9.mail.store.ImapResponseParser.readResponse(ImapResponseParser.java:29)
at
com.fsck.k9.mail.store.ImapStore$ImapConnection.executeSimpleCommand(ImapStore.java:2832)
at
com.fsck.k9.mail.store.ImapStore$ImapFolder.executeSimpleCommand(ImapStore.java:868)
at
com.fsck.k9.mail.store.ImapStore$ImapFolderPusher$1.run(ImapStore.java:3068)
at java.lang.Thread.run(Thread.java:841)

----

The open IMAP(S) connections of my phone - according to my WLAN router

root@openwrt:~# netstat-nat | grep imap
tcp android.lan:37342 my.university.com:imaps
ESTABLISHED
tcp android.lan:38222 157.56.251.214:imaps
ESTABLISHED
tcp android.lan:55973 wi-in-f108.1e100.net:imaps
ESTABLISHED


-------------------------------------
My interpretation:

For some reason, the connection to GMail and Office 365 times out.
If this happens it takes at least one or two pull refresh cycles untill
push works again.

The push connection to my univerity's mail server works because the server
wakes the phone up every 2min when he sends [OK, still here]. (which is
also not realy good because it uses battery - but I can't change it)


Hope this helps to fix the bug.

Logcat: http://pastebin.com/9cX8i77v

[k9m...@googlecode.com]

Comment #24 on issue 5707 by lukas.fr...@gmail.com: IMAP PUSH not working

in k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

Note: Issue 2596 seems to be a duplicate of this.

[k9m...@googlecode.com]

Comment #25 on issue 5707 by jhnmlkv...@gmail.com: IMAP PUSH not working in

k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

Thanks for the link. Issue 2596 is over three years old! wow. So maybe this
has never really worked on k9. Still a nice client though. If you really
don't care for imap push.

[k9m...@googlecode.com]

Comment #26 on issue 5707 by upandatt...@gmail.com: IMAP PUSH not working

in k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

Workaround for me for the last 2 years of using K9 is to disable push on
the account, then exit k9, kill the process, then restart K9 and go and
re-enable push on the account. EVEN IF push is already configured, I found
that removing it, quitting the app, killing the process, then starting it
up again and re-enabling fixes the issue until you change any account
settings again. Then you have to do it again. Pretty annoying and
surprising that it isn't fixed yet, but whatever.

-cl0ud 0x29A

[k9m...@googlecode.com]

Comment #27 on issue 5707 by jhnmlkv...@gmail.com: IMAP PUSH not working in

k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

@cl0ud thanks for the reply, but this workaround does not seem do anything
for me on my Nexus 4 running Android 4.4.2 with K9 4.802.

[k9m...@googlecode.com]

Comment #28 on issue 5707 by ladderma...@gmail.com: IMAP PUSH not working

in k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

tengo el mismo problema funcionaba y dejo de hacerlo..

[k9m...@googlecode.com]

Comment #29 on issue 5707 by emouroug...@gmail.com: IMAP PUSH not working

in k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

@upand...@gmail.com (#26), this workaround worked for me. I'll stay
tuned to see if it's still ok later in the day.

[k9m...@googlecode.com]

Comment #30 on issue 5707 by cketti: IMAP PUSH not working in k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

@lukas.fruhstuck: Your problem seems to be different. Please create a new
issue.

@jhnmlkvch9: From the log it looks like you have Push enabled but no folder
is configured for Push. Please uninstall K-9 Mail, install it again, and
use the default settings for Push. If that doesn't work please create a new
debug log with K-9 Mail using those default settings.

[k9m...@googlecode.com]

Comment #31 on issue 5707 by tommy.tr...@gmail.com: IMAP PUSH not working

in k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

I have used k9 for years, on several phones, under several versions of
Android, but I never experienced this issue until Verizon upgraded my
Galaxy S3 to Android 4.3 in early January. I just tried the workaround in
comment #26 and am hoping it will continue to work for awhile.

I tried activating polling for one account assuming if it was due to the
phone sleeping too deeply, maybe a poll would "wake up" the PUSH for other
accounts, however it didn't help. Any accounts not set to poll on a
schedule would update only when the phone was active, and sometimes after
only a very long delay.

[k9m...@googlecode.com]

Comment #32 on issue 5707 by jhnmlkv...@gmail.com: IMAP PUSH not working in

k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

@cketti please see attached log. It contains everything from the moment I
download to what happens (or rather does not happen) when push is enabled.
I have enabled push on my account and the folders are set to 1st class push
(which are the default settings as you suggested).

Then I tried to change idle timeout and even tried the suggestion in #26,
which I don't understand, because when push is disabled and I quit k9,
there is no running k9 process to kill.

All the people suggesting to reinstall, kill process etc., while I
appreciate the suggestions, I think have a different issue than what I am
facing, because I was unable to get imap push to work with k9mail, ever.
Now it may be something with the server, but the fact that Aquamail imap
push works, suggests this may not be the case.

At any rate, I've attached the log and if I can be of further assistance to
debug/help-fix this issue, please let me know.

Attachments:
logcat-log.txt 109 KB

[k9m...@googlecode.com]

Comment #33 on issue 5707 by jhnmlkv...@gmail.com: IMAP PUSH not working in

k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

Once again, I am using stock Android 4.4.2 on Nexus 4.

[k9m...@googlecode.com]

Comment #34 on issue 5707 by jhnmlkv...@gmail.com: IMAP PUSH not working in

k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

updated log file

Attachments:
logcat-log.txt 109 KB

[k9m...@googlecode.com]

Comment #35 on issue 5707 by jhnmlkv...@gmail.com: IMAP PUSH not working in

[k9m...@googlecode.com]

Comment #36 on issue 5707 by jhnmlkv...@gmail.com: IMAP PUSH not working in

k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

@cketti please see attached log. It contains everything from the moment I
download to what happens (or rather does not happen) when push is enabled.
I have enabled push on my account and the folders are set to 1st class push
(which are the default settings as you suggested).

Then I tried to change idle timeout and even tried the suggestion in #26,
which I don't understand, because when push is disabled and I quit k9,
there is no running k9 process to kill.

All the people suggesting to reinstall, kill process etc., while I
appreciate the suggestions, I think have a different issue than what I am
facing, because I was unable to get imap push to work with k9mail, ever.
Now it may be something with the server, but the fact that Aquamail imap
push works, suggests this may not be the case.

At any rate, I've attached the log and if I can be of further assistance to
debug/help-fix this issue, please let me know.

Once again, I am using stock Android 4.4.2 on Nexus 4.

[k9m...@googlecode.com]

Comment #37 on issue 5707 by cketti: IMAP PUSH not working in k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

@jhnmlkvch9: It looks like your server doesn't advertise the IDLE
capability. In that case K-9 Mail won't even try to use it.

[k9m...@googlecode.com]

Comment #38 on issue 5707 by jhnmlkv...@gmail.com: IMAP PUSH not working in

k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

@cketti Is advertising IDLE capability different from the server supporting
IDLE connections? I'm using IDLE on multiple clients (like Thunderbird) on
my desktops and Aquamail on Android.

Is using IDLE even when it is not advertised a feature of these clients?
If so, would it be possible for k-9 to have a similar feature?

thanks!

[k9m...@googlecode.com]

Comment #39 on issue 5707 by jhnmlkv...@gmail.com: IMAP PUSH not working in

k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

@cketti Is advertising IDLE capability different from the server supporting
IDLE connections? I'm using IDLE on multiple clients (like Thunderbird) on

my desktops and Aquamail on Android with no problems.

Is using IDLE; even when it is not advertised by the server; a feature of
these clients?

If so, would it be possible for k-9 to have a similar feature? In other
words, can I force k9 to try and use IDLE even when the server does not
advertise this capability. That would be great!

[k9m...@googlecode.com]

Updates:
Status: WontFix

Comment #40 on issue 5707 by cketti: IMAP PUSH not working in k9 mail 4.401
http://code.google.com/p/k9mail/issues/detail?id=5707

The specification states:
"If the server does not advertise the IDLE capability, the client MUST NOT
use the IDLE command and must poll for mailbox updates."

We're not going to test if the server supports the IDLE command even though
it didn't advertise support for it because that violates the specification.
It's not a widespread bug that would justify special treatment, either.

[k9m...@googlecode.com]

Comment #41 on issue 5707 by rhoci...@gmail.com: IMAP PUSH not working in
k9 mail 4.401
https://code.google.com/p/k9mail/issues/detail?id=5707

I still think there is a bug in k9. Cyrus-IMAPd advertises IDLE after
authentication, but not before. See the authentication sequence between
K9-5.001 and a cyrus server:

[pid 25188] read(13, "* OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID STARTTLS
AUTH=PLAIN AUTH=LOGIN SASL-IR COMPRESS=DEFLATE] mbox-server Cyrus IMAP
v2.3.16-Fedora-RPM-2.3.16-13.el6_6 server ready\r\n", 16384) = 168
[pid 25188] write(13, "1 AUTHENTICATE PLAIN\r\n", 22) = 22
[pid 25188] read(13, "+ \r\n", 16384) = 4
[pid 25188] write(13, "username-password\r\n", 58) = 58
[pid 25188] read(13, "1 OK [CAPABILITY IMAP4 IMAP4rev1 LITERAL+ ID
LOGINDISABLED COMPRESS=DEFLATE ACL RIGHTS=kxte QUOTA MAILBOX-REFERRALS
NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY
SORT SORT=MODSEQ THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE
CATENATE CONDSTORE SCAN IDLE LISTEXT LIST-SUBSCRIBED X-NETSCAPE URLAUTH]
Success (no protection)\r\n", 16384) = 356

[k9m...@googlecode.com]

Comment #42 on issue 5707 by rhoci...@gmail.com: IMAP PUSH not working in

k9 mail 4.401
https://code.google.com/p/k9mail/issues/detail?id=5707

I should specify that after the protocol negotiation above, K9 never sends
the server the IDLE command.