random(?) message status changes

[Paddy Healy]

Hello,

I'm using Outlook + Davmail + mbsync to transport my email to mu4e.  At times emails that I have read (and sometimes filed) end up being marked unread again.  Other times the emails are marked as trashed, requiring an untrash operation.  Although this sometimes seems to happen due to me reading the email later on Outlook I am certain that this is not the only time that the status changes in mu4e.

What is particularly frustrating is when I need to refile the thread a second time due to this random(?) status change.  Now I can have two repeated threads in my subdirectory.

Am I alone in experiencing this behaviour?  I don't know where it arises.

Thanks.

[Kourosh Kalayeh]

I am experiencing similar thing with my setup: mu4e 1.12.1, mbsync 1.4.4 and Gmail on macoS. Specifically, I have noticed sometimes emails marked to be archive goes to trash and vice versa (I think this usually happens when I have both marks at the same time, but I am not sure).

The other thing that I noticed is that some of the emails that I marked to move to archive are completely gone from the database; neither mu4e nor mu can find them. They are still on Gmail server though. This starts to happening after upgrading to 1.12.1. I was trying to find the root cause (being mu4e or mbsync) but no luck so far. Today, I deleted my maildir and sync and index everything from scratch but still there are emails on the server that cannot be found with mu (or mu4e).

Thanks,
Kourosh

[Dirk-Jan C. Binnema]

On Thursday Mar 14 2024, Kourosh Kalayeh wrote:

> I am experiencing similar thing with my setup: mu4e 1.12.1, mbsync 1.4.4 and
> Gmail on macoS. Specifically, I have noticed sometimes emails marked to be
> archive goes to trash and vice versa (I think this usually happens when I have
> both marks at the same time, but I am not sure).
>
> The other thing that I noticed is that some of the emails that I marked to move
> to archive are completely gone from the database; neither mu4e nor mu can find
> them. They are still on Gmail server though. This starts to happening after
> upgrading to 1.12.1. I was trying to find the root cause (being mu4e or mbsync)
> but no luck so far. Today, I deleted my maildir and sync and index everything
> from scratch but still there are emails on the server that cannot be found with
> mu (or mu4e).

> On Mar 14, 2024 Paddy Healy <heal...@gmail.com> wrote:
>
>> Hello,
>>
>> I'm using Outlook + Davmail + mbsync to transport my email to mu4e. At
>> times emails that I have read (and sometimes filed) end up being marked
>> unread again. Other times the emails are marked as trashed, requiring an
>> untrash operation. Although this sometimes seems to happen due to me
>> reading the email later on Outlook I am certain that this is not the only
>> time that the status changes in mu4e.
>>
>> What is particularly frustrating is when I need to refile the thread a
>> second time due to this random(?) status change. Now I can have two
>> repeated threads in my subdirectory.
>>
>> Am I alone in experiencing this behaviour? I don't know where it arises.

In the past at least, such problems could be traced back to the way the
mail retrieval works (mbsync in this case). Perhaps misconfiguration or
at least surprising behavior, and the mapping of GMail and its virtual
folders to local maildir folders can add to the surprise.

So personally I just use fetchmail to get the mail, and don't bother to
have the same folder structure in e.g. gmail as I have local.

That being said, if you want to see exactly what mu is doing in terms of
moving /removing message files, you can set:
(setq mu4e-mu-debug t)
in your configuration and read the logs to see exactly what is going on.

Kind regards,
Dirk.

--
Dirk-Jan C. Binnema Helsinki, Finland
e:dj...@djcbsoftware.nl w:www.djcbsoftware.nl
gpg: 6987 9CED 1745 9375 0F14 DA98 11DD FEA9 DCC4 A036

[Paddy Healy]

Thank you.  I've set debugging on and I will investigate.  If I find anything that I think might be of general interest I will post here.

[Kourosh Kalayeh]

On Mar 15, 2024 "Dirk-Jan C. Binnema" <dj...@djcbsoftware.nl> wrote:
> So personally I just use fetchmail to get the mail, and don't bother to
> have the same folder structure in e.g. gmail as I have local.

Is your setting available online that we can use?

> That being said, if you want to see exactly what mu is doing in terms of
> moving /removing message files, you can set:
> (setq mu4e-mu-debug t)
> in your configuration and read the logs to see exactly what is going on.

I was able to replicate email deletion by marking them for refiling. I tried to isolate the log file with related information, see below. Sorry if they are meaningless, this is right at the edge of my knowledge.

2024-03-15 13:12:42.994 <- (:update (:size 11180 :list-post ((:email mu-di...@googlegroups.com)) :reply-to ((:email mu-di...@googlegroups.com)) :date (26099 8683 0) :from ((:email heal...@gmail.com :name Paddy Healy)) :list mu-discuss.googlegroups.com :message-id 329b66c6-53b4-41bb...@googlegroups.com :priority low :subject random(?) message status changes :to ((:email mu-di...@googlegroups.com :name mu-discuss)) :path /Users/kouroshm/Mail/k.ka...@gmail.com/[Gmail].All Mail/cur/1710522762.c86e63a71f6af9d6ae.mac-y6f3c3vgh0.local:2,RS :changed (26100 33162 0) :flags (replied seen list) :maildir /k.ka...@gmail.com/[Gmail].All Mail :docid 70011) :move t :maybe-view t)
2024-03-15 13:12:42.976 -> (mkdir :path "/Users/kouroshm/Mail/k.ka...@gmail.com/[Gmail].All Mail" :update nil)
2024-03-15 13:12:42.976 -> (move :docid 70011 :msgid nil :flags "-N" :maildir "/k.ka...@gmail.com/[Gmail].All Mail" :rename t :no-view nil)
2024-03-15 13:12:42.976 -> (mkdir :path "/Users/kouroshm/Mail/k.ka...@gmail.com/[Gmail].All Mail" :update nil)
2024-03-15 13:12:42.976 -> (move :docid 70215 :msgid nil :flags "-N" :maildir "/k.ka...@gmail.com/[Gmail].All Mail" :rename t :no-view nil)
2024-03-15 13:12:42.976 -> (mkdir :path "/Users/kouroshm/Mail/k.ka...@gmail.com/[Gmail].All Mail" :update nil)
2024-03-15 13:12:42.976 -> (move :docid 70311 :msgid nil :flags "-N" :maildir "/k.ka...@gmail.com/[Gmail].All Mail" :rename t :no-view nil)
2024-03-15 13:12:42.993 * Received 2927 byte(s)

Thanks,
Kourosh

[Dirk-Jan C. Binnema]

That all looks fine, but (with mu4e-mu-debug set), the mu server logs
would be interesting as well; on Linux you might get those in the
systemd journal; on MacOS you might find a "mu.log" somewhere near where
you have the mu database. That log file should show the file-system
operations, and allows you to see if there's anything fishy there.

[Kourosh Kalayeh]

Below is the server logs from another example with same behavior:

2024-03-15 18:30:09 (process:53732): mu-DEBUG: 18:30:09.074: moving /Users/kouroshm/Mail/k.ka...@gmail.com/Inbox/new/1710536747.46883_19.mac-y6f3c3vgh0,U=19:2, --> /Users/kouroshm/Mail/k.ka...@gmail.com/Inbox/cur/1710536747.46883_19.mac-y6f3c3vgh0,U=19:2,S (assume-remote:false)
2024-03-15 18:30:09 (process:53732): mu-DEBUG: 18:30:09.075: moved /Users/kouroshm/Mail/k.ka...@gmail.com/Inbox/new/1710536747.46883_19.mac-y6f3c3vgh0,U=19:2, -> /Users/kouroshm/Mail/k.ka...@gmail.com/Inbox/cur/1710536747.46883_19.mac-y6f3c3vgh0,U=19:2,S
2024-03-15 18:30:09 (process:53732): mu-DEBUG: 18:30:09.103: sw: query: 'i:20240315204237....@marketing.lyftmail.com'; (related:no; threads:no; ngrams:no; max-size:∞): finished after 3.3 ms
2024-03-15 18:30:09 (process:53732): mu-DEBUG: 18:30:09.104: moving /Users/kouroshm/Mail/k.ka...@gmail.com/[Gmail].All Mail/new/1710536868.46883_2692.mac-y6f3c3vgh0,U=2671:2, --> /Users/kouroshm/Mail/k.ka...@gmail.com/[Gmail].All Mail/cur/1710536868.46883_2692.mac-y6f3c3vgh0,U=2671:2,S (assume-remote:false)
2024-03-15 18:30:09 (process:53732): mu-DEBUG: 18:30:09.104: moved /Users/kouroshm/Mail/k.ka...@gmail.com/[Gmail].All Mail/new/1710536868.46883_2692.mac-y6f3c3vgh0,U=2671:2, -> /Users/kouroshm/Mail/k.ka...@gmail.com/[Gmail].All Mail/cur/1710536868.46883_2692.mac-y6f3c3vgh0,U=2671:2,S
2024-03-15 18:30:09 (process:53732): mu-DEBUG: 18:30:09.109: sw: view_handler (indexing: false): finished after 35.2 ms

Thanks,
Kourosh

[Dirk-Jan C. Binnema]

Okay, so that is that mu claims it is doing; can you confirm (by
checking the actual files from the command-line) if those moves actually
happen?

I.e., if after that move from /

/Users/kouroshm/Mail/k.ka...@gmail.com/[Gmail].All Mail/new/1710536868.46883_2692.mac-y6f3c3vgh0,U=2671:2,

to

/Users/kouroshm/Mail/k.ka...@gmail.com/[Gmail].All Mail/cur/1710536868.46883_2692.mac-y6f3c3vgh0,U=2671:2,S

that the target exists and the source is gone?

And what's the exact problem you see with this message in mu4e? Is it
gone there?

(try to do everything _without_ mbsync, then run mbsync and see if
anything changes).

[Kourosh Kalayeh]

Thanks for quick response. Some weird things are happening that I cannot make sense. Below is the relevant lines from mu.log when I mark an email for archiving (refile) and execute:

2024-03-16 16:36:31 (process:39005): mu-DEBUG: 16:36:31.329: moving /Users/kouroshm/Mail/k.ka...@gmail.com/Inbox/new/1710620374.38908_23.mac-y6f3c3vgh0,U=51:2, --> /Users/kouroshm/Mail/k.ka...@gmail.com/Inbox/cur/1710620374.38908_23.mac-y6f3c3vgh0,U=51:2,S (assume-remote:false)
2024-03-16 16:36:31 (process:39005): mu-DEBUG: 16:36:31.330: moved /Users/kouroshm/Mail/k.ka...@gmail.com/Inbox/new/1710620374.38908_23.mac-y6f3c3vgh0,U=51:2, -> /Users/kouroshm/Mail/k.ka...@gmail.com/Inbox/cur/1710620374.38908_23.mac-y6f3c3vgh0,U=51:2,S
2024-03-16 16:36:31 (process:39005): mu-DEBUG: 16:36:31.383: sw: query: 'i:1207403003.7483076.1710613180862@e2bc458242d0'; (related:no; threads:no; ngrams:no; max-size:∞): finished after 2.1 ms
2024-03-16 16:36:31 (process:39005): mu-DEBUG: 16:36:31.383: moving /Users/kouroshm/Mail/k.ka...@gmail.com/[Gmail].All Mail/new/1710620375.38908_53.mac-y6f3c3vgh0,U=2712:2, --> /Users/kouroshm/Mail/k.ka...@gmail.com/[Gmail].All Mail/cur/1710620375.38908_53.mac-y6f3c3vgh0,U=2712:2,S (assume-remote:false)
2024-03-16 16:36:31 (process:39005): mu-DEBUG: 16:36:31.384: moved /Users/kouroshm/Mail/k.ka...@gmail.com/[Gmail].All Mail/new/1710620375.38908_53.mac-y6f3c3vgh0,U=2712:2, -> /Users/kouroshm/Mail/k.ka...@gmail.com/[Gmail].All Mail/cur/1710620375.38908_53.mac-y6f3c3vgh0,U=2712:2,S
2024-03-16 16:36:40 (process:39005): mu-DEBUG: 16:36:40.403: moving /Users/kouroshm/Mail/k.ka...@gmail.com/Inbox/cur/1710620374.38908_23.mac-y6f3c3vgh0,U=51:2,S --> /Users/kouroshm/Mail/k.ka...@gmail.com/[Gmail].All Mail/cur/1710621400.6d0bf6199e7fb04ee.mac-y6f3c3vgh0.local:2,S (assume-remote:false)

Everything is working fine locally and I can see the email is in [Gmail].All Mail indexed by mu. I can also find 1710621400.6d0bf6199e7fb04ee.mac-y6f3c3vgh0.local in maildir/k.ka...@gmail.com/[Gmail].All Mail/cur/. Then, I ran mbsync, the email is moved to trash on the server side, which is also reflected in mu4e i.e., the email is in [Gmail].Trash locally. The curious thing is that 1710621400.6d0bf6199e7fb04ee.mac-y6f3c3vgh0.local is still in maildir/k.ka...@gmail.com/[Gmail].All Mail/cur/.

I don't know whether it is related or not but I am getting the following warnings too:
2024-03-16 16:56:44 (process:39005): mu-WARNING **: 16:56:44.632: xapian_try: xapian error 'Data ran out unexpectedly when reading posting list'
2024-03-16 16:56:44 (process:39005): mu-WARNING **: 16:56:44.633: xapian_try: xapian error 'Data ran out unexpectedly when reading posting list'

And the following errors in Emacs:
error in process filter: mu4e-error: [mu4e] Error 113: failed to run query: failed to read database

error in process filter: [mu4e] Error 113: failed to run query: failed to read database

Thanks,
Kourosh

[Dirk-Jan C. Binnema]

On Saturday Mar 16 2024, Kourosh Kalayeh wrote:

[ snip: nothing unexpected there I think, at a quick glance ]

> Everything is working fine locally and I can see the email is in [Gmail].All
> Mail indexed by mu. I can also find
> 1710621400.6d0bf6199e7fb04ee.mac-y6f3c3vgh0.local in
> maildir/k.ka...@gmail.com/[Gmail].All Mail/cur/. Then, I ran mbsync, the email
> is moved to trash on the server side, which is also reflected in mu4e i.e., the
> email is in [Gmail].Trash locally. The curious thing is that
> 1710621400.6d0bf6199e7fb04ee.mac-y6f3c3vgh0.local is still in
> maildir/k.ka...@gmail.com/[Gmail].All Mail/cur/.

Don't know if it's the case here, but Gmail has "virtual folders" (basically
like mu4e's queries), but when you use mbsync / offlineimap, those
virtual folders are mapped to real maildirs, each with its own copy of
messages that are part of multiple virtual folders.

> I don't know whether it is related or not but I am getting the following warnings too:
> 2024-03-16 16:56:44 (process:39005): mu-WARNING **: 16:56:44.632: xapian_try:
> xapian error 'Data ran out unexpectedly when reading posting list'
> 2024-03-16 16:56:44 (process:39005): mu-WARNING **: 16:56:44.633: xapian_try:
> xapian error 'Data ran out unexpectedly when reading posting list'
>
> And the following errors in Emacs:
> error in process filter: mu4e-error: [mu4e] Error 113: failed to run query: failed to read database
>
> error in process filter: [mu4e] Error 113: failed to run query: failed to read database

Those last things suggest some database corruption... I'd recommend
doing a `mu init --reinit` to restore & re-indexing.

[Kourosh Kalayeh]

I think I made some progress, I set mu4e-change-filenames-when-moving to nil and it has been 2 days that everything is working fine (except getting mbsync error: UID is beyond highest assigned UID and mbsync duplicate UID errors).  Now, I am not sure if this has anything to do with the cause but thought to share it here in case it gives you some clues. 

Thanks, 

Kourosh

--
You received this message because you are subscribed to the Google Groups "mu-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mu-discuss+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mu-discuss/87il1lc1uk.fsf%40djcbsoftware.nl.

[Dirk-Jan C. Binnema]

On Sunday Mar 17 2024, Kourosh Kalayeh wrote:

> I think I made some progress, I set mu4e-change-filenames-when-moving to
> nil and it has been 2 days that everything is working fine (except
> getting mbsync error: UID is beyond highest assigned UID and mbsync
> duplicate UID errors). Now, I am not sure if this has anything to do with
> the cause but thought to share it here in case it gives you some clues.

Yeah, mbsync treats your Maildir as its kingdom, and has various
idiosyncratic expectations that other tools (like mu) don't necessarily
follow or even know about, but mu tries..
mu4e-change-filenames-when-moving can help indeed.

[Kourosh Kalayeh]

I can confirm that when I set mu4e-change-filenames-when-moving to nil, the refiling and deleting work as expected but when I set it to true refling causes messages to be deleted (among other weird behaviors).  As mentioned, however, setting it to nil cause issues with UID errors and every couple of hours I have to follow http://tiborsimko.org/mbsync-duplicate-uid.html to rename the affected files. 

Does anyone have a solution or work around?  I have been paralyzed without mu4e and I hate using webmail. 

Thanks, 

Kourosh 

--
You received this message because you are subscribed to the Google Groups "mu-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mu-discuss+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mu-discuss/87r0g7533a.fsf%40djcbsoftware.nl.

[Jerry Lee Daniel]

Loans, Project and Digital Investment financing available up to $150m.
Have a Business Plan, Fundable Project and Collateral.

Contact: danielj...@gmail.com
Whatsapp: ‪+1 (918) 707‑0858