Strange emacs behavior after upgrade to bullseye

[Rainer Dorsch]

Hello,

I hit a strange emacs issue, which appeared after upgrading to bullseye (I
think):

I have a virtualbox filesystem mounted using the standard virtualbox
mechanisms:

rd@Testing:~$ mount |grep dor1rt
dor1rt on /mnt/dor1rt type vboxsf (rw,nodev,relatime)
rd@Testing:~$

rd@Testing:~$ ls -l /mnt/dor1rt/Local/Managed/sb.blog
-rwxrwxrwx 1 root root 47086 Apr 19 13:40 /mnt/dor1rt/Local/Managed/sb.blog
rd@Testing:~$

While e.g. the standard KDE editor has no problems opening, editing, and
saving the file, emacs opens the file in read-only mode. I can disable read-only
mode, but emacs refuses to write, I get "Unlocking file: Operation not
permitted." Even worse, even if I try to save to ~/ I get the same unlocking
error.

Here is what I get in the *Messages* buffer:

Loading /etc/emacs/site-start.d/00debian.el (source)...done
Loading /etc/emacs/site-start.d/50asymptote.el (source)...done
Loading /etc/emacs/site-start.d/50autoconf.el (source)...done
Loading /etc/emacs/site-start.d/50cmake-data.el (source)...done
Loading /etc/emacs/site-start.d/50dictionaries-common.el (source)...
Loading debian-ispell...
Loading /var/cache/dictionaries-common/emacsen-ispell-default.el
(source)...done
Loading debian-ispell...done
Loading /var/cache/dictionaries-common/emacsen-ispell-dicts.el (source)...done
Loading /etc/emacs/site-start.d/50dictionaries-common.el (source)...done
Loading /etc/emacs/site-start.d/50festival.el (source)...done
Loading /etc/emacs/site-start.d/50latex-cjk-common.el (source)...
Loading /usr/share/emacs/site-lisp/latex-cjk-common/cjk-enc.el (source)...done
Loading /etc/emacs/site-start.d/50latex-cjk-common.el (source)...done
Loading /etc/emacs/site-start.d/50latex-cjk-thai.el (source)...done
Loading /etc/emacs/site-start.d/50latexmk.el (source)...done
Loading /etc/emacs/site-start.d/50texlive-lang-english.el (source)...done
Loading /home/rd/.emacs.d/lisp/plantuml-helpers.el (source)...done
Loading /home/rd/.emacs.d/lisp/melpa.el (source)...done
For information about GNU Emacs and the GNU system, type C-h C-a.
File exists, but cannot be read
funcall-interactively: Buffer is read-only: #<buffer sb.blog>
C-M-g is undefined
completing-read-default: Command attempted to use minibuffer while in minibuffer
Quit
Read-Only mode disabled in current buffer
You can run the command ‘read-only-mode’ with C-x C-q
Read-Only mode disabled in current buffer
Saving file /home/rd/local/Managed/sb.blog...
basic-save-buffer-2: Unlocking file: Operation not permitted, /mnt/dor1rt/Local/
Managed/sb.blog
set-visited-file-name: Unlocking file: Operation not permitted, /mnt/dor1rt/
Local/Managed/sb.blog
Mark set

Any idea or hint how to fix or work around this issue is welcome.

Thanks
Rainer


--
Rainer Dorsch
http://bokomoko.de/

[to...@tuxteam.de]

On Mon, Apr 19, 2021 at 06:48:41PM +0200, Rainer Dorsch wrote:
> Hello,
>
> I hit a strange emacs issue, which appeared after upgrading to bullseye (I
> think):
>
> I have a virtualbox filesystem mounted using the standard virtualbox
> mechanisms:
>
> rd@Testing:~$ mount |grep dor1rt
> dor1rt on /mnt/dor1rt type vboxsf (rw,nodev,relatime)
> rd@Testing:~$
>
> rd@Testing:~$ ls -l /mnt/dor1rt/Local/Managed/sb.blog
> -rwxrwxrwx 1 root root 47086 Apr 19 13:40 /mnt/dor1rt/Local/Managed/sb.blog
> rd@Testing:~$

Perhaps Emacs is trying to write a backup file to the directory.
Does it have write access to the containing directory?

Cf. the variable `make-backup-files' and those linked in its doc
(for this, do C-h v make-backup-files).

Cheers
- t

[Rainer Dorsch]

Just another update which makes emacs behavior even stranger:

Even though emacs reports when saving

basic-save-buffer-2: Unlocking file: Operation not permitted,
/mnt/dor1rt/Local/ Managed/sb.blog

the file gets saved!

I think somehow emacs gets out of sync with the real system.

Rainer


Am Dienstag, 20. April 2021, 12:00:05 CEST schrieb Rainer Dorsch:

> There is nothing which would not allow emacs to write a backup file in that
> directory
>
> rd@Testing:~/local/Managed$ touch test.txt
> rd@Testing:~/local/Managed$ ls -l test.txt
> -rwxrwxrwx 1 root root 0 Apr 20 11:48 test.txt
> rd@Testing:~/local/Managed$ rm -rf test.txt
> rd@Testing:~/local/Managed$
>
> For me the crucial message is

>
> basic-save-buffer-2: Unlocking file: Operation not permitted,
> /mnt/dor1rt/Local/ Managed/sb.blog
>

> (~/local is a symlink to /mnt/dor1rt/Local/)
>
> What does this message exactly mean and what is emacs trying to do here?
>
> Other editors (vi, kate) don't report any issue when performain an edit
> operation. Is emacs trying to derive permissions in a different way?

[Rainer Dorsch]

Am Montag, 19. April 2021, 22:25:44 CEST schrieb to...@tuxteam.de:

There is nothing which would not allow emacs to write a backup file in that
directory

rd@Testing:~/local/Managed$ touch test.txt
rd@Testing:~/local/Managed$ ls -l test.txt
-rwxrwxrwx 1 root root 0 Apr 20 11:48 test.txt
rd@Testing:~/local/Managed$ rm -rf test.txt
rd@Testing:~/local/Managed$

For me the crucial message is

basic-save-buffer-2: Unlocking file: Operation not permitted, /mnt/dor1rt/Local/
Managed/sb.blog

(~/local is a symlink to /mnt/dor1rt/Local/)

What does this message exactly mean and what is emacs trying to do here?

Other editors (vi, kate) don't report any issue when performain an edit
operation. Is emacs trying to derive permissions in a different way?

[to...@tuxteam.de]

On Tue, Apr 20, 2021 at 12:00:05PM +0200, Rainer Dorsch wrote:
> Am Montag, 19. April 2021, 22:25:44 CEST schrieb to...@tuxteam.de:

[...]

> > Perhaps Emacs is trying to write a backup file to the directory.
> > Does it have write access to the containing directory?

[...]

> There is nothing which would not allow emacs to write a backup file in that
> directory
>
> rd@Testing:~/local/Managed$ touch test.txt
> rd@Testing:~/local/Managed$ ls -l test.txt
> -rwxrwxrwx 1 root root 0 Apr 20 11:48 test.txt
> rd@Testing:~/local/Managed$ rm -rf test.txt
> rd@Testing:~/local/Managed$

Strange setup: How does test.text belong to root if you are not root?
I'd understand that it belongs to group root, that's what the setgid
bit is for. But _user_ root?

> For me the crucial message is
>
> basic-save-buffer-2: Unlocking file: Operation not permitted, /mnt/dor1rt/Local/
> Managed/sb.blog

Anyway, this is a good hint. See

"18.3.4 Protection against Simultaneous Editing"

in the Emacs user manual (or, if you prefer reading in a browser,
here [1].

But your permissions set up is... strange. The above behaviour
doesn't look plausible to me. Unless rd is actually root.

Cheers

[1] https://www.gnu.org/software/emacs//manual/html_node/emacs/Interlocking.html

- t

[Greg Wooledge]

Or /mnt/dor1rt/Local/ is on a non-Unix file system. Perhaps it's a
removable USB device with an NTFS or FAT type file system. Or perhaps
it's some sort of network file system whose underlying implementation
is not Unix-based.

[to...@tuxteam.de]

Yes, non-Unix file system os definitely a reasonable option: the
/mnt/ part (and the 777 modes everywhere) could be seen as a
hint :-)

Cheers
- t

[Rainer Dorsch]

Yes, correct, its mounted through virtualbox (vboxsf) and the host is a window
system which uses NTFS (I think). From a permission perspective 777 should be
sufficient though. The question is why does emacs think that is not enough, and
opens it as read-only? And even if I toggle the read-only mode, it complains
while writing...

thanks

[Greg Wooledge]

On Tue, Apr 20, 2021 at 03:59:00PM +0200, Rainer Dorsch wrote:
> > > > > basic-save-buffer-2: Unlocking file: Operation not permitted,
> > > > > /mnt/dor1rt/Local/ Managed/sb.blog

> Yes, correct, its mounted through virtualbox (vboxsf) and the host is a window
> system which uses NTFS (I think). From a permission perspective 777 should be
> sufficient though. The question is why does emacs think that is not enough, and
> opens it as read-only? And even if I toggle the read-only mode, it complains
> while writing...

Because the error says it cannot use LOCKS.

Because you can't do Unix file locking on a non-Unix file system.

The error does NOT say "Permission denied".

It says "Operation not permitted". That's a COMPLETELY different thing.
It has nothing to do with Unix file permissions, If you had an error
related to Unix file permissions, it would have said "Permission denied".

You get "Operation not permitted" whenever you try to do something you
can't do, for any reason OTHER than Unix file permissions. For example,
if you try to listen on a port below 1024 when you don't have that
capability. Or if you try to mount a file system when you don't have
that capability.

Or if you try to use Unix file locking on a non-Unix file system.

[Stefan Monnier]

> Because the error says it cannot use LOCKS.
> Because you can't do Unix file locking on a non-Unix file system.
> The error does NOT say "Permission denied".

FWIW, the error comes from Emacs's own locking code which doesn't seem
to use unix file locking, so the problem comes from elsewhere.

Emacs implements its locking protocol using symlinks with names
that look like `.#<FILENAME>` and whose content looks like
`US...@HOST.PID:BOOT_TIME`.


Stefan "still not sure exactly where it goes wrong"

[Greg Wooledge]

On Tue, Apr 20, 2021 at 11:08:57AM -0400, Stefan Monnier wrote:
> > Because the error says it cannot use LOCKS.
> > Because you can't do Unix file locking on a non-Unix file system.
> > The error does NOT say "Permission denied".
>
> FWIW, the error comes from Emacs's own locking code which doesn't seem
> to use unix file locking, so the problem comes from elsewhere.
>
> Emacs implements its locking protocol using symlinks with names
> that look like `.#<FILENAME>` and whose content looks like
> `US...@HOST.PID:BOOT_TIME`.

Ah, good old dot-locking. Well, perhaps the OP can test whether it's
possible to create a symlink in that directory.

[Stefan Monnier]

>> Emacs implements its locking protocol using symlinks with names
>> that look like `.#<FILENAME>` and whose content looks like
>> `US...@HOST.PID:BOOT_TIME`.
>
> Ah, good old dot-locking. Well, perhaps the OP can test whether it's
> possible to create a symlink in that directory.

That's probably part of the problem, yet the error doesn't seem to come
from the code which would *create* the symlink but rather the one that
would delete it :-(


Stefan

[David Wright]

I would assume that, because the file is on a foreign system, emacs
wants to use file-locking to protect against two programs making
changes at the same time. For whatever reason, its attempts to lock
the file fail in such a way to (perhaps) think that the file is
already locked, or just open the file readonly out of an abundance
of caution (to use that trending phrase).

When you toggle the file to read/write, you're effectively overriding
either the lock or the caution. As pointed out, the complaint upon
writing is probably emacs trying to lock the file while it writes
the buffer, or to do the unlocking.

If you want to chase this down further, you could try strace, but
do send the output to a file because emacs polls continually,
which spews strace output even while you sit on your hands.

Cheers,
David.

[Rainer Dorsch]

Hi Stefan,

many thanks for that excellent diagnosis, that really helped to trace it down:

- indeed vboxsf does not support symlinks (at least if ntfs is shared). But
instead of the symlinks there were real files. I assume a prior version of
emacs created them and used this as backup if the symlinks did not work on a
filesystem

- These files had permission 555 instead of 777. Changing this back (?) to 777
resolved the problem.

How that happened I cannot tell. I do not even understand what these
permissions mean on vboxsf, apparently they do at least something.

Maybe one explanation model:

-> I noticed that the timestamp of the .# lock files were super old (2019) on
the vboxsf file system even for files which I touched many times with emacs
since then
-> I noticed that if I edit these files with emacs the .# lock files are not
created anymore on the vboxsf filesystem by bullseye's emacs, but they get
removed if I close bullseye's emacs

So I speculate

-> stretch emacs used .# files instead of .# symlinks, if the file system did
not support symlinks. The content of the file contains the information which is
otherwise encoded in the symlink target

-> buster emacs did not care at all about .# on filesystems which do not
support symlinks. Somehow the permissions changed while the system was on
buster, possibly by virtualbox or by me myself in an accidential or intended
chmod -R a-w or something similar. I noticed the error but reverted it only
for the visible files. But since buster emacs did not care nobody noticed.

-> bullseye emacs not does not create the .# files on file systems, if they
don't have symlink support. But it finds them and puts the file in read-only
mode, if it is there. In addition, it tries to cleanup and delete these files,
which failed due to the wrong permission.

Even on a file system which supports symlinks, the problem can be easily
reproduced by

rd@h370:~$ touch test.txt
rd@h370:~$ touch .#test.txt
rd@h370:~$ chmod a-w .#test.txt
rd@h370:~$ emacs test.txt

What is somewhat ugly is that after toggling read-only mode and editing the
file, I cannot leave emacs anymore, it hangs.

I am undecided if that is a bug and I should report it or a real corner case
which is not worth to invest more time.

Thanks again, that was a great help.

[David Wright]

On Tue 20 Apr 2021 at 20:43:46 (+0200), Rainer Dorsch wrote:
> Am Dienstag, 20. April 2021, 17:08:57 CEST schrieb Stefan Monnier:
> > > Because the error says it cannot use LOCKS.
> > > Because you can't do Unix file locking on a non-Unix file system.
> > > The error does NOT say "Permission denied".
> >
> > FWIW, the error comes from Emacs's own locking code which doesn't seem
> > to use unix file locking, so the problem comes from elsewhere.
> >
> > Emacs implements its locking protocol using symlinks with names
> > that look like `.#<FILENAME>` and whose content looks like
> > `US...@HOST.PID:BOOT_TIME`.
>

> many thanks for that excellent diagnosis, that really helped to trace it down:
>
> - indeed vboxsf does not support symlinks (at least if ntfs is shared). But
> instead of the symlinks there were real files. I assume a prior version of
> emacs created them and used this as backup if the symlinks did not work on a
> filesystem

My comments are only on buster. I don't use vboxsf.

On a vfat USB stick, emacs can create locks and autosave files as
usual, where the lock, .#filename, is a file containing the target
of what would normally be a symlink.

(BTW I'd avoid using the term "backup", and say instead, perhaps,
"alternative method".)

> - These files had permission 555 instead of 777. Changing this back (?) to 777
> resolved the problem.

Write-protecting the contents of lockfiles seems reasonable in that
their username, hostname, PID, and process start time are invariant.
However, I've been testing on a single system with a vfat filesystem,
mounted so as to give 777 permissions (to test sharing), and haven't
observed them being set that way, or changing.

I have no idea what the role of vboxsf/Windows/NTFS would be, or
would have been at the time (2019, you say) these files were created.

> How that happened I cannot tell. I do not even understand what these
> permissions mean on vboxsf, apparently they do at least something.
>
> Maybe one explanation model:
>
> -> I noticed that the timestamp of the .# lock files were super old (2019) on
> the vboxsf file system even for files which I touched many times with emacs
> since then
> -> I noticed that if I edit these files with emacs the .# lock files are not
> created anymore on the vboxsf filesystem by bullseye's emacs, but they get
> removed if I close bullseye's emacs
>
> So I speculate
>
> -> stretch emacs used .# files instead of .# symlinks, if the file system did
> not support symlinks. The content of the file contains the information which is
> otherwise encoded in the symlink target

I don't think this is contentious.

> -> buster emacs did not care at all about .# on filesystems which do not
> support symlinks. Somehow the permissions changed while the system was on
> buster, possibly by virtualbox or by me myself in an accidential or intended
> chmod -R a-w or something similar. I noticed the error but reverted it only
> for the visible files. But since buster emacs did not care nobody noticed.

I can't reproduce that. I get locks (that are files) and autosave
files, both these being listed in the usual manner in each user's own
respective ~/.emacs.d/.saves-<PID>-<HOSTNAME>~ files.
However, AIUI locks only get created by the user who owns the directory
(I assume), and not by others. In your case, the owner was root, and
you were not running emacs as root.

Also bear in mind that locks aren't created until a need arises.
Opening a file in emacs is not enough: you have to modify the buffer.

Upon attempting to save a file being edited by several users, I get
the expected response with the user@host and PID of any valid lock.
I also observe the use of ~/.emacs.d/%backup%~ when an instance won't
overwrite the normal backup file, filename~, which it realises it
didn't write itself, perhaps.

> -> bullseye emacs not does not create the .# files on file systems, if they
> don't have symlink support. But it finds them and puts the file in read-only
> mode, if it is there. In addition, it tries to cleanup and delete these files,
> which failed due to the wrong permission.
>
> Even on a file system which supports symlinks, the problem can be easily
> reproduced by
>
> rd@h370:~$ touch test.txt
> rd@h370:~$ touch .#test.txt
> rd@h370:~$ chmod a-w .#test.txt
> rd@h370:~$ emacs test.txt
>
> What is somewhat ugly is that after toggling read-only mode and editing the
> file, I cannot leave emacs anymore, it hangs.
>
> I am undecided if that is a bug and I should report it or a real corner case
> which is not worth to invest more time.

When I start (buster) emacs with a fake, empty lock(file), even one
that I own like the above, it's not refreshed, but merely ignored.
Nor is the fake lock removed when emacs exits. I would assume that
a lock only works if the information it holds is valid, as far as
can be determined. I haven't bothered to check what happens with
manifestly stale locks (ie holding verifiable information).

I don't know how vboxsf is implemented, but I think it would be
necessary to disentangle the effects of the underlying components,
filesystem, OS, access method, before attributing strange behaviour
to emacs.

Cheers,
David.

[Rainer Dorsch]

Hi David,

I did some more testing, you can see the effect on bullseye without vboxsf even
on a ext4 filesystem.

Am Mittwoch, 21. April 2021, 20:49:14 CEST schrieb David Wright:
[...deleted a lot of history]

> > -> buster emacs did not care at all about .# on filesystems which do not
> > support symlinks. Somehow the permissions changed while the system was on
> > buster, possibly by virtualbox or by me myself in an accidential or
> > intended chmod -R a-w or something similar. I noticed the error but
> > reverted it only for the visible files. But since buster emacs did not
> > care nobody noticed.
> I can't reproduce that. I get locks (that are files) and autosave
> files, both these being listed in the usual manner in each user's own
> respective ~/.emacs.d/.saves-<PID>-<HOSTNAME>~ files.
> However, AIUI locks only get created by the user who owns the directory
> (I assume), and not by others. In your case, the owner was root, and
> you were not running emacs as root.
>
> Also bear in mind that locks aren't created until a need arises.
> Opening a file in emacs is not enough: you have to modify the buffer.
>
> Upon attempting to save a file being edited by several users, I get
> the expected response with the user@host and PID of any valid lock.
> I also observe the use of ~/.emacs.d/%backup%~ when an instance won't
> overwrite the normal backup file, filename~, which it realises it
> didn't write itself, perhaps.

I did go to a buster system with btrfs:

rd@master:~$ touch test.txt
rd@master:~$ touch .#test.txt
rd@master:~$ chmod a-w .#test.txt
rd@master:~$ emacs test.txt

works without emacs complaining about anything.

.# files are created as symlinks. I had no filesystem which does not support
symlinks on that system.

On the bullseye system with ext4 (but I do not expect that ext4 vs btrfs makes
a difference):

rd@h370:~$ touch test.txt
rd@h370:~$ touch .#test.txt
rd@h370:~$ chmod a-w .#test.txt
rd@h370:~$ emacs test.txt

opens test.txt as read only and if I modify it and try to leave emacs it even
hangs.

[Curt]

On 2021-04-20, Rainer Dorsch <m...@bokomoko.de> wrote:
>
> Other editors (vi, kate) don't report any issue when performain an edit
> operation. Is emacs trying to derive permissions in a different way?

There's this bug, which may or may not be pertinent.

https://gnu.emacs.bug.narkive.com/LoN17xVM/bug-37884-27-0-50-cannot-write-to-a-file-in-virtualbox-shared-directory

> Thanks
> Rainer
>


--