detecting deleted file that is still open and appending without error in Go?

[Jason E. Aten]

I'm debugging a weird situation where my open binary log file

has been deleted and replaced by an

identical copy by my doing a git commit (and maybe git rebase)

on it.  The Go process is still running, still has the origional

file handle open, and is still "writing" to the 

deleted log file to no effect (no further appends are happening).

The /proc listing shows the Go process's file handle to the file as (deleted). See below.

My question is: is there a way to have the Go process detect if the file it is writing to has been deleted by another process (git in this case) so that attempting to append to the file is no longer effective?  This example is on Linux on xfs filesystem; Ubuntu 18.04.  I suppose I could Sync then Stat the file before and after and if it is not growing by the appropriate number of bytes, take corrective action... but I wonder if there is a more elegant way?

go version go1.21.2 linux/amd64

  Fortunately I noticed this before too much had gone missing to the binary log, and was able to confirm that it was going missing by comparing to a textual version of the log that I also keep.

I would be happy to check the file descriptor on each append operation, to ensure that the logging will be effective. But writes are succeeding to the deleted file: I am checking the error returned by the os.File.Write() operation, and it is always nil indicating no error on the Write.

~~~

myuser@host /proc/94913/fd $ ls -al

total 0

dr-x------ 2 myuser myuser  0 Dec 10 10:02 .

dr-xr-xr-x 9 myuser myuser  0 Dec 10 10:02 ..

lrwx------ 1 myuser myuser 64 Dec 10 10:02 0 -> /dev/pts/8

lrwx------ 1 myuser myuser 64 Dec 10 10:02 1 -> /dev/pts/8

lrwx------ 1 myuser myuser 64 Dec 10 10:02 10 -> 'anon_inode:[eventpoll]'

...

lrwx------ 1 myuser myuser 64 Dec 10 10:02 8 -> '/mnt/b/data/logs/my.binarylog.host (deleted)' ## << how to detect this?

...

myuser@host /proc/94913/fd $ mount|grep mnt/b

/dev/sdb on /mnt/b type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)

myuser@host /proc/94913/fd $ uname -a

Linux host 5.4.0-126-generic #142~18.04.1-Ubuntu SMP Thu Sep 1 16:25:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

myuser@host /proc/94913/fd $ cat /etc/lsb-release 

DISTRIB_ID=Ubuntu

DISTRIB_RELEASE=18.04

DISTRIB_CODENAME=bionic

DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"

myuser@host /proc/94913/fd $

~~~

[Jan Mercl]

On Sun, Dec 10, 2023 at 5:41 PM Jason E. Aten <j.e....@gmail.com> wrote:

> My question is: is there a way to have the Go process detect if the file it is writing to has been deleted by another process (git in this case) so that attempting to append to the file is no longer effective?

It is effective and [most] operations on the file continue to work as
usual. "Removing" a file, like in `os.Remove` is just removing one
reference to it. Only when the reference count drops to zero is the
file deleted.

This is, AFAIK, how nixes work and it's IMO actually a neat feature.
It enables, for example, updating files without disrupting processes
that have those files opened. Less advanced operating systems, to
achieve the same effect, have to reboot etc.

[Jason E. Aten]

Thanks Jan.

Right. I'm familiar with the reference counting of files on unixies and the default file locking on Windoze without specifying that sharing should be allowed.

My use case is keeping a scientific notebook on disk. The notebook file is (or should be!) append-only. And I'm trying very hard

to preserve its contents at points in time; for example by also committing it to git for version control. 

So imagine my surprise when my notebook starts dropping appends on the floor. Yikes.  Simply because I've

also added the notebook to version control. The opposite of what I hoped to achieve has occurred: instead of preserving

updates, now updates are being dropped. Ouch.

I'm trying my original suggestion now: doing a Stat before and after the write, and noticing if the new bytes are not hitting disk.

It seems cumbersome, but its better than losing an experiment setup and its outcome.

[TheDiveO]

You basically already showed how to do it on Linux: inside your Go prog, you know the file descriptor number (f.Fd()). Then do an os.Readlink(fmt.Sprintf("/proc/self/fd/%d", f.Fd()) and check the result (if no error) with strings.HasSuffix(link, "(deleted)").

[Robert Engels]

Not sure how you are detecting that the append it not working - it usually does because a client can relink the inode to another file name using a syscall

I think what you want to do it either

1. Open the file exclusively so git cannot do this.
2. Check that the file create time is the same as when you first opened the file (you have to make the fstat() by name)

> On Dec 10, 2023, at 10:59 AM, Jan Mercl <0xj...@gmail.com> wrote:

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAA40n-VZDOdwMOqqmAc-RgpGyNkOi0LvyFR%2BKbzem4PqNNwrYQ%40mail.gmail.com.

[Jason E. Aten]

Thanks Robert.

On Sunday, December 10, 2023 at 6:30:11 PM UTC Robert Engels wrote:

Not sure how you are detecting that the append it not working - 

I noticed that the binary log was not growing, and its update timestamp was not changing. I have an alias, lh, that is

alias lh='ls -alFtrh|tail'

that I run regularly in the course of work to see what has changed last in a directory. It is very useful, if you do not use something like it.

Only by luck did I happen to notice that the file was not being grown, which concerned me and so I investigated via

/proc/pid/fd, saw the deletion and then looked at the git log for the file. The updates stopped hitting the file (because the updated time on the file stopped changing) right after I added it to git.

On Sunday, December 10, 2023 at 6:30:11 PM UTC Robert Engels wrote:

> it usually does because a client can relink the inode to another file name using a syscall

Interesting. What syscalls relink it? 

Ah... https://serverfault.com/questions/168909/relinking-a-deleted-file seems to suggest that it used to be possible, but not after 2011 / linux kernel 2.6.39 because of security concerns.

> I, of course, did test this before posting my solution and at that time it actually worked for me. What I wasn't aware of is that it only worked on tmpfs filesystems but not on e.g. ext3. Furthermore this feature got completely disabled in 2.6.39, see the commit. (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=aae8a97d3ec30788790d1720b71d76fd8eb44b73 ) So therefore this solution won't work with kernel 2.6.39 or newer anymore and in earlier versions it depends on the filesystem. 

– 

tnimeu

 Jan 26, 2012 at 14:04

where the link is to a commit:  "fs: Don't allow to create hardlink for deleted file" from 2011.

Anyway, the Stat approach using the original filepath seems to be working: I can recognize the file shrinking or not growing when it should, and re-create it from memory.

Thanks All.

[Jan Mercl]

On Sun, Dec 10, 2023 at 8:34 PM Jason E. Aten <j.e....@gmail.com> wrote:

> I noticed that the binary log was not growing, and its update timestamp was not changing.

I think the file was still growing as intended. It was only no more
associated with the _new_ entry/name in the directory. I'm pointing it
out because it means no data is being lost until the FD of the
no-more-linked file is closed.

IOW, I think observing the file "no more growing" and "its time stamp
not updating" is now observing a different file than the one you're
writing the log to. That also means os.Stat("some_name") cannot help.
The "truth" is available at (*os.File).Stat().

HTH

-j

[Robert Engels]

As the links pointed out you can no longer relink but you can certainly access the data. So the append it working - you just can’t see it using ls. 

Anyway, I guess you have enough knowledge now to address. 

On Dec 10, 2023, at 1:34 PM, Jason E. Aten <j.e....@gmail.com> wrote:

Thanks Robert.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/e322acad-3070-4b51-ac23-c541b830c0aen%40googlegroups.com.

[Brian Candler]

On Sunday 10 December 2023 at 16:41:00 UTC Jason E. Aten wrote:

My question is: is there a way to have the Go process detect if the file it is writing to has been deleted by another process (git in this case) so that attempting to append to the file is no longer effective?

On Unix, I'd fstat the open file, stat the file on disk by name, and compare the inode numbers.

In Go, it's easy enough to get FileInfo but you need to dig a bit into Sys() to pull out the underlying Unix stat data.

https://stackoverflow.com/questions/28339240/get-file-inode-in-go

[Jason E. Aten]

Thanks Brian.