Windows 'Access is denied' for os.Remove after exec.Output()

[Atakan Çolak]

Hello dear fellow gophers,

I had a relatively simple yet quite inconvenient issue which I felt the need to ask here. In my main() function;

os.Remove("my.exe") // err is nil, my.exe is removed

works in Windows without any errors, but when I call exec beforehand, I get access is denied error;

buffer, err := exec.Command("my.exe", myArgs...).Output() // err is nil here, I get desired output
os.Remove("my.exe") // remove "C:\\.......\my.exe": Access is denied

I tried using cmd.Process.Kill(), cmd.Process.Wait(), cmd.Start()-ioutil.ReadlAll()-cmd.Wait() alternatives as well. I kept getting no errors until 'Access is denied'. 

I'm using go1.14 linux/amd64 for my compiler and Windows 10 Enterprise 10.0.18362.

Thank you.

[Tamás Gulácsi]

Windows locks the running program's file, you cannot delete it when it's running - use "taskkill /F" to kill it proper.

[atakanc...@gmail.com]

Thank you for your reply. Unfortunately taskkill returns 128

buffer, err := exec.Command("my.exe", myArgs...).Output() // err is nil here, I get desired output 

_, err := exec.Command("taskkill", "/F", "/im", "my.exe").Output() // err is exit code 128, tried same with /pid, cmd.Process.Pid as well

os.Remove("my.exe") // remove "C:\\.......\my.exe": Access is denied  

14 Ağustos 2020 Cuma tarihinde saat 12:11:56 UTC+3 itibarıyla Tamás Gulácsi şunları yazdı:

[jake...@gmail.com]

This works fine for me on Windows 10.

What is "my.exe" doing?

Do you have third party antivirus software? If so, try turning it off. They are notorious for causing this kind of problem.

[atakanc...@gmail.com]

Hello guys, I have solved the issue.

Turns out it takes some time to release the lock on the folder, so we should do some time.Sleep before the os.Remove, so that Windows can release the lock. 

Thank you both for replying.

14 Ağustos 2020 Cuma tarihinde saat 16:21:17 UTC+3 itibarıyla jake...@gmail.com şunları yazdı:

[jake...@gmail.com]

> Turns out it takes some time to release the lock on the folder, so we should do some time.Sleep before the os.Remove, so that Windows can release the lock. 

I do not believe that should be the case under normal Windows operation. Using a Sleep in a case like this is always a hack. Sometimes it is the only way, but it can fail randomly, especially under stress. There is likely something else going on. Could be poorly written antivirus software, or a finalizer that has not run in your go app, or something else. If it is ok for it to work "most of the time", then maybe your Sleep() solution is sufficient. But if you need real reliability, I suggest figuring out what is really going on.

[Bob Alexander]

Here's what I think is really going on:

At the end of a process's execution, 2 things happen:

  - The process's code finishes its execution -- wait returns.

  - The OS closes the executable file.

The second item always "comes after" the first. On Windows the delay might be a few milliseconds, which can cause a problem, since an attempt to delete the executable right after wait returns can fail because the executable is still open. On Unix, this is not a problem because it's OK to "remove" an open file -- the file doesn't actually get deleted until all openers have closed the file.

At one time, Go's os/exec for Windows had a built-in unconditional 5ms delay to prevent this from happening. Not sure if it still does that.

It seems that, on Windows, if a program wants to delete the executable right after its process has finished, if should put the delete in a little retry loop:

   loop a few times (10?)

      try do delete the executable

      if the delete succeeds

          exit this loop

      wait a short time (1 ms ?)

   announce an error -- executable could not be deleted in reasonable time after process completion

This retry loop would be the responsibility of any Windows program that wants to delete the executable file after its execution finishes.

In reality, this is not done in very many places, done only by some tools like "go run" (build, run, delete), and the occasional user-written tool. The vast majority of places where external processes are run leave the executable file alone after process completion.

Is it the responsibility of an OS like Windows the guarantee the the executable is closed when a process wait returns? I would say not, because that might cause a (small) delay in the vast majority of external process executions where the executable is not deleted immediately after.

[Bob Alexander]

Note that the "retry loop for deleting the executable" technique has zero wait time if the delete succeeds.

A year or so ago I submitted a bug report because I had a program that ran hundreds of external processes (one at a time), and my Go program was way slower that a Python program that did the same thing. Turns out that it was because of the unconditional 5ms delay that was built into the Windows incantation of os (Go\src\os\exec_windows.go). This is an excerpt from that file:

   // NOTE(brainman): It seems that sometimes process is not dead
   // when WaitForSingleObject returns. But we do not know any
   // other way to wait for it. Sleeping for a while seems to do
   // the trick sometimes.
   // See https://golang.org/issue/25965 for details.
   defer time.Sleep(5 * time.Millisecond)
   defer p.Release()

Seems like a great idea to have zero delay except in those few times that it is necessary.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/XglcNW0USuc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f0a33b3f-a342-4a9a-9ccc-b0265bd2d60ao%40googlegroups.com.

[atakanc...@gmail.com]

Thank you for your reply.

It was refreshing to hear that this issue had been discovered earlier and some work has been put into it. I guess I will keep a loop for n times and fail if still can't delete it for a while. 

Cheers.

16 Ağustos 2020 Pazar tarihinde saat 21:28:26 UTC+3 itibarıyla bobj...@gmail.com şunları yazdı:

[Bob Alexander]

Adding to my previous post...

My expectation of a "process wait" is that it returns when the process is finished executing. I don't think it necessarily means that the OS has released the executable file. In Windows, this is clearly the case. It's unfortunate that the Windows characteristic that an open file cannot be removed (as it can in Unix) causes build-execute-delete tools to take care that the delete succeeds and take some action if it did not. I think the most efficient solution is one that costs nothing in the more common case where the delete succeeds, and costs an inevitable short delay *only* in the less common case.