os.remove is slow

[gabejes...@gmail.com]

What takes 18 seconds in a perl command:

perl -e 'for(<*>){((stat)[9]<(unlink))}'

is taking almost 8 minutes with the following code. Any ideas how I can speed this up?

dir, err := os.Open("/var/spool/directory")
        if err != nil {
                fmt.Fprintf(w, "failed - " + err.Error())
                return
        }
        defer dir.Close()


        files, err := dir.Readdir(-1)
        if err != nil {
                fmt.Fprintf(w, "failed - " + err.Error())
                return
        }

        for _, file := range files {
                if file.Name() == "." || file.Name() == ".." {
                        continue
                }

                os.Remove("/var/spool/directory/" + file.Name())
        }


        fmt.Fprintf(w, "success")

[Hotei]

Gabriel - Thank you for reminding me that perl is a write-only language.  It's been a few years since I had to deal with it.  How many files are we talking about and on what media?

[Gabriel Forster]

As far as I understand, that perl line is by far the fastest way to delete files in linux.  We're talking over 500k files on hdd.

On Monday, December 4, 2017 at 12:50:54 PM UTC-5, Gabriel Forster wrote:

[Justin Israel]

On Tue, Dec 5, 2017, 7:13 AM Gabriel Forster <gabejes...@gmail.com> wrote:

As far as I understand, that perl line is by far the fastest way to delete files in linux.  We're talking over 500k files on hdd.

It may be the 500k stats that are killing you. From the looks of your code, the stat seems unnecessary as you only use it to get the file name. Can you switch to Readdirnames() and see how that goes? 

On Monday, December 4, 2017 at 12:50:54 PM UTC-5, Gabriel Forster wrote:

What takes 18 seconds in a perl command:

perl -e 'for(<*>){((stat)[9]<(unlink))}'

is taking almost 8 minutes with the following code. Any ideas how I can speed this up?

dir, err := os.Open("/var/spool/directory")
        if err != nil {
                fmt.Fprintf(w, "failed - " + err.Error())
                return
        }
        defer dir.Close()


        files, err := dir.Readdir(-1)
        if err != nil {
                fmt.Fprintf(w, "failed - " + err.Error())
                return
        }

        for _, file := range files {
                if file.Name() == "." || file.Name() == ".." {
                        continue
                }

                os.Remove("/var/spool/directory/" + file.Name())
        }


        fmt.Fprintf(w, "success")

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gabriel Forster]

Readdirnames wasn't much better. Now using globbing and syscall.Unlink which take ~1minute 30seconds. It is much better, but still a long way from perl's sub-20 seconds.

package main

import (
    "fmt"
        "path/filepath"
    "syscall"
)

func main() {

    upperDirPattern := "/var/spool/directory/*"  
    matches, err := filepath.Glob(upperDirPattern)

    if err != nil {
        fmt.Println(err)
    }
   
    for _, file := range matches {
        syscall.Unlink(file)
    }
}

On Monday, December 4, 2017 at 12:50:54 PM UTC-5, Gabriel Forster wrote:

[Patrick Smith]

Does it make a difference if you first change directory to /var/spool/directory, then glob * and unlink the resulting filenames, without prepending the directory?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscribe@googlegroups.com.

[Gabriel Forster]

Ding ding ding! We have a winner! Thank you so much.  That made a huge difference. It is now running ~20 seconds.

The key is being in the same directory.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

[Reto Brunner]

But why? Why does it matter if you are in the same dir or not when you glob?

[Dan Kortschak]

Have you tried os.RemoveAll("/var/spool/directory")?

https://golang.org/pkg/os/#RemoveAll

If that's slow, file an issue.