Unexpected error from os.DirEntry on Win10

[rob]

I've been writing some code to learn about the library changes in Go 1.16.

The following code uses filepath.Glob to get a slice of matching
filenames, then uses os.ReadDir to get more information about these
files.  I found that os.ReadDir reports a file not found error for the
filenames returned by filepath.Glob.  I do not understand why, as when I
use os.Lstat there are no errors.  This code is extracted and simplified
from a longer program I wrote.

This code has been compiled w/ go1.16.1 and I'm getting the same error
messages.

I do not understand why.  I would have expected that os.ReadDir can find
the filename just as os.Lstat does.

--rob solomon


// rddirpblm to isolate the problem w/ going from filepath.Glob to
os.ReadDir on Win10.

package main

import (
    "fmt"
    "os"
    "path/filepath"
    "runtime"
    "strconv"
)

const LastAltered = "11 Mar 2021"

/*
Revision History
----------------
11 Mar 21 -- Isolating Glob -> ReadDir issue.  Go 1.16 is deprecating
ioutil functions.
*/

func main() {
    var err error

    fmt.Print("rddirpblm LastAltered ", LastAltered, ", compiled using
", runtime.Version(), ".")
    fmt.Println()

    direntries := make([]os.DirEntry, 0, 500)

    sepstring := string(filepath.Separator)
    HomeDirStr, e := os.UserHomeDir() // HomeDir code used for
processing ~ symbol meaning home directory.
    if e != nil {
        fmt.Fprintln(os.Stderr, e)
        HomeDirStr = ""
    }
    HomeDirStr = HomeDirStr + sepstring
    fmt.Println(" HomeDirStr is", HomeDirStr)

    globfor := "*.txt"

    filenamesStringSlice, err := filepath.Glob(globfor)
    if err != nil {
        fmt.Fprintln(os.Stderr, err, "  Exiting.")
        os.Exit(1)
    }

    for i := 0; i < 20; i++ {
        fn := filenamesStringSlice[i]
        direntry, err := os.ReadDir(fn)
        if err != nil {
            fmt.Fprintln(os.Stderr, "fn =", fn, err)
            fmt.Fprintln(os.Stderr, "Will try os.Lstat")
            fi, er := os.Lstat(fn)
            if er != nil {
                fmt.Fprintln(os.Stderr, err)
            }
            fmt.Println("From os.Lstat: Name =", fi.Name(), ", Size =",
fi.Size(), ", IsDir =", fi.IsDir())
            fmt.Println()
            continue
        }
        if len(direntry) > 1 {
            fmt.Fprintln(os.Stderr, " expecting only 1 direntry, but
len(direntry) is", len(direntry), ".  Don't know what this means yet.")
        }
        direntries = append(direntries, direntry[0])
    }

    for _, d := range direntries {
        f, err := d.Info()
        if err != nil {
            fmt.Fprintln(os.Stderr, err, ".  No idea why this caused an
error.")
        }

        sizestr := strconv.FormatInt(f.Size(), 10)
        if f.Size() > 100000 {
            sizestr = AddCommas(sizestr)
        }
        fmt.Printf("%17s %s %s\n", sizestr, sizestr, d.Name())
    } // end for range direntries
} // end main

//--------------------------------------------------------------------
InsertByteSlice
func InsertIntoByteSlice(slice, insertion []byte, index int) []byte {
    return append(slice[:index], append(insertion, slice[index:]...)...)
} // InsertIntoByteSlice

//----------------------------------------------------------------------
AddCommas
func AddCommas(instr string) string {
    //var Comma []byte = []byte{','}  Getting error that type can be
omitted
    Comma := []byte{','}

    BS := make([]byte, 0, 15)
    BS = append(BS, instr...)

    i := len(BS)

    for NumberOfCommas := i / 3; (NumberOfCommas > 0) && (i > 3);
NumberOfCommas-- {
        i -= 3
        BS = InsertIntoByteSlice(BS, Comma, i)
    }
    return string(BS)
} // AddCommas

// ------------------------------- MyReadDir
-----------------------------------
func MyReadDir(dir string) []os.DirEntry {

    dirname, err := os.Open(dir)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        return nil
    }
    defer dirname.Close()

    names, err := dirname.Readdirnames(0) // zero means read all names
into the returned []string
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        return nil
    }

    direntries := make([]os.DirEntry, 0, len(names))
    for _, name := range names {
        d, err := os.ReadDir(name)
        if err != nil {
            fmt.Fprintln(os.Stderr, " Error from os.ReadDir. ", err)
            continue
        }
        if len(d) > 1 {
            fmt.Fprintln(os.Stderr, " expected len(d) == 1, but it's",
len(d), ", which I don't yet understand.")
        }
        direntries = append(direntries, d[0])
    }
    return direntries
} // MyReadDir

[Axel Wagner]

Hi,

I'm not entirely sure what is happening (it's hard to see without getting more information about the actual file tree and what files you get errors for) and what OS you are running on. But one observation is that you glob for *.txt and then call `ReadDir` on the result - that doesn't make much sense to me, the first would suggest the filenames are regular files, but calling `ReadDir` does not make sense unless you expect it to be a directory.

I would expect that to return a wrapper around syscall.ENOTDIR, but again, depending on the OS there might be idiosyncrasies I'm not expecting. But this would definitely explain why LStat works, while ReadDir doesn't - the file exists, it just isn't a directory.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f8e57e32-1f61-abe5-77fa-b48586c4524a%40fastmail.com.

[rob]

As the subject line says, this is on windows.
I'm getting the file not found error for every file returned by the glob function.

I guess I misunderstood the purpose of ReadDir, and I really need to call os.Lstat to retrieve the directory information for each individual file that matches the glob pattern on widows 10.

I'll check, but I'm able to call os.ReadDir on an individual file on Ubuntu 20.04. I'll confirm when I get home that I'm getting a FileInfo structure returned from this routine



--
rob
drro...@fastmail.com

[Axel Wagner]

I don't understand why you opened a new thread. But FWIW, it would still be useful to know

a) what the actual contents of the directory are you are globbing

b) which of those file names is then giving you an error

c) and which specific call is returning that error.

Your message "Unexpected error from os.DirEntry" is confusing, as the only method of os.DirEntry that could return an error, hardcodes that error to be nil, as far as I can tell:

https://github.com/golang/go/blob/b3896fc331c36a539f825f1f656cef3f9cdffd3f/src/os/dir_windows.go#L68

In the other thread, I also tried to provide a plausible explanation from the limited info - it seems strange that you are globbing for *.txt files and then call ReadDir with those filenames. Have you read that message?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/137cb626-90f1-455e-b0aa-7a6d476e3813%40www.fastmail.com.

[peterGo]

rob.

Function filepath.Glob returns the names of all files matching the pattern. The files may be directories, regular files, links, and so forth.

Function os.ReadDir expects the name of a directory. If the name is not a directory, it returns an error.

Your Glob pattern is "*.txt". It probably returns regular files, not directories.

A simple example,

package main

import (
    "fmt"
    "os"
    "path/filepath"

)

func GlobFiles(pattern string) ([]os.FileInfo, error) {
    var infos []os.FileInfo
    filenames, err := filepath.Glob(pattern)
    if err != nil {
        return nil, err
    }
    for _, filename := range filenames {
        info, err := os.Lstat(filename)
        if err != nil {
            continue
        }
        infos = append(infos, info)
    }
    return infos, nil
}

func main() {
    glob := "*.txt"
    infos, err := GlobFiles(glob)

    if err != nil {
        fmt.Fprintln(os.Stderr, err)

        os.Exit(1)
    }
    for _, info := range infos {
        fmt.Println(
            "From os.Lstat: Name =", info.Name(),
            ", Size =", info.Size(),
            ", IsDir =", info.IsDir(),
        )
    }
}

https://play.golang.org/p/9F7pFSpDhBJ

peter

[Brian Candler]

On Thursday, 11 March 2021 at 20:31:24 UTC rob wrote:

As the subject line says, this is on windows.
I'm getting the file not found error for every file returned by the glob function.

I guess I misunderstood the purpose of ReadDir, and I really need to call os.Lstat to retrieve the directory information for each individual file that matches the glob pattern on widows 10.

The purpose of ReadDir is to list the contents of a directory.

 

That is: if stat tells you that the entry is a directory, then you can ReadDir to list the files inside that directory.  This is one way to traverse a directory tree recursively.

ReadDir doesn't work if the entry is a file, but stat will tell you other things you may want to know about the file (such as its size).  If you want to read the file contents, then open it and read it as normal.

I'll check, but I'm able to call os.ReadDir on an individual file on Ubuntu 20.04.

That shouldn't work.  For me, with go 1.16 under Ubuntu 18.04:

--------

package main

import (

"fmt"

"os"

)

func main() {

_, err := os.ReadDir("readdir.go")

if err != nil {

fmt.Printf("Error: %v\n", err)

os.Exit(1)

}

}

--------

Gives:

Error: readdirent readdir.go: not a directory

[rob]

Sorry, I did not intend to open a new thread.

I don't know how to answer you about the actual contents of the directory of interest.  It's my c:\users\rob\Documents directory.

There are a lot of files there.  I'm writing my own version of the dir command, one that will sort and behave exactly as I want.  I decided to experiment w/ the new library definitions; and discovered that DirEntry is not a replacement for FileInfo.  I thought that was the intent of the Go team, but I guess did not understand.  In my code I went back to using Lstat, and instead of ioutil.ReadDir, I now use

f, err := os.Open(directoryname)

if err != nil { blah blah blah}

files, err := f.Readdir(0)

if err != nil { blah blah blah}

And this works for me.

--rob solomon

[Axel Wagner]

On Sat, Mar 13, 2021 at 11:52 PM rob <drro...@fastmail.com> wrote:

Sorry, I did not intend to open a new thread.

I don't know how to answer you about the actual contents of the directory of interest.  It's my c:\users\rob\Documents directory.

There are a lot of files there.  I'm writing my own version of the dir command, one that will sort and behave exactly as I want.  I decided to experiment w/ the new library definitions; and discovered that DirEntry is not a replacement for FileInfo.  I thought that was the intent of the Go team, but I guess did not understand.

It is, for most use-cases that want to list the contents of a file. It works around the fact that on many platforms, you already get *some* info in the readdir call but some info only by following it up with a Stat - returning a FileInfo requires you to do the stat, but most applications only need the things returned already by readdir itself and don't need to follow it up with a separate stat call. The DirEntry design splits up the info returned by readdir (namely the file names and the file mode) from the info returned by stat (the DirEntry.Info method).

I still don't understand why you think that ReadDir/DirEntry did not work for you - as I said, from your code the issue seems to simply be that you call ReadDir on regular files, instead of directories. OTOH, as long as you've find a solution that works for you, I guess that's fine.

[rob]

I need file timestamp and size, so I need a full FileInfo.

-- 

  rob

  drro...@fastmail.com