how to run filepath.Walkfunc as a goroutine

[Scott Turnbull]

I'm using filepath.Walk on a directory and processing each file I find there.  I'd like the filepath.Walkfunc that I define to run as a goroutine but I'm fairly new to go so I'm having some trouble mapping the examples I've seen to an anonymous function like the one I defined for my walkfunc. 

Are there examples of using a walkfunc as a goroutine somewhere and is it advised to do it in this way with that function in a filepath.Walk?

[Jan Mercl]

Instead of (run in this goroutine)

err := filepath.Walk(myRoot, myWalkFunc)

write

c := make(chan error)
go func() { c <- filepath.Walk(myRoot, myWalkFunc) }()
...
err := <-c // Walk done, check the error

which will run _both_ the 'Walk' function and the 'myWalkFunc' in a
new goroutine.

-j

[Scott Turnbull]

I may not be understanding what's going on here conceptually well enough.  What I would expect from that code is the entire Walk to run as a goroutine but I'm not sure how each individual instance of the WalkFunc would be run a go routine.

I modified the code to run with the example you provided but my benchmark tests don't really show any performance improvement.  Since I'm struggling with this a bit I must be doing something wrong.  I'm including the specific method I'm creating as an example.  When you get to the bottom of the method you'll see the Walk running as a go routine.  Above the visit function defines the callback and in that I'm taking the file I find and doing a fixity check on it and putting the filepath and fixity result into a map so I can use it elsewhere.  I'm trying to get the callback for fixity to run as a go routine to improve performance.

// Performs an add on every file under the directory supplied to the
// method.  Returns a map of the filename and its fixity falue and a
// list of errors.
func (p *Payload) AddAll(src string, hsh hash.Hash) (fxs map[string]string, errs []error) {

	fxs = make(map[string]string) // Mapping filepaths to fixity results.

        // Implementation of my Walkfunc
	visit := func(pth string, info os.FileInfo, err error) error {
		hsh.Reset()  // Reset the hash value everytime a new fixity starts.
		if err != nil {
			errs = append([]error{err})
		}
		if !info.IsDir() {
			dstPath := strings.TrimPrefix(pth, src)
			fx, err := p.Add(pth, dstPath, hsh)  // run add on the file.
			if err != nil {
				return err
			}
			fxs[dstPath] = fx
		}
		return nil

	}

	c := make(chan error)

	go func() {

		c <- filepath.Walk(src, visit)
	}()
	if err := <-c; err != nil {
		errs = append([]error{err})
	}

	return
}

[Jan Mercl]

On Wed, Jun 19, 2013 at 4:24 PM, Scott Turnbull <stream...@gmail.com> wrote:
> I may not be understanding what's going on here conceptually well enough.
> What I would expect from that code is the entire Walk to run as a goroutine
> but I'm not sure how each individual instance of the WalkFunc would be run a
> go routine.

Ah, I think I've misunderstood you question, sorry.

Well, no. Running myWalkFunc in a separate goroutine makes little
sense, because filepath.Walk is blocked until myWalkFunc returns - as
it needs the returned value to decide what to do next. IOW,
filepath.Walk implements a synchronous algorithm, which myWalkFunc is
a part of.

OTOH, collecting the walked paths in/by myWalkFunc and _afterwards_
running a goroutine for each path _does make_ sense.

Sorry for any confusion created by my previous post.

-j

[setha...@gmail.com]

On Wednesday, June 19, 2013 7:29:55 AM UTC-4, Scott Turnbull wrote:

I'm using filepath.Walk on a directory and processing each file I find there.  I'd like the filepath.Walkfunc that I define to run as a goroutine but I'm fairly new to go so I'm having some trouble mapping the examples I've seen to an anonymous function like the one I defined for my walkfunc. 

Are there examples of using a walkfunc as a goroutine somewhere and is it advised to do it in this way with that function in a filepath.Walk?

Not sure if this is exactly what you ware looking for or not. I know this is an old thread but I thought this might be helpful:

  <script src="https://gist.github.com/sethamclean/9475737.js"></script>

https://gist.github.com/sethamclean/9475737

[setha...@gmail.com]

On Wednesday, June 19, 2013 7:29:55 AM UTC-4, Scott Turnbull wrote:

I'm using filepath.Walk on a directory and processing each file I find there.  I'd like the filepath.Walkfunc that I define to run as a goroutine but I'm fairly new to go so I'm having some trouble mapping the examples I've seen to an anonymous function like the one I defined for my walkfunc. 

Are there examples of using a walkfunc as a goroutine somewhere and is it advised to do it in this way with that function in a filepath.Walk?

package main

import (
    "path/filepath"
    "fmt"
    "os"
)

func main(){
    location := "../gocode/"
    chann := GoWalk(location)
    for msg := range chann {
        fmt.Println(msg)
    }
    return
}

func GoWalk(location string) (chan string) {
    chann := make(chan string)
    go func(){
        filepath.Walk(location, func(path string, _ os.FileInfo, _ error)(err error){
            chann <- path
            return
        })
        defer close(chann)
    }()
        return chann 

}

https://gist.github.com/sethamclean/9475737