syscall.Mmap

[John]

Preface: I'm no expert on mmap, I've read(and not completely
understood) the man page for it. I've not used it before.

I'm attempting to get better performance over a file I'm reading in.
My problem state is:

I get the following on cmd line:
prog field1 regexp1 field2 regexp2 < file

The file will have the top line as field names, and every other line
as tab separated entries for those fields.

My understanding is that I will get better performance by using mmap.
At the moment I haven't even attempted getting the data on stdin, I'm
just trying to read a file using mmap, which doesn't seem that
successful. Having not done this in any language, and slightly in the
dark over here, I'm sure I'm doing something stupid.

=====
func GetFileBuffer(path string) (*bytes.Buffer, int64, error) {
f, err := os.Open(path)
if err != nil {
return nil, 0, err
}
fin, err := f.Stat()
if err != nil {
return nil, 0, err
}

mmap, err := syscall.Mmap(int(f.Fd()), 0, int(fin.Size()),
syscall.PROT_READ, syscall.MAP_PRIVATE)
if err != nil {
return nil, 0, err
}

buffer := bytes.NewBuffer(mmap)

return buffer, fin.Size(), nil
}

func main() {
runtime.GOMAXPROCS(1)

buffer, _, err := GetFileBuffer("./myFile");
if err != nil {
panic(err)
}

for data := range buffer.Next(1000) {
fmt.Print(string(data))
}
}
=====
I would expect to see the output of the file, however what I'm seeing
is:
!"#$%&'()*+,-./0123456789:;<=>?
@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefg.......

[Evan Shaw]

On Sun, May 27, 2012 at 9:35 AM, John <johns...@gmail.com> wrote:
> Preface:  I'm no expert on mmap, I've read(and not completely
> understood) the man page for it.  I've not used it before.
>
> I'm attempting to get better performance over a file I'm reading in.
> My problem state is:
>
> I get the following on cmd line:
> prog field1 regexp1 field2 regexp2 < file
>
> The file will have the top line as field names, and every other line
> as tab separated entries for those fields.

This doesn't sound like a good problem for mmap. Using mmap when
streaming a file probably won't help your performance. It really
shines when you want random access. I would suggest using a
bufio.Reader wrapping os.Stdin instead.

Your problem, however, has nothing to do with mmap.

> for data := range buffer.Next(1000) {
> fmt.Print(string(data))
> }

Here data is not a byte from the buffer, but an index into the buffer.
You're printing out the string conversion of the indexes 1, 2, 3, etc.
You want:

for _, data := range buffer.Next(1000) {
fmt.Print(string(data))
}

Or even better:

fmt.Print(string(buffer.Next(1000)))

- Evan

[lamvak]

W dniu poniedziałek, 28 maja 2012 04:45:30 UTC+2 użytkownik Evan Shaw napisał:

On Sun, May 27, 2012 at 9:35 AM, John <...> wrote:
> Preface:  I'm no expert on mmap, I've read(and not completely
> understood) the man page for it.  I've not used it before.
>
> I'm attempting to get better performance over a file I'm reading in.
> My problem state is:
>
> I get the following on cmd line:
> prog field1 regexp1 field2 regexp2 < file
>
> The file will have the top line as field names, and every other line
> as tab separated entries for those fields.

This doesn't sound like a good problem for mmap. Using mmap when
streaming a file probably won't help your performance. It really
shines when you want random access. I would suggest using a
bufio.Reader wrapping os.Stdin instead.

AFAIR, it can help.

Just for a short second of googling: http://lists.freebsd.org/pipermail/freebsd-questions/2004-June/050301.html and also http://www.gnu.org/software/libc/manual/html_node/Memory_002dmapped-I_002fO.html (see note about MADV_SEQUENTIAL )

 

Your problem, however, has nothing to do with mmap.

> for data := range buffer.Next(1000) {
>   fmt.Print(string(data))
> }

Here data is not a byte from the buffer, but an index into the buffer.
You're printing out the string conversion of the indexes 1, 2, 3, etc.
You want:

for _, data := range buffer.Next(1000) {
  fmt.Print(string(data))
}

Or even better:

fmt.Print(string(buffer.Next(1000)))

- Evan

Also, see http://labix.org/gommap 

l.