Loop through lines in a file
robfig
John Asmuth
Krzysztof Kowalik
John Asmuth
On Wednesday, January 25, 2012 9:59:18 PM UTC-5, John Asmuth wrote:
There's this: https://github.com/skelterjohn/exp/blob/master/iochan/iochan.goJust something I was fooling around with a while back. Let's you do this: https://github.com/skelterjohn/exp/blob/master/iochan/iochan_test.go
Jessta
> Hi,
>
> you can use bufio (http://weekly.golang.org/pkg/bufio/#Reader.ReadLine):
>
> r := bufio.NewReader(file)
> for line, _, err := r.ReadLine(); err != io.EOF {
> // ...
> }
>
That will work fine as long as you're certain no line will ever be
more than 4096 bytes(default bufio buffer size).
Ignoring the prefix return value isn't a good idea. Returning an error
value stating that the file was in an incorrect format is a good idea,
or at least panic if your assumptions are incorrect.
--
=====================
http://jessta.id.au
Krzysztof Kowalik
Martin Kanarr
Kyle Lemons
roger peppe
i usually use bufio.Reader.ReadString('\n'),
and strip off any trailing \r\n or \n as necessary.
i do think the current design of ReadLine is unfortunate - the
most common case (reading a line regardless of how
long it is, with \r\n or \n stripped) is satisfied by none
of the existing calls.
as ever, the design is complicated by the fact that the
final line may or may not be newline-terminated, and
we'd like to be able to preserve that information.
but in my experience most callers don't care about this.
so i'd be happy if the existing ReadLine was renamed to ReadLineSlice
and there were two additional calls:
// ReadLine reads a line and returns it as a string,
// not including the end-of-line bytes.
// If you need to know whether the final line is
// terminated, use ReadLineSlice instead.
func (r *Reader) ReadLine() (string, error)
// ReadLineBytes reads a line and returns it as a []byte,
// not including the end-of-line bytes.
// If you need to know whether the final line is
// terminated, use ReadLineSlice instead.
func (r *Reader) ReadLineBytes() ([]byte, error)
so the canonical "read lines" loop can look like this:
for {
line, err := r.ReadLine()
if err != nil {
break
}
doSomething(line)
}
which is a lot closer to ideal IMHO.
it's not going to happen for Go-1 though.
⚛
> > What is the "best practice" on looping through lines in a file?
>
> i usually use bufio.Reader.ReadString('\n'),
> and strip off any trailing \r\n or \n as necessary.
- return bufio.Reader's internal buffer (if the delimiter was found in
the internal buffer)
- automatically create and return a new []byte if it is impossible to
return bufio.Reader's internal buffer (that is: if the size is greater
than 4096 bytes)
roger peppe
> On Jan 26, 11:51 am, roger peppe <rogpe...@gmail.com> wrote:
>> > What is the "best practice" on looping through lines in a file?
>>
>> i usually use bufio.Reader.ReadString('\n'),
>> and strip off any trailing \r\n or \n as necessary.
>
> A problem is that bufio.Reader has no method which would
>
> - return bufio.Reader's internal buffer (if the delimiter was found in
> the internal buffer)
>
> - automatically create and return a new []byte if it is impossible to
> return bufio.Reader's internal buffer (that is: if the size is greater
> than 4096 bytes)
i think that's an optimisation that's unnecessary in most cases.
if you find that you're allocating too much in a particular
program, it's simple to write a function that does this.
func readLine(r *bufio.Reader) ([]byte, error) {
line, isPrefix, err := r.ReadLine()
if !isPrefix {
return line, err
}
buf := append([]byte(nil), line...)
for isPrefix && err == nil {
line, isPrefix, err = r.ReadLine()
buf = append(buf, line...)
}
return buf, err
}
although to be fair it's not possible to give this the same
semantics as my proposed ReadLine as there's no
way AFAICS to avoid the possibility of returning a line
and an error at the same time.
Sonia Keys
If by best practice though, you mean writing code that can handle unusual cases and is robust in production work, you want bufio.ReadLine in a for loop as Kyle showed, and you want to handle each return value however is appropriate for your application.
Check the error value, and for isPrefix, stop and think, what would it mean if you encountered a line longer than 4K? Usually it means the user has opened the wrong file and the appropriate action is to stop immediately. Do you happen to be reading a file where you expect lines up to 64K in length? Then create your bufio.Reader of that size. Do you really want unlimited line length? Unlimited, really? That's usually a bad idea and I think it's great that it is not the default behavior. If you want to build up long lines from 4K pieces, append makes that easy, but really you should be making the conscious decision to do that by writing a couple of lines of code. I also like that ReadLine doesn't allocate. You decide to allocate just what you want to allocate and there is less waste and churn.
There is no one simple way. The simple way that is right for one application will not be right for another.
Johann Höchtl
Johann