golang read file of integers into an array/slice && prepend vs append for slices

[Sankar]

Hi,

I have a file of integers that I want to read into a var x[]int

I cannot find a better way than reading the file contents, cast to string, strings.split with '\n', and iterate each entry and append to a slice.

Is there a better way ?

Also, is there a way to 'prepend' to a slice instead of append ? I can use some programming logic and allocate a array and then assign each element inside a loop. But I want to know if there is any other way to prepend instead of append items to a slice ?

Or, Is append not a costly operation in golang slices, that a need for prepend is not there at all ?

Please explain and help.

Thanks.

Sankar

[Sankar P]

I realized that since the slice is backed up by an array, if we
allocate the slice with enough length, then the append operation will
not be costly. If we just realloc for every append, then it will be a
costly operation.

--
Sankar P
http://psankar.blogspot.com

[Caleb Doxsey]

You may also want to use the bufio package's ReadLine method:

http://golang.org/pkg/bufio/#Reader.ReadLine 

Then you don't have to store the entire file contents in memory. Doing this you won't know how many numbers there are ahead of time... but you could do something like add 100 integers to the array when its full instead of just 1. (Kinda depends on what you're trying to do)

[Sankar P]

Thanks. This is useful. But for my case, reading at one stretch will be better.

[minux]

On Thu, Sep 6, 2012 at 9:37 PM, Caleb Doxsey <caleb....@gmail.com> wrote:

You may also want to use the bufio package's ReadLine method:

http://golang.org/pkg/bufio/#Reader.ReadLine 

FYI, Reader.ReadLine is too primitive and its use is discouraged, should consider

using Reader.ReadBytes('\n') or Reader.ReadString('\n') instead.

This is documented in the tip (should appear in Go 1.0.3):

http://tip.golang.org/pkg/bufio/#Reader.ReadLine

[Sonia Keys]

You have identified a good way to do this.  Read with ioutil.ReadFile, convert to string, split with strings.Split "\n", make []int of the desired (or at least sufficient) size, then assign the values in a loop.  I would use strconv.Atoi (and perhaps strings.TrimSpace first).  It might sound like a lot of code, but you are specifying exactly what the code should do, and the benefits are knowing that your program is efficient, and having complete control over your handling of invalid or unexpected data.

To answer questions about append though, append is an efficient way to build up a list, as long as you are appending to the end.  Append uses a scheme of over-allocating by increasing amounts so that it doesn't have to allocate every time.  Alternatively, as you noted, you can allocate it with sufficient capacity to begin with.

You can use append to build up a list from the front though.

    list = append(<list type>{newElement}, list...)

will do the job, but now you really are allocating for every element added, and generating lots of garbage in the process.  Sometimes it's convenient, but usually you would do better to find a different algorithm.  If nothing else, building the list from the end, then reversing it in place would be more efficient in most cases.

[Larry Clapp]

On Friday, September 7, 2012 9:59:43 AM UTC-4, Sonia Keys wrote:

If nothing else, building the list from the end, then reversing it in place would be more efficient in most cases.

This is a common idiom in Lisp[1]: Add items to the front of your list, then reverse it when you return from the function.  In Lisp, at least, it's pretty efficient, as I understand it.

(In Go, as you say, you might instead add items to the rear of your array or slice, and then reverse them when you return.)

-- Larry

[1] Common Lisp

[Sankar P]

thanks guys. I infact did allocate max size initially and then used
append so that there will be no realloc garbage etc.