growing a large slice without copy

[Eric Z]

Hello Everyone,

I am trying to grow a pretty large slice which takes more than half of the memory. Is there anyway I can do it easily without running into virtual memory?

When I read the Go Slices: usage and internals, it says "To increase the capacity of a slice one MUST create a new, larger slice and copy the contents of the original slice into it." However, in my case, the slice is so large that if I create another one for copy it will definitely end up in spinning disks for virtual memory. Any workaround to add a small capacity to a large slice?

Another related question, can I manually free a slice memory space without waiting for the garbage collector?

Thanks a lot for the help!

Cheers,
Eric

[Ian Lance Taylor]

On Tue, Oct 29, 2013 at 2:08 PM, Eric Z <hadoo...@gmail.com> wrote:
>
> I am trying to grow a pretty large slice which takes more than half of the
> memory. Is there anyway I can do it easily without running into virtual
> memory?
>
> When I read the Go Slices: usage and internals, it says "To increase the
> capacity of a slice one MUST create a new, larger slice and copy the
> contents of the original slice into it." However, in my case, the slice is
> so large that if I create another one for copy it will definitely end up in
> spinning disks for virtual memory. Any workaround to add a small capacity to
> a large slice?

No. A slice's backing array must be in contiguous memory, so there is
no way to grow it beyond its capacity without copying it.

If I had a slice that large, I would consider another data structure,
one that does not require contiguous memory. Perhaps a simple slice
of slices. Indexing into it would be less efficient, but appending to
it would be faster.

> Another related question, can I manually free a slice memory space without
> waiting for the garbage collector?

You can't free a slice, but you can call runtime.GC().

Ian

[Matthew Kane]

On Tue, Oct 29, 2013 at 5:08 PM, Eric Z <hadoo...@gmail.com> wrote:

Another related question, can I manually free a slice memory space without waiting for the garbage collector?

No, but you can manually run the garbage collector with runtime.GC() 

--
matt kane
twitter: the_real_mkb / nynexrepublic
http://hydrogenproject.com

[Emanuel Czirai]

I was wondering if it could be done at some lower level or in C, but it makes sense that it cannot:

"Since you always want each block of dynamically-allocated memory to be contiguous (so that you can treat it as if it were an array), you and realloc have to worry about the case where realloc can't make the old block of memory bigger `in place`, but rather has to relocate it elsewhere in order to find enough contiguous space for the new requested size. realloc does this by returning a new pointer. If realloc was able to make the old block of memory bigger, it returns the same pointer. If realloc has to go elsewhere to get enough contiguous memory, it returns a pointer to the new memory, after copying your old data there. (In this case, after it makes the copy, it frees the old block.) Finally, if realloc can't find enough memory to satisfy the new request at all, it returns a null pointer. Therefore, you usually don't want to overwrite your old pointer with realloc's return value until you've tested it to make sure it's not a null pointer."

quote from here: http://www.eskimo.com/~scs/cclass/notes/sx11c.html

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Eric Z]

Thank you everyone, above. You guys are pretty helpful. It now makes sense to me that why it is not allowed.

I will make slice of slice and set nil to slice I want to free.

Cheers,
Eric

[Dave Cheney]

If you use append, that will happen automatically for you. 

[Klaus Post]

Hi!

This triggered my curiosity. I can see my "solution" was mentioned elsewhere - a slice of slices. But just for my own fun I coded a sample:

http://play.golang.org/p/zGyjBrj-ud

It will allocate in 1 megabyte chunks, which of course you can extend and even shrink at your liking.

It will of course be slower for general access - the math operations are fast, but you are adding an additional lookup (including bounds check) for each value retrieval. If that is critical, you can rather easily add a streaming interface that keeps track of your current position relative to the next chunk and returns the next value. However I don't think the performance difference will be that big. If you work with multi-gigabyte sizes it will probably be easier on the GC if you allocate bigger chunks than the 1MB I do in the sample.

I think either way it will be a good approach, since single allocations on that size is bound to give you trouble at some point in any language.

I am a Go beginner, so please excuse any beginner mistakes - I just did this as an exercise for myself.