Maximum array size of 2G elements?

[barnex]

Hi all,

I stumbled upon a maximum array size of 2G elements, even though I use
6g on a 64-bit system with 12GB of memory. This limit corresponds to
the size of int being only 32 bits on my 64-bit system, which was a
surprise. Is there any workaround for this limitation?

code example:

make([]byte, uint(2*1024*1024*1024))
panic: runtime error: makeslice: len out of range


BTW: I'm doing physics simulations with Go+CUDA, using 4 GPUs with
1.5GB RAM each. So yes, I really need that much memory ;-) Also, I
don't mind recompiling 6g and friends if necessary.

Thanks for any suggestions,
Arne.

[Russ Cox]

> I stumbled upon a maximum array size of 2G elements, even though I use
> 6g on a 64-bit system with 12GB of memory. This limit corresponds to
> the size of int being only 32 bits on my 64-bit system, which was a
> surprise. Is there any workaround for this limitation?

There's no workaround. It's something we know we'll have to
address eventually, but for now that's the limit. Note that the
limit is on the number of array elements, not the total memory
usage or even the size of the allocated block, so you should
be able to allocate, say, four separate 1.5 GB byte slices or
one 1.5 billion element slice of uint32s just fine.

Russ

[barnex]

Thanks for the quick reply, Russ.
I just wanted to be sure I wasn't missing out on anything before I
start to distribute my data over separate arrays etc.
A 1.5 billion element float32 array will be fine for the moment, but
I'm trying to be "future proof" for when the 4GB GPUs arrive....

Cheers,
Arne.

[Russ Cox]

> I just wanted to be sure I wasn't missing out on anything before I
> start to distribute my data over separate arrays etc.
> A 1.5 billion element float32 array will be fine for the moment, but
> I'm trying to be "future proof" for when the 4GB GPUs arrive....

Isn't 1.5 billion float32s already 6 GB?

Russ

[barnex]

Yes, so that's literally perfect for the 6GB I have for the moment.
But we're considering buying 4 GPUs with 4 or 6GB each in the future
(TESLA C2070s or the like), so then things might become a bit more
difficult. But It's perfectly fine for me to distribute my data over
different Go arrays. Since it's vector data, a [][3]float32 should
already do the trick. I just wanted to be sure this 2G element limit
was inevitable before I start to re-write parts of the 25,000 lines of
code.

[Malcolm Greaves]

Why isn't the length encoded as an unsigned integer? How could the length of an array (and thus slice) ever be negative? We should be able to have arbitrarily long arrays. I love Go, but this is a serious problem. When will this be fixed?

[Rob Pike]

If the index type was uint stuff wouldn't work very well because Go
does not have C's type promotion rules. Plus sometimes it's useful to
have negative indices in APIs to indicate "no such element".

See issue 2188. No time frame yet, but some time during Go 1, I'm
pretty confident.

-rob

[Malcolm Greaves]

Pre: Wow, you are very fast Rob. I appreciate the alacrity. And thanks for pointing me to issue 2188.

Discussion: Wasn't one of your design principles with Go to have users explicitly deal with errors? I always think that whenever one would want to use a "-1" to indicate a failure (e.g. what's the index of a rune in a string? -1 means not present) one could easily adapt to the style of index,err where if err is nil then index is a valid answer.

I realize it might be a bad idea to try and change everything to use uint as this might break a lot of people's code. But I feel like go vet could be updated to automatically make this kind of change. For example:

// current: index is of type int

index := strings.Index(s,sep)

if index == -1 {

  handleError()

}

// proposal: index is type uint, err is type error

index,err := strings.Index(s,sep)

if err != nil {

 handleError()

}

What would it take to have slices and arrays indexed on uint? Too much?

- Malcolm

[Rob Pike]

Can't change now. Go 1 is locked down, and besides I still think it's
right because of the type issue. No one wants to have to write uint
declarations everywhere.

-rob

[Malcolm Greaves]

So the compiler couldn't take slice[i] where i is int and treat i as unsigned when it indexes into slice?

- Malcolm

[Andrew Gerrand]

On 31 August 2012 09:31, Malcolm Greaves <greaves...@gmail.com> wrote:
> So the compiler couldn't take slice[i] where i is int and treat i as
> unsigned when it indexes into slice?

And what happens if i is negative? This doesn't seem to solve any
problems, and as Rob says we can't change it now.

http://golang.org/doc/go1compat.html

Another thing: there's a lot of virtue in strings.Index (and the like)
returning a single value. In the many cases when you know a particular
substring is present, it's nice to be able to write something like:

b := a[:strings.Index(a, "foo")] // cut a at "foo"

Andrew

[Malcolm Greaves]

That's a good take on it:

Another thing: there's a lot of virtue in strings.Index (and the like)
returning a single value. In the many cases when you know a particular
substring is present, it's nice to be able to write something like:

  b := a[:strings.Index(a, "foo")] // cut a at "foo"

 

So perhaps there should be a new datatype that acts like a slice but is specifically intended for very large arrays....

[Andrew Gerrand]

On 31 August 2012 09:52, Malcolm Greaves <greaves...@gmail.com> wrote:
> That's a good take on it:
>>
>> Another thing: there's a lot of virtue in strings.Index (and the like)
>>
>> returning a single value. In the many cases when you know a particular
>> substring is present, it's nice to be able to write something like:
>>
>> b := a[:strings.Index(a, "foo")] // cut a at "foo"
>
>
> So perhaps there should be a new datatype that acts like a slice but is
> specifically intended for very large arrays....

Slices will soon support more items than you can conceivably store
(2^63) on amd64 systems. I think that's fine.

Andrew

[Malcolm Greaves]

That would be great. But if we use ints to index into slices, how can we index into the 2^63 element? I suppose I'll just wait and see (:

[Dan Kortschak]

On Thu, 2012-08-30 at 19:32 -0500, Malcolm Greaves wrote:
> That would be great. But if we use ints to index into slices, how can
> we
> index into the 2^63 element? I suppose I'll just wait and see (:
>

When ints are 64 bits wide.

[Malcolm Greaves]

Ah, right. So, essentially, the planned change is essentially to change from using int32 for length & indexing to int64, right?

[Andrew Gerrand]

On 31 August 2012 10:54, Malcolm Greaves <greaves...@gmail.com> wrote:
> Ah, right. So, essentially, the planned change is essentially to change from
> using int32 for length & indexing to int64, right?

The plan is to change the size of int on amd64 systems from 32 bits to
64 bits, as per the Go spec.

[Uli Kunitz]

There is already a patch. see here: http://code.google.com/p/go/source/detail?r=ad78acd6f0bb

On Friday, October 19, 2012 4:46:37 PM UTC+2, Hagen wrote:

The plan is to change the size of int on amd64 systems from 32 bits to
64 bits, as per the Go spec.

I just ran into the int32 issue myself, so I am very interested in a time frame for this change.

Or is there some nightly to play around with already?

Thanks!

Hagen

[Niklas Schnelle]

By Go 1 is locked down, do you mean Go 1.0 or Go 1.x? As I understand it the Go language specification explicitly says that int is at least 32 bit so in theory making it int64 on x86_64 isn't an issue?!
Nice to see that there is already a patch, I always saw this one of the biggest mistakes of Java that they have so many things in their language that are absolutely unchangeable and yet totally wrong. Though it's much easier for Go because there is no strictly defined byte code and we already got uintXXs, structs and a sane type system in general.

[minux]

On Oct 21, 2012 9:37 AM, "Niklas Schnelle" <niklas....@gmail.com> wrote:
> By Go 1 is locked down, do you mean Go 1.0 or Go 1.x? As I understand it the Go language specification explicitly says that int is at least 32 bit so in theory making it int64 on x86_64 isn't an issue?!
> Nice to see that there is already a patch, I always saw this one of the biggest mistakes of Java that they have so many things in their language that are absolutely unchangeable and yet totally wrong. Though it's much easier for Go because there is no strictly defined byte code and we already got uintXXs, structs and a sane type system in general.

on the default branch, int and uint are already 64-bit
on amd64.