Severe memory problems on 32bit Linux

[Erik Unger]

We are using Go for a real world website (http://startuplive.in) with
real world traffic and the Go process has been crashing every other
hour or so on 32bit Intel Linux. Automatic restarts were no solution
because it was such am memory hog, that it killed other processes too
before killing itself.

After a change to 64bit, all those problems are gone.

We are also running godoc -http behind Apache on another 32 Linux
machine, and the godoc process is basically killing the whole system
on a daily basis.

It seems to be Issue 909 (http://code.google.com/p/go/issues/detail?
id=909) or something similar, which makes Go absolutely unusable for
real world applications on 32bit Linux.

[Jan Mercl]

On Thursday, April 5, 2012 10:02:31 AM UTC+2, Erik Unger wrote:

It seems to be Issue 909 (http://code.google.com/p/go/issues/detail?
id=909) or something similar, which makes Go absolutely unusable for
real world applications on 32bit Linux.

For some unknown to me definition of 'absolutely'.

[⚛]

You are right about the fact that it is an issue for long-running 32-bit programs. It may be possible to improve the situation by adhering to the following rules throughout your program:

- avoid struct types which contain both integer and pointer fields

- avoid struct types containing arrays such as [100]byte (replace it with *[100]byte or with []byte)

- avoid data structures which form densely interconnected graphs at run-time

- avoid deep call chains at run-time

- replace pointer identity with value equivalence (this can lead to a more explicit memory management in your program)

- if a data structure contains both long-lived and short-lived fields, move the short-lived fields into a separate data structure or into local variables of a function

- avoid integer values which may alias at run-time to an address; make sure most integer values are fairly low (such as: below 10000)

- if you are using caches to speed up your program, apply the rules mentioned here to redesign the cache. It may also help to use strings instead of structs as map keys.

- lower the overall memory consumption of your program

- carefully speed up your program (this may lead to a lower memory consumption in certain situations)

- call runtime.GC()  (at the right moment)

On Thursday, April 5, 2012 10:02:31 AM UTC+2, Erik Unger wrote:

[⚛]

It is possible that you may be forgetting what the purpose of the golang-nuts forum is.

[Jan Mercl]

Let me guess. Discussing Go language, Go coding, problems encountered with using Go, sharing opinions about what is good and bad about Go and all of the previous, etc.?

What's wrong with not agreeing to a quite strong statement "Go absolutely unusable for real world applications on 32bit Linux" and saying so here? Is it OT or something?

As a meta topic now, I suggest to continue off-list, would you mind? ;-)

[André Moraes]

The 8g version of the compiler isn't the best choice to long-lived
process, at least right now, so you don't have much of a choice.
Stick to a 64 bit machine and things should work without much problem.

The godoc problem could be solved by auto-killing or running it
"on-demand", so when no request are being handled the process exits
and return memory, this could not be a solution for a high traffic
site but for something like a internnal godoc this should work.

> It seems to be Issue 909 (http://code.google.com/p/go/issues/detail?
> id=909) or something similar, which makes Go absolutely unusable for
> real world applications on 32bit Linux.

Not all real world applications are server side applications. There is
a considerable ammount of applications in Go that are client
applications (text-editor, gocode, games).

--
André Moraes
http://andredevchannel.blogspot.com/

[tom westberg]

On Thursday, April 5, 2012 4:13:45 AM UTC-4, Jan Mercl wrote:

On Thursday, April 5, 2012 10:02:31 AM UTC+2, Erik Unger wrote:

It seems to be Issue 909 ... which makes Go absolutely unusable for

real world applications on 32bit Linux.

For some unknown to me definition of 'absolutely'.

While you are literally correct that there are many real world applications for which go remains viable on 32-bit platforms, it's not really a reassuring attitude to engineers considering the use of go on a new project. Issue 909 has a priority of "Later" and comments which suggest that for long-running applications you should just get new hardware. The bug will be addressed in double-digit months.

Imagine yourself as an engineer who evangelized go inside an organization for an application that happens to fit this bad profile. Foolishly you did not read the entire open issues database, believing that christening a language "1.0" means that it works as a general-purpose language on all supported platforms. You deploy your system and.. it starts crashing. Advice is to switch to 64-bit hardware. In this case I hope we can forgive a bit of hyperbole from someone calling it "absolutely unusable". 

I find myself in a similar position (although I hadn't reached the actual implementation stage; I've just been intrigued by the go language and its features) in considering go for a platform which must be 32-bit (we ship atom-based appliances) on long-running applications (network connection daemons). It's conceivable that the memory usage profile for the project would not tickle the gc bug, but the language for avoiding the problem seems vague ("It may be possible to improve the situation.."). 

Would you be an advocate for such a project?

[Norbert Roos]

> We are using Go for a real world website (http://startuplive.in) with
> real world traffic and the Go process has been crashing every other
> hour or so on 32bit Intel Linux.

Is this problem related to x86 or does it apply to ARM as well?

And what about the option to disable gc with a compiler switch and doing
the memory management self using delete()? For future versions, of
course.. i've always been sceptical about gc.

Norbert

[Sebastien Binet]

André Moraes <and...@gmail.com> writes:

> The 8g version of the compiler isn't the best choice to long-lived
> process, at least right now, so you don't have much of a choice.

does gcc-go 32b expose the same behaviour ?
(otherwise, it could be a rather good choice)

-s

[André Moraes]

>
> And what about the option to disable gc with a compiler switch and doing the
> memory management self using delete()? For future versions, of course.. i've
> always been sceptical about gc.
>

I think we shouldn't blame the concept of GC because of a issue in one
implementation.

[André Moraes]

2012/4/5 Sebastien Binet <seb....@gmail.com>:

I never used the gcc-go, but it's a alternative.

[minux]

On Thu, Apr 5, 2012 at 11:19 PM, Norbert Roos <nr...@webware-experts.de> wrote:

We are using Go for a real world website (http://startuplive.in) with
real world traffic and the Go process has been crashing every other
hour or so on 32bit Intel Linux.

Is this problem related to x86 or does it apply to ARM as well?

I think ARM port suffer from this issue too. The main reason is that on 32-bit architectures,

normal data have a much large chance to look like a pointer, and current gc is conservative,

and lacking some type annotation (is this word possible pointer?).

And what about the option to disable gc with a compiler switch and doing the memory management self using delete()? For future versions, of course.. i've always been sceptical about gc.

Not possible yet. the runtime and std library needs gc themself.

[minux]

On Thu, Apr 5, 2012 at 7:04 PM, ⚛ <0xe2.0x...@gmail.com> wrote:

You are right about the fact that it is an issue for long-running 32-bit programs. It may be possible to improve the situation by adhering to the following rules throughout your program:

- avoid struct types which contain both integer and pointer fields

- avoid struct types containing arrays such as [100]byte (replace it with *[100]byte or with []byte)

- avoid data structures which form densely interconnected graphs at run-time

- avoid deep call chains at run-time

- replace pointer identity with value equivalence (this can lead to a more explicit memory management in your program)

- if a data structure contains both long-lived and short-lived fields, move the short-lived fields into a separate data structure or into local variables of a function

- avoid integer values which may alias at run-time to an address; make sure most integer values are fairly low (such as: below 10000)

- if you are using caches to speed up your program, apply the rules mentioned here to redesign the cache. It may also help to use strings instead of structs as map keys.

- lower the overall memory consumption of your program

- carefully speed up your program (this may lead to a lower memory consumption in certain situations)

- call runtime.GC()  (at the right moment)

one more advice, declare large array of struct with no pointer fields statically (i.e. don't use new), and the compiler

will place them in special places (.noptrdata and .noptrbss sections) that the GC won't even scan, and this can also

eliminate some fale positives (some data happen to look like pointers are regarded as pointers).

If you are brave enough, you can implement your own memory allocator on static allocated memory.

[Russ Cox]

On Thu, Apr 5, 2012 at 04:02, Erik Unger <ung...@gmail.com> wrote:
> We are using Go for a real world website (http://startuplive.in) with
> real world traffic and the Go process has been crashing every other
> hour or so on 32bit Intel Linux. Automatic restarts were no solution
> because it was such am memory hog, that it killed other processes too
> before killing itself.
>
> After a change to 64bit, all those problems are gone.

Great!

> We are also running godoc -http behind Apache on another 32 Linux
> machine, and the godoc process is basically killing the whole system
> on a daily basis.
>
> It seems to be Issue 909 (http://code.google.com/p/go/issues/detail?
> id=909) or something similar, which makes Go absolutely unusable for
> real world applications on 32bit Linux.

Very easy to believe. There's no easy software fix here.
The easy hardware fix is to switch to 64-bit.

Russ

[Ian Lance Taylor]

Sebastien Binet <seb....@gmail.com> writes:

Gccgo ought to be better on the margin, because it does not interpret
the entire data section as possibly containing pointers. However, it
does not address the base issue, which is that Go uses a conservative
garbage collector, and more values look like pointers in a 32-bit world.

The only real fix would be to improve the garbage collector's
understanding of which values are pointers and which are something else
(e.g., floating point numbers that happen to look like pointers). And
that is not an easy fix.

Ian

[Joubin Houshyar]

On Apr 5, 8:55 am, André Moraes <andr...@gmail.com> wrote:
...

> Not all real world applications are server side applications. There is
> a considerable ammount of applications in Go that are client
> applications (text-editor, gocode, games).

That is true, André, but Go is advertised as a "systems" PL. I for
one would like to see the Go PL stick to promise. Long running server
apps are very much a "systems PL" concern.

[David Symonds]

On Fri, Apr 6, 2012 at 1:14 PM, Joubin Houshyar <jhou...@gmail.com> wrote:

> That is true, André, but Go is advertised as a "systems" PL.

It's not. It used to be, but it's not any more.

http://golang.org/

[Joubin Houshyar]

On Apr 5, 11:42 pm, David Symonds <dsymo...@golang.org> wrote:

> On Fri, Apr 6, 2012 at 1:14 PM, Joubin Houshyar <jhoush...@gmail.com> wrote:
> > That is true, André, but Go is advertised as a "systems" PL.
>
> It's not. It used to be, but it's not any more.

/That is news/. So just "general PL", then? (With due apologies) I
have no idea if you are a member of set "Go Authors" as your sig
suggests, but If yes, what is the 'purpose'/'mission statement' of Go,
as of today?


>
> http://golang.org/

[Kyle Lemons]

There are lots of authors :).

 

>
> http://golang.org/

[Leon Szpilewski]

> Go absolutely unusable for real world applications on 32bit Linux.

I have been using Go on a 32bit server for a pretty frequently visited Blog (30k uniques/day). And there were no memory problems whatsoever.

I guess you have to over think your software design ...

cheers,

Leon

[Archos]

On Apr 5, 8:29 pm, Russ Cox <r...@golang.org> wrote:

> On Thu, Apr 5, 2012 at 04:02, Erik Unger <unge...@gmail.com> wrote:
> > We are also running godoc -http behind Apache on another 32 Linux
> > machine, and the godoc process is basically killing the whole system
> > on a daily basis.
>
> > It seems to be Issue 909 (http://code.google.com/p/go/issues/detail?
> > id=909) or something similar, which makes Go absolutely unusable for
> > real world applications on 32bit Linux.
>
> Very easy to believe.  There's no easy software fix here.
> The easy hardware fix is to switch to 64-bit.

So, Go is not suitable to build desktop long-running applications on
32-bit, at least that you want say to the users that switch to 64-bit.

[Paul]

hmmm... if you are writing applications for a specific hardware
platform, in this case Intel Atom, you probably can't just switch to
64bit hardware so easily. Likely it will screw up the whole business
model.

I actually don't know what the future is of 32bit, but it appears that
Intel is still selling it, at least for small devices.

I think it would help to have something like a "Read.me" file for the
Go- language which lists known issues.

The next thing that comes to mind is: now that Go 1 is out, would this
not be good time to devote some time and resources to developing a
improved runtime and garbage collector, and take it to the next level?

[Hotei]

Paul,

A look at recent posts in golang-dev might be enlightening...

[⚛]

On Friday, April 6, 2012 3:55:29 PM UTC+2, Hotei wrote:

A look at recent posts in golang-dev might be enlightening...

Which recent posts do you mean?

[⚛]

On Friday, April 6, 2012 5:08:44 PM UTC+2, Peter Thrun wrote:

Here's a sampling of recent threads about updates to the garbage collector:

https://groups.google.com/d/topic/golang-dev/9F1L4hBCkFs/discussion

https://groups.google.com/d/topic/golang-dev/-55InBXhAgA/discussion

https://groups.google.com/d/topic/golang-dev/Q_j12RbO7Zk/discussion

https://groups.google.com/d/topic/golang-dev/hYhB-jbzaXc/discussion

In my opinion, they are unrelated to GC issues on 32-bit CPUs.

[Devon H. O'Dell]

Op 6 april 2012 11:16 heeft ⚛ <0xe2.0x...@gmail.com> het volgende
geschreven:

Yes, these changes make the GC parallel, so it doesn't pause, but they
do not change the fact that it is a conservative GC.

--dho

[Uriel]

On Fri, Apr 6, 2012 at 3:39 PM, Paul <2pau...@googlemail.com> wrote:
> hmmm... if you are writing applications for a specific hardware
> platform, in this case Intel Atom, you probably can't just  switch to
> 64bit hardware so easily. Likely it will screw up the whole business
> model.
>
> I actually don't know what the future is of 32bit, but it appears that
> Intel is still selling it, at least for small devices.

1) For most workloads and applications Go works fine on 32bits

2) All recent Atom chips can do 64bits, I'm not sure if old
32-bit-only parts are still sold by Intel, but I doubt they will be
around for much longer.

[Aram Hăvărneanu]

> 2) All recent Atom chips can do 64bits, I'm not sure if old
> 32-bit-only parts are still sold by Intel, but I doubt they will be
> around for much longer.

Let's hope x32 doesn't catch on: http://en.wikipedia.org/wiki/X32_ABI

--
Aram Hăvărneanu

[andre...@gmail.com]

Sure looks like the FAQ advertises Go as a "systems" PL. Quoting from http://golang.org/doc/go_faq.html

   No major systems language has emerged in over a decade ... We believe it's worth trying again with [Go]

   ...

   By its design, Go proposes an approach for the construction of system software on multicore machines.

                                Andrew

                                da...@dalkescientific.com

[Russ Cox]

We removed the word 'systems' because it was too limiting.
Go is more general than that. It is still a great language for
writing systems.

Russ

[kortschak]

For fun I've been doing the coursera cryptography course, and I've noticed something that might be related.

One of the assignments is to perform a Birthday Attack on the LSB₅₀ of SHA256 - the Go core library and built-in data types make putting this together trivial and quite fun. What I've found though is that repeated iterations of the attack (required to guarantee a collision generation) result in continually increasing memory consumption over successive iterations (i.e. the second iteration of the attack does not start with a memory footprint reduced from the level at the end of the first iteratoion even if runtime.GC() is called and the lookup table is nil'd - this results in thrashing on my laptop by the middle of the third iteration). The solution to this is running the attack repeatedly either manually or from a script, which seem inelegant.

Should I expect continually increasing memory consumption from repeated large block allocation and release? Is there someway to force freeing ob blocks like this? GOARCH=amd64.

I can't put up the code right now due to the ToS of the course, but the due date for the assignment (extra credit project) is the 9th, so I should be OK to put it up then - and the approach is pretty much exactly what the algorithm suggest, using a map for the table. Note that h and m are random in {0,1}⁵⁰ and {0,1}⁶³ respectively and m might generate byte sequences that look like pointers when stored (as an [8]byte).

[Hotei]

Box,

You read more into my post than was there.  I was pointing out to Paul that the go developers do have a plan for updating the garbage collector and currently it's not focused on 32 bit fixes.  That's neither good nor bad, it just is.  It's also my opinion that failing to perform adequately in one specific area does not make go a bad "systems language".  It means that implementations on some platforms are lagging a bit.  Rather like saying C is a bad systems language because it doesn't perform well on a 6502 as it does on a Z80.    (yes, that's non-fiction.)  The go developers saying "Go is a good language for many things, including systems programming." is not a promise that it will work for every situation on every platform.  If it doesn't fit the person's business model for a specific problem there's always the option to use something else that does.  It might be nice to post a "warning sign" on the 32-bit road, but the time since the first official language release is still being measured in days not months or years. It's a little premature to write-off go though I can understand the OP's problem.

[Russ Cox]

You should be able to mount the attack in 0 allocations per iteration.  Once your assignment is over, if you're still having memory problems, show us the code.

Russ

[Dan Kortschak]

Thanks Russ.

I've successfully completed the assignment, but a less naive approach
would be interesting. I'll repost when the time times.

Dan

[Archos]

On Apr 7, 12:18 am, Hotei <hotei1...@gmail.com> wrote:
> Box,
> You read more into my post than was there.  I was pointing out to Paul that
> the go developers do have a plan for updating the garbage collector and
> currently it's not focused on 32 bit fixes.  That's neither good nor bad,
> it just is.  It's also my opinion that failing to perform adequately in one
> specific area does not make go a bad "systems language".  It means that
> implementations on some platforms are lagging a bit.  Rather like saying C
> is a bad systems language because it doesn't perform well on a 6502 as it
> does on a Z80.    (yes, that's non-fiction.)  The go developers saying "Go
> is a good language for many things, including systems programming." is not
> a promise that it will work for every situation on every platform.  If it
> doesn't fit the person's business model for a specific problem there's
> always the option to use something else that does.  It might be nice to
> post a "warning sign" on the 32-bit road, but the time since the first
> official language release is still being measured in days not months or
> years. It's a little premature to write-off go though I can understand the
> OP's problem.

Go was primarily targeted at server-side applications and it's right
but then it should be said in the FAQ, or into a visible place so
people don't feel frustrated or deceived due to wrong expectactives.

These things are not good for a young language like Go because it
starts to getting bad posts and comments:

http://www.abtinforouzandeh.com/2012/04/08/Do-Not-Use-Go-For-32bit-Development.html
http://news.ycombinator.com/item?id=3814020
http://news.ycombinator.com/item?id=3805302

And, as recommendation, it would be good if somebody of the Go team
writes a post about this problem. There is people like me who have
inverted time on this language thinking in use it to developing
desktop applications and this is a great problem for such development
if it not could be fixed. Thanks!

[Archos]

On Apr 8, 6:45 pm, Archos <raul....@sent.com> wrote:
> Go was primarily targeted at server-side applications and it's right
> but then it should be said in the FAQ, or into a visible place so
> people don't feel frustrated or deceived due to wrong expectactives.

To stay it very clear, I mean about the problem in architectures of 32-
bits

[Michael Jones]

It can be fixed. Everything single thing is possible, generally, at the expense of something else. My sense is that priorities have been elsewhere so far.

--

Michael T. Jones | Chief Technology Advocate  | m...@google.com |  +1 650-335-5765

[Paul]

I did look at some of the posts about GC on golang-dev, although I
don't follow that forum as continuously as I do this one. I have been
aware of the work going on in the area of parallel GC.

I finally had a chance to look at the actual issue today:
http://code.google.com/p/go/issues/detail?id=909

It was surprising to me, apparently it was surprising to some others
as well. I do feel that there should be a warning sign on that road.

Apparently the issue may also be relevant on 64bit systems, although
it will not become obvious there, I don't know if it has security
implications.

The Unicode package for example seems to aggravate the problem,
although there seems to be a fix in comment 20 of issue 909 that
targets that area (not sure...).

While its a matter of design (conservativ GC), I have the feeling that
some stop-gap patches could be possible to mitigate the problem.
Decriptions of practical workarounds will likely also help. The issue
seems to have sufficient severity.

[Norbert Roos]

> It seems to be Issue 909 (http://code.google.com/p/go/issues/detail?
> id=909) or something similar,

After reading this thread, the issue discussion and a discussion on the
internet, im am now rather confused.

Could someone who understood the problem please summarize it?

- What is actually the problem [1]?

- Which operating systems are affected [2]?

- Does this behaviour affect all long-running 32-bit programs, or only
especially those who allocate a lot of memory?

- Can i somehow detect if my program is slowly running into trouble, e.g
with top or ps, or is it completely hidden inside the go runtime
environment?

- When the program crashes, can it take down other programs as well?
(when the memory is completely exhausted and the os starts killing
processes randomly)?

[1] This thread is about crashing after a while, other discussions
sounded like it was a problem right at the start, when a DLL is loaded
in the middle of the virtual address space...

[2] other discussions were only related to Windows

Thank you for clarification!

Norbert

[Kyle Lemons]

Greatly simplified:

On Wed, Apr 11, 2012 at 9:57 AM, Norbert Roos <nr...@webware-experts.de> wrote:

It seems to be Issue 909 (http://code.google.com/p/go/issues/detail?
id=909) or something similar,

After reading this thread, the issue discussion and a discussion on the internet, im am now rather confused.

Could someone who understood the problem please summarize it?

- What is actually the problem [1]?

Normal numbers are more likely to look like pointers in 32-bit, because the gc has very limited knowledge of what is in memory

 

- Which operating systems are affected [2]?

None in particular, just 32-bit toolchains.

 

- Does this behaviour affect all long-running 32-bit programs, or only especially those who allocate a lot of memory?

The more memory is allocated, the greater the chance that some number (often in a lookup table or something) will coincide with its address and prevent the garbage collector from freeing it

 

- Can i somehow detect if my program is slowly running into trouble, e.g with top or ps, or is it completely hidden inside the go runtime environment?

Not really, but if profiling looks like things aren't being freed which should be, you might be seeing it.

 

- When the program crashes, can it take down other programs as well? (when the memory is completely exhausted and the os starts killing processes randomly)?

In linux, the OOM killer has some interesting strategies, but the short answer is yes, the OOM killer could kill other processes before yours.

[⚛]

On Wednesday, April 11, 2012 6:57:37 PM UTC+2, Norbert Roos wrote:

- Does this behaviour affect all long-running 32-bit programs, or only
especially those who allocate a lot of memory?

This is hard to tell. 32-bit programs which use a lot of memory are affected more likely than programs which use small amounts of memory.

If you know which code patterns to avoid, the probability of a normal 32-bit Go program running out of memory is close to zero.

Below are some concrete examples of code patterns to avoid. An arbitrary binary value is a value which can range from 0 to (1<<32 - 1):

  type S struct {

     a, b, c T

     binaryData [1000]byte   // Replace with *[1000]byte or []byte

  }

  type S struct {

     a, b, c T

     arbitraryBinaryValue int

  }

  var m map[int]T   // If the int keys are arbitrary values

  func f() {

      var arbitraryBinaryValue int = something

      ...

      allocateLotOfMemory()   // Trigger a garbage collection

      ...

  }

[minux]

On Thu, Apr 12, 2012 at 1:23 AM, ⚛ <0xe2.0x...@gmail.com> wrote:

If you know which code patterns to avoid, the probability of a normal 32-bit Go program running out of memory is close to zero.

Below are some concrete examples of code patterns to avoid. An arbitrary binary value is a value which can range from 0 to (1<<32 - 1):

  type S struct {

     a, b, c T

     binaryData [1000]byte   // Replace with *[1000]byte or []byte

  }

If you choose to allocate array of S statically, and never copy it to heap (i.e. pass index/pointer around to access

an entry), then this code pattern won't give you problems. If you allocate struct S dynamically, i.e. with &S{}, or S{},

or new, then you *might* be trapped by the conservative GC.

Any pure data (i.e. no pointer involved) allocated statically will be handled correctly by the current gc toolchain.

[Gustavo Niemeyer]

On Wed, Apr 11, 2012 at 14:23, ⚛ <0xe2.0x...@gmail.com> wrote:
>      binaryData [1000]byte   // Replace with *[1000]byte or []byte

How're the replacements any better than [1000]byte in this case? If
people follow the advice but make([]byte, 1000) dynamically, that may
end up in the heap and they'll get to the same place, right?

--
Gustavo Niemeyer
http://niemeyer.net
http://niemeyer.net/plus
http://niemeyer.net/twitter
http://niemeyer.net/blog

-- I'm not absolutely sure of anything.

[Kyle Lemons]

On Wed, Apr 11, 2012 at 10:35 AM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:

On Wed, Apr 11, 2012 at 14:23, ⚛ <0xe2.0x...@gmail.com> wrote:
>      binaryData [1000]byte   // Replace with *[1000]byte or []byte

How're the replacements any better than [1000]byte in this case?  If
people follow the advice but make([]byte, 1000) dynamically, that may
end up in the heap and they'll get to the same place, right?

Any pointer anywhere in the byte array, no matter how it's allocated, will pin it.  Having it as a pointer/reference in the struct keeps the struct from also being pinned.

[⚛]

On Wednesday, April 11, 2012 7:35:42 PM UTC+2, Gustavo Niemeyer wrote:

On Wed, Apr 11, 2012 at 14:23, ⚛ <0xe2.0x...@gmail.com> wrote:
>      binaryData [1000]byte   // Replace with *[1000]byte or []byte

How're the replacements any better than [1000]byte in this case?

It is better because Go's GC may fail if the struct contains [1000]byte and also contains pointer fields.

The GC can handle make([]byte,1000) correctly.

[minux]

On Thu, Apr 12, 2012 at 1:49 AM, Kyle Lemons <kev...@google.com> wrote:

On Wed, Apr 11, 2012 at 10:35 AM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:

On Wed, Apr 11, 2012 at 14:23, ⚛ <0xe2.0x...@gmail.com> wrote:
>      binaryData [1000]byte   // Replace with *[1000]byte or []byte

How're the replacements any better than [1000]byte in this case?  If
people follow the advice but make([]byte, 1000) dynamically, that may
end up in the heap and they'll get to the same place, right?

Any pointer anywhere in the byte array, no matter how it's allocated, will pin it.  Having it as a pointer/reference in the struct keeps the struct from also being pinned.

Only dynamically allocated byte arrays have this problem.

If you declare it will something like:

var someVar [10000]S

at the package level, then it won't cause you any problem (in fact gc won't ever scan it looking for

pointers).

[Gustavo Niemeyer]

On Wed, Apr 11, 2012 at 14:53, ⚛ <0xe2.0x...@gmail.com> wrote:
> It is better because Go's GC may fail if the struct contains [1000]byte and
> also contains pointer fields.
>
> The GC can handle make([]byte,1000) correctly.

Sorry, I misunderstood what you meant. I thought you were concerned
about pointers *to* that data causing problems sooner, since the
sample structure doesn't contain visible pointers either and thus
would get FlagNoPointers by itself.

[⚛]

On Wednesday, April 11, 2012 8:23:00 PM UTC+2, Gustavo Niemeyer wrote:

On Wed, Apr 11, 2012 at 14:53, ⚛ <0xe2.0x...@gmail.com> wrote:
> It is better because Go's GC may fail if the struct contains [1000]byte and
> also contains pointer fields.
>
> The GC can handle make([]byte,1000) correctly.

Sorry, I misunderstood what you meant. I thought you were concerned
about pointers *to* that data causing problems sooner, since the
sample structure doesn't contain visible pointers either and thus
would get FlagNoPointers by itself.

You are right. I should have used *T instead of just T in order to be more explicit about what I meant.

[Gustavo Niemeyer]

On Wed, Apr 11, 2012 at 14:30, minux <minu...@gmail.com> wrote:
> Any pure data (i.e. no pointer involved) allocated statically will be
> handled correctly by the current gc toolchain.

Pure data in the heap also seems to be handled correctly.

[minux]

On Thu, Apr 12, 2012 at 2:40 AM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:

On Wed, Apr 11, 2012 at 14:30, minux <minu...@gmail.com> wrote:
> Any pure data (i.e. no pointer involved) allocated statically will be
> handled correctly by the current gc toolchain.

Pure data in the heap also seems to be handled correctly.

Yeah, I think the recommendation really should be: never mix pointers and large array

of arbitrary binary data in the same struct, because FlagNoPointers only apply to the

type as a whole. Is that right?

[Gustavo Niemeyer]

On Wed, Apr 11, 2012 at 15:44, minux <minu...@gmail.com> wrote:
> Yeah, I think the recommendation really should be: never mix pointers and
> large array of arbitrary binary data in the same struct, because FlagNoPointers
> only apply to the type as a whole. Is that right?

This sounds like a good workaround to try once someone has issues, but
I'm sure if we should be providing such recommendations at this point,
as it's hard to really tell what is the cause of such a problem. A
silly edge example: if you have 1GB of contiguously allocated data,
the real issue isn't whether this data has pointers or not, but the
fact that you have a good deal of the address space taken. Then, a
couple of types involved in a conservative leak could hold all of that
data in memory.

If people start having a lot of real issues (this thread doesn't show
that), it's time to fix the implementation IMO, rather than
recommending code bending techniques.

[Andy Balholm]

On Wednesday, April 11, 2012 12:26:11 PM UTC-7, Gustavo Niemeyer wrote:

If people start having a lot of real issues 

I've been having some issues with this problem, but it seems like some simple "code-bending" has taken care of most of them for me.

The project I'm working on is a long-running server process (an ICAP server). Its memory allocation patterns would be ideal for a generational garbage collector: it builds over 100 MB of data structures from its config files at startup. This includes several large maps, and a huge tree for Aho-Corasick string matching. This data is never modified again. It is only read. Then each request allocates a bunch of memory, but all of this memory is temporary. Once the request has been served and logged, it is never accessed again.

On one of my beta-testers' systems (an old Pentium 4 running ClearOS, which is 32-bit), it runs out of memory and crashes after a few weeks of operation. 

Calling runtime.GC() after every tenth request seems to keep it from crashing. After running a long time, it still has an RSS of only 175 MB. Apparently if it gets GC'd often enough, the heap never grows far enough to run into the addresses that are pinned by the pseudo-pointers. 

This result would suggest to me that lowering the default memory usage threshold for running the GC on 32-bit systems would eliminate some of the problems that people are having with conservative GC. The GC would run more often, but its pause times would be shorter, since the heap would stay smaller.

My own system has a new enough processor to support running a 64-bit kernel. ClearOS doesn't come in a 64-bit version, but I recently installed a 64-bit kernel from CentOS, leaving the userland 32-bit. I compiled my server with CGO_ENABLED=0 on a 64-bit machine, and it works fine on the mixed system. Another win for static linking!

[Norbert Roos]

On 04/11/2012 07:07 PM, Kyle Lemons wrote:

> Greatly simplified:

Thank you very much, and the others, too!

Norbert

[Dan Kortschak]

Hi Russ,

I didn't manage to figure out how to do what you're suggesting unless I
don't require that I can return the colliding values (i.e. I can easily
generate collision without allocations if I don't care what they are -
the assignment asked that we give the colliding values).

The linked code <https://gist.github.com/2371244> should generate a
collision with P = ~1023/1024, but in practice during the second run
memory is exhausted (amd64 4GB laptop), so I need to stop the program
and restart (actually the version I used for the assignment does not
have the j loop, and succeeds with P = ~0.5 and I manually ran until
success).

thanks
Dan

On Fri, 2012-04-06 at 19:39 -0400, Russ Cox wrote: