Severe memory problems on 32bit Linux

13,976 views
Skip to first unread message

Erik Unger

unread,
Apr 5, 2012, 4:02:31 AM4/5/12
to golang-nuts
We are using Go for a real world website (http://startuplive.in) with
real world traffic and the Go process has been crashing every other
hour or so on 32bit Intel Linux. Automatic restarts were no solution
because it was such am memory hog, that it killed other processes too
before killing itself.

After a change to 64bit, all those problems are gone.

We are also running godoc -http behind Apache on another 32 Linux
machine, and the godoc process is basically killing the whole system
on a daily basis.

It seems to be Issue 909 (http://code.google.com/p/go/issues/detail?
id=909) or something similar, which makes Go absolutely unusable for
real world applications on 32bit Linux.

Jan Mercl

unread,
Apr 5, 2012, 4:13:45 AM4/5/12
to golan...@googlegroups.com
On Thursday, April 5, 2012 10:02:31 AM UTC+2, Erik Unger wrote:
It seems to be Issue 909 (http://code.google.com/p/go/issues/detail?
id=909
) or something similar, which makes Go absolutely unusable for
real world applications on 32bit Linux.

For some unknown to me definition of 'absolutely'.

unread,
Apr 5, 2012, 7:04:14 AM4/5/12
to golan...@googlegroups.com
You are right about the fact that it is an issue for long-running 32-bit programs. It may be possible to improve the situation by adhering to the following rules throughout your program:

- avoid struct types which contain both integer and pointer fields

- avoid struct types containing arrays such as [100]byte (replace it with *[100]byte or with []byte)

- avoid data structures which form densely interconnected graphs at run-time

- avoid deep call chains at run-time

- replace pointer identity with value equivalence (this can lead to a more explicit memory management in your program)

- if a data structure contains both long-lived and short-lived fields, move the short-lived fields into a separate data structure or into local variables of a function

- avoid integer values which may alias at run-time to an address; make sure most integer values are fairly low (such as: below 10000)

- if you are using caches to speed up your program, apply the rules mentioned here to redesign the cache. It may also help to use strings instead of structs as map keys.

- lower the overall memory consumption of your program

- carefully speed up your program (this may lead to a lower memory consumption in certain situations)

- call runtime.GC()  (at the right moment)

On Thursday, April 5, 2012 10:02:31 AM UTC+2, Erik Unger wrote:

unread,
Apr 5, 2012, 7:09:06 AM4/5/12
to golan...@googlegroups.com
It is possible that you may be forgetting what the purpose of the golang-nuts forum is.

Jan Mercl

unread,
Apr 5, 2012, 7:20:40 AM4/5/12
to golan...@googlegroups.com
Let me guess. Discussing Go language, Go coding, problems encountered with using Go, sharing opinions about what is good and bad about Go and all of the previous, etc.?

What's wrong with not agreeing to a quite strong statement "Go absolutely unusable for real world applications on 32bit Linux" and saying so here? Is it OT or something?

As a meta topic now, I suggest to continue off-list, would you mind? ;-)

André Moraes

unread,
Apr 5, 2012, 8:55:32 AM4/5/12
to Erik Unger, golang-nuts
The 8g version of the compiler isn't the best choice to long-lived
process, at least right now, so you don't have much of a choice.
Stick to a 64 bit machine and things should work without much problem.

The godoc problem could be solved by auto-killing or running it
"on-demand", so when no request are being handled the process exits
and return memory, this could not be a solution for a high traffic
site but for something like a internnal godoc this should work.

> It seems to be Issue 909 (http://code.google.com/p/go/issues/detail?
> id=909) or something similar, which makes Go absolutely unusable for
> real world applications on 32bit Linux.

Not all real world applications are server side applications. There is
a considerable ammount of applications in Go that are client
applications (text-editor, gocode, games).

--
André Moraes
http://andredevchannel.blogspot.com/

tom westberg

unread,
Apr 5, 2012, 10:08:05 AM4/5/12
to golan...@googlegroups.com
On Thursday, April 5, 2012 4:13:45 AM UTC-4, Jan Mercl wrote:
On Thursday, April 5, 2012 10:02:31 AM UTC+2, Erik Unger wrote:
It seems to be Issue 909 ... which makes Go absolutely unusable for
real world applications on 32bit Linux.

For some unknown to me definition of 'absolutely'.

While you are literally correct that there are many real world applications for which go remains viable on 32-bit platforms, it's not really a reassuring attitude to engineers considering the use of go on a new project. Issue 909 has a priority of "Later" and comments which suggest that for long-running applications you should just get new hardware. The bug will be addressed in double-digit months.

Imagine yourself as an engineer who evangelized go inside an organization for an application that happens to fit this bad profile. Foolishly you did not read the entire open issues database, believing that christening a language "1.0" means that it works as a general-purpose language on all supported platforms. You deploy your system and.. it starts crashing. Advice is to switch to 64-bit hardware. In this case I hope we can forgive a bit of hyperbole from someone calling it "absolutely unusable". 

I find myself in a similar position (although I hadn't reached the actual implementation stage; I've just been intrigued by the go language and its features) in considering go for a platform which must be 32-bit (we ship atom-based appliances) on long-running applications (network connection daemons). It's conceivable that the memory usage profile for the project would not tickle the gc bug, but the language for avoiding the problem seems vague ("It may be possible to improve the situation.."). 

Would you be an advocate for such a project?

Norbert Roos

unread,
Apr 5, 2012, 11:19:26 AM4/5/12
to golan...@googlegroups.com
> We are using Go for a real world website (http://startuplive.in) with
> real world traffic and the Go process has been crashing every other
> hour or so on 32bit Intel Linux.

Is this problem related to x86 or does it apply to ARM as well?

And what about the option to disable gc with a compiler switch and doing
the memory management self using delete()? For future versions, of
course.. i've always been sceptical about gc.

Norbert

Sebastien Binet

unread,
Apr 5, 2012, 11:52:56 AM4/5/12
to André Moraes, Erik Unger, golang-nuts
André Moraes <and...@gmail.com> writes:

> The 8g version of the compiler isn't the best choice to long-lived
> process, at least right now, so you don't have much of a choice.

does gcc-go 32b expose the same behaviour ?
(otherwise, it could be a rather good choice)

-s

André Moraes

unread,
Apr 5, 2012, 11:53:55 AM4/5/12
to Norbert Roos, golan...@googlegroups.com
>
> And what about the option to disable gc with a compiler switch and doing the
> memory management self using delete()? For future versions, of course.. i've
> always been sceptical about gc.
>

I think we shouldn't blame the concept of GC because of a issue in one
implementation.

André Moraes

unread,
Apr 5, 2012, 11:56:58 AM4/5/12
to Sebastien Binet, Erik Unger, golang-nuts, ia...@google.com
2012/4/5 Sebastien Binet <seb....@gmail.com>:

I never used the gcc-go, but it's a alternative.

minux

unread,
Apr 5, 2012, 12:29:02 PM4/5/12
to Norbert Roos, golan...@googlegroups.com
On Thu, Apr 5, 2012 at 11:19 PM, Norbert Roos <nr...@webware-experts.de> wrote:
We are using Go for a real world website (http://startuplive.in) with
real world traffic and the Go process has been crashing every other
hour or so on 32bit Intel Linux.

Is this problem related to x86 or does it apply to ARM as well?
I think ARM port suffer from this issue too. The main reason is that on 32-bit architectures,
normal data have a much large chance to look like a pointer, and current gc is conservative,
and lacking some type annotation (is this word possible pointer?).

And what about the option to disable gc with a compiler switch and doing the memory management self using delete()? For future versions, of course.. i've always been sceptical about gc.
Not possible yet. the runtime and std library needs gc themself.

minux

unread,
Apr 5, 2012, 12:35:55 PM4/5/12
to golan...@googlegroups.com
On Thu, Apr 5, 2012 at 7:04 PM, ⚛ <0xe2.0x...@gmail.com> wrote:
You are right about the fact that it is an issue for long-running 32-bit programs. It may be possible to improve the situation by adhering to the following rules throughout your program:

- avoid struct types which contain both integer and pointer fields

- avoid struct types containing arrays such as [100]byte (replace it with *[100]byte or with []byte)

- avoid data structures which form densely interconnected graphs at run-time

- avoid deep call chains at run-time

- replace pointer identity with value equivalence (this can lead to a more explicit memory management in your program)

- if a data structure contains both long-lived and short-lived fields, move the short-lived fields into a separate data structure or into local variables of a function

- avoid integer values which may alias at run-time to an address; make sure most integer values are fairly low (such as: below 10000)

- if you are using caches to speed up your program, apply the rules mentioned here to redesign the cache. It may also help to use strings instead of structs as map keys.

- lower the overall memory consumption of your program

- carefully speed up your program (this may lead to a lower memory consumption in certain situations)

- call runtime.GC()  (at the right moment)
one more advice, declare large array of struct with no pointer fields statically (i.e. don't use new), and the compiler
will place them in special places (.noptrdata and .noptrbss sections) that the GC won't even scan, and this can also
eliminate some fale positives (some data happen to look like pointers are regarded as pointers).

If you are brave enough, you can implement your own memory allocator on static allocated memory.

Russ Cox

unread,
Apr 5, 2012, 3:29:01 PM4/5/12
to Erik Unger, golang-nuts
On Thu, Apr 5, 2012 at 04:02, Erik Unger <ung...@gmail.com> wrote:
> We are using Go for a real world website (http://startuplive.in) with
> real world traffic and the Go process has been crashing every other
> hour or so on 32bit Intel Linux. Automatic restarts were no solution
> because it was such am memory hog, that it killed other processes too
> before killing itself.
>
> After a change to 64bit, all those problems are gone.

Great!

> We are also running godoc -http behind Apache on another 32 Linux
> machine, and the godoc process is basically killing the whole system
> on a daily basis.
>
> It seems to be Issue 909 (http://code.google.com/p/go/issues/detail?
> id=909) or something similar, which makes Go absolutely unusable for
> real world applications on 32bit Linux.

Very easy to believe. There's no easy software fix here.
The easy hardware fix is to switch to 64-bit.

Russ

Ian Lance Taylor

unread,
Apr 5, 2012, 6:00:51 PM4/5/12
to Sebastien Binet, André Moraes, Erik Unger, golang-nuts
Sebastien Binet <seb....@gmail.com> writes:

Gccgo ought to be better on the margin, because it does not interpret
the entire data section as possibly containing pointers. However, it
does not address the base issue, which is that Go uses a conservative
garbage collector, and more values look like pointers in a 32-bit world.

The only real fix would be to improve the garbage collector's
understanding of which values are pointers and which are something else
(e.g., floating point numbers that happen to look like pointers). And
that is not an easy fix.

Ian

Joubin Houshyar

unread,
Apr 5, 2012, 11:14:33 PM4/5/12
to golang-nuts


On Apr 5, 8:55 am, André Moraes <andr...@gmail.com> wrote:
...
> Not all real world applications are server side applications. There is
> a considerable ammount of applications in Go that are client
> applications (text-editor, gocode, games).

That is true, André, but Go is advertised as a "systems" PL. I for
one would like to see the Go PL stick to promise. Long running server
apps are very much a "systems PL" concern.

David Symonds

unread,
Apr 5, 2012, 11:42:55 PM4/5/12
to Joubin Houshyar, golang-nuts
On Fri, Apr 6, 2012 at 1:14 PM, Joubin Houshyar <jhou...@gmail.com> wrote:

> That is true, André, but Go is advertised as a "systems" PL.

It's not. It used to be, but it's not any more.

http://golang.org/

Joubin Houshyar

unread,
Apr 5, 2012, 11:52:46 PM4/5/12
to golang-nuts


On Apr 5, 11:42 pm, David Symonds <dsymo...@golang.org> wrote:
> On Fri, Apr 6, 2012 at 1:14 PM, Joubin Houshyar <jhoush...@gmail.com> wrote:
> > That is true, André, but Go is advertised as a "systems" PL.
>
> It's not. It used to be, but it's not any more.

/That is news/. So just "general PL", then? (With due apologies) I
have no idea if you are a member of set "Go Authors" as your sig
suggests, but If yes, what is the 'purpose'/'mission statement' of Go,
as of today?


>
> http://golang.org/

Kyle Lemons

unread,
Apr 6, 2012, 12:51:51 AM4/6/12
to Joubin Houshyar, golang-nuts
There are lots of authors :).
 
>
> http://golang.org/

Leon Szpilewski

unread,
Apr 6, 2012, 3:14:26 AM4/6/12
to golan...@googlegroups.com
> Go absolutely unusable for real world applications on 32bit Linux.

I have been using Go on a 32bit server for a pretty frequently visited Blog (30k uniques/day). And there were no memory problems whatsoever.

I guess you have to over think your software design ...

cheers,
Leon

Archos

unread,
Apr 6, 2012, 5:10:48 AM4/6/12
to golang-nuts

On Apr 5, 8:29 pm, Russ Cox <r...@golang.org> wrote:
> On Thu, Apr 5, 2012 at 04:02, Erik Unger <unge...@gmail.com> wrote:
> > We are also running godoc -http behind Apache on another 32 Linux
> > machine, and the godoc process is basically killing the whole system
> > on a daily basis.
>
> > It seems to be Issue 909 (http://code.google.com/p/go/issues/detail?
> > id=909) or something similar, which makes Go absolutely unusable for
> > real world applications on 32bit Linux.
>
> Very easy to believe.  There's no easy software fix here.
> The easy hardware fix is to switch to 64-bit.
So, Go is not suitable to build desktop long-running applications on
32-bit, at least that you want say to the users that switch to 64-bit.

Paul

unread,
Apr 6, 2012, 9:39:28 AM4/6/12
to golang-nuts
hmmm... if you are writing applications for a specific hardware
platform, in this case Intel Atom, you probably can't just switch to
64bit hardware so easily. Likely it will screw up the whole business
model.

I actually don't know what the future is of 32bit, but it appears that
Intel is still selling it, at least for small devices.

I think it would help to have something like a "Read.me" file for the
Go- language which lists known issues.

The next thing that comes to mind is: now that Go 1 is out, would this
not be good time to devote some time and resources to developing a
improved runtime and garbage collector, and take it to the next level?

Hotei

unread,
Apr 6, 2012, 9:55:29 AM4/6/12
to golan...@googlegroups.com
Paul,
A look at recent posts in golang-dev might be enlightening...

unread,
Apr 6, 2012, 10:58:30 AM4/6/12
to golan...@googlegroups.com
On Friday, April 6, 2012 3:55:29 PM UTC+2, Hotei wrote:
A look at recent posts in golang-dev might be enlightening...

Which recent posts do you mean?
Message has been deleted

unread,
Apr 6, 2012, 11:16:42 AM4/6/12
to golan...@googlegroups.com
On Friday, April 6, 2012 5:08:44 PM UTC+2, Peter Thrun wrote:
Here's a sampling of recent threads about updates to the garbage collector:



In my opinion, they are unrelated to GC issues on 32-bit CPUs.

Devon H. O'Dell

unread,
Apr 6, 2012, 11:18:02 AM4/6/12
to ⚛, golan...@googlegroups.com
Op 6 april 2012 11:16 heeft ⚛ <0xe2.0x...@gmail.com> het volgende
geschreven:

Yes, these changes make the GC parallel, so it doesn't pause, but they
do not change the fact that it is a conservative GC.

--dho

Uriel

unread,
Apr 6, 2012, 11:37:39 AM4/6/12
to golang-nuts
On Fri, Apr 6, 2012 at 3:39 PM, Paul <2pau...@googlemail.com> wrote:
> hmmm... if you are writing applications for a specific hardware
> platform, in this case Intel Atom, you probably can't just  switch to
> 64bit hardware so easily. Likely it will screw up the whole business
> model.
>
> I actually don't know what the future is of 32bit, but it appears that
> Intel is still selling it, at least for small devices.

1) For most workloads and applications Go works fine on 32bits

2) All recent Atom chips can do 64bits, I'm not sure if old
32-bit-only parts are still sold by Intel, but I doubt they will be
around for much longer.

Aram Hăvărneanu

unread,
Apr 6, 2012, 12:11:37 PM4/6/12
to Uriel, golang-nuts
> 2) All recent Atom chips can do 64bits, I'm not sure if old
> 32-bit-only parts are still sold by Intel, but I doubt they will be
> around for much longer.

Let's hope x32 doesn't catch on: http://en.wikipedia.org/wiki/X32_ABI

--
Aram Hăvărneanu

andre...@gmail.com

unread,
Apr 6, 2012, 12:41:05 PM4/6/12
to golan...@googlegroups.com
Sure looks like the FAQ advertises Go as a "systems" PL. Quoting from http://golang.org/doc/go_faq.html

   No major systems language has emerged in over a decade ... We believe it's worth trying again with [Go]
   ...
   By its design, Go proposes an approach for the construction of system software on multicore machines.


                                Andrew
                                da...@dalkescientific.com

Russ Cox

unread,
Apr 6, 2012, 12:53:59 PM4/6/12
to andre...@gmail.com, golan...@googlegroups.com
We removed the word 'systems' because it was too limiting.
Go is more general than that. It is still a great language for
writing systems.

Russ

kortschak

unread,
Apr 6, 2012, 6:57:23 PM4/6/12
to golan...@googlegroups.com
For fun I've been doing the coursera cryptography course, and I've noticed something that might be related.

One of the assignments is to perform a Birthday Attack on the LSB₅₀ of SHA256 - the Go core library and built-in data types make putting this together trivial and quite fun. What I've found though is that repeated iterations of the attack (required to guarantee a collision generation) result in continually increasing memory consumption over successive iterations (i.e. the second iteration of the attack does not start with a memory footprint reduced from the level at the end of the first iteratoion even if runtime.GC() is called and the lookup table is nil'd - this results in thrashing on my laptop by the middle of the third iteration). The solution to this is running the attack repeatedly either manually or from a script, which seem inelegant.

Should I expect continually increasing memory consumption from repeated large block allocation and release? Is there someway to force freeing ob blocks like this? GOARCH=amd64.

I can't put up the code right now due to the ToS of the course, but the due date for the assignment (extra credit project) is the 9th, so I should be OK to put it up then - and the approach is pretty much exactly what the algorithm suggest, using a map for the table. Note that h and m are random in {0,1}⁵⁰ and {0,1}⁶³ respectively and m might generate byte sequences that look like pointers when stored (as an [8]byte).

Hotei

unread,
Apr 6, 2012, 7:18:42 PM4/6/12
to golan...@googlegroups.com
Box,
You read more into my post than was there.  I was pointing out to Paul that the go developers do have a plan for updating the garbage collector and currently it's not focused on 32 bit fixes.  That's neither good nor bad, it just is.  It's also my opinion that failing to perform adequately in one specific area does not make go a bad "systems language".  It means that implementations on some platforms are lagging a bit.  Rather like saying C is a bad systems language because it doesn't perform well on a 6502 as it does on a Z80.    (yes, that's non-fiction.)  The go developers saying "Go is a good language for many things, including systems programming." is not a promise that it will work for every situation on every platform.  If it doesn't fit the person's business model for a specific problem there's always the option to use something else that does.  It might be nice to post a "warning sign" on the 32-bit road, but the time since the first official language release is still being measured in days not months or years. It's a little premature to write-off go though I can understand the OP's problem.

Russ Cox

unread,
Apr 6, 2012, 7:39:40 PM4/6/12
to kortschak, golan...@googlegroups.com

You should be able to mount the attack in 0 allocations per iteration.  Once your assignment is over, if you're still having memory problems, show us the code.

Russ

Dan Kortschak

unread,
Apr 6, 2012, 7:44:14 PM4/6/12
to r...@golang.org, golan...@googlegroups.com
Thanks Russ.

I've successfully completed the assignment, but a less naive approach
would be interesting. I'll repost when the time times.

Dan

Archos

unread,
Apr 8, 2012, 1:45:17 PM4/8/12
to golang-nuts

On Apr 7, 12:18 am, Hotei <hotei1...@gmail.com> wrote:
> Box,
> You read more into my post than was there.  I was pointing out to Paul that
> the go developers do have a plan for updating the garbage collector and
> currently it's not focused on 32 bit fixes.  That's neither good nor bad,
> it just is.  It's also my opinion that failing to perform adequately in one
> specific area does not make go a bad "systems language".  It means that
> implementations on some platforms are lagging a bit.  Rather like saying C
> is a bad systems language because it doesn't perform well on a 6502 as it
> does on a Z80.    (yes, that's non-fiction.)  The go developers saying "Go
> is a good language for many things, including systems programming." is not
> a promise that it will work for every situation on every platform.  If it
> doesn't fit the person's business model for a specific problem there's
> always the option to use something else that does.  It might be nice to
> post a "warning sign" on the 32-bit road, but the time since the first
> official language release is still being measured in days not months or
> years. It's a little premature to write-off go though I can understand the
> OP's problem.
Go was primarily targeted at server-side applications and it's right
but then it should be said in the FAQ, or into a visible place so
people don't feel frustrated or deceived due to wrong expectactives.

These things are not good for a young language like Go because it
starts to getting bad posts and comments:

http://www.abtinforouzandeh.com/2012/04/08/Do-Not-Use-Go-For-32bit-Development.html
http://news.ycombinator.com/item?id=3814020
http://news.ycombinator.com/item?id=3805302

And, as recommendation, it would be good if somebody of the Go team
writes a post about this problem. There is people like me who have
inverted time on this language thinking in use it to developing
desktop applications and this is a great problem for such development
if it not could be fixed. Thanks!

Archos

unread,
Apr 8, 2012, 1:47:39 PM4/8/12
to golang-nuts

On Apr 8, 6:45 pm, Archos <raul....@sent.com> wrote:
> Go was primarily targeted at server-side applications and it's right
> but then it should be said in the FAQ, or into a visible place so
> people don't feel frustrated or deceived due to wrong expectactives.
To stay it very clear, I mean about the problem in architectures of 32-
bits

Michael Jones

unread,
Apr 8, 2012, 1:59:48 PM4/8/12
to Archos, golang-nuts
It can be fixed. Everything single thing is possible, generally, at the expense of something else. My sense is that priorities have been elsewhere so far.
--
Michael T. Jones | Chief Technology Advocate  | m...@google.com |  +1 650-335-5765

Paul

unread,
Apr 9, 2012, 1:20:13 PM4/9/12
to golang-nuts
I did look at some of the posts about GC on golang-dev, although I
don't follow that forum as continuously as I do this one. I have been
aware of the work going on in the area of parallel GC.

I finally had a chance to look at the actual issue today:
http://code.google.com/p/go/issues/detail?id=909

It was surprising to me, apparently it was surprising to some others
as well. I do feel that there should be a warning sign on that road.

Apparently the issue may also be relevant on 64bit systems, although
it will not become obvious there, I don't know if it has security
implications.

The Unicode package for example seems to aggravate the problem,
although there seems to be a fix in comment 20 of issue 909 that
targets that area (not sure...).

While its a matter of design (conservativ GC), I have the feeling that
some stop-gap patches could be possible to mitigate the problem.
Decriptions of practical workarounds will likely also help. The issue
seems to have sufficient severity.

Norbert Roos

unread,
Apr 11, 2012, 12:57:37 PM4/11/12
to golan...@googlegroups.com
> It seems to be Issue 909 (http://code.google.com/p/go/issues/detail?
> id=909) or something similar,

After reading this thread, the issue discussion and a discussion on the
internet, im am now rather confused.

Could someone who understood the problem please summarize it?

- What is actually the problem [1]?

- Which operating systems are affected [2]?

- Does this behaviour affect all long-running 32-bit programs, or only
especially those who allocate a lot of memory?

- Can i somehow detect if my program is slowly running into trouble, e.g
with top or ps, or is it completely hidden inside the go runtime
environment?

- When the program crashes, can it take down other programs as well?
(when the memory is completely exhausted and the os starts killing
processes randomly)?

[1] This thread is about crashing after a while, other discussions
sounded like it was a problem right at the start, when a DLL is loaded
in the middle of the virtual address space...

[2] other discussions were only related to Windows

Thank you for clarification!

Norbert

Kyle Lemons

unread,
Apr 11, 2012, 1:07:37 PM4/11/12