Severe memory problems on 32bit Linux

[Erik Unger]

We are using Go for a real world website (http://startuplive.in) with
real world traffic and the Go process has been crashing every other
hour or so on 32bit Intel Linux. Automatic restarts were no solution
because it was such am memory hog, that it killed other processes too
before killing itself.

After a change to 64bit, all those problems are gone.

We are also running godoc -http behind Apache on another 32 Linux
machine, and the godoc process is basically killing the whole system
on a daily basis.

It seems to be Issue 909 (http://code.google.com/p/go/issues/detail?
id=909) or something similar, which makes Go absolutely unusable for
real world applications on 32bit Linux.

[Jan Mercl]

On Thursday, April 5, 2012 10:02:31 AM UTC+2, Erik Unger wrote:

It seems to be Issue 909 (http://code.google.com/p/go/issues/detail?
id=909) or something similar, which makes Go absolutely unusable for
real world applications on 32bit Linux.

For some unknown to me definition of 'absolutely'.

[⚛]

You are right about the fact that it is an issue for long-running 32-bit programs. It may be possible to improve the situation by adhering to the following rules throughout your program:

- avoid struct types which contain both integer and pointer fields

- avoid struct types containing arrays such as [100]byte (replace it with *[100]byte or with []byte)

- avoid data structures which form densely interconnected graphs at run-time

- avoid deep call chains at run-time

- replace pointer identity with value equivalence (this can lead to a more explicit memory management in your program)

- if a data structure contains both long-lived and short-lived fields, move the short-lived fields into a separate data structure or into local variables of a function

- avoid integer values which may alias at run-time to an address; make sure most integer values are fairly low (such as: below 10000)

- if you are using caches to speed up your program, apply the rules mentioned here to redesign the cache. It may also help to use strings instead of structs as map keys.

- lower the overall memory consumption of your program

- carefully speed up your program (this may lead to a lower memory consumption in certain situations)

- call runtime.GC()  (at the right moment)

On Thursday, April 5, 2012 10:02:31 AM UTC+2, Erik Unger wrote:

[⚛]

It is possible that you may be forgetting what the purpose of the golang-nuts forum is.

[Jan Mercl]

Let me guess. Discussing Go language, Go coding, problems encountered with using Go, sharing opinions about what is good and bad about Go and all of the previous, etc.?

What's wrong with not agreeing to a quite strong statement "Go absolutely unusable for real world applications on 32bit Linux" and saying so here? Is it OT or something?

As a meta topic now, I suggest to continue off-list, would you mind? ;-)

[André Moraes]

The 8g version of the compiler isn't the best choice to long-lived
process, at least right now, so you don't have much of a choice.
Stick to a 64 bit machine and things should work without much problem.

The godoc problem could be solved by auto-killing or running it
"on-demand", so when no request are being handled the process exits
and return memory, this could not be a solution for a high traffic
site but for something like a internnal godoc this should work.

> It seems to be Issue 909 (http://code.google.com/p/go/issues/detail?
> id=909) or something similar, which makes Go absolutely unusable for
> real world applications on 32bit Linux.

Not all real world applications are server side applications. There is
a considerable ammount of applications in Go that are client
applications (text-editor, gocode, games).

--
André Moraes
http://andredevchannel.blogspot.com/

[tom westberg]

On Thursday, April 5, 2012 4:13:45 AM UTC-4, Jan Mercl wrote:

On Thursday, April 5, 2012 10:02:31 AM UTC+2, Erik Unger wrote:

It seems to be Issue 909 ... which makes Go absolutely unusable for

real world applications on 32bit Linux.

For some unknown to me definition of 'absolutely'.

While you are literally correct that there are many real world applications for which go remains viable on 32-bit platforms, it's not really a reassuring attitude to engineers considering the use of go on a new project. Issue 909 has a priority of "Later" and comments which suggest that for long-running applications you should just get new hardware. The bug will be addressed in double-digit months.

Imagine yourself as an engineer who evangelized go inside an organization for an application that happens to fit this bad profile. Foolishly you did not read the entire open issues database, believing that christening a language "1.0" means that it works as a general-purpose language on all supported platforms. You deploy your system and.. it starts crashing. Advice is to switch to 64-bit hardware. In this case I hope we can forgive a bit of hyperbole from someone calling it "absolutely unusable". 

I find myself in a similar position (although I hadn't reached the actual implementation stage; I've just been intrigued by the go language and its features) in considering go for a platform which must be 32-bit (we ship atom-based appliances) on long-running applications (network connection daemons). It's conceivable that the memory usage profile for the project would not tickle the gc bug, but the language for avoiding the problem seems vague ("It may be possible to improve the situation.."). 

Would you be an advocate for such a project?

[Norbert Roos]

> We are using Go for a real world website (http://startuplive.in) with
> real world traffic and the Go process has been crashing every other
> hour or so on 32bit Intel Linux.

Is this problem related to x86 or does it apply to ARM as well?

And what about the option to disable gc with a compiler switch and doing
the memory management self using delete()? For future versions, of
course.. i've always been sceptical about gc.

Norbert

[Sebastien Binet]

André Moraes <and...@gmail.com> writes:

> The 8g version of the compiler isn't the best choice to long-lived
> process, at least right now, so you don't have much of a choice.

does gcc-go 32b expose the same behaviour ?
(otherwise, it could be a rather good choice)

-s

[André Moraes]

>
> And what about the option to disable gc with a compiler switch and doing the
> memory management self using delete()? For future versions, of course.. i've
> always been sceptical about gc.
>

I think we shouldn't blame the concept of GC because of a issue in one
implementation.

[André Moraes]

2012/4/5 Sebastien Binet <seb....@gmail.com>:

I never used the gcc-go, but it's a alternative.

[minux]

On Thu, Apr 5, 2012 at 11:19 PM, Norbert Roos <nr...@webware-experts.de> wrote:

We are using Go for a real world website (http://startuplive.in) with
real world traffic and the Go process has been crashing every other
hour or so on 32bit Intel Linux.

Is this problem related to x86 or does it apply to ARM as well?

I think ARM port suffer from this issue too. The main reason is that on 32-bit architectures,

normal data have a much large chance to look like a pointer, and current gc is conservative,

and lacking some type annotation (is this word possible pointer?).

And what about the option to disable gc with a compiler switch and doing the memory management self using delete()? For future versions, of course.. i've always been sceptical about gc.

Not possible yet. the runtime and std library needs gc themself.

[minux]

On Thu, Apr 5, 2012 at 7:04 PM, ⚛ <0xe2.0x...@gmail.com> wrote:

You are right about the fact that it is an issue for long-running 32-bit programs. It may be possible to improve the situation by adhering to the following rules throughout your program:

- avoid struct types which contain both integer and pointer fields

- avoid struct types containing arrays such as [100]byte (replace it with *[100]byte or with []byte)

- avoid data structures which form densely interconnected graphs at run-time

- avoid deep call chains at run-time

- replace pointer identity with value equivalence (this can lead to a more explicit memory management in your program)

- if a data structure contains both long-lived and short-lived fields, move the short-lived fields into a separate data structure or into local variables of a function

- avoid integer values which may alias at run-time to an address; make sure most integer values are fairly low (such as: below 10000)

- if you are using caches to speed up your program, apply the rules mentioned here to redesign the cache. It may also help to use strings instead of structs as map keys.

- lower the overall memory consumption of your program

- carefully speed up your program (this may lead to a lower memory consumption in certain situations)

- call runtime.GC()  (at the right moment)

one more advice, declare large array of struct with no pointer fields statically (i.e. don't use new), and the compiler

will place them in special places (.noptrdata and .noptrbss sections) that the GC won't even scan, and this can also

eliminate some fale positives (some data happen to look like pointers are regarded as pointers).

If you are brave enough, you can implement your own memory allocator on static allocated memory.

[Russ Cox]

On Thu, Apr 5, 2012 at 04:02, Erik Unger <ung...@gmail.com> wrote:
> We are using Go for a real world website (http://startuplive.in) with
> real world traffic and the Go process has been crashing every other
> hour or so on 32bit Intel Linux. Automatic restarts were no solution
> because it was such am memory hog, that it killed other processes too
> before killing itself.
>
> After a change to 64bit, all those problems are gone.

Great!

> We are also running godoc -http behind Apache on another 32 Linux
> machine, and the godoc process is basically killing the whole system
> on a daily basis.
>
> It seems to be Issue 909 (http://code.google.com/p/go/issues/detail?
> id=909) or something similar, which makes Go absolutely unusable for
> real world applications on 32bit Linux.

Very easy to believe. There's no easy software fix here.
The easy hardware fix is to switch to 64-bit.

Russ

[Ian Lance Taylor]

Sebastien Binet <seb....@gmail.com> writes:

Gccgo ought to be better on the margin, because it does not interpret
the entire data section as possibly containing pointers. However, it
does not address the base issue, which is that Go uses a conservative
garbage collector, and more values look like pointers in a 32-bit world.

The only real fix would be to improve the garbage collector's
understanding of which values are pointers and which are something else
(e.g., floating point numbers that happen to look like pointers). And
that is not an easy fix.

Ian

[Joubin Houshyar]

On Apr 5, 8:55 am, André Moraes <andr...@gmail.com> wrote:
...

> Not all real world applications are server side applications. There is
> a considerable ammount of applications in Go that are client
> applications (text-editor, gocode, games).

That is true, André, but Go is advertised as a "systems" PL. I for
one would like to see the Go PL stick to promise. Long running server
apps are very much a "systems PL" concern.

[David Symonds]

On Fri, Apr 6, 2012 at 1:14 PM, Joubin Houshyar <jhou...@gmail.com> wrote:

> That is true, André, but Go is advertised as a "systems" PL.

It's not. It used to be, but it's not any more.

http://golang.org/

[Joubin Houshyar]

On Apr 5, 11:42 pm, David Symonds <dsymo...@golang.org> wrote:

> On Fri, Apr 6, 2012 at 1:14 PM, Joubin Houshyar <jhoush...@gmail.com> wrote:
> > That is true, André, but Go is advertised as a "systems" PL.
>
> It's not. It used to be, but it's not any more.

/That is news/. So just "general PL", then? (With due apologies) I
have no idea if you are a member of set "Go Authors" as your sig
suggests, but If yes, what is the 'purpose'/'mission statement' of Go,
as of today?


>
> http://golang.org/

[Kyle Lemons]

There are lots of authors :).

 

>
> http://golang.org/

[Leon Szpilewski]

> Go absolutely unusable for real world applications on 32bit Linux.

I have been using Go on a 32bit server for a pretty frequently visited Blog (30k uniques/day). And there were no memory problems whatsoever.

I guess you have to over think your software design ...

cheers,

Leon

[Archos]

On Apr 5, 8:29 pm, Russ Cox <r...@golang.org> wrote:

> On Thu, Apr 5, 2012 at 04:02, Erik Unger <unge...@gmail.com> wrote:
> > We are also running godoc -http behind Apache on another 32 Linux
> > machine, and the godoc process is basically killing the whole system
> > on a daily basis.
>
> > It seems to be Issue 909 (http://code.google.com/p/go/issues/detail?
> > id=909) or something similar, which makes Go absolutely unusable for
> > real world applications on 32bit Linux.
>
> Very easy to believe.  There's no easy software fix here.
> The easy hardware fix is to switch to 64-bit.

So, Go is not suitable to build desktop long-running applications on
32-bit, at least that you want say to the users that switch to 64-bit.

[Paul]

hmmm... if you are writing applications for a specific hardware
platform, in this case Intel Atom, you probably can't just switch to
64bit hardware so easily. Likely it will screw up the whole business
model.

I actually don't know what the future is of 32bit, but it appears that
Intel is still selling it, at least for small devices.

I think it would help to have something like a "Read.me" file for the
Go- language which lists known issues.

The next thing that comes to mind is: now that Go 1 is out, would this
not be good time to devote some time and resources to developing a
improved runtime and garbage collector, and take it to the next level?

[Hotei]

Paul,

A look at recent posts in golang-dev might be enlightening...

[⚛]

On Friday, April 6, 2012 3:55:29 PM UTC+2, Hotei wrote:

A look at recent posts in golang-dev might be enlightening...

Which recent posts do you mean?

[⚛]

On Friday, April 6, 2012 5:08:44 PM UTC+2, Peter Thrun wrote:

Here's a sampling of recent threads about updates to the garbage collector:

https://groups.google.com/d/topic/golang-dev/9F1L4hBCkFs/discussion

https://groups.google.com/d/topic/golang-dev/-55InBXhAgA/discussion

https://groups.google.com/d/topic/golang-dev/Q_j12RbO7Zk/discussion

https://groups.google.com/d/topic/golang-dev/hYhB-jbzaXc/discussion

In my opinion, they are unrelated to GC issues on 32-bit CPUs.

[Devon H. O'Dell]

Op 6 april 2012 11:16 heeft ⚛ <0xe2.0x...@gmail.com> het volgende
geschreven:

Yes, these changes make the GC parallel, so it doesn't pause, but they
do not change the fact that it is a conservative GC.

--dho

[Uriel]

On Fri, Apr 6, 2012 at 3:39 PM, Paul <2pau...@googlemail.com> wrote:
> hmmm... if you are writing applications for a specific hardware
> platform, in this case Intel Atom, you probably can't just  switch to
> 64bit hardware so easily. Likely it will screw up the whole business
> model.
>
> I actually don't know what the future is of 32bit, but it appears that
> Intel is still selling it, at least for small devices.

1) For most workloads and applications Go works fine on 32bits

2) All recent Atom chips can do 64bits, I'm not sure if old
32-bit-only parts are still sold by Intel, but I doubt they will be
around for much longer.

[Aram Hăvărneanu]

> 2) All recent Atom chips can do 64bits, I'm not sure if old
> 32-bit-only parts are still sold by Intel, but I doubt they will be
> around for much longer.

Let's hope x32 doesn't catch on: http://en.wikipedia.org/wiki/X32_ABI

--
Aram Hăvărneanu

[andre...@gmail.com]

Sure looks like the FAQ advertises Go as a "systems" PL. Quoting from http://golang.org/doc/go_faq.html

   No major systems language has emerged in over a decade ... We believe it's worth trying again with [Go]

   ...

   By its design, Go proposes an approach for the construction of system software on multicore machines.

                                Andrew

                                da...@dalkescientific.com

[Russ Cox]

We removed the word 'systems' because it was too limiting.
Go is more general than that. It is still a great language for
writing systems.

Russ

[kortschak]

For fun I've been doing the coursera cryptography course, and I've noticed something that might be related.

One of the assignments is to perform a Birthday Attack on the LSB₅₀ of SHA256 - the Go core library and built-in data types make putting this together trivial and quite fun. What I've found though is that repeated iterations of the attack (required to guarantee a collision generation) result in continually increasing memory consumption over successive iterations (i.e. the second iteration of the attack does not start with a memory footprint reduced from the level at the end of the first iteratoion even if runtime.GC() is called and the lookup table is nil'd - this results in thrashing on my laptop by the middle of the third iteration). The solution to this is running the attack repeatedly either manually or from a script, which seem inelegant.

Should I expect continually increasing memory consumption from repeated large block allocation and release? Is there someway to force freeing ob blocks like this? GOARCH=amd64.

I can't put up the code right now due to the ToS of the course, but the due date for the assignment (extra credit project) is the 9th, so I should be OK to put it up then - and the approach is pretty much exactly what the algorithm suggest, using a map for the table. Note that h and m are random in {0,1}⁵⁰ and {0,1}⁶³ respectively and m might generate byte sequences that look like pointers when stored (as an [8]byte).

[Hotei]

Box,

You read more into my post than was there.  I was pointing out to Paul that the go developers do have a plan for updating the garbage collector and currently it's not focused on 32 bit fixes.  That's neither good nor bad, it just is.  It's also my opinion that failing to perform adequately in one specific area does not make go a bad "systems language".  It means that implementations on some platforms are lagging a bit.  Rather like saying C is a bad systems language because it doesn't perform well on a 6502 as it does on a Z80.    (yes, that's non-fiction.)  The go developers saying "Go is a good language for many things, including systems programming." is not a promise that it will work for every situation on every platform.  If it doesn't fit the person's business model for a specific problem there's always the option to use something else that does.  It might be nice to post a "warning sign" on the 32-bit road, but the time since the first official language release is still being measured in days not months or years. It's a little premature to write-off go though I can understand the OP's problem.

[Russ Cox]

You should be able to mount the attack in 0 allocations per iteration.  Once your assignment is over, if you're still having memory problems, show us the code.

Russ

[Dan Kortschak]

Thanks Russ.

I've successfully completed the assignment, but a less naive approach
would be interesting. I'll repost when the time times.

Dan

[Archos]

On Apr 7, 12:18 am, Hotei <hotei1...@gmail.com> wrote:
> Box,
> You read more into my post than was there.  I was pointing out to Paul that
> the go developers do have a plan for updating the garbage collector and
> currently it's not focused on 32 bit fixes.  That's neither good nor bad,
> it just is.  It's also my opinion that failing to perform adequately in one
> specific area does not make go a bad "systems language".  It means that
> implementations on some platforms are lagging a bit.  Rather like saying C
> is a bad systems language because it doesn't perform well on a 6502 as it
> does on a Z80.    (yes, that's non-fiction.)  The go developers saying "Go
> is a good language for many things, including systems programming." is not
> a promise that it will work for every situation on every platform.  If it
> doesn't fit the person's business model for a specific problem there's
> always the option to use something else that does.  It might be nice to
> post a "warning sign" on the 32-bit road, but the time since the first
> official language release is still being measured in days not months or
> years. It's a little premature to write-off go though I can understand the
> OP's problem.

Go was primarily targeted at server-side applications and it's right
but then it should be said in the FAQ, or into a visible place so
people don't feel frustrated or deceived due to wrong expectactives.

These things are not good for a young language like Go because it
starts to getting bad posts and comments:

http://www.abtinforouzandeh.com/2012/04/08/Do-Not-Use-Go-For-32bit-Development.html
http://news.ycombinator.com/item?id=3814020
http://news.ycombinator.com/item?id=3805302

And, as recommendation, it would be good if somebody of the Go team
writes a post about this problem. There is people like me who have
inverted time on this language thinking in use it to developing
desktop applications and this is a great problem for such development
if it not could be fixed. Thanks!

[Archos]

On Apr 8, 6:45 pm, Archos <raul....@sent.com> wrote:
> Go was primarily targeted at server-side applications and it's right
> but then it should be said in the FAQ, or into a visible place so
> people don't feel frustrated or deceived due to wrong expectactives.

To stay it very clear, I mean about the problem in architectures of 32-
bits