OpenGL GL_OUT_OF_MEMORY

[Tom Willcocks]

Hi all,
Some of our machines have started getting this error message quite
frequently:

Exception in thread "MTGame Renderer" javax.media.opengl.GLException:
glGetError() returned the following error codes after a call to
glTexImage2D(): GL_OUT_OF_MEMORY

http://tom01278.pastebin.com/EdJVW5YL

Does it mean the memory on the graphics card?
And is there a fix that isn't simply having fewer models/smaller
textures?


Also, I notice that that in the Client Monitor, the red Maximum line
is at 512MB, and the Blue and Green lines seldom go that high; so why
does Process Explorer say it's using up to and over a GB or memory?

If our computers have 6GB of RAM, will OWL use as much as it needs, or
will it be limited by that 512MB red line?

Thanks much,
Tom


--
Subscription settings: http://groups.google.com/group/openwonderland/subscribe?hl=en

[Jonathan Kaplan]

The GL_OUT_OF_MEMORY error is indeed due to memory issues on the graphics card.  Large textures are the most likely culprit, so reducing the texture sizes should make a difference. From the log, it doesn't look like you are moving around the world too much. If it does happen after using placemarks or portals to travel around the world, then another possibility would be that textures aren't unloading properly.

Java typically places its own limits on the amount of memory the VM can use, independent of the operating system. You can configure the maximum available to the Wonderland client by modifying wonderland/web/webstart/web/Wonderland.jnlp.  There is a line that looks like this:

<j2se version="1.6+" initial-heap-size="128m" max-heap-size="512m" java-vm-args=""/>

You can tune the maximum heap size by changing the value of "max-heap-size".  For more information on Java memory tuning, see:

http://java.sun.com/performance/reference/whitepapers/tuning.html

[Tom Willcocks]

Thanks,
I'll have a go at tuning the heap size.

Hmm, all our textures are 512x512, including the ones in the evolver files*.
It seems to be that the error occurs when moving from place to place.  Especially when you start running back before everything is unloaded properly.  Could that be the problem?

Thanks,
Tom



* The EVM files we got back from Evolver had six 2k images in them!  Had to go in with photoshop and downsize them. :/

[Tom Willcocks]

Is the Total Size in Client Monitor > JME Buffers the amount of memory being used? 
Also what are the Buffer Counts on that page? 
Could they help us to tune our world and get rid of these crashes?
Thanks again,
Tom

[Tom Willcocks]

Tried a couple of different heap sizes, but only managed to make it less stable.
Increasing heap size seems to make it much more likely to run out of graphics memory. (And decreasing it makes it more likely to run out of normal memory, obviously)

Cheers,
Tom

[Jonathan Kaplan]

The fact that increasing the heap makes it run out of memory more often is an interesting result. I assume that means that data on the graphics card is being left there until Java garbage collection decides to clean it up (which would be less often if the heap is bigger).  There are lots and lots of knobs to tune in the garbage collector:

http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html

If you have a chance to try out some of the more interesting tunings (like changing the GC algorithm) and report back, that would be helpful.

[Jonathan Kaplan]

Also, a few more subjective tests would be interesting too. Is the problem related to the geometry in world, or the avatars? What if you switch from evolver avatars to cartoon avatars?

[Tom Willcocks]

Hi,
What about "java.lang.OutOfMemoryError: Direct buffer memory"?
If google serves me correctly, direct buffer memory is not set from max-heap-size, but using -XX:MaxDirectMemorySize=256M .
However there seems to be no way to use that argument with web start...

Or am I barking up the wrong tree here?
Thanks,
Tom

[Jonathan Kaplan]

You are right about the direct buffer memory (which is used extensively by JME as well as the Darkstar IO system).  Fortunately, it looks like in Java 6 they added the ability to configure MaxDirectMemorySize via JNLP, according to this doc:

http://java.sun.com/javase/6/docs/technotes/guides/javaws/developersguide/syntax.html#resources

You should be able to modify Wonderland.jnlp to add:

<resources>

    <j2se version="1.6+" initial-heap-size="128m" max-heap-size="512m" java-vm-args="-XX:MaxDirectMemorySize=256M"/>

    ...

[Tom Willcocks]

Ah, thanks, that's good news.
I'll give that a try at some point, and see if I can get rid of some outofmemoryexceptions :)
Cheers,
Tom

[jagwire]

After switching from an on board intel card to a much new NVidia card, we ran into the following stack traces:

WARNING 1:32:54 PM org.jdesktop.mtgame.Renderer processCommitList
MTGame: Exception Caught in renderer commit: javax.media.opengl.GLException: glGetError() returned the following error codes after a call to glTexSubImage2D(): GL_OUT_OF_MEMORY 
javax.media.opengl.GLException: glGetError() returned the following error codes after a call to glTexSubImage2D(): GL_OUT_OF_MEMORY 
	at javax.media.opengl.DebugGL.checkGLGetError(DebugGL.java:12715)
	at javax.media.opengl.DebugGL.glTexSubImage2D(DebugGL.java:9136)
	at com.jme.renderer.jogl.JOGLRenderer.updateTextureSubImage(JOGLRenderer.java:1986)
	at com.jmex.awt.swingui.JOGLImageGraphics.update(JOGLImageGraphics.java:186)
	at org.jdesktop.wonderland.modules.appbase.client.DrawingSurfaceImageGraphics$UpdateProcessor.commit(DrawingSurfaceImageGraphics.java:338)
	at org.jdesktop.mtgame.Renderer.processCommitList(Renderer.java:1399)
	at org.jdesktop.mtgame.Renderer.run(Renderer.java:954)
WARNING 1:32:54 PM org.jdesktop.mtgame.Renderer processCommitList
MTGame: Exception Caught in renderer commit: javax.media.opengl.GLException: glGetError() returned the following error codes after a call to glTexSubImage2D(): GL_OUT_OF_MEMORY 
javax.media.opengl.GLException: glGetError() returned the following error codes after a call to glTexSubImage2D(): GL_OUT_OF_MEMORY 
	at javax.media.opengl.DebugGL.checkGLGetError(DebugGL.java:12715)
	at javax.media.opengl.DebugGL.glTexSubImage2D(DebugGL.java:9136)
	at com.jme.renderer.jogl.JOGLRenderer.updateTextureSubImage(JOGLRenderer.java:1986)
	at com.jmex.awt.swingui.JOGLImageGraphics.update(JOGLImageGraphics.java:186)
	at org.jdesktop.wonderland.modules.appbase.client.DrawingSurfaceImageGraphics$UpdateProcessor.commit(DrawingSurfaceImageGraphics.java:338)
	at org.jdesktop.mtgame.Renderer.processCommitList(Renderer.java:1399)
	at org.jdesktop.mtgame.Renderer.run(Renderer.java:954)
WARNING 1:32:54 PM org.jdesktop.mtgame.Renderer processCommitList
MTGame: Exception Caught in renderer commit: javax.media.opengl.GLException: glGetError() returned the following error codes after a call to glTexSubImage2D(): GL_OUT_OF_MEMORY 
javax.media.opengl.GLException: glGetError() returned the following error codes after a call to glTexSubImage2D(): GL_OUT_OF_MEMORY 
	at javax.media.opengl.DebugGL.checkGLGetError(DebugGL.java:12715)
	at javax.media.opengl.DebugGL.glTexSubImage2D(DebugGL.java:9136)
	at com.jme.renderer.jogl.JOGLRenderer.updateTextureSubImage(JOGLRenderer.java:1986)
	at com.jmex.awt.swingui.JOGLImageGraphics.update(JOGLImageGraphics.java:186)
	at org.jdesktop.wonderland.modules.appbase.client.DrawingSurfaceImageGraphics$UpdateProcessor.commit(DrawingSurfaceImageGraphics.java:338)
	at org.jdesktop.mtgame.Renderer.processCommitList(Renderer.java:1399)
	at org.jdesktop.mtgame.Renderer.run(Renderer.java:954)
WARNING 1:32:54 PM org.jdesktop.wonderland.client.cell.CellCacheBasicImpl$CellStatusChanger run
Exception thrown in Cell.setStatus unable to create new native thread
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
	at java.util.concurrent.AbstractExecutorService.submit(Unknown Source)
	at org.jdesktop.wonderland.modules.sharedstate.client.SharedMapImpl.waitForInit(SharedMapImpl.java:432)
	at org.jdesktop.wonderland.modules.sharedstate.client.SharedStateComponent.get(SharedStateComponent.java:125)
	at org.jdesktop.wonderland.modules.sharedstate.client.SharedStateComponent.get(SharedStateComponent.java:67)
	at org.jdesktop.wonderland.modules.spaces.client.SharedStateCell.setStatus(SharedStateCell.java:54)
	at org.jdesktop.wonderland.modules.spaces.client.SpacesCell.setStatus(SpacesCell.java:106)
	at org.jdesktop.wonderland.client.cell.CellCacheBasicImpl.setCellStatus(CellCacheBasicImpl.java:448)
	at org.jdesktop.wonderland.client.cell.CellCacheBasicImpl.access$700(CellCacheBasicImpl.java:80)
	at org.jdesktop.wonderland.client.cell.CellCacheBasicImpl$CellStatusChanger.run(CellCacheBasicImpl.java:962)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
WARNING 1:32:54 PM org.jdesktop.wonderland.client.jme.JmeClientMain$1 uncaughtException
Uncaught exception
javax.media.opengl.GLException: glGetError() returned the following error codes after a call to glClear(): GL_OUT_OF_MEMORY 
	at javax.media.opengl.DebugGL.checkGLGetError(DebugGL.java:12715)
	at javax.media.opengl.DebugGL.glClear(DebugGL.java:466)
	at com.jme.renderer.jogl.JOGLRenderer.clearBuffers(JOGLRenderer.java:502)
	at org.jdesktop.mtgame.OnscreenRenderBuffer.clear(OnscreenRenderBuffer.java:161)
	at org.jdesktop.mtgame.DefaultBufferController.renderBuffer(DefaultBufferController.java:177)
	at org.jdesktop.mtgame.DefaultBufferController.renderScene(DefaultBufferController.java:167)
	at org.jdesktop.mtgame.Renderer.run(Renderer.java:929)
WARNING 1:33:22 PM org.jdesktop.wonderland.client.jme.JmeClientMain$1 uncaughtException
Uncaught exception
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Unknown Source)
	at org.jdesktop.wonderland.modules.avatarbase.client.jme.cellrenderer.GestureHUD$4.run(GestureHUD.java:175)

It's odd that switching to a better card yields these results.

-jW

[Jos Flores]

Long shot here Ryan, but any chance the card is not working correctly?

cheers,
José

[Maggie Leber]

On Wed, Oct 17, 2012 at 2:49 PM, Jos Flores <josmas...@gmail.com> wrote:
> Long shot here Ryan, but any chance the card is not working correctly?

Or is the driver perhaps a bit downlevel? The quality of nVidia
drivers can be...variable.

[Roland Sassen]

Ryan maybe you can disable your intel on board card in the bios, or with a jumper,
disable the intel driver, disable shared memory( video and RAM)
Roland

No virus found in this message.
Checked by AVG - www.avg.com
Version: 2012.0.2221 / Virus Database: 2441/5336 - Release Date: 10/16/12

[jagwire]

It's definitely possible the card is working incorrectly. Though, it seems to load the operating system and get into Wonderland "okay". I was told the driver is already up-to-date from NVidia though the jury is still out on the on-board card.

-jW

[Maggie Leber]

History repeats itslef: http://www.java.net/node/700301

[Ric Moore]

On 10/17/2012 03:06 PM, Roland Sassen wrote:
> Ryan maybe you can disable your intel on board card in the bios, or with
> a jumper,
> disable the intel driver, disable shared memory( video and RAM)
> Roland

That was my first thought, make sure that all traces of the original
onboard chip are gone in the bios. Make sure you have your nVidia driver
correctly installed. If running Linux you will have to use the nVidia X
Server Settings tool and save your corrected /etc/X11/xorg.conf file.
You might have to delete your ~/.java directory then reboot. Ric



--
My father, Victor Moore (Vic) used to say:
"There are two Great Sins in the world...
..the Sin of Ignorance, and the Sin of Stupidity.
Only the former may be overcome." R.I.P. Dad.
http://linuxcounter.net/user/44256.html

[jagwire]

I guess I'm a bit confused. The worlds ran relatively "okay". That is to say, we didn't see the GL_OUT_OF_MEMORY error when using the on board card. If for whatever reason it was still being used, why would that change now?

-jW

[Ric Moore]

On 10/19/2012 08:39 AM, jagwire wrote:
> I guess I'm a bit confused. The worlds ran relatively "okay". That is to
> say, we didn't see the GL_OUT_OF_MEMORY error when using the on board
> card. If for whatever reason it was still being used, why would that
> change now?

Usually the onboard chipsets use system ram as Video ram. If the onboard
ship is being recognized while the added video card announces it has
it's own memory, you might have a conflict regarding just how much
memory is actually available. So, you have to disable the onboard video
completely, remove all reference to it in config files, install the
nVidia setup program and allow it to write to your /etc/X11/xorg.conf
file the correct configuration. You might wish to read-up on your
motherboard documentation for video-setup bios settings. Ric