-mem 8000m doesn't pre-reserve system memory?

[Scott Heavner]

I'm not seeing the behavior I expect when I pass -mem 8000m to the play production startup script in 2.3.7.

The JVM starts with -Xmx 8000 and -Xms 8000, but the memory doesn't look like it is used on ubuntu 14.04.    I see something that looks like it grabbed 2G instead of 8G.

Some googling yielded a suggestion to include -XX:+AlwaysPreTouch which does create the behavior I expect.  I see top and free both reporting that half my machine's memory has been consumed by java.

So is -Xms really going out and reserving system memory at startup?   Is this just some linux artifact that doesn't show the memory as used even though it's reserved.   Or some open-jdk specific behavior?   The big question: can some other process go out and get the memory reserved for this process?   If so, should play -mem include the AlwaysPreTouch flag?

play 2.3.7 | ubuntu 14.04 | openjdk-6-64bit

java version "1.6.0_34"
OpenJDK Runtime Environment (IcedTea6 1.13.6) (6b34-1.13.6-1~deb7u1)
OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode)

[Igmar Palsenberg]

 

I'm not seeing the behavior I expect when I pass -mem 8000m to the play production startup script in 2.3.7.

The JVM starts with -Xmx 8000 and -Xms 8000, but the memory doesn't look like it is used on ubuntu 14.04.    I see something that looks like it grabbed 2G instead of 8G.

Some googling yielded a suggestion to include -XX:+AlwaysPreTouch which does create the behavior I expect.  I see top and free both reporting that half my machine's memory has been consumed by java.

So is -Xms really going out and reserving system memory at startup?   Is this just some linux artifact that doesn't show the memory as used even though it's reserved.   Or some open-jdk specific behavior?   The big question: can some other process go out and get the memory reserved for this process?   If so, should play -mem include the AlwaysPreTouch flag?

Memory only get's really allocated by the kernel if written to. So, you can mmap() a 16 GB area, but as long as you don't write it, it should be fine. This is callled the overcommit flag. Really reserving the memory is a very bad idea in general : The kernel does a far better job than any application will do. The only exception is pinning secure pages in memory to prevent them from being swapped out.

If you do touch it all, the next allocation could end up that you application being killed by the OOM killed. If that happens, it's end of story.

In other word : Don't do this.

Igmar

[James Roper]

I'd disagree here.  The JVM won't garbage collect until the heap is somewhat full.  So inevitably, the JVM is going to write to all of this memory before it does it's first GC (or rather its first full GC).  The JVM makes no effort to keep memory usage low, you tell it to take 8G, it takes the full 8G, and uses that full 8G to the max, it never returns it to the OS, and makes no attempt to confine its usage of it to only a small part.  So, by using the always pretouch flag, you ensure that what will inevitably happen before your app first full GCs, that is, all memory requested being allocated, doesn't cause a problem.  That means, if you have granted the JVM more memory than the system can allocate, you get an out of memory error immediately when the app first starts, rather than inevitably getting one a few hours later, which is much better in my opinion.

Programs that manage their own memory, like the JVM, don't play very well with OS over committing.  The only time my blog ever made it to the front page of hacker news, a misconfiguration in the reverse proxy on my machine caused the Linux kernel to decide to kill my JVM instance that was running the blog (since it was taking the most memory, but of course that's the worst possible thing it could do) to make room for more reverse proxy processes (which of course was pointless, can't proxy to a killed app server).  The best thing to do is to ensure that your JVM app is the only thing running on that machine.

Igmar

--
You received this message because you are subscribed to the Google Groups "play-framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to play-framewor...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

James Roper
Software Engineer

Typesafe – Build reactive apps!
Twitter: @jroper

[Igmar Palsenberg]

I'd disagree here.

On which part ?

 

  The JVM won't garbage collect until the heap is somewhat full.  So inevitably, the JVM is going to write to all of this memory before it does it's first GC (or rather its first full GC).  The JVM makes no effort to keep memory usage low, you tell it to take 8G, it takes the full 8G, and uses that full 8G to the max, it never returns it to the OS, and makes no attempt to confine its usage of it to only a small part.  So, by using the always pretouch flag, you ensure that what will inevitably happen before your app first full GCs, that is, all memory requested being allocated, doesn't cause a problem.  That means, if you have granted the JVM more memory than the system can allocate, you get an out of memory error immediately when the app first starts, rather than inevitably getting one a few hours later, which is much better in my opinion.

Correct, but if the system is low on memory, and starts somehow requesting above that 8GB (what I've seen), the OOM killer kicks in and your app goes down. If you're doing a pre-alloc, you should be sure that the kernel's overcommit flag is disabled. If it isn't, you're in for some nasty surprises. Oh, and also make sure you don't have any swap enabled : That's a huge performance killer. Memory is cheap, disk access isn't.

 

Programs that manage their own memory, like the JVM, don't play very well with OS over committing.  The only time my blog ever made it to the front page of hacker news, a misconfiguration in the reverse proxy on my machine caused the Linux kernel to decide to kill my JVM instance that was running the blog (since it was taking the most memory, but of course that's the worst possible thing it could do) to make room for more reverse proxy processes (which of course was pointless, can't proxy to a killed app server).  The best thing to do is to ensure that your JVM app is the only thing running on that machine.

I only have a jvm, nginx (for some static files / reverse proxy), and a rsyslogd running. That works well.

Igmar

[James Roper]

On 23 April 2015 at 17:49, Igmar Palsenberg <ig...@palsenberg.com> wrote:

I'd disagree here.

On which part ?

 

  The JVM won't garbage collect until the heap is somewhat full.  So inevitably, the JVM is going to write to all of this memory before it does it's first GC (or rather its first full GC).  The JVM makes no effort to keep memory usage low, you tell it to take 8G, it takes the full 8G, and uses that full 8G to the max, it never returns it to the OS, and makes no attempt to confine its usage of it to only a small part.  So, by using the always pretouch flag, you ensure that what will inevitably happen before your app first full GCs, that is, all memory requested being allocated, doesn't cause a problem.  That means, if you have granted the JVM more memory than the system can allocate, you get an out of memory error immediately when the app first starts, rather than inevitably getting one a few hours later, which is much better in my opinion.

Correct, but if the system is low on memory, and starts somehow requesting above that 8GB (what I've seen), the OOM killer kicks in and your app goes down. If you're doing a pre-alloc, you should be sure that the kernel's overcommit flag is disabled. If it isn't, you're in for some nasty surprises. Oh, and also make sure you don't have any swap enabled : That's a huge performance killer. Memory is cheap, disk access isn't.

But my point is, that's going to happen regardless of whether you pre allocate or not - you tell the JVM it can take 8GB, and it will take and use the full 8GB, it's just that without prealloc enabled, it will just take a little longer to take it (depending on the load on your app, this could be anywhere from a few minutes to a few days before the first full GC is done).

Let me put it this way, lets say your app with no load takes 100MB of heap, and then each request creates 100KB of garbage.  You allocate 8GB of heap.  After the app has served 1000 requests, it's now written to 200MB of memory.  If it did a GC, that would go back down to 100MB.  But it doesn't do that, because the JVM only GCs when the heap is full.  After 10000 requests, it's now at 1100MB of memory that it's written to, of which 1000MB is garbage.  It still doesn't GC.  After 79000 requests, the JVM has now written to the full amount of heap, and so the kernel has allocated the whole lot.  ONLY THEN does the JVM say "oh, I've used my whole heap, I better GC now".  At this point, the memory is all allocated by the kernel, you're in exactly the same situation as you would be if you used prealloc, it's just that it took you a little longer to get there.

Now, in practice, due to generational GC, it's more complex than that, the JVM will do a partial GC every time its eden space is full.  Each time it does that, a little bit of memory that is not yet but soon to be garbage makes its way into the old space.  Over time, that builds up, until you're in the same situation as you would have been if you prealloced all the memory, then a full GC is done.  In practice, the systems I administer tend to do their first full GC after a few hours.  In that situation, the only effect prealloc has is it changes the time that the OS allocates the full memory to a few hours earlier, which means some problems I find out about immediately, rather than after I've gone home.

All this happens regardless of whether the system has enough memory to support it or not.  You give 8GB of heap to a JVM on a system that only has 6GB of memory, the question isn't whether the JVM will write to the full 8GB of memory forcing some of the heap into swap, that is guaranteed, the only question is how long it will take to do so.

So, the JVM writing to all the memory, and therefore causing the OS to allocate it, is not something that only happens if the JVM needs it, it's something that is *inevitable*, regardless of whether you specify prealloc or not.  prealloc only changes when that's done.  Not using prealloc has *no* impact on whether your process is going to go to swap, or whether it's going to be killed by the linux OOM killer.  But it does help to find some types of problems sooner rather than later.

Programs that manage their own memory, like the JVM, don't play very well with OS over committing.  The only time my blog ever made it to the front page of hacker news, a misconfiguration in the reverse proxy on my machine caused the Linux kernel to decide to kill my JVM instance that was running the blog (since it was taking the most memory, but of course that's the worst possible thing it could do) to make room for more reverse proxy processes (which of course was pointless, can't proxy to a killed app server).  The best thing to do is to ensure that your JVM app is the only thing running on that machine.

I only have a jvm, nginx (for some static files / reverse proxy), and a rsyslogd running. That works well.

Igmar

--
You received this message because you are subscribed to the Google Groups "play-framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to play-framewor...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nathan Williams]

On Thursday, April 23, 2015 at 6:27:17 PM UTC-6, James Roper wrote:

On 23 April 2015 at 17:49, Igmar Palsenberg <ig...@palsenberg.com> wrote:

I'd disagree here.

On which part ?

 

  The JVM won't garbage collect until the heap is somewhat full.  So inevitably, the JVM is going to write to all of this memory before it does it's first GC (or rather its first full GC).  The JVM makes no effort to keep memory usage low, you tell it to take 8G, it takes the full 8G, and uses that full 8G to the max, it never returns it to the OS, and makes no attempt to confine its usage of it to only a small part.  So, by using the always pretouch flag, you ensure that what will inevitably happen before your app first full GCs, that is, all memory requested being allocated, doesn't cause a problem.  That means, if you have granted the JVM more memory than the system can allocate, you get an out of memory error immediately when the app first starts, rather than inevitably getting one a few hours later, which is much better in my opinion.

Correct, but if the system is low on memory, and starts somehow requesting above that 8GB (what I've seen), the OOM killer kicks in and your app goes down. If you're doing a pre-alloc, you should be sure that the kernel's overcommit flag is disabled. If it isn't, you're in for some nasty surprises. Oh, and also make sure you don't have any swap enabled : That's a huge performance killer. Memory is cheap, disk access isn't.

But my point is, that's going to happen regardless of whether you pre allocate or not - you tell the JVM it can take 8GB, and it will take and use the full 8GB, it's just that without prealloc enabled, it will just take a little longer to take it (depending on the load on your app, this could be anywhere from a few minutes to a few days before the first full GC is done).

Not true.  If you pre-allocate you dirty a bunch of pages, causing the memory to be allocated by the OS.  Then, if your application doesn't actually use those pages, the OS (for any OS that uses a modern memory model with overcommit, which is every OS your familiar with) will page them back to disk, and page them back in whenever they are referenced again.  At this point, if there isn't enough available memory to bring them back in, the OOM killer will wipe out your app, regardless of the fact that at one point in time the application and OS had the amount of memory requested.

 

Let me put it this way, lets say your app with no load takes 100MB of heap, and then each request creates 100KB of garbage.  You allocate 8GB of heap.  After the app has served 1000 requests, it's now written to 200MB of memory.  If it did a GC, that would go back down to 100MB.  But it doesn't do that, because the JVM only GCs when the heap is full.

That's not true either.  GC happens *ALL* the time, not just when the heap hits the limit.  In the early Java 1.1 days that was the case, but it hasn't been that way since you were a teenager. :)

  After 10000 requests, it's now at 1100MB of memory that it's written to, of which 1000MB is garbage.  It still doesn't GC.  After 79000 requests, the JVM has now written to the full amount of heap, and so the kernel has allocated the whole lot.  ONLY THEN does the JVM say "oh, I've used my whole heap, I better GC now".  At this point, the memory is all allocated by the kernel, you're in exactly the same situation as you would be if you used prealloc, it's just that it took you a little longer to get there.

Now, in practice, due to generational GC, it's more complex than that, the JVM will do a partial GC every time its eden space is full. 

Now we're talking realistic behavior, which makes most of the above statements moot.

 

Each time it does that, a little bit of memory that is not yet but soon to be garbage makes its way into the old space.  Over time, that builds up, until you're in the same situation as you would have been if you prealloced all the memory, then a full GC is done.  In practice, the systems I administer tend to do their first full GC after a few hours.  In that situation, the only effect prealloc has is it changes the time that the OS allocates the full memory to a few hours earlier, which means some problems I find out about immediately, rather than after I've gone home.

Your systems must have no other processes running on them except for your application, since that would only explain never having to compete for memory resources.  That is not the case for many/most applications and systems.  In my experience, most organizations can't afford to dedicate any entire OS image for each application, so we must share resources across multiple JVMs, which means that we penalize the applications and OS by pre-allocating the memory ahead of time, causing un-necessary paging (and sometime swapping) of unused pages out by pre-allocating memory.

 

All this happens regardless of whether the system has enough memory to support it or not.  You give 8GB of heap to a JVM on a system that only has 6GB of memory, the question isn't whether the JVM will write to the full 8GB of memory forcing some of the heap into swap, that is guaranteed, the only question is how long it will take to do so.

 

And the point of this is it makes errors happen sooner rather than later?  I don't see any way pre-allocation of memory would cause errors to occur sooner vs. later.  Unless you are trying to track down JVM and/or OS bugs, how memory is allocated/page shouldn't affect you, unless your application needs 8GB of memory just to run, and in that case, it'll swap itself to death regardless of whether you pre-allocate or wait for it to hit the 8GB memory limit after program startup.

 

So, the JVM writing to all the memory, and therefore causing the OS to allocate it, is not something that only happens if the JVM needs it, it's something that is *inevitable*, regardless of whether you specify prealloc or not.  prealloc only changes when that's done.  Not using prealloc has *no* impact on whether your process is going to go to swap, or whether it's going to be killed by the linux OOM killer.  But it does help to find some types of problems sooner rather than later.

Programs that manage their own memory, like the JVM, don't play very well with OS over committing.

I call bull on that.  OS developers take into account programs that manage their memory, and spend time tuning for this very behavior.

Nate - OS dude in a former life...