Memory leak on my server?

[Eric]

So I have a test server running my custom "Humanities World" map going. I am testing technical limitations etc. before I go "live" on a larger scale, probably next year. I had some recent problems with severe lag on the server, and the excellent combined efforts of the programming team and some fellow members of this forum led to an improvement in the Edu code that eliminated most all Edu lag - for that I am very grateful!

But a new problem has come to my attention -- a memory leak. My server constantly rises in its memory usage, and after less than 24 hours of running has far exceeded the 1GB server RAM limits and causing terrible lag. I have to reset the server every night to try and avoid the lag. 

I have talked to the hosting company and they said they think it's a memory leak. But another user on here has a server running as well, and no memory problems. He never needs to reset his server. He is the one that provided me with the necessary command lines and settings to get the server running  in the first place, so our servers should be identical. Why am I having massive RAM issues while he is not?

So I thought maybe it could be the map. Is it possible for a map to cause a memory leak? The map is only 52MB big - it's not some crazy 2GB map or anything. Could this be another problem with the Edu code that could be fixed? 

Here is the link to my map again if you need it: http://dl.dropbox.com/u/53773950/Humanities2012.zip

That link is a little outdated, but the "memory leak" problem has been going since I first set up my server. Please take a look, and hopefully this will lead to yet another improvement in the Edu code. Thanks!

[Michael Rose]

Eric, I'm off on holiday tomorrow but (if I have time tonight) I'll
upload your world to my server and see what the performance is like.

Anywhere in particular I should explore?
m.

[EduElfie]

I have never had that with any of my maps, even more so after the recent fix that you mention. I too will load up your world over the weekend and see if it gets to the point where the RAM usage goes sky high. Update to come later.
Elfie.

[Joel Levin]

I've had a temporary map running on a linux server for over two weeks now with no memory problems.  I will load your map onto the server later today and see how it goes.

I'm glad we have a big, complex map such as your to do some "stress testing" of MinecraftEdu.  Last time, as you said, it led to a significant performance increase.

~Joel

[Eric]

Thanks so much, everyone! Great to have such an awesome team to look at this stuff! 

I have not noticed any particular areas that could be causing it. It would be pretty tough to pinpoint, as students can run off in any direction. The memory usage just goes up and up and up, and never down. I never have more than 8 or so students on at a time, and it's usually just 2 or 3. I have tried using the -Xincgc command, and that seems to keep the memory down, but for some reason adds tons of lag for me, so it's a bit self-defeating.

Keep me updated!

[Michael Rose]

Eric, as I mentioned the memory has been fine on my remote setup, I'll
take a look at your map tonight but I have a feeling it'll be the
server not the map.

It's my understanding that the memory should go up and up, but only to
the point it's limited to (in your setting and the memory variable you
put into the run command. Then it should dump out old blocks to
replace new. I'd try also changing the server-distance parameter.

m.

[Joel Levin]

Okay, I have a server running on my linux box with your map.  It's currently purring nicely at ~2% CPU utilization and 443megs of RAM.  By comparison, I have a Bukkit server running on the same machine that's at 13% and 1.3gigs.  I'll keep an eye on things today.

For reference here's how I ran it:

java -Djava.ext.dirs=../lib -Xmx1024M -Xms1024M -jar minecraftedu_server.jar nogui

If anyone is using the 0.981 beta version (1.2.5 compatible), feel free to hop on and tax the server a little:

mc.cgps.org:25568

Oh, but another thought I had (and this goes for everyone)... Eric, are you playing with the personal speed setting?  If you put yourself to 100% and outrun the servers chunk generation, you WILL crash the server.  We are actually thinking of lowering the maximum possible speed to prevent this from happening.

~Joel

[steve isaacs]

Trying to connect to server and getting an Internal exception: Java.net.SocketException: Connection reset error.  I'm sure it's something on my end.  Any suggestions?

Steve

--

 

Steven Isaacs

follow me on twitter: @mr_isaacs

http://wamstechtips.blogspot.com
steve...@gmail.com
sis...@bernardsboe.com

[Eric]

Big improvements! Not going to get too excited and call this a fix yet, but I am now running at 312megs with 3 players on, whereas with my old startup lines it would jump to 840megs with just me on it after a few minutes! Here's my new startup line:

screen java -Xmx768M -XX:ParallelGCThreads=8 -Xincgc -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSIncrementalPacing -XX:+AggressiveOpts -XX:+CMSParallelRemarkEnabled -XX:+DisableExplicitGC -XX:MaxGCPauseMillis=500 -XX:SurvivorRatio=16 -XX:TargetSurvivorRatio=90 -XX:+UseAdaptiveGCBoundary -XX:-UseGCOverheadLimit -Xnoclassgc -XX:UseSSE=3 -XX:PermSize=128m -XX:LargePageSizeInBytes=4m -Djava.ext.dirs=../lib -jar minecraftedu_server.jar nogui

Now, I have no idea what half of that means, but I got it from here: http://forums.bukkit.org/threads/tutorial-reducing-lag-and-possibly-fixing-corrupt-worlds.20911/

Seems to be working very well.

Thoughts?

[Eric]

PS - I have turned movement speed up and it's never been a problem. Yes, I've gotten ahead of the generation, but then I just wait for a while and it loads around me. It's never crashed the server. And isn't moving at that speed no different than using a teleporter block, when it comes to loading new land quickly?

[Joel Levin]

Wow!  That's one beast of a command line!

As for getting ahead of chunk generation, it has certainly been a cause for server crashes for me with pre-.981 versions of the server.  But things have been better now.

~Joel

[Eric]

Here's the latest version of my world, if you're interested (updated to Anvil format for 1.2.5):

http://dl.dropbox.com/u/53773950/Humanities2012.zip

And here's the latest version of the World Map to help you find your way around:

http://dl.dropbox.com/u/53773950/World%20of%20Humanities%20Map%20-%20May%202012.jpg

[steve isaacs]

I think my failure to connect error was a firewall issue.  Slowly learning :)

Steve

[Joel Levin]

Steve, the server is running the latest beta version of MinecraftEdu which is why you probably had problems connecting.  But anyone on this list should let us know if you'd like access to it so you can try the new features.  The big items are 1.2.5 compatibility, WorldEdit tools, and significantly improved server performance.

Eric, the server ran all day without problems.  The one area that I'm suspicious of is the giant errupting volcano.  My memory usage doubled and stayed there when I went near it.  How are you getting all that lava to stay suspended in the air?  Is it possible that the server is trying to figure out where everything should flow constantly?  I know I've accidentaly broken maps after pasting large areas of water.

Anyway, I am now running the latest version of the map on my server.  Let me tell you yet again that the map is an absolute work of art and an inspiration of what's possible when teaching with Minecraft.  If I find the time I would love to do a video tour for my youtube channel, if that's okay with you.

Again, any beta testers should feel free to hop on and check it out at:

mc.cgps.org:25568

Oh, and the teacher password is "asdasd".  Be sure to use the teleport menu and see the sights!

~Joel

[Michael Rose]

On 4 May 2012 20:19, Joel Levin <joel....@gmail.com> wrote:

 

Let me tell you yet again that the map is an absolute work of art and an inspiration of what's possible when teaching with Minecraft.

Absolutely. Well said! Inspiration for us all.

[Eric]

Thank you so much Joel. Your approval is a gift. I have dedicated 100+ hours, but I hope this time will guide other teachers and classrooms in the excitement and wonder of learning. All I have ever done is imagine myself as a middle schooler - what would be cool at that age? Of course, this is the same tactic I have used as a teacher. Thank  you for providing the opportunity;. I am glad I can provide some service back to you.

[Eric]

I'm sorry I didn't answer your question. Yes, it would be an honor to be featured on your channel. 

And also --- if you have any interest, let's meet up this summer! I'm finishing my Master's, but would be free in August. I'll be back in the States, and flying anywhere in the US shouldn't be a problem. New York? Where would you like to meet? I would love to talk "futuristic shop"! 

[Eric]

More weirdness here - there still seems to be extreme lag, but only for me as a client when I'm connected! I have a Putty window open looking at server chat, and when I type something in, it almost immediately appears on the server log, but takes forever to appear in my actual game client. Other students are seeing it almost immediately too, because they will often have replied before my first chat line ever shows up in my client. Could Edu just be failing to send out info to some clients, making it seem like the whole server is lagging when it's really not? You would think I just have a bad internet connection - but I don't (3/5 bars in the player list)... and how would it show up immediately in my server window and not in my client??

Joel's server feels the same way to me, very laggy. Could there be something wrong with my client? But why do I get this lag only sometimes, and not after a server has been restarted? Weird things going on here. 

Good thing is, it seems the students aren't getting the lag I am, at least not to my knowledge.

[Joel Levin]

Eric, I noticed that you logged on to the server and said you still had performance problems.  I just tried and everything was quite fast.  Not sure what to think.

Have you tried connecting from different computers?  Maybe it's you... but I have no idea how/why.

~Joel

[Eric]

Yes, and it's not just different computers, but different times I log in. Sometimes when I log into my own server from my own computer, it is quite fast. It really must be something with my internet connection, I guess. But it's strange how the Putty window can update so fast while the Edu client can take 10+ seconds to show chat lines sometimes. I think disconnecting/reconnecting is the best thing to try for now. 

[EduElfie]

This might be very stupid, but are you sure you are running the latest jar on your machine, perhaps you are running a slightly older beta jar as I know Kulttuuri updated the minecraft.jar as well as the server jar in the latest beta, also did you copy the worldedit jar over as if it cannot find that you might run into trouble. Just a thought.
Elfie.

[Eric]

The beta zip doesn't have a minecraft.jar in it, it only affects the MinecraftEdu folder. I have everything updated. If you try to connect with an older version, it says Java error. Thanks for the idea, but that wouldn't explain why SOMETIMES it's fast and other times very laggy. I think it's just a Kuwait internet connection thing. So far my huge command line seems to have fixed the memory issues. RAM is holding at about 420megs now after 3+ days of running. Much better than the 1300megs after a few hours. I highly recommend others try launching with my command line and see how it works for them. Even if your server performance is already good (and I can't for the life of me figure out why my server, on default command lines, was using so much more ram than everyone else), it may make your performance even better.

[Warren Bezuidenhout]

Hi Guys

Where can I go to get access to the newest Beta?

Thanks

Warren

[Eric]

OK I think I have it figured out. With those command lines, the server has basically no lag anymore. Celebration!

But every so often, when I connect, I have terrible lag. Chat takes 30 seconds to appear, land stops loading for minutes at a time, etc. Again, by monitoring on Putty, I can see that when I enter chat it goes to the server almost immediately, but it takes forever for data to be sent back to me as a client.

I can then disconnect, reconnect, and the lag is gone again. It seems like sometimes when I connect, I just do not establish a good connection, and I don't receive info from the server for a long time. I don't know if this is something to do with my internet connection, it might, but is it possible that Edu sometimes does not establish a good connection with a client and doesn't prioritize to send them data? I haven't really heard of this problem before with vanilla Minecraft, so I'm wondering if it's a weird Edu problem. 

It seems to happen about 1/5 times I connect. It has always been fixed by disconnecting and reconnecting (knock on wood). I haven't had students complain about this, so again, there are chances it is just something weird with my own internet. But why are random connections stuck as laggy and bad for receiving data, but send data just fine, and can be fixed by reconnecting? Strange stuff.

[steve isaacs]

What is the address for this server?  I'd like to join people on it if possible.

Steve

[Eric]

Sorry, my server is for students only. Joel posted his test server that you could join in an earlier message.