performance questions

[Dan]

I'm working on a game using Meteor and I've run into a few problems.  Any advice would be appreciated.  The game is http://dominusgame.net

Can't load more than around 600 hexes at once.  Each hex of the map is in the database.  I figure out what hexes the player can see and publish those to the client.  The more hexes I send to the client the longer it takes to load.  I've found that going over 600 takes too long.  I'd like to be able to load hexes that are offscreen so that when you scroll around you don't see them update or have an option to zoom out and see more of the map but it's not possible.

Map loading freezes the client.  While hexes are updating the client's computer freezes.  Updating the hexes takes a few seconds so scrolling around the map is very painful.  How can I get it to not freeze the client's computer?  Ideally I would like to load twice as many hexes as the client can see and have them load in the background so that the client never sees them update.

Server performance.  I have a server that clients can't connect to that runs jobs.  I'm using oplog.  One job takes about 20 seconds to finish.  While this job is running all servers are close to 100% cpu.  

[Arunoda Susiripala]

Hi,

Not exactly sure what's happening. What is a Job and what's happening there? 

May be you'll find something interesting here: https://kadira.io/academy/

---

Arunoda Susiripala

I curate Meteor Weekly - Check it out!

--
You received this message because you are subscribed to the Google Groups "meteor-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to meteor-talk...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dan]

I think the main problem I need to fix is the client's browser freezing while their subscription to the hex db is updating.  If their browser didn't freeze then I could load more at once and if it takes longer that is ok because they won't notice.

The server performance I can probably work on, thanks for the link Arunoda.  The browser freezing I have no idea how to fix.

[Arunoda Susiripala]

Okay. While the subs are loading. Use chromes performance tab to do a profile and see what's happening. That might help.

You might be doing some computations or something wrong with how you use Deps.

---

Arunoda Susiripala

I curate Meteor Weekly - Check it out!

On Tue, Jun 10, 2014 at 12:25 PM, Dan <dan...@gmail.com> wrote:

I think the main problem I need to fix is the client's browser freezing while their subscription to the hex db is updating.  If their browser didn't freeze then I could load more at once and if it takes longer that is ok because they won't notice.

The server performance I can probably work on, thanks for the link Arunoda.  The browser freezing I have no idea how to fix.

--

[Gadi Cohen]

I think this could be the issue we saw in "Finding source of performance issues (Slow subscribe after sign in?)".  To quote Slava in this post:

I looked at the app briefly and I think the "freezing" is the combination of two things:

1) you get a lot of updates in a short amount of time (right after you log in) and Meteor doesn't batch these updates.

2) every updates triggers changes to minimongo's sorted query (which is written in a very naive way at this point and runs a linear number of shifts for each insert),

and it also triggers Blaze to reflect the change in the collection to the screen (which is not super expensive but well, it adds up).

Just to clarify, when you wrote "One job takes about 20 seconds to finish.  While this job is running all servers are close to 100% cpu.", I'm assuming you meant that the clients were close to 100%, right?

On the client, are you sorting the query, and if you don't sort it, does it improve performance?  Just to see if this is the problem.

[Avital Oliver]

If the issue is performance when logging in, there is a workaround: Wrap all subscriptions in a Deps.autorun that also calls Meteor.userId():

Deps.autorun(function () {

  // just register a dependency on userid. this way the subscriptions gets re-run when a user logs in or out.  

  Meteor.userId();

  Meteor.subscribe(...)

});

Then, due to new subscriptions only exposing data once the subscription is first "ready", it should have the same performance as loading the page initially.

I'm curious if this is indeed the case and if this helps. It's something we should fix by making DDP connections have one userId, so that this behavior will be what happens always when a user logs in or out.

--

PS. I genuinely *really* like receiving feedback of any kind. Leave it anonymously at http://www.admonymous.com/avital

[Dave Workman]

Does the tile template rely on a session variable(s)? I recently had a problem because of a "selected" session variable, having 300-500 items relying on the session variable caused freezing problems when new data came in, or the session variable changed. I was able to change my pagination logic to throw out data that's been scrolled out of view so there's only ever 40 on the page at once, that might not be a solution for a game map like this though.

[Stefano Diem]

On a non related question, what are you using for rendering the html5 svg? I am learning Phaser atm moment, which seems pretty cool, and also doing a hex based came (albeit more like settlers of catan).

Abraços,

Stefano Diem Benatti

phone: 55 11 9343 0994 skype:  teonimesic

--

[Dan]

Here's what I meant about server cpu.  This is from Modulus.  The bottom graph is using two servos.  The top graph is the job server.

 

 Notice that when the big job runs the other servers go to 100%.  Also notice that project requests is not that high but cpu is.  Server facts says that there are 14 sessions at the moment (that might be for one servo?) and google analytics says there are 9 people online.  The big job that spikes the cpu is looping through every worker (resource miner in the game) and giving each player resources that were mined.  It updates the users db a lot.

The map is svg Stefano, it doesn't use a library like Phaser, just plain svg.

I tried adding a {limit: 10} to the client's Hexes.find() so that the server still sent all 600 hexes but the client didn't update.  The freeze still happens.

I don't think adding a Meteor.userId() would help.  The freeze happens every time someone scrolls the map.  Scrolling the map causes the client to get 600 new hexes.  (I wish I could think of a way to only get the new hexes)  Which causes the freezing.

I went through all of the Deps.autoruns and none of them rely on the hexes db.  It does have to do some calculations for each hex it draws like figure out it's coordinates on the screen but I don't think that's the problem.  After trying {limit: 10} on the client like I mentioned above it still happens.

I am using session variables but nothing that changes often.

[Dan]

After reading Slava's post that Gadi mentioned I took out sorting the hexes on the client.  It was Hexes.find({}, {sort: {y: 1, x: 1}}) now it's just Hexes.find().  I think that helped.

[Dan]

Yep I think removing sorting helped.  The freeze is still there, you can't continually scroll around the map like google maps but it's better.

[Dave Workman]

"I am using session variables but nothing that changes often."

If your tile templates rely on it, every time you load 600 new tiles, it needs to run that helper 600 times, regardless of whether the value of the session variable is different.

[Stefano Diem]

Maybe you could add a server subscribe which receives the boundaries of the screen, such as minX, maxX, minY, maxY, and only subscribe to the hexes inside of that?

Something like:

Meteor.publish 'hexes', (minX, maxX, minY, maxY) ->

  Hexes.find({x: {$lt: maxX, $gt: minX}, y: {$lt: maxY, $gt: minY}})

And on the client you would have something like:

Deps.autorun ->

    Meteor.subscribe 'hexes', Session.get('minX'), Session.get('maxX'), Session.get('minY'), Session.get('maxY')

And a scroll event that updates those session variables?

Abraços,

Stefano Diem Benatti

phone: 55 11 9343 0994 skype:  teonimesic

On Tue, Jun 10, 2014 at 1:17 PM, Dan <dan...@gmail.com> wrote:

Yep I think removing sorting helped.  The freeze is still there, you can't continually scroll around the map like google maps but it's better.

--

[Stefano Diem]

The idea behind this, is that when you would move out of the current hexes, they wouldn't be loaded or update anymore, freeing your memory. Unless you do it, you will only add more and more hexes in memory which receives more and more events, eventually leading to a freeze.

Abraços,

Stefano Diem Benatti

phone: 55 11 9343 0994 skype:  teonimesic

[Dan]

That's how it currently works Stefano.  When you release the mouse button when dragging it subscribes to hexes with min/max x/y.  I think it's sending 600 new hexes each time that happens though? 

I'll try removing session variables tonight, thanks Dave.

[Arunoda Susiripala]

Hey Dave,

You talked about background process. Can you share us the sample code.

Without looking at it, it is pretty hard to tell the reason.

---

Arunoda Susiripala

I curate Meteor Weekly - Check it out!

--

[Arunoda Susiripala]

Dave,

If you can replicated the load locally, try running a CPU profile.

Use look: https://github.com/baryshev/look

It's very easy to use. Don't use this production unless you've a pretty big Box. Eats a lot of CPU.

---

Arunoda Susiripala

I curate Meteor Weekly - Check it out!

[Dave Workman]

@Arunoda, I believe you mean Dan not me correct?

[Arunoda Susiripala]

yeah! sry for that.

---

Arunoda Susiripala

I curate Meteor Weekly - Check it out!

[Dan]

I'll try look tonight, thanks.

[Gadi Cohen]

cc'ing Slava & Dave since they were part of the discussion on the other thread.

Just to clarify, we have a number of different issues here:

1. Client temp freeze / 100% CPU after map position changed (update of 600 records at once, subscription change)
   
2. Client temp freeze / 100% CPU while server background job runs (update <= 600 records at once, from subscription)
   
3. Server near 100% CPU while server background job runs

Right?

I guess #1 & #2 are essentially the same thing / same cause.

Regarding #3; does your query in the publish function use any sorting?  If it does, see also if removing that makes a difference (again, just to get a handle on things).

I believe (and MDG will correct me if I'm wrong) that this relates back to that the fact that each update is being acted on as soon as it arrives (and triggering resort, rerender, etc after each individual change), and not being batched or debounced or otherwise.  Slava/Dave, any thoughts?

Avital, that's a useful trick which I hope people keep in mind for that situation... I also hope we recommended it to Jerry back then :)

And of course, I think the ideas and requests for more info from Arunoda and Dave Workman will be helpful too.

Gadi

[Dan]

Thanks Gadi. Here are the issues.

1. Client freeze after map position changed (update of 600 records at once, subscription change)  It's smaller now that I removed sorting but still there.  It's no longer very annoying but it would be great if there were no freeze like google maps.

2. Client freeze gets much worse if I load over 600 hexes at once.  I haven't tried it since I removed sorting.  I would like to be able to load twice as many hexes as the player can see but going over 600 starts to really slow down the game.

3. I'm using one server for background jobs and other servers for the game.  I'm using oplog.  While the background job server is at 100% the other servers are at 100%.  The background job is now 5 minutes long and runs every 30 min.  It was longer than 20 sec when I originally posted and it's grown.  It's basically what makes the game run.  It loops through everything then updates users.  I'm going to try storing the changed users' values in a variable then when the job is done looping through them and updating them all at once.  Currently one user could be updated many times during the job.  The number of people I'm able to get one a server is dissappointing and I'm worried about being able to continue running the game.

[Arunoda Susiripala]

About #3.

Going through all the users and updating all of them is a bit of a task. I don't know how you do it. If you are getting data to the server and then do some math and updating again. Then you'll be definitely have a high CPU usage. Even in Mongo.

It's better if you can rethink and update the game's logic to avoid a the background as much as possible.

---

Arunoda Susiripala

I curate Meteor Weekly - Check it out!

--

[Andrew Mao]

One way to do this would be to publish an indexed hexes collection instead of storing the hexes in the users collection. With fine-grained publication and reactivity, you should be able to support a lot of people. It's also possible you have some redundant dependencies on the users collection (such as a global publish somewhere) that is making these global updates hit a lot of merge boxes and cause a ton of CPU and memory usage. See http://stackoverflow.com/a/21835534/586086 for what I mean.

[Dan]

I've made a lot of improvements and performance has gotten a lot better.  Arunoda's subscription manager made the map work a lot better and Kadira really helps see what is going on.

I still have one issue.  I have a panel that lists a players units.  Big players might have a thousand units.  When they open this panel performance really drops.  I think it's because all of the units are sorted alphabetically.

Also because Meteor.userId() and Meteor.user() are functions I thought they were smarter simply doing a findOne and find.  I thought they cached the user's record.  I have them all over the place.  It would be really nice if they did.