Grease and the network, numpy, etc.

[oeyo...@gmail.com]

Casey Duncan:

I understand that you've moved on from game programming with Python and/or entity based game programming but I'm just now getting started with it. Do you think it's still a valid pursuit?

I am attempting to build a multi-player rogue-like game with Libtcod's Python bindings. I also have looked at Twisted's AMP protocol (TCP) for the network portion. I saw in a previous grease user post that you were toying at the idea of backing grease with Numpy. I was thinking the same thing as the 2D gridbased game play of rogue-likes and Numpy's blazing speed seem like a perfect match! It looks like that around the time that you lost interest in continuing with grease. If you don't mind me asking, what were your findings with integrating Numpy with grease? I also was thinking Redis may be a nice match as it is blazing fast and has all the data structures needed by Grease/ES game programming.

Had you ever given any thought to adding network support to grease? I saw there is a Twisted reactor for use with pyglet, but Libtcod is SDL based (though it does also have an OpenGL renderer). I figure I'll have to divorce Grease of pyglet, but that shouldn't prove too difficult as what you've done is a great foundation to work on. Would I create a network component or would the Network layer be coded 'outside' of the ES system? My thought was to create a network component that would just update component data directly (position, etc). Then when lower systems ran it would use the new data. However, Twisted "taking over" the event loop via it's reactor and callback programming is a tough paradigm to wrap my head around. There is also gevent which is a green thread (co-routine) based I/O library. Pretty neat that it has the same API as threads/processes but is designed for I/O based use cases. I've used it before and it worked well.

I have to say that looking through the source of grease has taught me a ton about API design, testing, and has given me a standard to attempt to meet when releasing my own libraries! I understand you're probably really busy so if you don't have the time to respond that's cool too. I really just wanted to make contact and let you know I'm really enjoying Grease's source.

Thanks,

Dave

[Casey Duncan]

Hi Dave,

I did take a break from game programming for a while, but in fact I've
started dusting off my chops lately. Mostly I've been improving my
Perlin noise library to use for world generation, for ironically
enough a rogue-like 4x style game I'm designing.

Thanks for pointing out Libtcod actually, I wasn't familiar with it.
Definitely looks like some really interesting games are being made
with it. I'll definitely be checking it out.

As far as whether I think entity-based programming is worthwhile, I
would say that my experiments were very encouraging. Particularly for
writing games in a high-level and slow language like Python. It gives
you great opportunities to push grunt work down into native code where
it belongs. Plus, and probably more importantly, it sets up an
organizational framework other than deep inheritance, which just
doesn't scale or lend itself to maintainability in the long run. Plus
it just makes me itchy generally ;^). And it gives you clean
separation of data, logic and drawing code out of the box.

I did work on backing Grease with numpy and there is significant
progress with that in a branch on github. Doing it generally, as I was
doing, is a considerable undertaking since the data structures need to
be heavily micromanaged to allow you to do fast operations in bulk
across entities. I think using numpy in a more specific application
would be significantly easier. numpy is extremely powerful once you
master its world-view.

Grease was a bit of a victim of being popped off the stack too long. I
needed a good 2D geometry library and I decided to write one for it as
a separate thing. That became planar, which is good on its own, but
sort of lost the goal of doing everything in bulk somewhere along the
way. Designing a good geometry API isn't super hard (though there are
a lot of bad ones IMO), but designing a good bulk-style geometry API
is considerably harder. I also pushed my perfectionism to the max with
planar, and frankly I just couldn't keep it up, eventually I couldn't
make progress fast enough to keep me interested on my own. I attempted
to just go back to Grease to do the numpy work, but my old bones just
can't sit at a computer 20 hours a day anymore, it was starting to
make it really difficult to do my day job so I had to scale back for a
while. But I am still here, and paying attention, and interested 8^).

I think using redis as a backing store for a multiplayer game could be
really good. I caution to keep your expectation realistic for a
one-man (assuming it is just you) project at first. Being ambitious is
good, but you should probably approach it in stages. However, that
said it is difficult to bolt-on multiplayer to an existing game. The
game really needs to designed for it, though it is considerably
simpler for turn based games it's still very complex overall.

As for divorcing it from Pyglet, I'm totally in favor of that. In fact
there is a somewhat recent github fork working on that. Though I'm not
in contact with its author, so I don't know what his goals are with
it. Having separate backends was an intended feature from the start,
though I was focusing on Pyglet only to start with.

Twisted is a little tough to use with games, as you say, because it
has its own event loop, which can be challenging to reconcile with the
event loop of the game engine. There was always a intention to make
Grease have networking, and I think the data-driven entity-based
design really lends itself to that. It was intended that data in
components would be tagged with incrementing generations so that they
could be synchronized efficiently between machines. However, no
development of this has happened. It might even be possible to
implement network syncing of components as a system that runs early to
refresh the component data for the remaining systems to use. It could
work asynchronously doing the network updates outside of the game
steps. If it were me, I wouldn't want to depend on a massive library
like twisted, something lighter weight like gevent would be more my
speed.

Thanks for the kind words about the project. It's not really dead,
just hibernating until I get a fire burning under me to work on it. My
current game idea will probably benefit from an ES approach. I'll take
a closer look at Libtcod later this week, and if I like it I might be
inclined to help get Grease to play well with it. That could be a
starting direction, particularly if we are both working on rogue-likes
8^)

-Casey

> --
> You received this message because you are subscribed to the Google Groups
> "Grease Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to grease-users...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[oeyo...@gmail.com]

Hey Casey,

Wow, thanks for the reply! It is pretty funny that we're both tackling rogue-likes. Libtcod is a pretty neat library. Brogue is a somewhat popular 'modern' rogue-like that uses it. Ultima Ratio Regum is a hugely ambitious rogue-like currently under development and Goblin Camp got some good press a few years ago, being billed as a more approachable Dwarf Fortress. The author's own demos are pretty impressive as well. There is a noise generation module included and Perlin noise is one of the options (among lots of other goodies). The best part is it's all C++ so it's speedy. One quick question, but when you say "native" code do you mean C or C++? I've been reading up on calling C/C++ code from Python and (I think) cffi is the new fangled way of doing so that's compatible with Pypy. I'm not sure if cffi works with C++ though.

There are a few undocumented goodies hiding in the Python bindings. The ConsoleBuffer class and the console_fill_* functions. It's interesting that the console_fill_* functions can also take numpy arrays. Looks like ConsoleBuffer just uses python lists for color data, but that could easily be extended to use numpy arrays. It would end up making the renderer pretty dead-simple.

Over the last few weeks I've been slowly working on learning twisted and had a basic client/server going with map data and movement working. I could tell that my code was already starting to become mangled, even at such an early stage. I posted to r/gamedev and through that started reading about component/entity based programming. I found the t=machine blog and then found grease. Now I'm here! :)

So last night I blasted out a bare-bones entity system using the guidelines on the t=machine wikie with Libtcod as the renderer. My code definitely "feels" better than when I was using tradition OOP. I'm glad that you think ES is still a valid pursuit. I really like how it can be thought of MVC and also SQL tables. 

I'm pretty excited to try and get networking working using this new paradigm. Have you seen PodSixNet? It's built on Python's asyncore module and looks dead simple to use, however I think I might give Gevent a shot since the (behind the scenes) event loop runs in C or C++. Just use a new greenlet per connection. I did something similar messing with Gevent, SocketIO and Flask.

I've spent a few hours looking through the numpy branch of grease. Fields are 1 dimensional numpy arrays with a dtype corresponding to what type of value residing in the field. Each field is a Block which is basically a resizable numpy array (like a python list). How are updates applied in bulk? Is the generation portion of the entity_id the same generation you spoke about with regards to syncing component data between computers? I couldn't find how the generation was used anywhere outside of including it in the entity_id.

It was intended that data in components would be tagged with incrementing generations so that they could be synchronized efficiently between machines. However, no development of this has happened. It might even be possible to implement network syncing of components as a system that runs early to refresh the component data for the remaining systems to use.

Could you expand on this a bit more (if you don't mind). Perhaps I don't fully grasp the issues with multiplayer game programming. I did read Valve's networking guide in regards to their source engine and there were far far more issues than my naive brain ever thought about. There product however is a twitch based FPS where pinpoint accuracy and microseconds make all the difference. Your second sentence was what I had in my head about how networking would work. Anytime data came in (since it's async) it would update component data and the game would just keep chugging along with whatever data was in the components at the time.

I have to say you're totally right about being a little too ambitious. I'm still in the prototyping/brain storming stage so my thoughts are all over the place. I'm hoping that'll settle down here in a week or two as I research all the options and finally decide on a path. Again, thanks for the reply!

Dave

[Casey Duncan]

Hi Dave,

Brogue, haha. I've seen URR, which looks really incredible/insane and
it has some inspiration on my design as well. I saw the noise
implementation in libtcod, looks like they are doing alot of the same
things I want to do in my game. I have new version of my perlin noise
lib ready, which should be out shortly fwiw. It's amazing how much you
can do with that as a start.

By native code I mean anything that compiles to native machine
language. In my case its C, but C++ also counts.

Those numpy hooks look pretty useful, thanks. Always looking for ways
to pass data around efficiently.

The generation number in the numpy branch is not to mark the data
generation, but to allow entity "slots" to be recycled. The idea was
to keep entity creation costs low by minimizing memory allocations,
particularly in games or simulations where many entities are created
and destroyed over time. If you look at the _EntityIdGenerator class
in world.py it may make more sense. The whole thing needs some
explanation, but since its not fully baked I haven't gotten to it 8^).
The overriding concept though is that like entities are kept next to
each other in blocks, and that a single entity has the same index in
the same block id across all components. That way common blocks across
components can be treated as units for bulk operations.

Multiplayer is a big subject, but if you are dealing with a small
enough turn-based game, you can probably start by just syncing all the
state from the "server" to the clients each turn. The data generations
would be more useful for a real-time/simulation type game where it
needs to keep going regardless of transient network conditions. As
with everything, try the simplest thing you can think of and only make
it more complex if that doesn't work.

-Casey