Map<Class<Component>, Component>

[Ashiq A.]

Hi,

I'm trying to make an entity-component system. I have a base Component class (sparse), and an Entity class.  I would like to have a map/dictionary of component classes to those components.

Is there any way to do this?

I tried using HashMap, ObjectMap, and Map, and all of them fail to compile for different reasons. (I think HashMap required me to add a hashcode function onto Class<Component>; one of the other two failed a constraint check that Class<Component> is not { })

[OvermindDL1]

Making a hashing function for a class is not too hard, over the complete name might be fine, however it is not performant, the pointer would be faster and 'should' not change, but unsure how well that would work on all platforms...

If you want an efficient ECS that is already built for haxe and uses underlying storage types very efficiently, I'd recommend ECX:  https://github.com/eliasku/ecx

[PSvils]

Hey!

I'll get over the self-plug at first: https://github.com/PDeveloper/eskimo/tree/next
I'm trying to finish off the API for Eskimo 'next' into Master, but everything is working, and I don't expect radical changes. Eskimo is super fast, but I'm trying to focus on keeping the API simple and flexible as well.

For Eskimo, I get the name of the Class name at runtime with reflection, and then I have a Map<String, Component>, which works fine across all platforms. The thing is, this isn't a very fast process. So what Eskimo does, and as far as I have read, the best practice for ECS anyways, is having a central component store/manager. I have a container for each component type, and components in each container are indexed by the Entity ID. What this allows, is that systems know which components they'll need to access, and so the Reflection and String map access happens only once at setup.
For example, a system needs access to the Transform and Physics components, when the system is initialized, it uses Reflection, and gets references to both Transform and Physics containers, and beyond that, all access is a single look-up into an Array.

I hope this helps in your own ECS construction :)

[OvermindDL1]

Oh hey it's you!  I looked at your library a few times when I was looking for a good ECS lib for Haxe a bit over a year ago but it did not really meet my requirements at the time, it was amazingly slow at the time but I've heard it is faster now, but I'm curious just how much faster?

The reason I finally decided on ECX was because, well, it was not done yet and the dev was still working on it so I...encouraged him to design it more like my own C++ ECS library, which he did, and got substantial speed improvements from the redesign, so I had a vested interest (and too lazy to port my C++ version to Haxe).  ^.^

But at the time ECX Vs Eskimo was this:  https://eliasku.github.io/ecx_benchmarks.html

Those are *very* old benchmarks and after some improvements ECX was substantially speed up more to be this (this just compares ECX to a very raw manually done ECS with no overhead of any library):  https://eliasku.github.io/ecx2.html

And benchmarks of the newer ECX comparing other ECS libraries he found (Eskimo was not included in this one because it took so long to run compared to everything else that it made benchmarking a bit painful, same with seagal, which was dropped too, plus one of the libraries basically billed itself as a faster eskimo):  https://eliasku.github.io/ecx2_versus.html

Should I update the benchmarks and run them with the latest Eskimo again? I'm really curious if you've managed to exceed my decade-old-long design.  :-)

For note, ECX uses component tables that can be custom made and packed in whatever way you want (I have a QuadTree one for one of the component types in my system for example, but most are array's) and it has a default implementation that should be 'good enough' for most purposes.  There is no reflection, there are some macro's, and it uses platform dependent structures whenever possible to maximize speed if you use the 'HOT' setups.

[PSvils]

On my own machine, Eskimo runs faster than all other ECS in the benchmark, EXCEPT the two "_hot" ones, but I think those benefits could be applied to Eskimo as well!

Updating the benchmarks would be awesome, I'm interested to see myself where it stands now, on that machine / those benchmarks. From a performance standpoint, I'm not sure how to speed up Eskimo at this point, since component setting and getting compiles to direct Array access! The API on the other hand feels like working with normal objects that have components as properties.

0.1.x Eskimo to 0.2.x Eskimo, there is an over 100x times improvement :D (Just because 0.1.x was so slow)

var view = new View<Transform, Physics>(entity_manager);
var entity = view.create();
entity.transform = new Transform();
entity.physics = new Physics();

Here you can treat entities as objects with properties.

Give the benchmarks a shot, comparing with old Eskimo, it's definitely worth a look :)
API is finalized on my end, will be a few changes from current `next`, but should be on Master branch soon enough.

[OvermindDL1]

That is really cool!  A *substantial* improvement from when I saw it last!  :-D

For note, the ecx person has made a lot of libraries, one of those abstracts out the efficient arrays and vectors and so forth of the various platforms, that library is what is used for the 'hot' setup, you could use it as well in yours and it would indeed help.  :-)

I'm curious, how easy is to to make a new component storage type?  Like say instead of an array I want to use a RedBlack-Tree because I'm only expecting, say, 8 things to have this component instead of a million?  Or an Quadtree component that is internally in an Quadtree instead of an array?  I use both of those in my own things through using ecx so if I were to port to test I'm curious how easy they'd be to implement?

How fast is sparse iteration, like say I have a million entities (not uncommon in my mini-games, though most do not 'do' a whole lot calculation-wise), and I have a component that is scattered across around 10,000 of them fairly randomly (user controlled), are these families cached for efficient iteration or does every entity have to be iterated through to test?  If cached (I'm guess your `View` does this) is it a randomly sorted list based on component added order, or is it in entity ID order or is the internal storage of the cache configurable (I'm partial to an ordered set in a couple cases)?

I'm curious about the component setting, you are showing a `entity.transform = new Transform();`, does this do any memory allocation or do you have some macro-magic converting it into something like `system.getStorage<Transform>().set()` or so?  I ask because I tend to use a *LOT* of components as very simple storage and/or empty-flags so I'm curious how that would hit memory if so.

Also, are the list of Systems static at compile-time or can they be dynamically added?  Must the components be specific as static at compile-time or can they be dynamically added?  Do you have an event system in or do you represent events by adding/removing components as appropriate?

Is there an API for working with the components in a more traditionally component table style or is the object-style required (the object style would not at all work for my data-driven loading from definition files without hard-coding every-single-component-setter)?

I'm very much liking how its developed though, it looks amazing!  Do you have benchmark app setup for it somewhere already so I can do some pre-benchmarking until I get time (hopefully this weekend) to update the main benchmarks here to compare to all?

[PSvils]

Oh boy, good list of questions!

So by design, I've setup Eskimo to support component-specific storage in the future, but right now that's not really implemented, though technically you COULD set it manually yourself ... I'll take a look at this after the next release! Views do indeed "cache" entities, and currently it is a randomly sorted list, since currently I can't see a benefit to having it ordered in any way. You can manually sort it, though I'll also look into supporting sorted insert in the View itself.

`entity.transform = new Transform()` - how do you mean, allocating memory? Haxe works with references, so in any case you're simply setting a reference. But this does compile into accessing a container, and setting it, in the current situation, a simple Array. Something like `transformComponents[entity.id] = new Transform()`

Systems and components can both be added or removed at runtime, there's nothing preventing that. A specific use-case of my own is supporting live coding, where I dynamically reload Systems, and load new ones as well, with new components, etc. An event system is something you could hook on yourself, currently there is none, but this is another feature I'm looking into, and how close it fits into an actual ECS library. (I guess the main goal for me, is to have an ECS "library" with some helpers, rather than a Framework. As such, there really isn't any magic behind just the fancy, macro interface!)

As for working with components and their containers directly, check out the ComponentManager class, that should have all the access functions you need (by class name for example) etc. No app for benchmarking currently :(

A good read, I'll review what you wrote in more depth later on as I'm finalizing the new release, and will give a thinker for the new features. For the specific component storage types - is using a redblack-tree simply a memory consideration?

P.

[OvermindDL1]

On Friday, April 21, 2017 at 8:52:46 AM UTC-6, PSvils wrote:

Oh boy, good list of questions!

So by design, I've setup Eskimo to support component-specific storage in the future, but right now that's not really implemented, though technically you COULD set it manually yourself ... I'll take a look at this after the next release! Views do indeed "cache" entities, and currently it is a randomly sorted list, since currently I can't see a benefit to having it ordered in any way. You can manually sort it, though I'll also look into supporting sorted insert in the View itself.

Usually not required at all yeah, but in some cases like heavy orderered work like is appropriate for physics calculations and optimizing over SSE or so it has provided a *HUGE* speed boost for (an over 12x speed boost), but yeah, most of the time it does not help a lot, though the cache ordering does help speed a little, not really worth the insert cost if things are constantly changing, just have to benchmark to see where it is useful and when not on a per-thing-basis.  :-)

On Friday, April 21, 2017 at 8:52:46 AM UTC-6, PSvils wrote:

`entity.transform = new Transform()` - how do you mean, allocating memory? Haxe works with references, so in any case you're simply setting a reference. But this does compile into accessing a container, and setting it, in the current situation, a simple Array. Something like `transformComponents[entity.id] = new Transform()`

Actually you do not always need to set a reference, plus that scatters memory around hard.  Like take a component that holds a Tile position, so an integer x/y position, that can be represented by an in-line integer array striped in sets of 2 for the x/y.  Even with a struct you can store that in-line in many cases as well without any memory allocation beyond the raw array storage itself as well.  I've always gone through the work to ensure that all of my component storages, whether arrays, trees, mappings, etc... are all able to be memcpy'd, thus no references, no pointers, etc...  The lack of pointer indirections helps speed and cache coherency significantly over pointer lookups in many ordered cases and still noticeable in non-ordered cases (since less memory access).  For example, if I have, say, some component that is a structure of, say, an integer, 2 floats, and a string, I'll pack them in an in-line array on arch's that support it and stripe it out among multiple primitive arrays on arch's that don't with the strings pointing to an in-memory character array via holding an index where this character array is also in the storage table, all hidden behind the unified interface.  This allows me to just memcpy out the entire memory block each very efficiently and quickly on arch's that support it and still is significantly faster than following pointers all over the place for a more naive serialization method.

I pulled out part of my C++ ECS system out of the engine it was built inside of into a standalone project (mostly for benchmarking purposes and not really for use, it is lacking the event system and some niceties but is otherwise fully functional) that you can take a look at to see how I originally did this style (and have 'mostly' duplicated into ECX):  https://github.com/OvermindDL1/OverECS/blob/master/main.cpp

On Friday, April 21, 2017 at 8:52:46 AM UTC-6, PSvils wrote:

Systems and components can both be added or removed at runtime, there's nothing preventing that. A specific use-case of my own is supporting live coding, where I dynamically reload Systems, and load new ones as well, with new components, etc. An event system is something you could hook on yourself, currently there is none, but this is another feature I'm looking into, and how close it fits into an actual ECS library. (I guess the main goal for me, is to have an ECS "library" with some helpers, rather than a Framework. As such, there really isn't any magic behind just the fancy, macro interface!)

If you are curious about an event system my old C++ engine had one for it's ECS that had hot-paths that were inline optimized (via C++ templates) and everything falls back gracefully to a generic path that worked fantastically both in C++ and for binding to a scripting language to mod in (in my case Lua and Terra) with the connections able to be cached for faster creation/removal of events as well.  I created an Atom type that converted a short string to an integer and vice-versa (5-bit packing, and you can see the code for it at https://github.com/OvermindDL1/OverECS/blob/master/StringAtom.hpp and usage is exampled in the comment at the bottom), which is still used all over my C++ engine for things like event ID's and lookups and such (you can 'switch' on a string even, since it is not compiled to a string), so for example a frame update event might be "UPDATE", which you type in C++ as "UPDATE"_atom64, which gets compiled to an integer so you can switch over it, efficiently hash it, etc...  I have only a pale shadow of the event system in the ecs example code (a couple of the hot-paths are kind of baked in manually) but I can easily describe the main setup if you are curious, the design has served me very well for almost 2 decades.

On Friday, April 21, 2017 at 8:52:46 AM UTC-6, PSvils wrote: 

As for working with components and their containers directly, check out the ComponentManager class, that should have all the access functions you need (by class name for example) etc. No app for benchmarking currently :(

Ah that looks useful!

No problem, was just curious if there was an benchmark out already or not.  :-)

On Friday, April 21, 2017 at 8:52:46 AM UTC-6, PSvils wrote: 

A good read, I'll review what you wrote in more depth later on as I'm finalizing the new release, and will give a thinker for the new features. For the specific component storage types - is using a redblack-tree simply a memory consideration?

I've benchmarked that a redblack tree to store <12 elements is faster than a hashmap, and a hashmap is better up to a certain amount of collisions, then after that an indexed sparse array 'if' the data is packed in sets (it often is for many components, but is not for some others), and beyond that just a normal array.  The red-black tree I choose if there are very few elements thanks to its speed at those sizes, it just falls off fast if there are more than a few though, and it is easy to represent entirely in a single array so no pointer indirections are needed, like a hashmap can do as well (the standard containers for RB trees and hashmaps are *NOT* inline memory though, these are custom creations), so a hashmap or so would be fine for low memory on not-often-used-components, but I just choose different structures based on the purpose and usage for speed reasons, I often just use a plain list in many cases as well, or a bitset for data-less flag components, or a quadtree for efficient spatial lookup, etc...

To be honest, by asking these questions and showing these uses I'm trying to help steer the development of eskimo to include more useful features for my own use cases (which tend to push engines pretty hard).  ^.^;

[OvermindDL1]

For note, I just ran the benchmark of my OverECS set to 1 million entities but otherwise the same as the github code, I just got this output (this is what I compared ecx too, ecx was close to this, but still did not meet the same speed, also this computer I ran this on is *OLD*, not even remotely fast, the ecx benchmarks on this machine often test at a 3rd the speed of what is shown on github on my desktop here):

All pre-caching is disabled so everything is grown on the fly, alignment is disabled except in eigen tests since SSE has to be aligned, the loops are very basic and 'direct', many systems would be optimized for their specific use-cases, build_destroy just tests creating the million entities with the 2 sets of components (with all families updated on the fly and all) then destroying them all, and the update tests test iterating over the different families and doing some operations that cannot be optimized out by the compiler, I think they all use an array for storage currently, pos_vel is a very generic style that anyone might normally make, eigen uses the eigen library to do SSE work over the physics adding and so forth (and it operates over everything, set on an entity or not, because it is just that much faster in SSE that I don't care about working on 'only the necessary things' at this point, just have to be sure to set proper defaults of course for non-assigned entities):

benchmarking pos_vel build_destroy

collecting 100 samples, 1 iterations each, in estimated 57.5043 s

mean: 562.127 ms, lb 561.737 ms, ub 562.546 ms, ci 0.95

std dev: 2.06429 ms, lb 1807.32 μs, ub 2.38953 ms, ci 0.95

found 4 outliers among 100 samples (4%)

variance is unaffected by outliers

benchmarking pos_vel update

collecting 100 samples, 1 iterations each, in estimated 107.066 s

mean: 42.0272 ms, lb 41.4819 ms, ub 42.5277 ms, ci 0.95

std dev: 2.66538 ms, lb 2.30199 ms, ub 3.08624 ms, ci 0.95

found 9 outliers among 100 samples (9%)

variance is severely inflated by outliers

benchmarking pos_vel_eigen build_destroy

collecting 100 samples, 1 iterations each, in estimated 59.9239 s

mean: 586.812 ms, lb 586.258 ms, ub 587.381 ms, ci 0.95

std dev: 2.86057 ms, lb 2.54152 ms, ub 3.3163 ms, ci 0.95

found 0 outliers among 100 samples (0%)

variance is unaffected by outliers

benchmarking pos_vel_eigen update

collecting 100 samples, 1 iterations each, in estimated 101.601 s

mean: 424.621 μs, lb 414.544 μs, ub 436.613 μs, ci 0.95

std dev: 55.9113 μs, lb 47.7781 μs, ub 67.414 μs, ci 0.95

found 3 outliers among 100 samples (3%)

variance is severely inflated by outliers