My neurons dont improve well. Can I get some tips?

[bog tom]

Hey, I'm new with neat.

And would like to hear some tips.

I'm using unityNEAT for my game(a port of sharpNEAT to unity)

First this is my game:

Made it with friends for a competition called ludumdare, The game is a sumo game where you have to push other players.

I've always wanted to try machine learning, so I took it as a project, creating an intelligent bot with neat for our game.

At first I've deleted the fences, and the powerups, to simplify it.

Also to speed things up, instead of one fight at a time, I've made an implantation of IGenomeListEvaluator that makes many fights at the same time together.

(each fight is in parallel world, so they don't interrupt each other)

my inputs are x,y,z, direction, speed of the player and of the opponent(x,y,z relative to the center). also it has the direction and distance from the player to the opponent.

and there are 3 outputs, whether to stop or move, move forward or backward, move left or right.

I've made a simulation where the opponent doesn't move,

and the closer he get to the opponent, the bigger the fitness.

I've tried at the beginning a normal neat genome factory which did quite well with targets that don't move,

but failed to do well with targets that moved, it missed it.

So i tried to change it to cpnn, and added some of my own activation functions(Rectifier, Sign, Tan, Pow, Quad and a few more..)

I think that it actually worse now.

I've tried various populations, and changing the NeatGenomeParameters..

I've heard about hyperNEAT, but from what I saw it works when each input index represent a different position, and here it doesn't represent anything.

Should I change my inputs, that'll be a sensor of each position?

Or maybe I just should be more patient and wait for it to evolve? I usually stop it at generation 200.

Also from what I saw in the walker example that comes with the neat library, it's not improving so well too.

Any ideas?

[Colin Green]

Hi,

My first suggestion would be to look at the coordinate system...

The cartesian coordinate system isn't a 'natural' one for neural nets to use. In order to use x,y,z cartesian coords the neural net has to learn to do trigonometry to use those inputs. You say you're also inputting direction and distance, so that's good, but it depends how you're encoding direction.

I would suggest dividing the radius around your player into N segments, with an input node for each segment which goes on when the opponent is in that segment. You can then combine this with a single distance input and delete the x,y,z inputs.

You could further expand this scheme to have M inputs for the distance parameter, so that you could have one node each for different distance ranges. So you'd want more nodes representing close ranges, and just a few to distinguish between 'far away' and 'very far away', i.e you don't need the extra resolution once the opponent is outside of your immediate proximity.

I would do that first, and start with NEAT not hyperNEAT. You should get some progress with plain NEAT, and from there you can experiment with hyperNEAT to see if you can improve further, but first I think you should focus on the issues with the position encoding scheme.

Take a look at the prey capture problem domain that ships with SharpNEAT, this does something very similar to what I'm describing above, so you can probably lift some of the basic trigonometry from that code.

Hope that helps,

Colin

[bog tom]

Thanks a lot Colin! You're really quick, you also fixed the issue I opened today :)

I'm transfering the direction in a few different ways, to let it choose for itself what's best for it.

in degree(from 0 to 360), and in normalized x,y, z. I can also try radians instead.

When I used normal neat(not cppn), it used the normalized direction of the player to the enemy x, z, and checked if z is positive move forward, otherwise move backward and this way it learned to move the player.

With cpnn for somereason it didnt come up yet with this method.

About the radius segments, you mean that my inputs should be something like that? 

the player is the red, the enemy is green, which represented as one, and 0 is soil, -1 is air.

My problem with this method, is that it'd ignore the player and the enemy's direction and speed.

I could also make the same radius for speed and direction tho..Where green will represent the next position of the enemy instead of the current.

And also you say I should go back to neat, and not using cppn?

בתאריך יום שלישי, 27 בדצמבר 2016 בשעה 20:33:12 UTC+2, מאת Colin Green:

[Colin Green]

On 27 December 2016 at 19:08, bog tom <bog...@gmail.com> wrote:

 

I'm transfering the direction in a few different ways, to let it choose for itself what's best for it.

I would advise against that and to experiment with one input scheme at a time. 

 

About the radius segments, you mean that my inputs should be something like that? 

Yeh, you can of course use a different number of sectors/radials; 8 worked find for the prey capture though, so is probably at a sweet spot where there's enough sectors to work, but not so many to make learning difficult. I suppose one could have multiple sets of inupts, for e.g. low, mid and high resolution (e.g. quarters, eighths, 16ths..).

 

the player is the red, the enemy is green, which represented as one, and 0 is soil, -1 is air.

Yeh I would have one set on inputs for the enemy, and another for air. Air is basically the boundary your bot needs to know it is approaching so it doesn't fall off the edge, so you don't need another set of inputs for 'ground'.

 

My problem with this method, is that it'd ignore the player and the enemy's direction and speed.

The position inputs should work fine i think, unless there's a reason to believe very fine speed and direction inputs are needed.

And also you say I should go back to neat, and not using cppn?

For starters, yes. You're just layering on more complexity before you've got the thing working, which just makes it that much harder to reason about why it isn't working.

Colin

[Colin Green]

To clarify, I'm recommending this scheme...

I think the distance as a separate set of inputs makes sense for first attempt. This should be easier for evolution to start using quickly. You could also activate the distance inputs so that, e.g. if the enemy is at distance 5, then we also set inputs 6 and 7 to on (i.e. the 'correct' input, and all other inputs beyond that one). Again, that's about making it easier for evolution to use the inputs.

You can then repeat the above scheme for detection of the world boundary (the air). In that case you may have multiple direction inputs on at the same time, so I would suggest setting the distance input to be whichever the closest distance is in all of the directions.

In general evolution is more likely to work when you encode inputs in a way that are easy to use, and this means using lots of binary inputs, each rerpesenting one specific thing, rather than continuous inputs (e.g. distance, angle). This is essentially how biological sensory encoding work, but with vastly more inputs. However, with NEAT we also want to keep the number of inputs as low as possible, because it's a direct encoding that has to discover every connection individually, so there's a balance to be had, and I think the above scheme is a good balance.

Final point - I just noticed that my prey capture example tilts the above segments 22.5 degrees, so that the due North, East, South, West directions are in the center of a sector's range, rather than being directly on the boundary between two sectors; which i think makes sense in your case too.

Colin

[bog tom]

Hello again Colin, I really appreciate your support and help.

I've made this radar in the picture below, it's a 12*3(12 angles, 3 radius) inputs of segments where each one is -1 0 or 1 as I said before, meaning 36 inputs.

Later on I tried 8*3, 24 inputs. and then I changed to as you said, two different sets of inputs for air and enemy.

Making it 48 inputs, First 24 - 0 no enemy on segment, 1 - enemy on segment, and the rest 24 - 0 middle of segment is not edge, 1 middle of segment is edge.

I tried the last method for 4 hours. With population size of 500, and 50 species.

It did 243 generations.

It stopped getting better after 123 generations. The best had only 13 connections(with no hidden units), which is way less then what I would except.

This is the result of the best:

The drawing has only 24 inputs to simplify it, the air and enemy combined, but it actually has 48 inputs as said.

Btw, My fitness scoring works like so:

Every evaluation I give him 9 static targets with different angles(all with same distance) and make average of their scoring. 

• Against each target, he starts his scoring with 1000 scores,
  
• If he dies he'll loose 200 scores. 
  
• If he touch his opponent, he gets 10 scores. 
  
• If he kills him, he gets 150 scores and additional bonus score on how fast he did it.
• Every 0.1 seconds he gets rewarded about how close he got to the opponent relative to the previous time, if he got closer he gets 0.1 score, otherwise he looses 0.1 score.
  I've also tried now that every frame the closer he gets, the more points he get relative to the distance he had on the beggining(Fitness += startDistance-currentDistance),
  Which gave much better results with only 79 generations. For some angles, it even kills him. but it's not good enough yet.(it has 31 connections, 3 hidden units)

To optimize it, it always stops after 5 seconds, which is the time that'll take an optimal bot touch and kill him, and if someone didnt move after a second the round began, it skips him.

I also thought about making the arena smaller, because it'll make him do quicker evaluations because he'll die or touch his opponent faster, and I think he can get to the same conclusions with smaller one, at least for his close range inputs, or I can make the whole sensor relatively smaller.

My NeatGenomeParameters are the default parameters, I think I might try increasing the connection mutation probability.

Are you sure that I shouldn't try hyperNEAT yet? It'll make it learn much faster I think with symmetric of the segments(if i got right what hyperNEAT does).

When they'll fight each other, Their speed will be quite necessary, because the one who is faster is the one who pushes, so it'll help him decide if he should push or dodge.

Thanks again,

Tom.

בתאריך יום שלישי, 27 בדצמבר 2016 בשעה 21:49:08 UTC+2, מאת Colin Green:

[Colin Green]

A slightly more advanced scheme would use the same inputs as a I just described, but rather than pure binary on/off inputs, the input level increases gradually as thing being sensed approaches the input's range. I'd call this a wavelet approach, since an input value will vary like so...

[bog tom]

I answered to your previous message a few minutes ago, and saw this message just now, 

I understand what you say, though the enemy distance inputs will work well against one enemy, but in the future when I'll have more then one I think it'll cause some problems,

maybe I'll try with 6 angles instead of 8, and only 2 ranges instead of 3 as a start.

Thank you colin

בתאריך יום רביעי, 28 בדצמבר 2016 בשעה 15:08:38 UTC+2, מאת Colin Green:

[bog tom]

so for each angle 1 means he is close, 0.5 means he is quite far, and 0 means he is out of range. this way I'll have only 8 inputs for each angle, instead of 8*3 for each range?

בתאריך יום רביעי, 28 בדצמבר 2016 בשעה 15:24:01 UTC+2, מאת Colin Green:

[Colin Green]

 

> though the enemy distance inputs will work well against one enemy, but in the future when I'll have more then one I think it'll cause some problems,

Ok, understood. 

I think it will be useful to experiment with the scheme I described playing against just one enemy. If you can make that work then move on to multiple enemies; however, if you can't then you're wasting your time on the more complex task before you've got the simpler task working. 

Colin

[Colin Green]

> so for each angle 1 means he is close, 0.5 means he is quite far, and 0 means he is out of range. this way I'll have only 8 inputs for each angle, instead of 8*3 for each range?

It's the same number of inputs, and the same scheme overall. But instead of an input going directly from 0 to 1 as the enemy moves into a sector, the level would increase gradually, going above zero before the enemy is in that sector, and reaching one when the enemy is in the center of that sector's range.

I would just stick with the pure binary method for now, it should work with the pure binary inputs, but i think the wavelet inputs idea is again more akin to how natural/biological sensors work, and intuitively it feels to me like it would be an improvement over the pure binary scheme. But I think your problems lie elsewhere at the moment.

Colin

[bog tom]

Okay, I'm trying now only 6 binary inputs, each one represent if it is in its angle range or not, and let's see how well they'll do before we progress to more advanced inputs.

I'll update you soon how it's going.

בתאריך יום רביעי, 28 בדצמבר 2016 בשעה 15:39:17 UTC+2, מאת Colin Green:

[Colin Green]

On 28 December 2016 at 13:17, bog tom <bog...@gmail.com> wrote:

Btw, My fitness scoring works like so:

Every evaluation I give him 9 static targets with different angles(all with same distance) and make average of their scoring. 

• Against each target, he starts his scoring with 1000 scores,
  
• If he dies he'll loose 200 scores. 
  
• If he touch his opponent, he gets 10 scores. 
  
• If he kills him, he gets 150 scores and additional bonus score on how fast he did it.
• Every 0.1 seconds he gets rewarded about how close he got to the opponent relative to the previous time, if he got closer he gets 0.1 score, otherwise he looses 0.1 score.
  I've also tried now that every frame the closer he gets, the more points he get relative to the distance he had on the beggining(Fitness += startDistance-currentDistance),
  Which gave much better results with only 79 generations. For some angles, it even kills him. but it's not good enough yet.(it has 31 connections, 3 hidden units)

Sounds a little bit complicated.

How about.

1) Start with zero.

2) Run for fixed amount of time, or until the player dies.

3) Award points per unit time (i.e. 10 points per second).

4) If the enemy is killed then award extra points and restart a new game. Keep playing until overall the fixed amount of time expires.

5) If the player is killed then subtract points and restart a new game.

However, you want to avoid assigning a zero score, so the subtracted points in rule 5 shouldn't be too harsh, or (as you have done) start with a positive number of points.

Fitness schemes are very tricky to get right. You can spend a lot of time tweaking schemes, or trying out completely different schemes. 

Try and keep it simple.

Colin

[bog tom]

I'll try it. only problem is with 3, that it'll reward bots that don't move. So maybe I'll stop the game once they stop moving

Thank you Colin!

[Colin Green]

On 28 December 2016 at 14:11, bog tom <bog...@gmail.com> wrote:
>
> I'll try it. only problem is with 3, that it'll reward bots that don't move.
> So maybe I'll stop the game once they stop moving
>

It depends on the game. I would expect not moving would result in
being killed by the enemy(?). If not then not moving is a valid
strategy (avoids being killed or falling off the edge!) and maybe the
game needs changing, not the fitness scheme. Just a thought.

[bog tom]

Yes it would make him getting killed in the real game. but to simplify it, I'm letting them fight against a static target that doesn't move as a start. so they'll just learn to kill it, and later when they do it well, I'll let them fight my coded bot, or fight each other.
Not moving is indeed a valid strategy for my simplified game where the other player doesn't move, but I'm rewarding them to move(in my previous fitness scheme they got more points as they got closer to the target), so they'll gain more fitness if they move.

You think I should let them fight against a coded bot right away instead of my simplified static target?

בתאריך יום רביעי, 28 בדצמבר 2016 בשעה 16:28:28 UTC+2, מאת Colin Green:

[bog tom]

Hey, I've made some progress,

I changed the output to be binary as well. (until now they had 2 outputs, that could produce any direction).

so I tried 4 outputs and it takes the biggest value each one represent different direction, same as in your prey example.

and only 4 inputs, of which segment the player is in, each one represent different angle.

Also I drew it, with extension called uneaty, that draws your neural network in unity.

And modified it a little so the drawing will represent the domain problem.

I found out, that my outputs and inputs weren't sync to the same coordinates, I had to rotate my segments inputs, so the input forward and output forward will mean the same. They had before inputs of top right, bottom right, top left, bottom right and outputs of top, bottom, left, right.

And the bots solved it now with only 13 generations! they attacked the player.

that's what his brain looks like:

green ball = input, red ball = output, blue ball = bios.

red connection = negative weight,

white connection = positive weight.

the bigger the connection, the bigger the weight.

The inputs and outputs are positioned as you would except, meaning left input, would have left output on top of it, and the right input and output would be on the opposite direction.

This is the best brain they produced after 13 generations, in the picture above.

It works very well, I would except it to have 4 white connections, each one with his top one. but they did something that worked too, they made negative connection to the opposite direction, so that'd have the same effect I guess.

Now let's see how well they do with 8 inputs and outputs.

בתאריך יום רביעי, 28 בדצמבר 2016 בשעה 16:28:28 UTC+2, מאת Colin Green:

On 28 December 2016 at 14:11, bog tom <bog...@gmail.com> wrote:

[bog tom]

So now with 8 inputs and 8 outputs it takes them between 20 to 60 generations to attack the opponent well(I doubled the population to 400 and 20 species. with 4 inputs I had 200 and 15 species).

This is the best brain after 20 generation:

His brain structure seems a little clumsy, but it works well for all angles somehow. (I think there is maybe one angle in generation 20 that worked a little worse than the rest, but it got fixed after some later generations).

Now I'll try separating the direction to 2 ranges. making it 16 inputs. hope it'll still work well.

בתאריך יום חמישי, 29 בדצמבר 2016 בשעה 16:05:52 UTC+2, מאת bog tom: