Animat example with 16 states and 4 actions

[FireDragonGameStudio]

What would you like to submit? (put an 'x' inside the bracket that applies)

• question
• bug report
• feature request

Issue description
Hy,
I am currently working on the QLearning Animat sample. When running test1 with 256 states, 8 sensors and 4 actions everything works. After reducing the sensors to 4 and the states to 16 the learning will not succeed, even if I change the number of iterations to 1000. Can somebody explain, why this is not working? I attached the changed code, maybe this will help.

Best regards :)

...
qLearning = new QLearning(16, 4, new TabuSearchExploration(4, new EpsilonGreedyExploration(explorationRate)));
...
private int GetStateNumber(int x, int y) {
    int c1 = (map[y - 1, x] != 0) ? 1 : 0;
    int c2 = (map[y, x + 1] != 0) ? 1 : 0;
    int c3 = (map[y + 1, x] != 0) ? 1 : 0;
    int c4 = (map[y, x - 1] != 0) ? 1 : 0;

    return c1 | (c2 << 1) | (c3 << 2) | (c4 << 3);
}

[cesarsouza]

Hi @FireDragonGameStudio,

Thanks for opening the issue!

Well, I think I would kind of expect that removing the robot's ability to see in all diagonals would adversely affect training. Probably before the change, the learning algorithm could rely on the diagonals to identify corners. But with the changes, when the robot is in a situation like this:

it will "think" it is in a corridor, because it cannot detect that there is a open space on its top-left. If it believes it is in a corridor, I guess the only moves it would consider is to move alongside this corridor (up and down). But then the problem is that when the robot comes back from the corridor and sees in this situation:

is also the same it would had seen if it were here (disconsidering blocks in the diagonal which it cannot see):

and in one the correct action would be to go up and in the other, to go down. Probably TabooSearch/EpsilonGreedy exploration are too simple to consider this. However, I also have to say that I am not an expert on active learning. The Q-learning, Tabu search and the other policies have been implemented the original author of the AForge.NET Framework which this project is based upon (@andrewkirillov) who was much more familiar with them and way more skilled with their robotics applications than me.

Regards,
Cesar

[FireDragonGameStudio]

Hy @cesarsouza,
thank you for the excellent explanation. I'll try to dig deeper into the QLearning topic and maybe come up with additional information regarding this special case.

Best regards :)

[FireDragonGameStudio]

Hy @cesarsouza,
I had a closer look into QLearning, especially the source code of AForge.NET. To get a proper understanding of how everything works, I came up with my own (but still similar) version of QLearning. The interface was also adapted, as the AForge version was a little confusing for me ^^ It works for sample2, where Sarsa and QLearning fail, but not really for test3 (because the learning takes very long Oo), where Sarsa and QLearning succeed nearly instant. Additionally I realized that there is no method in the current implementation which chooses the action, based on the highest, learned q-value when showing the learned solution. Please have a look at the attached gifs to see the difference (for sample2). I'll keep you up to date.

Best regards :)

QLearning

SARSA

MyQLearn

[cesarsouza]

Hi @FireDragonGameStudio,

Wow, that's really, really great!! Is there any chance I could know what the changes were exactly? (and do you have any plans to contribute them to framework as a PR in the near future? 😉)

PS: Also if I may ask - which software did you use to record those gifs? I'd love to use it to create some tutorials/demos on how to execute sample applications!

Regards,
Cesar

[FireDragonGameStudio]

Hy @cesarsouza,
after I finished the code clean up and refactoring (this will be in the next few days), I'll summarize the changes here and create a PR asap. Hopefully this will be a useful contribution. :)

For the recording I used OBS Studio, because of its very handy window recording mode and converted the video files to gif with an online converter called ezgif. If you're planning to record gifs on a larger scale, maybe one of the tools mentioned on filmora is the right choice for you.

Best regards :)