Help with AI Nodes

Hello,

I am currently working on a project that requires an AI. I thought that my AI was working, but upon checking it wasn't actually learning anything, and I think I've found the issue. For some reason, the individual nodes are not moving towards a better solution for some reason. They are supposed to move towards whatever gives them a higher fitness, but I seem to have missed something here. Could someone please have a look at my current testing script to see where I have gone wrong?

Please note that the name of the file is "NEAT", so the NEAT class will run and initialize the Node class. Also, the use of the "Tracker" object is simply to prevent 1 node connected to another node from creating an infinite loop. In this example with only 2 nodes, the tracker object is irrelevant:

Code (CSharp):

1. using System.Collections.Generic;
2. using System.IO;
3. using UnityEngine;
4.  
5. public class Node {
6.     public float ID;
7.  
8.     public enum Status { Input,Normal,Output,Unassigned };
9.     public Status nodeStatis;
10.  
11.     public Dictionary<Node, float> connectedNodes = new Dictionary<Node, float>();
12.     public Dictionary<Node, float> prevNodes = new Dictionary<Node, float>();
13.     public Node self = null;
14.  
15.     public bool NodeStructureStable = true;
16.     public float inputVal;
17.  
18.     float prevFitness = Mathf.NegativeInfinity;
19.     float movementVar = 1;
20.  
21.     public float NodeOutput(List<Node> Tracker = null) {
22.         if (nodeStatis == Status.Input) { return inputVal; }
23.         if (nodeStatis == Status.Unassigned) { throw new System.Exception("Attempt to get output of undefined node!"); }
24.  
25.         if (Tracker == null) { Tracker = new List<Node>(); }
26.         if (Tracker.Contains(self)) { NodeStructureStable = false; return Mathf.NegativeInfinity; }
27.  
28.         float result = 0.00f;
29.         foreach (KeyValuePair<Node, float> i in connectedNodes) {
30.             Debug.Log("Weight: " + i.Value.ToString());
31.             Tracker.Add(self);
32.  
33.             float temp = i.Key.NodeOutput(Tracker);
34.             if (temp == Mathf.NegativeInfinity) { return Mathf.NegativeInfinity; }
35.             result += temp * i.Value;
36.         }
37.      
38.         return result;
39.     }
40.  
41.     public void AlterWeights(float fitness) {
42.         if (nodeStatis == Status.Input) { throw new System.Exception("Attempt to alter input node weights!"); }
43.         if (nodeStatis == Status.Unassigned) { throw new System.Exception("Attempt to alter undefined node weights!"); }
44.      
45.         if (fitness > prevFitness) {
46.             prevFitness = fitness;
47.             prevNodes = connectedNodes;
48.         } else {
49.             connectedNodes = prevNodes;
50.         }
51.  
52.         //Now that everything is as it should be, randomly alter the weights
53.         Dictionary<Node, float> temp = new Dictionary<Node, float>(connectedNodes);
54.  
55.         foreach (KeyValuePair<Node, float> i in temp) {
56.             float offset = Random.Range(-0.01f, 0.01f) * movementVar;
57.             connectedNodes.Remove(i.Key);
58.             connectedNodes.Add(i.Key, i.Value + offset);
59.         }
60.     }
61. }
62.  
63. public class NEAT : MonoBehaviour {
64.     Node myTestInputNode = new Node();
65.     Node myTestOutputNode = new Node();
66.  
67.     void Start() {
68.         myTestInputNode.nodeStatis = Node.Status.Input;
69.         myTestOutputNode.nodeStatis = Node.Status.Output;
70.  
71.         myTestInputNode.self = myTestInputNode;
72.         myTestOutputNode.self = myTestOutputNode;
73.  
74.         myTestOutputNode.connectedNodes = new Dictionary<Node, float> { { myTestInputNode, 0.5f } };
75.     }
76.  
77.     void Update() {
78.         myTestInputNode.inputVal = 0.5f;
79.         float output = myTestOutputNode.NodeOutput();
80.         float fitness = 1 / (0.5f - output);
81.  
82.         myTestOutputNode.AlterWeights(fitness);
83.         print("Fitness: " + fitness.ToString());
84.     }
85. }

I'm not sure why the node isn't trying to reach a more optimal solution. In this example I've literally just got 1 input and 1 output node. The fitness here is just "How far away from an output of 0.5f are you?", so it's a super simple test to see if it works. I've been working on the script this fits into for about a week now and I just need a fresh set of eyes to solve this issue.

Thanking you in advance,
Andrey

 

Bumping because I really need an answer

 

well, I'm not sure. the original NEAT is not a trivial endeavor and has many moving parts and some tight math, that shouldn't be taken lightly. definitely not something you write in an afternoon and then ask random people with a deficit in the attention span.

it seems that you have transcribed this from some other language (python?), namely you don't need that 'self' reference, in C# you use system-provided keyword this. it's really an eye-sore.

it also seems that the fitness can only go up from negativeinfinity, and nothing prevents your output (line 79) from becoming 0.5f which would immediately blow up to positiveinfinity in line 80, and this would then completely choke the system from that point on.

I'm really not sure what's going on, what seems to be the issue, what should happen instead, and where did you get this code from. most likely you have written it wrongly, but I could be wrong. I've also checked for the simple errors, and couldn't see any. maybe if you know more about the internals you can guide me through it.

I know a little about the neural networks, and this looks like a very basic half of a neuron with some logistical boilerplate attached to it. I say one half, because even the simplest of neurons have at least a couple of weights to be called that, or at least a sophisticated feedback.

in other words, this looks unsophisticated EXACTLY because you're running it with only two nodes. it's supposed to be a network, for a reason. that quantity is the source of all required complexity. but you got to make sure it's perfect, line by line.

 

I've actually built this completely from scratch, from the ground up myself. I completely understand how the NEAT algorithms work in theory, it's just turning that theoretical knowledge into an actual working AI that's proving tricky and a fun challange. I could've used a pre-existing NEAT AI system, but there were a few features (Like saving a trained structure and reloading it later, as well as an interface for the code) that I need to add later on, which not many online NEAT scripts have (Probably for good reason, but I'm determined to make this work nomatter how long it takes).

I tried using "this" earlier and it was throwing errors, but I think I just misunderstood how to use it at the time. That has been corrected thankyou!

I'll have to keep that in consideration for later. While it isn't an immediate problem, I will be fixing that as soon as I can to stop issues later on.

I have setup a whole system to handle the structure of nodes, as well as a network of structures. It's just that everything boils down to the singular node that isn't operating correctly. I could drop the entire code for everything here if you want me to, but I think you'll find it isn't gonna help much past the original node script.

The current node setup is simply 2 nodes. An input, and the output that's doing the learning. IN THEORY what is supposed to happen is that the node realises that the closer to a weight of 1 it gets (Or just the larger it makes that weight), the higher fitness it gets. That's it. It should overtime approach a higher value, but it simply doesn't. It's not even a complex calculation. "Alter the weight slightly in a random direction. Did it increase fitness? If no, revert back to the better working one, then try again. If yes, keep it and try again."

I know that I can make the nodes WAY more efficient by determining the direction and amount to change the weights by overtime by doing some pretty aggressive calculations, but just for now even a super simple node like this should work. I'm just curious why it isn't learning at all. It doesn't seem to be reverting to a version of itself that worked better. I'll try to figure out why now that it's been a couple weeks off working on other projects. If I can get this to work well, then once I've optimized the code, I'll be sharing it

 

maybe you've got a basic error somewhere and I cannot see it. that's a possibility.

there is also a possibility that it finds the equilibrium immediately and negligibly oscillates around it, simply because the fitness feedback has too much weight when you use just one other node. a synthetic latency plays a major role in neural networks, that's why connectivity matters, and why they always include several hidden layers to densely pack a massive grid in order to defer feedback transmission. in other words, maybe it works as expected.

neural networks are weird in that they behave like people in social settings do. like making a wave on a stadium, right? one man's opinion doesn't mean much, but together their relative positioning is conductive to some abstract meaning.

after a while they will begin storing fluctuation residues of this random conduction, based on their own geography, and this is used as a resistance for the next generation feedback. I mean it's really weird if you think about it, social systems are de facto neural networks.

anyway, I'll be glad to help if I can. ping me if you make another iteration of this, I'll take a look.

 

I'VE FOUND IT! You weren't far off, and it was a pretty simple error hiding in plain sight! Line 47 and 49 I've put "prevNodes = connectedNodes;" and "connectedNodes = prevNodes;" respectively.

By doing this, both prevNodes and connectedNodes is referring to the exact same dictionary object. So, if I change something in connectedNodes, it also changes it for prevNodes. SO, when the software notes that the fitness was lower, it says "Set connectedNodes to the same object as prevNodes", which it already was.

What I needed to say instead isn't "prevNodes refers to the connectedNodes object", but instead "prevNodes should be a COPY of the connectedNodes object" and vice versa. Suddenly, the code instantly starts to work!

So I just change it from:
connectedNodes = prevNodes;
To:
connectedNodes = new Dictionary<Node, float> (prevNodes);

And it works! I've still got to do some work on the node structure software, but the project is actually super close to completion now that that issue is solved!

 

Great!
I couldn't see that one, but now that you've put it on a plate, it's kind of obvious

I just don't like that you're allocating a new dictionary every time, you should optimize that somehow down the line.
Of course, after you've done with the bare bones.

 

Yeah, I still got quite a way to go, but this is designed to be an easy to use generic NEAT AI that can save and load trained structures so that you don't have to retrain every time you turn off your PC. Optimizations will come later DEFINITELY, but for now I want to get it working.

By having it generic, it should be able to learn pretty much any task, and be easy enough to interface with that even a beginner could learn to use my AI script. It's a one-script AI. Everything in the one script. Multiple classes (One for the node, one for the evolutionary structure, and one for the network of structures) sure, but all in one script. I shall return in a couple weeks once it's done and share my code to the world!

 

And here I am just "a couple weeks later" (Exactly 666 days later XD) with my result. Still need to actually rewrite NEAT_AI.Node.GetNodeOutput() so you can have different types of nodes, but to anyone who was wondering, that's my code so far