Obversation struggle - Observing an 2 dimensional array int's with 5 statuses

Hey there!

So I have a question concerning the ML Agent's observation of an 2d array of integers.
As can be seen in the screenshot I created a set up for training an agent containing some randomly positioned "rooms" and the agents task is supposed to move and order them in a certain way, that they are arranged and neighbouring as asked (seen in visually represented by colors).
However my question is concerning the Observations, not the Reward. Because I was using ML Agents 8 for a long time without updating up to newer versions I wasn't aware about the introduction of the Grid Observations.
So I basically came up with my "own" grid I was planning to also manage the observations with.

My plan was to "simply" feed the int[,] of the "voxles/pixels" as observations in the network. Reading the documentation I stumbled across the AddOneHotObservation() function, which seems fitting for that.

So my approach was to simply iterate over all the values of the int[,] and add them on after an other:

Code (CSharp):

1.     public override void CollectObservations(VectorSensor sensor)
2.     {
3.         //observation of rooms "color"-status
4.  
5.         for (int i = 0; i < field.VoxelPositionArray.GetLength(0); i++)
6.         {
7.             for (int j = 0; j < field.VoxelPositionArray.GetLength(1); j++)
8.             {
9.                 sensor.AddOneHotObservation(field.VoxelPositionArray[i, j].Status, statusCount);
10.             }
11.         }
12.  
13.     }

But there is still one sentence in the documentation irritating me:

"Both Agent.CollectObservations() and ObservableAttributes produce vector observations, which are represented at lists of floats. ISensors can produce both vector observations and visual observations, which are multi-dimensional arrays of floats."

Later in the Documentation for the Grid Sensor the following is stated:

"During observations, the sensor detects the presence of detectable objects in each cell and encode that into one-hot representation. The collected information from each cell forms a 3D tensor observation and will be fed into the convolutional neural network (CNN) of the agent policy just like visual observations."

https://github.com/Unity-Technologi...nts.md#isensor-interface-and-sensorcomponents

So basically the question is, if it does make any difference for the Network, if I iterate over all observations and feed them in in a "List", respectively an 1DArray via the "one hot Observation" function or somehow collect the observations with one of the ISensors functions?
Or asked diffently: can I feed in an two dimensional array in the Network or did I understand it correctly, that it is anyways transormed into a "list"/1D array, when fed in?

And there is an other porblem: with my current Setup I would have 8000 observations and when I use them with the "normal" Vector observations they are all directly fed into a fully connected Network instead of a CNN I guess? Does that matter a lot? Is 8000 Inputs for a network without convolutions to many?

So what would be the best way in ML Agents in Unity to collect Observations in this particular case without fully rewriting the Setup for using the Grid Sensor or without using a Camera Sensor?
I could of course use a visual Observation with a Camera Sensor, but on one side I would like to extent that "experiment" later on also in 3 spatial dimensions which exludes using a Grid sensor and so I also don't really see a point in rendering an Image of it, although I already have the "raw" data I would need for it.

If anyone can help me with that, that would be really great!

 

@mahon123 How can one implement a two-dimensional tensor with .CollectObservations? I want to try to play with a custom grid sensor instead of using a default GridSensor component