ML-Agents v1.0 and Roadmap Update (Mar 19, 2020) - Discussion Thread

Congratulations to the team on such constant rapid improvements, and on a job well done so far! (you guys put out updates like they're going out of style!)

I'm looking forward to migrating everything into sensor classes for better organization. My ultimate goals will involve very highly dimensional and multi-modal observation spaces, and the existing iSensors are already starting to make that easier to manage.

Regarding C# refactoring, I'm not sure there's too much room for improvement in terms of simplicity, but one thing that might be useful is to include some optimized array functions for c# side data handling.

Lately I have become concerned about the scalability of visual observations. Deploying in-game cameras involves aspects of rendering that I don't at all understand. If possible, some guidance on best practices when using cameras (how to make rendering cheaper and efficient) would be very helpful (perhaps some options on the iSensor camera component that can override the camera object settings to ensure speed at the cost of detail or effects).

It would also be excellent if some generic filters could be made optional in the camera sensor (such as a depth based grey-scale). Using visual observations is one of the more intriguing parts of ml-agents, but it is also difficult to get right, and resource intensive if not done efficiently.

 

In older versions of ML-Agents it was possible to access the observations in the (back then: ) "Heuristic Brain".

In 0.14 I had to make m_Observations in VectorSensor public, in order to make heuristic decions based on obervations.
Is there a more elegant way than this? The examples all seem to use direct controller-input.

I would find it useful to have access to those observations in the first release candidate again. If I'm just missing something here: disregard everything I said ;-)

 

There are online video tutorials for ML Agents, but they place Academy and Brain objects on the scene attached as MonoBehaviours. They also discuss various Brain types, which is no longer the case. This also happens on blog posts in Unity blog.

I think documentation should mention such things where appropriate, e.g.:

• Starting with version 0.14, you no longer place Academy onto the scene. Instead, it is now a singleton, which only can be accessed via code.

• Starting with version 0.11, you no longer create Brains as either ScriptableObjects or attached scripts. Instead, you attach Behavior Parameters component to an agent and give them a name. If you want for several agents to share a brain, give Behavior Paramters the same name.

This was really confusing when I first started using ML Agents.

 

Yay! Since 0.3 I was looking forward to 1.0 release. Really excited for this.

I would love to see more attention given to the onboarding process of unity-ml users, especially if we assume you are not targeting RL experts. Some ideas could be a list of "gotchas". Things like, if you only give the position information every frame, the agent can not figure out the velocity, because it doesn't have a sense of memory. (I see this pretty often especially in games that don't use rigid bodies) Or how to write code that stays stable when the simulation speed increases. That is usually forgotten and causes things to go badly.

Is the team looking to support using different learning backend as a first-class feature? That is to say, the env runs through unity-ml but the neural network training is done in python written by some other entity. I always find myself having to write some code to the unityml library base to get this workflow to function, but I also understand this is an edge case.

I also usually expose more of the variables on the C# side of things, I don't recall what exactly but I think it was related to rewards, or steps taken. (might not be either, but what I was looking for wasn't too exotic either)

Finally, I think just general stability and maybe a bit more helpful error messages. In my last pass of the library, I had issues related to this and this. Some might be user error, some might be genuine bugs, but the cases where you anticipate a user error pointing to what might be the problem would be super useful.

I would be down if you are looking to have a more official qualitative data collection session from your users.

Overall, thank you for work! Unity ML got me into reinforcement learning and now I am an addict!

 

Apparently, 80% or more of the performance in neural networks comes from 20% or less of the nodes and synapses (in ml). Some architectures exploit this for efficiency at deployment time by finding ways to atrophy and "drop out" the unnecessary connections.

Another similar function can be used to harden the weights of the important synapses (instead of dropping our the rest), such that transfer learning wont destroy the initially learned task.

Dropout could be especially useful for making very effective and efficient networks for deployment....

Just food for thought!

 

This is awesome! The ML-A framework is simply amazing, thank you!

 

Wish it was more easy to run this ML agents on enviroments such as docker and google collab

 

I just merged https://github.com/Unity-Technologies/ml-agents/pull/3825 which will be in the next release. You can now call Agent.GetObservations() to get a read-only list of the observations that were made in CollectObservatoins.

 

Hey I've been using ML Agents since it was first released, the progress has been awesome and many of the newest features are helpful. I think there are a handful of things that would make it easier to use:

• counting how many vector observations can be tedious when making iterative changes to an agent, the warnings about expecting a number of observations and receiving another number are helpful, but if this could be more automatic it would be helpful.

• I use this Unity package Squiggle to better understand the observations I'm sending my agent, https://assetstore.unity.com/packages/tools/utilities/squiggle-21970 without this I feel like development would be extremely more difficult. Some kind of native tool to debug both the inputs and outputs to the network would be really great.

• Maybe this is already a feature and I've somehow overlooked it but it would be great if there was a way to log custom data into the TensorBoard without getting into the python API, for example, logging the average number of goals scored in a game or some other environment specific diagnostic. My biggest time sink when developing with ML Agents is training environments that are seemingly performing well according to the training statistics, but not behaving as intended. Having more insights like this during training could help communicate something has gone wrong to the developer and allow them to stop training and fix the problem before sinking 12-24 hours into training.

• More error checking in general would be great, caching NAN observations or rewards is good but if you accidentally divide by zero think it still accepts Infinity as an observation , then eventually the network starts outputting NAN.

 

They added a function to count the number of observations in 1.0