Search Unity

Bug Inference Failing with Error: "AssertionException: Assertion failure. Values are not equal."

Discussion in 'ML-Agents' started by ajdm432, Mar 21, 2023.

  1. ajdm432

    ajdm432

    Joined:
    Jul 8, 2019
    Posts:
    1
    Hello,
    I'm training a PPO network and had successfully trained and tested multiple networks already. I just completed my 6th training run and wanted to see how the network would perform on inference. I added the .onnx file from training to the project, put it on the agent prefab, switched the prefab to "inference only" and tried to run. I got the following error:

    AssertionException: Assertion failure. Values are not equal.
    Expected: 105 == 106
    UnityEngine.Assertions.Assert.Fail (System.String message, System.String userMessage) (at <86acb61e0d2b4b36bc20af11093be9a5>:0)
    UnityEngine.Assertions.Assert.AreEqual[T] (T expected, T actual, System.String message, System.Collections.Generic.IEqualityComparer`1[T] comparer) (at <86acb61e0d2b4b36bc20af11093be9a5>:0)
    UnityEngine.Assertions.Assert.AreEqual[T] (T expected, T actual, System.String message) (at <86acb61e0d2b4b36bc20af11093be9a5>:0)
    UnityEngine.Assertions.Assert.AreEqual (System.Int32 expected, System.Int32 actual) (at <86acb61e0d2b4b36bc20af11093be9a5>:0)
    Unity.Barracuda.UnsafeArrayCPUOps.Dense (Unity.Barracuda.Tensor X, Unity.Barracuda.Tensor W, Unity.Barracuda.Tensor B, Unity.Barracuda.Layer+FusedActivation fusedActivation) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/Core/Backends/BarracudaUnsafeArrayCPU.cs:1456)
    Unity.Barracuda.StatsOps.Unity.Barracuda.IOps.Dense (Unity.Barracuda.Tensor X, Unity.Barracuda.Tensor W, Unity.Barracuda.Tensor B, Unity.Barracuda.Layer+FusedActivation fusedActivation) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/Core/Backends/StatsOps.cs:84)
    Unity.Barracuda.GenericWorker+<StartManualSchedule>d__33.MoveNext () (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/Core/Backends/GenericWorker.cs:251)
    Unity.Barracuda.GenericWorker.Execute () (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/Core/Backends/GenericWorker.cs:160)
    Unity.Barracuda.GenericWorker.Execute (System.Collections.Generic.IDictionary`2[TKey,TValue] inputs) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/Core/Backends/GenericWorker.cs:145)
    Unity.MLAgents.Inference.ModelRunner.DecideBatch () (at Library/PackageCache/com.unity.ml-agents@2.0.1/Runtime/Inference/ModelRunner.cs:213)
    Unity.MLAgents.Policies.BarracudaPolicy.DecideAction () (at Library/PackageCache/com.unity.ml-agents@2.0.1/Runtime/Policies/BarracudaPolicy.cs:125)
    Unity.MLAgents.Agent.DecideAction () (at Library/PackageCache/com.unity.ml-agents@2.0.1/Runtime/Agent.cs:1360)
    Unity.MLAgents.Academy.EnvironmentStep () (at Library/PackageCache/com.unity.ml-agents@2.0.1/Runtime/Academy.cs:578)
    Unity.MLAgents.AcademyFixedUpdateStepper.FixedUpdate () (at Library/PackageCache/com.unity.ml-agents@2.0.1/Runtime/Academy.cs:43)


    I attempted to do a very short training run and run the resulting model, thinking that I had accidentally changed something in the environment, however I received exactly the same error running inference on the new model. I tried running inference on an old model which had worked a few days ago and got this separate error:

    IndexOutOfRangeException: Index was outside the bounds of the array.
    Unity.Barracuda.Tensor.set_Item (System.Int32 b, System.Int32 ch, System.Single value) (at Library/PackageCache/com.unity.barracuda@2.0.0/Barracuda/Runtime/Core/Tensor.cs:2091)
    Unity.MLAgents.Sensors.ObservationWriter.AddList (System.Collections.Generic.IList`1[T] data, System.Int32 writeOffset) (at Library/PackageCache/com.unity.ml-agents@2.0.1/Runtime/Sensors/ObservationWriter.cs:151)
    Unity.MLAgents.Sensors.RayPerceptionSensor.Write (Unity.MLAgents.Sensors.ObservationWriter writer) (at Library/PackageCache/com.unity.ml-agents@2.0.1/Runtime/Sensors/RayPerceptionSensor.cs:326)
    Unity.MLAgents.Inference.ObservationGenerator.Generate (Unity.MLAgents.Inference.TensorProxy tensorProxy, System.Int32 batchSize, System.Collections.Generic.IList`1[T] infos) (at Library/PackageCache/com.unity.ml-agents@2.0.1/Runtime/Inference/GeneratorImpl.cs:276)
    Unity.MLAgents.Inference.TensorGenerator.GenerateTensors (System.Collections.Generic.IReadOnlyList`1[T] tensors, System.Int32 currentBatchSize, System.Collections.Generic.IList`1[T] infos) (at Library/PackageCache/com.unity.ml-agents@2.0.1/Runtime/Inference/TensorGenerator.cs:173)
    Unity.MLAgents.Inference.ModelRunner.DecideBatch () (at Library/PackageCache/com.unity.ml-agents@2.0.1/Runtime/Inference/ModelRunner.cs:204)
    Unity.MLAgents.Policies.BarracudaPolicy.DecideAction () (at Library/PackageCache/com.unity.ml-agents@2.0.1/Runtime/Policies/BarracudaPolicy.cs:125)
    Unity.MLAgents.Agent.DecideAction () (at Library/PackageCache/com.unity.ml-agents@2.0.1/Runtime/Agent.cs:1360)
    Unity.MLAgents.Academy.EnvironmentStep () (at Library/PackageCache/com.unity.ml-agents@2.0.1/Runtime/Academy.cs:578)
    Unity.MLAgents.AcademyFixedUpdateStepper.FixedUpdate () (at Library/PackageCache/com.unity.ml-agents@2.0.1/Runtime/Academy.cs:43)


    Here is my version information:
    ml-agents: 0.26.0,
    ml-agents-envs: 0.26.0,
    Communicator API: 1.5.0,
    PyTorch: 1.13.0+cu116

    I would really appreciate some help with this issue. The error seemed to pop up out of nowhere and is affecting inference on models that were previously working. Please let me know if any other info would be useful.

    Thanks in advance!