Question OnActionRecieved() Discrete Actions Always Returns Zero

I have been struggling with this issue for a while now. My OnActionRecieved() function only returns zeros. The continuous actions work okay, but the discrete actions are where this issue arises. I am using the latest python version(30.0). I am manually stepping the environment and requesting decisions.

Below is my config file. The only thing I have changed in an attempt to fix this issue is set 'normalize' to true:

behaviors:
ShootingMLAgent:
trainer_type: ppo
hyperparameters:
batch_size: 10
buffer_size: 100
learning_rate: 0.0003
beta: 0.0005
epsilon: 0.2
lambd: 0.99
num_epoch: 3
learning_rate_schedule: linear
beta_schedule: constant
epsilon_schedule: linear
network_settings:
normalize: true
hidden_units: 128
num_layers: 2
reward_signals:
extrinsic:
gamma: 0.99
strength: 1.0
max_steps: 500000
time_horizon: 64
summary_freq: 10000​

 

Since I haven't received any responses, I will use continuous actions instead and clamp them. Is this an okay thing to do, or will it impact performance?

 

what's your branch size? should be 2.

 

I'm actually so dumb. Thank you so much!

 

I have the same problem and branch size didn't fix it. Discrete actions always 0 ???

 

are you sure your agent is actively learning and it's not running in heuristic mode?
have you masked any actions?
show us your behaviour params and the agent code if it's still not working

 

public override void OnActionReceived(ActionBuffers actionBuffers)
// public override void OnActionReceived(ActionBuffers actions)
{
// Perform actions based on the received vectorAction
shouldFire = actionBuffers.DiscreteActions[0] == 1;



Debug.Log("Discrete Action [0] = " + actionBuffers.DiscreteActions[0]);

// shouldFire = actions.DiscreteActions[0] == 1;

if (shouldFire)
{

for (int i = 0; i < Missiles.Length; i++)

 

public override void OnActionReceived(ActionBuffers actionBuffers)
// public override void OnActionReceived(ActionBuffers actions)
{
// Perform actions based on the received vectorAction
shouldFire = actionBuffers.DiscreteActions[0] == 1;



Debug.Log("Discrete Action [0] = " + actionBuffers.DiscreteActions[0]);

// shouldFire = actions.DiscreteActions[0] == 1;

if (shouldFire)
{

for (int i = 0; i < Missiles.Length; i++)
{
GameObject missileObject = Missiles;
if (missileObject.transform.position.y > 1 && missileObject.tag == "Missile")
{

// Calculate the direction vector from the missile to the agent
Vector3 missileToAgentDirection = (me.transform.position - missileObject.transform.position).normalized;

// Calculate the dot product between the missile's velocity and the missile-to-agent direction
float missileDirectionDot = Vector3.Dot(missileObject.GetComponent<Rigidbody>().velocity, missileToAgentDirection);

Vector3 objectToPlayer = me.transform.position - missileObject.transform.position;
Vector3 objectVelocity = missileObject.GetComponent<Rigidbody>().velocity;

missileObject.GetComponent<Launch>().dot_Product = Vector3.Dot(objectToPlayer.normalized, objectVelocity.normalized);
if (missileObject.tag == "Missile") // Randomly Takes a shot
{


if (missileObject.GetComponent<Launch>().dot_Product > 0.5) // Towards
{
DrawLine(me.transform.position, missileObject.transform.position, Color.green, 0.05f);
//missileObject.tag = "Shot";
AddReward(1.5f);
missileObject.tag = "Correctly Shot";
}

if (missileObject.GetComponent<Launch>().dot_Product < 0 && missileObject.tag == "Missile") // Away from
// if (missileObject.GetComponent<Launch>().dot_Product < 0) // Away from
{
// DrawLine(me.transform.position, missileObject.transform.position, Color.red, 0.05f);
AddReward(-0.5f);
missileObject.tag = "Wasted Shot";
//wasted_shots++;
}

if (missileObject.GetComponent<Launch>().dot_Product > 0 && missileObject.GetComponent<Launch>().dot_Product < 0.5 && missileObject.tag == "Missile") // Adjacent to
{
// DrawLine(me.transform.position, missileObject.transform.position, Color.yellow, 0.05f);
missileObject.tag = "Wasted Shot";
AddReward(-0.2f);
// wasted_shots++;
}


Vector3 center = dome.transform.position;
// float radius = dome.transform.localScale.x / 1.5f; // Assuming a uniform scale
float radius = dome.transform.localScale.x / 2; // Assuming a uniform scale
// Calculate the distance between the point and the center of the sphere
float distance = Vector3.Distance(missileObject.transform.position, center);
if (distance <= radius)
{
close = true;
}

if (close == true) // Towards
{
DrawLine(me.transform.position, missileObject.transform.position, Color.blue, 0.05f);
AddReward(1.0f);
missileObject.tag = "Correctly Shot";
close = false;
}

}




} // Above Ground and Tag = Missile



correctly_shot = GameObject.FindGameObjectsWithTag("Correctly Shot");
wasted_shot = GameObject.FindGameObjectsWithTag("Wasted Shot");
landed = GameObject.FindGameObjectsWithTag("landed");
hit_defended = GameObject.FindGameObjectsWithTag("Hit_Defended");


}



//// Calculate the distance between the missile and the target
//float distance = Vector3.Distance(missile.position, target.position);

//if (distance <= firingDistance)
//{
// // Reward the agent for firing on an incoming missile
// AddReward(1f);
//}
//else
//{
// // Penalize the agent for firing on a missile moving away
// AddReward(-1f);
//}
}

// Perform any other necessary actions in the environment based on the agent's decision
}


public override void Heuristic(in ActionBuffers actionsOut)
{
// Allow manual control of the agent for testing purposes
actionsOut.DiscreteActions.Array[0] = Input.GetKey(KeyCode.Space) ? 1 : 0;
}

 

Your branch is 1. That would only be 1 value for both branches, which is 0. You need branch 1 to have a size of two and that would give you a 0 or 1.

The first number is how many branches (Discrete actions) you need.
The second number is how many values each of the discrete actions have.

 

I tried 1 branch with value of 2 that still gave value of 0 every time. I also tried 2 branches with 2 values in each, that also produced 0 every time. The Heuristic does work though !

 

I do see you access the buffer in Heuristic as actionsOut.DiscreteActions.Array[0] (I've never used the Array) and in the OnActionRecieved you use actionBuffers.DiscreteActions[0] (I use this when I do it)

Maybe try using .Array in OnActionRecieved or vice-versa?

 

I've tried both
public override void OnActionReceived(ActionBuffers actionBuffers)
public override void OnActionReceived(ActionBuffers actions)

And using an int to take the value but it's always 0 - it's like the Decision Requester is always returning 0 ?

Still can't figure it out !!

 

those are same lines of code, just different variable names for the Action Buffer.

actionsOut.DiscreteActions.Array[0] = Input.GetKey(KeyCode.Space) ? 1 : 0;
in your Heuristic should be
actionsOut.DiscreteActions[0] = Input.GetKey(KeyCode.Space) ? 1 : 0;

(I dont use this way)
or maybe try in your OnActionReceived change
actionBuffers.DiscreteActions[0] to be actionBuffers.DiscreteActions.Array[0]

 

It's still 0

See image

 

My project appears to have this issue. It outputs 0 for the first few decisions, but after that works correctly... You can try to make a few fake calls at the start and see if after that it will output something other than 0.

 

What do you mean by a few fake calls ? How do you do that ?

I've tried the Heuristic, and it works fine. It's the Decision maker / Discrete actions that's not working ?

 

Call agent's RequestDecision. (Assuming you don't already have it on auto-step and you've made a few fake calls anyway. Otherwise this may not be your issue.)

However, this is an issue I am experiencing with discrete action requests.

 

yes I've tried calling RequestDecision in a FixedUpdate loop, but no effect

 

Just to be clear, you have Behavior Parameters updated? It seems like it has to be that. Should be...
Continuous Actions = 0
Discrete Branch = 2
Branch 0 Size = 2
Branch 1 Size = 2