Search Unity

Archers, and the arrow problem!

Discussion in 'ML-Agents' started by m4l4, Sep 4, 2020.

  1. m4l4

    m4l4

    Joined:
    Jul 28, 2020
    Posts:
    81
    Hi everyone, i'm working on an archer agent, and the simulation is presenting some interesting problems.

    in my previous projects, i've always used raycast to shoot at targets. you click, and boom, target gets destroyed and you get the reward. With arrows, there's quite a delay between click and get rewarded, so the agent has an hard time understanding what gave him points. (the credit assignment problem).

    i'm trying with visual observation, agent is stationary, 1 arrow, it can only rotate 45deg left and right, and aim forward up to 90deg. target is spawned, once the arrow hits something, it passes the reward to the agent based on it's distance from target and EndEpisode();

    it's the first time i use a camera sensor, and i don't know if i'm doing something wrong, but the agent after half milion step, hasn't showed ANY sign of progress, still shooting randomly. I'm no expert, but it's the first time i experience zero progress
    I've read that visual observation should be used only if the situation is difficult "to explain" with vectors, because it's slower and unreliable in some cases.

    With vectors, the problem persist, the arrows flies up, outside the range of a 3d raycast sensor, the agent looses track of it. I could pass distance and direction of the arrow as inputs, but i don't know if that would solve the problem.

    How would you approach that?
     
  2. wshigham

    wshigham

    Joined:
    Dec 30, 2019
    Posts:
    14
    I haven't tried this type of problem, but a couple of things I'm thinking of
    - In a difficult problem like this you might need to use curriculum learning, starting with rewarding the agent just for shooting in the general direction of the target? How about a small reward at the time of firing for how close to the ideal fire angle they selected, and then a large reward if they hit?
    - Can you directly also pass the agent the position information of the arrow, or a vector of the arrow relative to the agent?

    To be honest if I was trying this problem I wouldn't have thought to use cameras as input but instead
    - Vector to the nearest enemy
    - Velocity of the nearest enemy
    - Agent Velocity (if moving)

    Or with your raycast solution for enemy detection, that should work too, but maybe pass the relative position of arrow instead of relying on that being detected by the rays.
     
  3. m4l4

    m4l4

    Joined:
    Jul 28, 2020
    Posts:
    81
    the idea of using cameras as inputs, comes from the limitation of the raycast sensor. if i want the agent to perceive a target on a wall, i need another raycast sensor placed somewhere over its head (like in the walljumper example), but then, if the agent is on the wall, it doesn't see what's below, and so on. The camera would have solved the issue of identifying things in a 3d space, instead of in a 2d plane like raycast does.
    Also, i'd really like to give a try at robotic implementation, and learning how to use pixel inputs seems rather important.
     
  4. wshigham

    wshigham

    Joined:
    Dec 30, 2019
    Posts:
    14
    With the raycast I think you cast a sphere with a fair sized radius to make it less of 2D vision and more like area vision.

    If you want to experiment with pixel inputs, I think that's great, I just don't know that this game idea is the place to start. It seems quite complex for pixel input. How about start with something simple like a tic tac toe game with a camera above the board? I thought of that as it is kind of similar to the Unity example of camera input (X vs O detection) so should be possible.
     
  5. mbaske

    mbaske

    Joined:
    Dec 31, 2017
    Posts:
    473
    I don't think your agent needs to observe the arrow's flight path at all, because it's completely deterministic. Which is to say the flight path is implicit in the arrow's initial direction and velocity when it is shot. Have you tried observing these values, in addition to the target direction and distance from your agent? That would be all vector observations, no raycasts or cameras.
     
  6. koirat

    koirat

    Joined:
    Jul 7, 2012
    Posts:
    2,070
    It's a parabolic aim at moving object. Equation is not easy do calculate if you don't know what you are doing so better search internet for it.
     
  7. m4l4

    m4l4

    Joined:
    Jul 28, 2020
    Posts:
    81
    spherecast can be useful to detect object in a wider space, but the rays will still return just a distance and a tag, it won't tell you if the target is up or down.

    i think you might be right... that doesn't solve the issue of detecting object up or down a stair, but, target distance, direction, and bow angle (arrow initial velocity is constant, no pulling strings for now) should be enough to solve the problem