Search Unity

  1. Improved Prefab workflow (includes Nested Prefabs!), 2D isometric Tilemap and more! Get the 2018.3 Beta now.
    Dismiss Notice
  2. The Unity Pro & Visual Studio Professional Bundle gives you the tools you need to develop faster & collaborate more efficiently. Learn more.
    Dismiss Notice
  3. Improve your Unity skills with a certified instructor in a private, interactive classroom. Watch the overview now.
    Dismiss Notice
  4. Want to see the most recent patch releases? Take a peek at the patch release page.
    Dismiss Notice

Alpha Pilot AI Drone Racing could Unity be used to build an AI for this?

Discussion in 'General Discussion' started by Arowx, Sep 13, 2018.

  1. Arowx

    Arowx

    Joined:
    Nov 12, 2009
    Posts:
    6,462


    So the Drone Racing League race super fast drones driven in FPS mode by human pilots and Lockheed Martin have started a challenge for teams to make AI pilots that can outrace the best humans.

    The thing is if the visual processing/scanning system is not a major factor then (Nvidia are sponsoring the event and could provide Turing level visual processing capabilities) then the only thing you need to make is an AI capable of flying as fast as possible around a course.

    In theory, it sounds like Unity could be a good tool to develop and test an AI Drone Racer, what do you think?

    Is Unity fast enough (DRL style drones can fly at over 179 mph about 80.2m/s)?



    Is Unity ML fast and smart enough to compete?

    Challenge Site For More Info -> https://www.lockheedmartin.com/en-us/news/events/ai-innovation-challenge.html
     
  2. ShilohGames

    ShilohGames

    Joined:
    Mar 24, 2014
    Posts:
    2,033
    Why would you want to use Unity for that? It would just be extra overhead.
     
    BlankDeed likes this.
  3. Kiwasi

    Kiwasi

    Joined:
    Dec 5, 2013
    Posts:
    15,536
    Unity does have the libraries to handle 3D spaces. It would be faster to use Unity then to develop your own 3D space handling system. Its not a perfect fit, but the extra overhead really shouldn't be an issue here.

    Note: You wouldn't use ML for this. A hard coded AI would do the job better.
     
  4. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    1,427
    For fast moving drones, you are likely want to work with high responsive systems. Which more likely you want to work on micro controller directly for controlling rotors.
    However, I could see Unity use, for path planning and general purpose AI, which communicates with drone, to send commands. Even could use swarm control with visualization via Unity. I see no issue with that.
     
  5. ShilohGames

    ShilohGames

    Joined:
    Mar 24, 2014
    Posts:
    2,033
    I would suggest designing an embedded system that either ran code bare metal or possibly on top of a minimal Linux build of some kind. Unity could definitely be used to visualize what was going on remotely. But I would definitely not recommend using Unity as the core processing solution for a device located in the drone.
     
    zombiegorilla likes this.
  6. neoshaman

    neoshaman

    Joined:
    Feb 11, 2011
    Posts:
    3,415
    questions:
    - is the track fixed?
    - is there many tracks?
    - is the track reveal at competition time (so essentially random)
    - is teh track always looping?
    - if it loops, how many loop?
    - will there be always bright neon indicator?
    - is the neon sufficient for blind navigation or there is a need to know the sequence of traversal if the track intersect itself (for example goes below the arch, the hole on top after the curve sequence).
    - is all races in indoor environment with industrial gray decay, or there is a variety of environment from green golf outdoor, to parking mall in a windy area?
    - is the drone hardware fixed or you build yourself?
    - is there specs to respect to build the hardware?
    - what's the length of a race?
    - how many camera are allowed?
    - can camera or compute decoupled from teh drone?

    It can be as simple as using openCV under some condition, though I think ML can assist to finesse somethings like air turbulence management (things you won't have simulate in unity) rather that take care of everything or a full image processing.
     
  7. bluescrn

    bluescrn

    Joined:
    Feb 25, 2013
    Posts:
    431
    You could perhaps use Unity to build a simulation in which to test very early versions of your AI....

    Does seem an interesting challenge... do you try to pack enough processing power onto the drone itself (where you have direct access to and HD camera and other sensors), or try to stream enough camera/sensor data from the drone to a big powerful PC and control it remotely?

    From what I've seen of FPV drone video feeds, they're usually analog/SD and very prone to noise/interference, so not a great input to a computer video system. And you'd be very limited by weight/size for other sensors (along with the drone's very fast rate of movement/rotation)
     
  8. Arowx

    Arowx

    Joined:
    Nov 12, 2009
    Posts:
    6,462
    I'm guessing but I would expect the drones are the same as those used by the players so that you only get a video feed and controller style inputs. The thing is you would need some next level visual processing AI on par with Nvidias Smart car hardware, therefore I would expect that is built into the challenge as it takes a lot of training data and processing power to train a visual recognition system.
     
  9. ShilohGames

    ShilohGames

    Joined:
    Mar 24, 2014
    Posts:
    2,033
    I would actually assume the AI based drones would have a completely embedded solution instead of sending the video back to a desktop. If any ML was used, it would be trained offline.
     
  10. Arowx

    Arowx

    Joined:
    Nov 12, 2009
    Posts:
    6,462
    I don't think there is ML hardware small enough, fast enough and with low enough power usage to process the video onboard, e.g. Nvidia's AGX processor is a small PCI board with memory and IO chips far too large for a DRL drone.



    105 mm x 105 mm

    Which is probably larger and heavier than the DRL Drone (800 grams).

    And there are already RF video solutions that the pilots use.

    Also adding any kind of LIDAR or additional sensors would probably weaken the drone's performance massively.
     
  11. Murgilod

    Murgilod

    Joined:
    Nov 12, 2013
    Posts:
    3,619
    Why would you use full video processing instead of an IR blaster array
     
  12. Arowx

    Arowx

    Joined:
    Nov 12, 2009
    Posts:
    6,462
    A TV remote controller?
     
  13. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    1,427
    You are likely loose on important data, due to low resolution. Kinect 1 had IR array. Is ok for short distance, otherwise you need a lot of dense array IR nodes.

    You could use easily camera with filter, to provide just gray scale image. Is more likely reduce amount of computation.
    I don't know about NVidia solutions, but ARM micro controllers are pretty powerful beasts. You probably don't need high frequency of capturing images. Gyroscope provides necessary high responsive position/orientation feedback. Together with camera imaging, probably you can resolve navigation issue.
     
  14. ShilohGames

    ShilohGames

    Joined:
    Mar 24, 2014
    Posts:
    2,033
    There is no reason to place that board in a drone, but there are plenty of embedded options that would be much smaller and lighter that could work for the task.

    The key would be to find out for sure what the rules are.