Alpha Pilot AI Drone Racing could Unity be used to build an AI for this?

So the Drone Racing League race super fast drones driven in FPS mode by human pilots and Lockheed Martin have started a challenge for teams to make AI pilots that can outrace the best humans.

The thing is if the visual processing/scanning system is not a major factor then (Nvidia are sponsoring the event and could provide Turing level visual processing capabilities) then the only thing you need to make is an AI capable of flying as fast as possible around a course.

In theory, it sounds like Unity could be a good tool to develop and test an AI Drone Racer, what do you think?

Is Unity fast enough (DRL style drones can fly at over 179 mph about 80.2m/s)?

Is Unity ML fast and smart enough to compete?

Challenge Site For More Info -> https://www.lockheedmartin.com/en-us/news/events/ai-innovation-challenge.html

 

Why would you want to use Unity for that? It would just be extra overhead.

 

Unity does have the libraries to handle 3D spaces. It would be faster to use Unity then to develop your own 3D space handling system. Its not a perfect fit, but the extra overhead really shouldn't be an issue here.

Note: You wouldn't use ML for this. A hard coded AI would do the job better.

 

For fast moving drones, you are likely want to work with high responsive systems. Which more likely you want to work on micro controller directly for controlling rotors.
However, I could see Unity use, for path planning and general purpose AI, which communicates with drone, to send commands. Even could use swarm control with visualization via Unity. I see no issue with that.

 

I would suggest designing an embedded system that either ran code bare metal or possibly on top of a minimal Linux build of some kind. Unity could definitely be used to visualize what was going on remotely. But I would definitely not recommend using Unity as the core processing solution for a device located in the drone.

 

questions:
- is the track fixed?
- is there many tracks?
- is the track reveal at competition time (so essentially random)
- is teh track always looping?
- if it loops, how many loop?
- will there be always bright neon indicator?
- is the neon sufficient for blind navigation or there is a need to know the sequence of traversal if the track intersect itself (for example goes below the arch, the hole on top after the curve sequence).
- is all races in indoor environment with industrial gray decay, or there is a variety of environment from green golf outdoor, to parking mall in a windy area?
- is the drone hardware fixed or you build yourself?
- is there specs to respect to build the hardware?
- what's the length of a race?
- how many camera are allowed?
- can camera or compute decoupled from teh drone?

It can be as simple as using openCV under some condition, though I think ML can assist to finesse somethings like air turbulence management (things you won't have simulate in unity) rather that take care of everything or a full image processing.

 

You could perhaps use Unity to build a simulation in which to test very early versions of your AI....

Does seem an interesting challenge... do you try to pack enough processing power onto the drone itself (where you have direct access to and HD camera and other sensors), or try to stream enough camera/sensor data from the drone to a big powerful PC and control it remotely?

From what I've seen of FPV drone video feeds, they're usually analog/SD and very prone to noise/interference, so not a great input to a computer video system. And you'd be very limited by weight/size for other sensors (along with the drone's very fast rate of movement/rotation)

 

I'm guessing but I would expect the drones are the same as those used by the players so that you only get a video feed and controller style inputs. The thing is you would need some next level visual processing AI on par with Nvidias Smart car hardware, therefore I would expect that is built into the challenge as it takes a lot of training data and processing power to train a visual recognition system.

 

I would actually assume the AI based drones would have a completely embedded solution instead of sending the video back to a desktop. If any ML was used, it would be trained offline.

 

I don't think there is ML hardware small enough, fast enough and with low enough power usage to process the video onboard, e.g. Nvidia's AGX processor is a small PCI board with memory and IO chips far too large for a DRL drone.



105 mm x 105 mm

Which is probably larger and heavier than the DRL Drone (800 grams).

And there are already RF video solutions that the pilots use.

Also adding any kind of LIDAR or additional sensors would probably weaken the drone's performance massively.

 

Why would you use full video processing instead of an IR blaster array

 

A TV remote controller?

 

You are likely loose on important data, due to low resolution. Kinect 1 had IR array. Is ok for short distance, otherwise you need a lot of dense array IR nodes.

You could use easily camera with filter, to provide just gray scale image. Is more likely reduce amount of computation.
I don't know about NVidia solutions, but ARM micro controllers are pretty powerful beasts. You probably don't need high frequency of capturing images. Gyroscope provides necessary high responsive position/orientation feedback. Together with camera imaging, probably you can resolve navigation issue.

 

There is no reason to place that board in a drone, but there are plenty of embedded options that would be much smaller and lighter that could work for the task.

The key would be to find out for sure what the rules are.