Need Ideas & Direction for complex environment (e.g: wind, currents, smell, etc...)

Greetings!

i have got some complex agents that are starting to take shape, and I find myself in need of several environment mechanics that are non-trivial to implement (mathematically, and programmatically).

My agents are muscle-controlled and physics based, and one of their major senses is smell (smell is represented with an N dimensional vector, where each dimension is a particular chemical concentration). Smells can be high level, or nuanced, (e.g: hard coded channels for superficial smells like "food" or "apple" or "predator" instead of using a chemical signature for unique smells (which would consist of a certain ratio of chemical concentrations).

This is all well and good, but implementing the actual smell system presents challenges. Firstly, if only using a function of the distance between a nose and smell emitting object to determine chemical concentration, then this creates an unrealistic and noiseless signal (and one that travels through walls). In the real world, scent is carried around with wind currents and atmospheric mixing, and this creates scent trails, among other effects (this kind of added dynamism is the kind of complexity I would like my agents to work in).

Secondly, once we start implementing hundreds of smell emitting objects, and hundreds of noses, the amount of distance calculations (like a fully connected network) becomes a computational bottleneck...

As a solution, I am imagining some sort of 3d grid based system as a way to granularize and calculate smells in the environment. Each smell could deposit into the local grid coordinate it is located in, and the grids could trade and equalize their chemical concentrations over time (a little bit on each tick). This way, when a nose requests the current smell from the smell manager, it just needs to look at the smell emitters that are also in the local grid space, and also the neighboring grids...

This kind of grid space would also let me implement basic temperature and currents (which could affect smell transfer). The grids could trade temperature (with a sunlight input to the system, conduction + convection within it, and thermal radiation back out into *space*), and temperature would create pressure, which could generate wind. It would be somewhat crude, and I'm not sure just how computationally viable it is for a very large environment (it has the scaling problem of environment size/granularity rather than agent count).

I'm interested in both wind and water currents, especially where it will allow my agents to actually create wind and air currents of their own due to their actions. Before comitting to what I've described above, I felt it necessary to ask for advice and opinions about the approach, and what alternatives I might have.

Many of the agents I plan to make will depend highly on smell, and also need to deal with strong wind or water currents, so whatever I can do do make it has efficient or realistic (or interesting) as possible is something I need to explore.

Below is one of the agents I'm working on. It's a muscle and physics driven fish (lots of senses, including vision and smell). Since vision is sort of expensive, and since fish have very poor eyesight to begin with, I really want to get smell and the current forces right (see" fish + lateral line")...

 

This is beyond the realm of ML Agents, but I used to study this in grad school, so I'll try to give some pointers.

You're basically getting into numerical simulation for partial differential equations. The simplest form for what you're describing is the convection-diffusion equation, which would dictate how the quantities like concentration move in the currents (convection) and spread out (diffusion). The wikipedia article isn't that great for learning how to implement, but hopefully that's a good starting point to google other sources. One other thing to be careful of is the CFL condition, which relates how your spatial grid and your timesteps have to be scaled to avoid instability.

It's been a few years, so I'm not sure if there are better textbooks for this nowadays, but I really liked A First Course in the Numerical Analysis of Differential Equations by Iserles.

Once you get into simulating currents themselves, that gets a lot trickier since you're dealing with the Navier-Stokes equations. These look scary, but if you understand the convection-diffusion equations, they're basically saying that the velocity of the fluid also gets convected and diffused. For water, you also need to have some additional incompressibility constraints; for air, you don't need this but it might make things simpler and it's not unreasonable for things like wind.

"Stable Fluids" by Jos Stam is the best reference that I remember for this in terms of implementation. Here's another followup by the same author that has C code and (IIRC) is a bit more game-focused.

Sorry I don't have anything more recent; there might be a few advancements since then, but it's still fundamentally a hard problem (both mathematically and computationally)...

 

These sources are amazing to read because I actually have experience with this type of simulation!

I made a 3D grid based thermodynamics simulator that incorporated convection, conduction, and also radiation! (absorption and emission). Reading the CFL condition is absolutely hilarious, because I have vivid memories of long and frustrating nights spent confronting the same type instability. In the end, my solution to the CFL condition was first to make all the influence exchanges synchronous (by having the exchanges only apply deltas to a new buffer value, and having all nodes update to their new working value only after all exchanges are one for that tick), because this makes the simulation behave symmetrically (the order of exchanges can alter results and lead to strange flows and asymmetries if this is not done). The only catch is that the magnitude of the time-step needs to be small enough or the rate of exchange slow enough) to ensure that equilibrium points aren't skipped over (imagine one cube surrounded by 6 cubes (one for each face). If the center cube is very cold, and the outer cubes are very hot, then they outer cubes will each dump a large amount of heat into the center cube on the next tick. If the amount they dump is greater than 1/6th of the temperature difference, then because the middle cube is getting heat from 6 sources at once, its temperature would become unrealistically high). My simulator was pretty darn accurate too despite the granularity. It was to simulate glass and steel space stations (with or without mylar), and the gas inside them. Here is a big cube made of smaller cubes,with light striking it from the top



I wrote this in JS, but it was actually a passion project to help out with a Unity game called Stationeers . I learned a ridiculous amount of physics in the process! What stuns me is just how similar this is to the basic solver approach that so many others have converged toward.

------

I do realize that this is beyond the scope of ML-Agents, but that is only true for a limited time

The most rewarding long term potential of the tool-kit is that its architecture (and how it gets used) will evolve toward greater complexity. Creating ecosystems can never be good unless the system itself is complex and dynamic (we don't need bona fide butterfly effects, but the way agents can affect their environment, and the non-linear ramifications this can bring, is a central part of giving agents more interesting information and problem spaces (room for creativity), and also a non-obvious part of our ability to actually see/recognize and evaluate learned results).

While my needs are somewhat specific (currents, smells, temperature, etc...), a basic solver system for environment dynamics could serve an uncountable number of multi-agent (or even single agent) scenarios. There may actually be many significant benefits from a strict RL point of view; this kind of solver system would introduce both noise and signal from the perspective of an individual agent; some current would just be turbulence or macro environmental trends, but others (light currents or oscillations) might have been generated by another fish (this is actually the main and most important way most fish perceive and identify each-other). Some of the currents and effects will also have been caused by the agent itself, which would also give it extra and noisy feedback about its own actions.

The links you provided give me some great starting points. I'm getting close to actually trying to pull it off in Unity, but I'm still assessing my options.