Contextually highlighting objects based on player focus

The title is somewhat obscure, but I'm basically trying to implement a UI for my interactables similar to Life Is Strange, in which interactable objects highlight or display labels based on the level of player attention: things near the camera's focus get a simple label, and approaching an object eventually results in a more detailed label with command prompts. Actually displaying said labels seems pretty trivial: give each interactable a labelState that is either Off, Simple, or Detail, and execute a switch(labelState) which displays the desired UI, and listens for command inputs if the state is set to Detail.

What I'm not at all sure about is how to determine which objects the player is paying attention to, in order to set that object's state appropriately. I attached a gif of Life Is Strange's implementation, and it looks like they're rendering simple labels based on proximity to the camera center (which is presumably what the player will aim at what they're paying attention to), and switching to the detailed display based on a combination of proximity to the object and camera orientation (you don't get command prompts for an object you're standing directly next to unless you're also looking right at it).

The issue is, how do you efficiently determine which objects the player is paying attention to? If there are 10-15 objects inside the view frustum, I don't want to test distance to play for each object each frame, and I can't be using casting to determine where the player is looking, as you don't need to look directly at something to trigger it. I seriously doubt that trigger colliders are being used either, as the camera orientation needed to trigger a label seems to grow and shrink contextually based on how many objects are in the area. Is there an extremely simple way to manage this that I'm not thinking of, or was this likelier to be a pretty complicated state machine that chewed up a nontrivial amount of CPU?

 

If you give them a tag, and use FindGameObjectsWithTag, and then use a squared distance, instead of using square root, it might not be bad.

 

I could probably avoid making repeated Finds of any kind by making a manager class with a cached reference to each object in the scene, so all I'd need to do is iterate through each object in the manager and set their labelState based on some criteria.

Although now that I think about it, I don't exactly like the idea of using a manager: it seems much cleaner and less dependent to put each object in charge of managing its own UI by running something like

Code (csharp):

1.  
2. void Update(){
3. if (renderer.isVisible)
4.      ManageLabelFlags();
5. }
6.  

 

Why not let the highly-optimized PhysX library handle it? Create a series of trigger colliders, one each for Detail, Simple, and any other levels of focus, attached to the camera (or something similar). Use OnTriggerEnter and OnTriggerExit to keep track of when objects enter each trigger collider. When an object enters the Detail trigger, activate its Detail UI and deactivate its Simple UI.

 

Oh hey, that's a great idea... that removes the need for a lot of code, and is significantly harder to break. Thanks, Tony!

 

I'm doing an adventure game, first person, but I've found a manager is a necessity because of the gui handling, also, why have all those updates running on every object. You might need it on some, but probably not all.

 

Happy to help, @Sendatsu_Yoshimitsu !

I agree. If you use Unity UI and world space canvases for each UI, you don't need any updates. Just activate and deactivate the UI GameObjects as necessary, and that's it.

 

Do you mean the manager is a necessity? Because I'm not sure it's needed with your idea of using physics- I followed your suggestion and just added a detail and simple trigger to the camera, and gave each one a tiny script that just toggles the UI on and off on trigger enter/exit, and that seems to be more than enough to handle the interactable UI.

This is a much more rudimentary question, but is there a simple way to ensure that my triggers don't interact with anything except for interactable objects? I tried making an Interaction layer, assigning both the triggers and the objects to that layer, then opening the layer collision matrix and unchecking everything in the interaction layer except for interaction, but the triggers still see everything that has physics. It's not a big deal for prototyping, but it'd be a nice optimization to ensure that the interaction triggers can only see interactable objects.

 

Sorry, I just re-read my post and it's really unclear. I was only agreeing that you don't need updates. You also don't need a manager.

Hmm, sounds like you did everything perfectly. Did the collision matrix maybe get reset? Or did the layers change?

 

I'm not really sure what was going on, but I fixed it by manually re-enabling every layer, restarting unity, and manually disabling each layer again, so it now hums along quite smoothly- thank you again for the idea, this is a massively simpler approach than I thought I'd be able to get away with.

 

Great! Glad I could help a bit.

 

So I belatedly realized something that I should've considered from the outset: I used to trigger interactables ("Press 'F' to eat") with an event that the object listened for when the player was looking right at it, but I don't want to force that level of precision: being anywhere close to the center of the screen (as determined by the detail trigger collider) should be enough to get the detail menu.

The problem is, I only want one object to display its detail menu at a time. The way I'm listening for input now, and please let me know if this is excessively inefficient, is by giving each interactable a bool showDetails that gets toggled by ontriggerenter/exit, and putting this in the root interactable script:

Code (csharp):

1.  
2. void Update(){
3.      if (showDetails){
4.           if (Input.GetButtonDown("Interact1"){
5.           //Some interaction
6.           }
7.           if (Input.GetButtonDown("Interact2"){
8.           //Some other interaction
9.           }
10.      }
11. }
12.  

This works fine if we assume that only one object gets showDetails, but right now it would result in multiple items in close proximity to all be used simultaneously. One solution would be for the DetailManager class (which currently lives in the trigger and toggles showDetails on trigger enter/exit) to update a list of every object inside the collider each time ontriggerenter/exit is called, then calculate which object on the list is closest to the center of the detail collider, but that would have to run very frequently so the UI didn't update with noticeable lag, and it would involve an awful lot of pointless calculations in the (many) cases where only one object was inside the detail collider.

 

The Dialogue System has a ProximitySelector example script that does something similar. (Don't go buying the Dialogue System just for this one feature; it's only an example script, and it doesn't do everything you've described in this thread.)

Since OnTriggerEnter and OnTriggerExit messages are sent to both colliders, the ProximitySelector handles them instead of the individual objects.

In OnTriggerEnter, ProximitySelector adds the new interactable to a list and "selects" it by enabling its "use-me" UI and disabling the previously-selected interactable's "use-me" UI.

In OnTriggerExit, ProximitySelector removes the interactable from the list and selects one of the remaining interactables in the list. (Your script could be a little smarter and select the closest interactable instead of a random one from the list.)

Only ProximitySelector has an Update method. The interactables themselves don't. When the player presses the Interact1 button, ProximitySelector calls an Interact method on the current selection.

Since everything's handled through ProximitySelector, you're guaranteed that only one interactable is enabled at a time.

 

Luckily I already own your Dialogue System (and, senseless plug, everyone else should too- it's one of the most consistently useful tools I've bought from the asset store), so I'm looking at it now, and I just want to clarify- this would go on something childed to the player (or in my case childed to the camera), and run based on that object's (trigger) collider, which in my case would be the detail collider, touching a useable object, right? I'm guessing that if I wanted to change the behavior from retaining the last collider entered to the one closest to the screen I'd want to edit CheckTriggerEnter and leave CheckTriggerExit untouched, but that returns me to the problem I originally solved by using colliders, which is programmatically specifying "the Useable closest to the center of the detail collider", as that would presumably be the one the player is trying to raise a prompt for.

I'm probably overthinking this, and the immediate if slightly clunky solution would be to just make the detail collider small enough that the player couldn't physically fit two colliders into it at once, but there must be a reasonably clean way to sort Useables based on their position inside of the collider- right now the detail collider is about the player's width and extends half a bodylength in front of them (to ensure you never have to fiddle around to get the menu you want), so I'm trying to make sure that if they walk up to, say, two Useable bottles and sway back and forth, they can smoothly switch item prompts without either item leaving the collider at any time.

 

(Thanks for the kind words about the Dialogue System!)

Since you've already culled most of the usables, it should be fine to use Update to run distance checks on the few that are in the detail collider. If the closest changes, call SetCurrentUsable on the new closest one. I wouldn't bother with distance-squared math; these days it provides little benefit over a regular distance computation.

Rather than computing distance, I wonder if it would be better to compute the angle degree from the camera's heading. If something is only 1 degree off center, I think I'd expect to select it over something that's slightly closer but 20 degrees off center.

 

Oh, that's an interesting idea... you're thinking something like just checking Vector3.Angle(useable,camera) for each index of usablesInRange? I'm assuming I would want to put this in some function SortUseableAngles() that runs in update (since the player can change what they want to look at without triggering ontriggerenter), do you think it'd be prohibitively slow to just do something like this? It looks awfully hefty for running every frame.

Code (csharp):

1.  
2.     private Useable SortUseableAngles(){
3.        float bestAngle = 0f;
4.        int bestIndex = 0;
5.  
6.        if (usablesInRange.Count > 0) {
7.          float bestAngle = 360f;
8.          int bestIndex = 0;
9.  
10.          for (int i = 0; i < usablesInRange.Count; i++){
11.            float newAngle =Vector3.Angle(usablesInRange[i], camera);
12.            if (newAngle < bestAngle){
13.              bestAngle = newAngle;
14.              bestIndex = i;
15.            }
16.          }
17.        }
18.        return usablesInRange [bestIndex];
19.      }
20.  

 

I don't think it'll be so bad.

1. You're only running it on objects inside the detail collider.

2. You can further optimize by only running it when the player rotates.

 

Oh that's a great idea for the optimization. I'll dump this in this afternoon and report back with hopefully unmitigated success, thank you for your time!

Oh yea, for the actual actions on the command prompt window, each object has a maximum of four verbs (look, take, break, etc) from a library of about twenty, do you think delegates are the way to go for customization? I was thinking it would be fastest to make an enum with each verb type, make one for each interaction button, have each command input just execute some delegate buttonDelegate, then have some function UpdateInputs() that just feeds the button's enum to a switch statement and runs something like Case(Command.Look): button1Delegate = ActionLibrary.Look

 

Your command pattern sounds good to me. If you're so inclined, post a screenshot or video of your final result. This has been an interesting discussion, and I'm curious to see the end product.

 

I just realized that I'm comparing the wrong thing with my Vector3.Angle, I'm doing (object.transform, camera.transform), which ignores heading entirely- would Angle(object.transform, camera.transform.forward) return the correct relationship of object position relative to camera heading?

 

If that doesn't work, try something like this:

Code (csharp):

1. var target = object.transform;
2. var origin = camera.transform;
3. var targetHorizontalPosition = new Vector3(target.position.x, origin.position.y, target.position.z);
4. float horizontalAngle = Vector3.Angle(origin.forward, Vector3.Normalize(targetHorizontalPosition - origin.position));

This computes the horizontal (left/right) angle; I'm not sure if that feels better than absolute angle, but you could change it if you don't like it.

 

Oh hey, I sat down and wrote my own implementation using ProximityDetector and everything we've discussed as a guide, and it works really well- horizontal angle feels pretty good, it's soft without being imprecise. The only problem is that I do definitely need to include some kind of vertical sensitivity, as right now it will always prefer whatever is closest to the camera, which makes it tough to target things higher or lower than the player's head. Math is very decidedly my weak suit, so I'm having trouble thinking up alternatives to separately computing horizontal and vertical angles: is there any way to effectively combine both measurements into a single float I could compare against?

I'd definitely be happy to upload a gif of this in action once I polish it up, too- I'm buckling down to learn myself some UGUI as I work on the angle, and once I have working labels instead of debug statements I'll make a recording.

 

The dot product the normalised camera forward and the normalised direction from the camera to the intractable could be used.

 

Code (csharp):

1. var angle = Vector3.Angle(object.transform.position - camera.transform.position, camera.transform.forward);

 

Hmm, this works perfectly in first-person mode, but in third-person (which uses the same camera and aiming point) it has a habit of offsetting by one object: I made a grid of nine identical objects to test, and in third person mode it will consistently guess that I'm trying to select one object to the left of the one the cursor is on.

 

In the Dialogue System, Selector.cs has some debug code in OnDrawGizmos that draws the raycast ray and hits in the Scene view. You could add something similar to see what's going in third person mode.

 

I think it's a problem with my messaging, not the actual selector, which is weird because they use the same function. For debugging purposes I drew a line every frame, which always pointed to the correct object:

Code (csharp):

1.  
2.     void OnDrawGizmos(){
3.        Gizmos.color = Color.red;
4.        testentity = FindBestMatch(usablesInRange);
5.        Gizmos.DrawLine (playerCamera.transform.position, testentity.transform.position);
6.      }
7.  

What isn't working is another call to the same method- this is supposed to make whatever entity I've targeted light up:

Code (csharp):

1.  
2. void Update(){
3.       if (Input.GetKeyDown (KeyCode.L)){
4.          testentity = FindBestMatch(usablesInRange);
5.          testentity.SendMessage("lightOn");
6.        }
7.  
8.        if (Input.GetKeyUp(KeyCode.L)){
9.         testentity.SendMessage("lightOff");
10.        }
11. }
12.  

This uses the same function FindBestMatch, but it consistently highlights the previous best object, and I can easily get this and the debug line to indicate different things.

To add high drama to mystery I just tried running sendmessage in OnDrawGizmos instead of stapling it to a keystroke, and when it runs in drawgizmos it messages the correct object. It's only when it runs from Input.GetKeyDown that it seems to identify the wrong thing. What's even weirder is that debug disagrees with it: a debug.log in FindBestMatch reports what item it thinks I want, and even when it's run by Input.GetKeyDown it returns the correct object in the log, but even though I sendmessage the same object it just returned, the wrong one gets lit up.

 

That's really weird. Can you cache the value in Update and use it in OnDrawGizmos? This way you're only calling FindBestMatch once, so both methods should consistently point to the same result. Something like this (although this calls FindBestMatch every frame):

Code (csharp):

1. private GameObject bestMatch = null;
2.  
3. void Update(){
4.     bestMatch = FindBestMatch(usablesInRange);
5.     if (bestMatch == null) return;
6.     if (Input.GetKeyDown(KeyCode.L)){
7.         bestMatch.SendMessage("lightOn");
8.     }
9.  
10.     if (Input.GetKeyUp(KeyCode.L)){
11.         bestMatch.SendMessage("lightOff");
12.     }
13. }
14.  
15. void OnDrawGizmos(){
16.     if (bestMatch == null) return;
17.     Gizmos.color = Color.red;
18.     Gizmos.DrawLine (playerCamera.transform.position, bestMatch.transform.position);
19. }

This won't fix the problem, but at least your debugging will be consistent with your Update method.

 

Caching it in update reveals a weird behavior that running it separately didn't: in first-person mode it works fine, as before, but with this code, when in third-person mode both the line and logged object will constantly flicker back and forth between two options. Further experimentation shows that it always alternates between the object the third-person camera is looking at, and the object it *would* be looking at if it were in first person mode. What's weird about this is that I only have a single camera for ivewing and it physically moves depending on which mode its in, so there shouldn't be any way FindBestMatch can figure out where the fps camera would be looking. For reference this is the current version, but I can't see any issues with it:

Code (csharp):

1.  
2.     private GameObject FindBestMatch(List<GameObject> usablesList){
3.  
4.        if (usablesList.Count > 0 ){
5.          Transform target;
6.          Transform origin = playerCamera.transform;
7.        
8.          float bestAngle = 360;
9.          int bestIndex = 0;
10.          for (int i = 0; i < usablesList.Count; i++){
11.            target = usablesList[i].gameObject.transform;
12.            float angle = Vector3.Angle(target.transform.position - origin.transform.position, origin.transform.forward);
13.            Debug.Log("" + usablesList[i].gameObject.name + " has an angle of " + angle);
14.            if (angle < bestAngle){
15.              bestAngle = angle;
16.              bestIndex = i;
17.            }
18.          }
19.          Debug.Log("The best angle is " + usablesList[bestIndex].gameObject.name + "who has an angle of " + bestAngle);
20.          return usablesList[bestIndex];
21.        }
22.        else{
23.          return null;
24.        }
25.      }
26.  
27.  

Edit: I think the flickering is caused by it somehow trying to use the weapons camera as a view camera. Disabling every camera in the scene except for the single camera I use for first and third person view gets rid of the flickering, but it now horizontally offsets items by one- when the third-person camera looks at the middle object of three, it selects the left-hand object.

Double Edit: On the downside, I'm a complete idiot, on the upside the problem is gone. There wasn't any code problem at all: what was happening was that the crosshair I was using to track facing wasn't calibrated properly, so what my crosshair was telling me was screen center was actually calibrated to first person center, and severely offset in third person. Having fixed that everything works, and Tony Li is an awesome dev for spending so much time helping me debug- thank you!! I'll upload a gif with this in action the instant I get labels finished

 

I finally recorded a demonstration of the system in action- sorry it took so long, getting my camera and controller working took far longer than I anticipated. I uploaded the demo to http://imgur.com/giPNTrz

The grey text is simple UI, and the red is detail UI. The gif doesn't give any indication of this, but whenever red text appears it also indicates that that item is listening for inputs: any contextual use item buttons I design will only be heard by the object with red text over it. Programmer art aside, eventually I'm definitely going to need to expand this with a routine that looks at every visible label and adjusts them to prevent occlusion, as right now it's down to manually positioning each canvas.

 

Looks good! It looks like it would feel very intuitive to select interactables.

I haven't played Life is Strange yet. Do you think it positions labels to prevent overlap, or did they just position interactables far enough apart that they don't overlap?

 

I just loaded up the game to check, and I'm pretty sure that careful placement of labels by staff artists is behind 90% of it: I managed to get labels to overlap each other a few times, and when it happens nothing adjusts them or switches one off, but things are positioned so that it's extremely difficult to do that- in the course of normal gameplay I doubt it would happen even a single time if the player wasn't trying to make it happen on purpose.

Since my use case requires more dense interactable compositions there are a couple of things I'm thinking about- the simplest solution which solves the issue immediately is only displaying the label for the object with the detail UI. It loses a bit of utility since the player can't determine what they can interact with at a glance, but that demo gif looks pretty crowded with only 3-5 labels active at a time, and a table covered in notes and instruments would look terrible.

A compromise would be to only display a handful of labels, one for the detail object and one for the 2-3 objects closest to the detail object. Since all canvases are in world space it would be pretty trivial to test for occlusion and lerp a canvas up or down to prevent them from overlapping, but that would require a lot of work to keep the corrective motion from being distracting.