[RELEASED] Realistic Eye Movements

@AcidArrow Yes, it can be made to work with FinalIK. Maybe I'll support it out of the box in a future release, but for now it's easy to do it yourself:

The head IK is currently updated in EyeAndHeadAnimator.cs in the function OnAnimatorIK. You can modify the code there to use Final IK's LookAtIK and move it to the beginning of LateUpdate in EyeAndHeadAnimator.cs, something like this:

LateUpdate()
{
if ( headWeight > 0 && lookAtIK != null )
{
float targetIKWeight = (lookTarget == LookTarget.StraightAhead || lookTarget == LookTarget.ClearingTargetPhase2 ||lookTarget == LookTarget.ClearingTargetPhase1 ) ? 0 : headWeight;
ikWeight = Mathf.Lerp( ikWeight, targetIKWeight, Time.deltaTime);
lookAtIK.solver.IKPositionWeight = ikWeight;
lookAtIK.solver.IKPosition = headTargetPivotXform.TransformPoint( Vector3.forward );
}
...*current LateUpdate code*...}

 

Cheers Faikus, that sounds straight forward enough.

 

Hi Faikus,

I've been working with the settings and I think I have the hang of the Look At and Stare Back and Notice Distance settings.... What I don't see any control over is something that would turn the head faster, like in an argument or an emergency. We can change how far the head turns, but not the "attack speed" of the head turn.

Could this maybe be an additional setting? The speed now is perfect for ambient or casual conversation, but I'd like to pick up the energy sometimes, and also have different characters respond at different speeds.... Maybe Nervousness could also effect the headturn, or maybe the head turn's own sensitivity threshold setting would work?

 

@wetcircuit Yes, a slider or some other setting for faster turns makes sense, I'll try to implement something along these lines in a future update. Until then, if you want you can add a few lines to EyeAndHeadAnimator.cs to get such a slider:

At the top, after "#region fields", add:

// Modification to add Head Speed slider
[Range(0.1f, 3)]
public float headSpeedModifier = 1;​

Then in the function StartHeadMovement, after the line

headVertDuration = kDuration + mDuration * vertDistance;​

(around line 930 or so), add these lines:

// Modification to add Head Speed slider
headMaxSpeedHoriz *= headSpeedModifier;
headMaxSpeedVert *= headSpeedModifier;
headHorizDuration /= headSpeedModifier;
headVertDuration /= headSpeedModifier;​

Let me know if there are any problems with this.

 

Just left a review for this amazing asset. It would be great to see this tech expanded for lip synching

 

@jaelove: thanks!

Lip synch is quite a different field, I think that would be a task for a different asset.

 

Bought this yesterday, was about to come here and post that a head turn speed slider is really needed and then I found that an update was released, today, that adds a speed slider

 

@AcidArrow You beat me to the headslider announcement

Here are the changes for the new version:

• For eyelid control with bones, the position of the bones is now saved as well (before only the rotation was saved). This makes it easier to set up rigs like those from Autodesk Character Generator.
• New demo scene: you can switch between a character made with MakeHuman, Autodesk Character Generator, and Mixamo.
• Head speed slider: lets you adjust the head turning speed.
• Head latency (how much later the head starts turning than the eyes when looking at a new target) is now exposed in the API.
• Fixed bug in look duration when calling LookAtPoiDirectly.
• Fixed bug with eyeWiden slider.

 

Version 1.9 is out. Here are the changes:

You can now export and import settings. Once you've set up a character, you can apply these settings (assignment of eyelid bones, how far eyes can rotate up etc.) to new characters by just importing the preset file you exported. There are some presets already. If you created your character with either Mixamo, Autodesk Character Generator or MakeHuman, you can set them up in a few seconds by just importing the corresponding preset. There is a preset for UMA as well and you can add Realistic Eye Movements to all your UMA characters from code with just a few lines.

Enjoy!

 

Just tested the new UMA integration feature and it works great. Good job!

 

Hi Faikus, I have problem with the "Look AT Player Ratio". Your documentation says, if I set the "Look AT Player Ratio" to e.g. 0.1 the NPC looks 10% of the time at the player and 90% randomly somewhere else or, if set, at a POI.

Can't get this work. Once the NPC notices the Player inside the "Notice Player Distance", the NPC look permanently at the Player. I can walk away, don't look at the NPC anymore, run over hills ... NPC stares at the Player.

 

@Slowbud: Can you send me a screenshot of your LookTargetController and EyeAndHeadAnimator components in the inspector so I can see what parameters you set? Also, are you using the latest version of the asset?

 

Sure Faicus, here's the screenshot. AFAIK I'm using version 1.9.

 

@Slowbud: I can't reproduce your problem with your settings. Does anything happen to the CameraNPC object? Were the scripts changed? Are you calling any of the API functions?

 

@Faikus Didn't change anything at the scripts (ich werde mich hüten ;-) ) and no API calls.
I testet a little more, even when I put the "Look at player ratio" and "Stare back factor" to zero, as soon as I'm reconized, the NPCs look at me and never turn their look away anymore. I've attached your scripts to 5 NPCs, all have the same behavior. "Personal Space Distance" works fine.
Maybe something to proof: I'm running Unity 5.2 in native VR mode and using the Oculus Rift DK2 Driver 0.7. Did you try to reproduce with these settings?
As far as I remember, but not 100% sure anymore, at the beginning of the development, with Unity 4.xx and external Oculus Driver PlugIn, everything of your scripts worked as it should.

BTW: The marvelous realistic eye movement in your "Coffee without words" motivated me some month ago to develop a VR world by myself

 

@Slowbud: Thanks, with the info about VR I think I got it, there needs to be a change in one of my scripts for your setup to work. The problem is that with your nervousness setting the NPC macro-saccades with the eyes frequently from one of the player's eyes to the next and this prevented the switching of the targets, because the scripts wait for 2 seconds after a macro saccade to switch targets, but this dead-time of 2 seconds should be shorter when macro saccades are more frequent due to nervousness. I will have the change in the next update, in the meantime you can change your script yourself (nur Mut :
In EyeAndHeadAnimator.cs, change line 270 which currently reads:

return Time.time-timeOfLastMacroSaccade >= 2f;
​

to:

return Time.time-timeOfLastMacroSaccade >= 2f * 1.0f/(1 + nervousness);
​

Let me know whether there are any more problems.

Good luck with your VR project, I hope to see Amara's eyes in action in your VR world some time!

 

@Faikus Thanks, works fine. Though it seemed first after code change it didn't help, after disableling and enable macro-saccades again it worked.
Sure you can see Amara's (and many more) eyes acting. I plan a first preAlpha release end of this week. (nach monaten von Entwicklerschweiss) But you should be open minded to some fetish. In detail, don't get frightend when meeting some muscled Amazons and Vampires. (no XXX, no blood, no kills). I can send you a PM on first free distribution if you like. Others interested are welcome too.

 

I've updated the Oculus VR demo for Realistic Eye Movements for the new Oculus runtime 0.7, it now uses the new Morph 3D character. If you have a DK2, try it out: https://drive.google.com/uc?export=download&id=0B8-Xo8vK3swYNDRWMkZEQTdKVGc

 

@Faikus, ++Good asset. You only have to think about a problem and you're already on it.

My problem is with Morph3D. I have one character and I am working on a load and save function, so I have one avatar but many morphs. The problem is after I apply a morph, the eyeball bone anchors show up in the right place for the unmorphed character. So if the character is morphed into someone shorter, they always are looking down, since their original unmorphed eyes are still original position above their head. I can tell this using the "Draw Sightlines in Editor" option. You can see that function thinks the start point is above the head.

Idle speculation: maybe Morph3D in their optimization perform the mesh transformation after the normal LateUpdate, meaning Mecamin sees a normal skeleton but it gets "warped" for rendering? I suppose I could use Gameobjects instead of bones, but I would loose blendshape eye blinks which works just fine, along with head and body turn. It could be that it is not a problem if I refresh the Morph3D character after each runtime morph setting, but I have to find that button on Morph3D. I know the problem is not in your code since with SightLines I can see its doing what it is supposed to do at that point in time, just for the original unmorphed character. However ... since you are using Morph3D could you look into it Or at least give me some ideas how to work an appropriate offset or maybe how to convince your code to do a eye position update.

 

@Faikus, think I have a solution. I made a "ReInitilize()" method for EyesAndHeadAnimator that does the core of export and import, sets isInitilized to false, then calls Initilize(). That seems to make it happy. I noticed that I could change the blendshapes in the editor and the eye anchors tracked appropriately. In either case, my problem seems to be solved for me now, just had to talk it out.

 

@kinodax Thanks for sharing your solution. I haven't looked into Morph3D very deeply yet, I've just used the unmorphed default character so far. I want to wait until their asset is a bit more stable, as there still seem to be a lot of issues with it. But the characters sure look great.

 

@Faikus Some time ago I made an interesting real world observation.
When someone changes his horizontal view angle more than a specific amount (including head turn), the person blinks slowly. I guess it's to avoid a motion blur. How about implenting this in your scripts? ;-)
Something like: If OldPOI to NewPOI > 45° -> blink slow.

 

@Slowbud Good observation. It's actually already implemented. The current angle to trigger a slow blink is 25 degrees. If you want you can play around with that angle to see whether you like a different value better. In EyeAndHeadAnimator.cs, in line 987:

const float kMinBlinkDistance = 25.0f;​

change the 25 to a value of your choice.

 

@Faikus LOL, OK, you're the specialist, but I didn't would write the lines, if I hadn't testet before. Maybe I just blinked when my characters blinked ;-) I'll test again. But what maybe be more interesting to play around with is the "slow blink time". Where can I find this?

 

@Slowbud In EyeAndHeadAnimator.cs, around line 87, you can increase the numbers for kBlinkCloseTimeLong, kBlinkKeepingClosedTime and kBlinkOpenTimeLong to make the character blink more slowly during long blinks.

When testing, keep in mind that it's easy to misjudge angles. What you feel is a 40 degrees angle shift might be much less. Activate draw lines in inspector to see.

 

@Slowbud is right that in the case of eyelids controlled by blendshapes the character doesn't do the slow blink. The fix will be in the next update. In the meantime, you can implement the fix in your code. In EyeAndHeadAnimator.cs, change line 984 from:

if ( useUpperEyelids || useLowerEyelids )​

to:

if ( useUpperEyelids || useLowerEyelids || controlData.eyelidControl == ControlData.EyelidControl.Blendshapes )​

Thanks for your help in finding the bug, Slowbud!

 

Hello @Faikus,

I'm confused. Things should be working. I just downloaded this 3D model monk that has a fully animated mecanim rig. Unfortunately in the Eye and Head Animator script when I try to use mecanim eye bones for the eye control it says "eye bones not found". When I looked into the rig avatar it had both eyes properly assigned, so I'm confused why this isn't working out. Have you ran into this problem before? If there is no easy way to fix this, is there a way to manually assign the eyes in the avatar? I'm in a bit of a bind right now, and I would appreciate your help. I look forward to using this amazing asset!

Thanks

 

Hi @tannerperson, is the character gameobject maybe deactivated? Is there an error message in the console? If yes, can you click on it to expand the stack trace and send the stack trace to me?

 

@Faikus,

Thank God, I figured out what the problem was. In the Rig import settings I set checked the "Optimize Game Objects" setting. This basically took away my ability to interact with the bones of the model. Once I unchecked the setting all my bones appeared in a hierarchy as children of my model object. Everything is working just fine now!

That's a good note for other users...I wonder if others have run into that problem.

 

@tannerperson

Good find! I will add a corresponding remark to the documentation. Thanks for the info!

 

@Faikus

So, I've been trying once again to make it work properly with FinalIK (because the LookAt stuff from FinalIK work so much better than the Unity Lookat (no weird neck positions)) and I implemented the code, roughly as you said in your post.

And it works, but it overrides the eye movement as well, so the Micro/Macro Saccades no longer work.

So it seems to me it's a tad more involved than just changing the lookat solver. So, no pressure, but I'm asking once again for "official" FinalIK support

 

@AcidArrow

It works for me with the code I posted. Did you maybe, instead of inserting the code at the beginning of LateUpdate and leaving the rest of LateUpdate intact, replace all the code in LateUpdate? Also, can you put EyeAndHeadAnimator.cs in the script execution order into last position, so it is executed after the FinalIK scripts and tell me whether you still have problems with the eye movement? Are there any error messages in the console?

 

I thought of the excursion order being an issue just as I was leaving the office. It might solve it.

I'm pretty sure I edited the script correctly.

In any case, I'll try your suggestions in the morning. Cheers.

 

@Faikus

Yep, it probably was the execution order. Maybe.

I did everything again from scratch with a clear head today and it's now working, so yay!

 

@AcidArrow good to hear!

 

@Faikus

Ran into another small issue. Realistic Eye Movement is not working for Windows Phone/Windows Store apps (at least on Unity 4.6.9).

Apparently on Phone/Store serialization is not very well supported. I keep getting this error when trying to build for Windows Store/Phone 8.1

"Assets\RealisticEyeMovements\Scripts\EyeAndHeadAnimator.cs(8,36): error CS0234: The type or namespace name 'Formatters' does not exist in the namespace 'System.Runtime.Serialization' (are you missing an assembly reference?)"

 

@AcidArrow Ah, Windows Phone doesn't seem to support this type of serialization. For now, you can put "#if !UNITY_WP8 && !UNITY_WSA" in front of and "#endif" after the parts of EyeAndHeadAnimator.cs that reference BinaryFormatter. Put those preprocessor directives around:

using System.Runtime.Serialization.Formatters.Binary; (at the top)

And also around the functions CanImportFromFile, ExportToFile and ImportFromFile.

I'll include this in the next update.

 

Cheers, that seems to work.

Just a note: Since I'm on 4.6.x I used

"#if !UNITY_WP8 && !UNITY_WP_8_1 && !UNITY_METRO"

Since UNITY_WSA is not supported in 4.6 I believe. (and UNITY_WP_8_1 is undocumented, but it works).

 

@AcidArrow Thanks for the info!

 

Hey again Faikus, I need some help again.

I use blendshapes a lot (for the eyes and also for expressions etc) so I edit the character model a lot and add more blendshapes.

A lot of times when I do that, I guess the order the blendshapes are written in the file change, since then realistic eye movement seems to get confused on which blendshapes I have set for closed eyes, looking up, looking down (it remembers the values, but it applies them to a different blendshape).

It's not that big a deal, since I can re set them, but on days where I want to add a lot of blendshapes and then quickly test them, it gets annoying real fast.

 

Hi, just a heads up. The new version of Mixamo Fuse was released by Adobe yesterday and appears to be free while it's a "preview" app, all I needed was an Adobe CC id to download, (I don't have a subscription)..., and the Mixamo site still works if you were a member before.... FYI there are some improvements, but it's not radically different. The main issue is that they've scrapped almost all the Fuse 1.3 figures and have new figures (some based on body scans?)

Anyway, this isn't a review.... I tested the new figure with Faikus' R.E.M. script and I had no eye movement yesterday (only head turning) , but this morning everything seems to be working perfectly – head and eyes.... (just Unity playing games with me... ).

 

@AcidArrow

Ah, yes, the blendshapes are currently saved by index. It would make more sense to save them by name instead. I'll try to include that in the next update.

 

@wetcircuit

Wow, that new Mixamo character looks better than the old ones, I hope the eyelids no longer look like they are peeling off the eyeballs. About your problem yesterday, are there any other components/scripts that could set eye movements? Maybe there is a race condition in which script gets to execute first.

 

I hadn't checked the eyes until your post...



The new model is definitely better. Eyes are positioned much better and the breasts are now actually breast shaped rather than scary domes with iron nipples....

One thing everyone will hate is that all the figures seem to have permanent underwear now. I can make it invisible but the normals are still there....

 

...and to answer your question, yes, I'm also using SALSA/RandomEyes for the speech and random face movement. REM and RandomEyes can be set up to play nice together, but for some reason I think you are correct, they might be competing somehow...

The eyes also aren't called "Eyes", they are called "default" for some dumb reason even though the rest of the model parts are named correctly.... I'll tell Mixamo because that just seems wrong.

 

...also, could you illuminate what "personal space distance" does? And "Stare Back Factor"... There's times I'm just guessing at these settings but I'm not really sure when I'm using them correctly.

 

Yes, the eyelids look better than the old ones. I hope other eye models fit better now, I'd like to use Eye Advanced. About RandomEyes, could you add EyeAndHeadAnimator.cs to the script execution order queue into last position?

 

The slider Stare Back Factor determines how quickly the character looks back at the player if the characters sees the player keep staring at him or her (so how sensitive the character is to being stared at).

If you set the slider Personal Space Distance to a value greater than zero and the player comes closer than this distance, the character looks away (avoiding the player in a „shy“ way).

Please let me know if that needs more clarification, others might have the same questions.

 

I think I get it.... It works best when the camera actively moves across R.E.M. trigger distances.... If the camera is static sometimes it takes a long time for the look focus to change (if it ever changes). I think that is why sometimes I am wondering why it is "not working"....

 

Hm, actually that should work even when not crossing trigger distances. In the demo scene, please set Personal Space Distance of the default character (the Autodesk characer) to 1 and press Play, she should look away from the main camera quite quickly. Is it different in your setup? About Stare Back Factor, there is a small bug that is fixed in the upcoming update that makes stare-back not working if Look-at-player-ratio is 0.