Unity 2018.3 and Hololens: holograms are not stable

Hi.

We've detected a quite serious issue with the 2018 releases of Unity (since 2018.2) with Hololens and hologram stability.

Basically, the holograms are never stable. The best way to see the effect is to look at an hologram and physically jump a bit. The holograms also appear to jump instead of staying in place. Debugging the stabilization plane shows that the plane appears in the correct place but it seems to have no effect.

Searching the internet, this issue seems to come up from time to time, but is mostly dismissed as not being unity's fault as most people use Mixed Reality Toolkit's implementation of stabilization plane.

We've been "stuck" in Unity 2017.4 LTS because of this, since the issue doesn't happen in this version (currently 2017.4.23f1)
We want to move on to Unity 2018.3, mostly because of the "nested prefabs" feature, latest Vuforia version, etc. But this issue keeps happening and it seems nothing is being done to fix it (not that I'm aware of). I hope this report changes this (I'm also writing a bug report).

We created 2 empty unity projects (Unity 2017.4.23f1 and Unity 2018.3.9f1). In both projects we created a cube and a sphere that appear in front of the Hololens (fixed position). Both projects have the "Enable Depth Buffer Sharing" checked and some other basic settings to run in Hololens. No other mechanism is used (no world anchors, no MRT, no Vuforia tracking, no manual setting of stabilization plane, literally nothing else).
After build/deploy, the final result of both projects can be compared and there's the obvious issue with the 2018.3 build.

Unity 2017.4.23f1 (LTS): (holograms are stable)


Unity 2018.3.9f1: (holograms are unstable)

Actual framerate in both apps is constant 60fps.

TLDR: Unity 2018.3 holograms are not stable. This doesn't happen in Unity 2017.4 LTS with the same settings. We're unable to update unity because of this and we really want/need to. It is most likely a Unity issue since no code or any other packages are being used.

 

Hi.

The bug report has the 2 projects I mentioned in the post and the videos that show the issue (raw captured videos instead of compressed gifs).

The info of the bug report:
Case: 1138810
Title: Unity 2018.3 and Hololens: holograms are not stable but they are in Unity 2017.4 LTS

Thank you.

 

It actually looks like it's not so stable compared to what we get with 17.4LTS.

Also, small up-down shifts while looking at the holograms is the best way to reproduce the issue we're having.
(that's what I'm doing in the videos attached to the bug report, which are the sources for the gifs posted here).

 

Hi @JasonCostanza,

Do you got any news regarding this? We are facing the exact same issue.

We first noticed holograms instability so we experimented different settings (2017.4, 2018.4, with and without MRTK, with and without WorldAnchor). Without using the MRTK, the holograms are way more stable in Unity 2017.4 whereas on 2018.4 they are jumping a lot.

As a result, we are currently stuck with 2017.4.

FYI, our HoloLens has been wiped yesterday and is updated to the lastest OS version (17763.615 - 09 July 2019).

Do you need any thing to help you identify the issue?

Edit: video files that demonstrate the stability issue - https://we.tl/t-elTLiLwgso

 

@JasonCostanza We are already using Shared Depth Buffer.

Please find attached a repro project. Inside the archive, you will find both the Unity project and the HoloLens built packages for Unity 2017.4 and Unity 2018.4

Link: https://we.tl/t-b7ZtYfldNv (will be available for a week)

When you are trying them, please jump or bend your knees quickly: this is the best way to notice the jitter on 2018.

Here is how the Unity project was made:
- Blank project (nothing inside, no MRTK)
- Project settings according to Microsoft Guidelines (https://docs.microsoft.com/en-us/windows/mixed-reality/recommended-settings-for-unity):

• No Realtime/Baked GI
• Camera Clear Flags set to solid color (Black)
  
• Camera position set to (0, 0, 0)
• Camera near/far range tuned down (0.2 - 20)
  
• XR Settings enabled for WMR
• Depth Buffer enabled and set to 16-bits
• Stereo rendering mode set to Single Pass (we tried both SP and SP Instanced)
• A cube set at position (0, -0.25, 1.75) and Unlit color (Green) material

 

Quick update

I noticed in 2019.1.9 patch notes the following line:
XR: Fixed Render Viewport Scale Jitter when render viewport scale changes continuously. (1153261, 1159974)

Do you know if it could be related to the bug we are currently facing, and if it is do you got any ETA for 2018.4 fix?

Edit: Well, I just tested with Unity 2019.1.12f1 and the issue is not solved. As such, only Unity 2017.4 seems to be a safe environment to develop at the moment.

 

I would also like to know this, we have been battling hololens jitter since early this year and its quite noticeable, none of the recommended steps alieviate it.

 

Popping in this thread to confirm that I'm seeing the same issues on Hololens 1 with the example projects.

2017 = Rock Steady Hologram Placement
2018 = Lots of Jitter

 

I can also confirm seeing the jitter present in 2018 but stability in 2017

 

Thanks for investigating! Should be easy to replicate with the provided projects
Let me know if you need anything more.

 

I am seeing the exact same behaviour as Jonathan if that helps.

 

Running on device, using latest SDK, latest HoloLens updates, Unity 2018.4.4f1. (Same on all 2018.4 versions that I have checked). Using in built pipeline, using mixed reality toolkit shaders (but is same regardless of shader as far as I can tell). This happens whether object is anchored or not, anchoring it does actually seem to make it a bit worse (hopefully again this may be helpful, my thoughts initially was that it is something incorrect under the hood with how world anchors locate themselves).

I dont develop without unity, but I have a friend who has used the same models in a non-unity hololens app (direct x I think) and it did not produce any jitter at all.

Jitter gets worse up close, and there was no (custom) code running in scene.

I think thats about everything?

Hopefully this will be of use, I wouldnt be able to provide any repro project from this PC as we are forbidden due to security to send out any project files even fresh projects not to do with work.

 

• I noticed the issue when running on the device, I'm never relying on simulation nor remoting.
  
• The project I uploaded is an upgraded version from 2017.4, but we initially spotted the issue when starting a fresh 2018.4 project so it's not relevant unfortunately
• As you can see, there is no third party tools in the project (no mrtk, no custom packages, no special assets)
• I confirm it's still occuring on 2019

Please let me know when you'll have tested my build so we can compare the results.

 

I'm also seeing this behaviour in 2018.x. We'd previously been targeting 2017.3f1 and have since rebuilt in 2018.4.3f1. I'd also second that it appears to get worse when close (~1m) to an object.

The other change on our end was to use depth buffer sharing rather than manually setting the focus plane as we did in 2017.x, but given that others are experiencing this I don't imagine that's the issue.

 

Additional Details- I used the examples provided by Jonathan, without changing anything. Saw the behavior described. Because I'm generally curious I also did a fresh project from scratch in 2018.4 with the exact same setup, only difference being the cube I used had different colored faces. Saw the same behavior.

I tried implementing MRTK as well, and used it's default setup options to see if there was any change and saw no difference. I also tried disabling the buffer sharing and setting the focal plane manually to align with the cube when focused, again with no change in behavior. All of these continued to jitter in 2018.4.

This was tested on 4 different 1st Gen Hololens' running on the latest Hololens OS.

I can provide this project with MRTK implemented and additional scene with manual focal plane palcement if you would like. But given that the issue appears in the simplest example, and none of my experiments resulted in noticeable changes, I don't think it will be very useful.

 

@JasonCostanza Ok, here is a step-by-step guide when starting from scratch (which I just did). Please note that everything that is not mentioned here has been left untouched from the default configuration.

Build Environment

PC Windows OS Version (from winver.exe): 1903 (18362.239)
Latest Windows SDK version: 10.0.18362.0
HoloLens 1st Gen build version (from device portal information): 17763.615.x86fre.rs5_release_svc_prod1.190630-1301
Visual Studio Version: 15.9.14

Procedure

• Install Unity 2018.4.5f1 (I do not use Unity HUB - and I never overwrite a previous installation; always install in a fresh new folder)
• Launch Unity
• Create a new project called "Jitter", with 3D as default mode
• Switch platform to UWP
• Prepare the material for the cube:
  • Create a folder named "Materials"
  • Create a new material in that folder, called "UnlitRed"
  • Set material shader to Unlit/Color
  • Set Main Color property to (255, 0, 0, 255)

• In the default scene that is provided:
  • Remove directional light
  • Set the camera:
    • Set Position to (0, 0, 0)
    • Set Rotation to (0, 0, 0)
    • Clear flags to Solid Color (0, 0, 0, 0)
    • Far clip plane: 20
    • Disable "Allow MSAA"
  • Place a cube in the scene:
    • Set position to (0, -0.25, 1.75)
    • Set scale to (0.15, 0.15, 0.15)
    • Set material to previously created "UnlitRed"
  • Open Lighting settings (Window -> Rendering -> Lighting Settings):
    • Clear Skybox Material and Sun Source property field (click on the currently assigned value and press "Del.")
    • Disable Baked Global Illumination (uncheck the box)
• Save the scene
• Add it to the builds settings (File -> Build settings -> Add open scenes)
• Open the package manager (Window -> Package Manager):
  • Select "In Project" from the top left top down
  • Remove all packages but Package Manager UI
  • Remove "Ads"
  • Remove "Analytics Library"
  • Remove "In App Purchasing"
  • Remove "TextMesh Pro"
  • Remove "Unity Collaborate"
  • Normally, only Package Manager UI should be there
• Open Quality Settings (Edit -> Project Settings -> Quality):
  • Delete all quality levels but Very Low
• Open Player Settings (Edit -> Project Settings -> Player):
  • Under XR Settings:
    • Check Virtual Reality Supported
    • Set Stereo Mode Rendering to Single Pass Instanced
• Open Build Settings (File -> Build Settings)
  • Click the Build button at the bottom
  • Create a new folder from Windows Explorer named "Builds"
  • Select the folder for the builds
• Once build is complete, open visual studio solution (Builds/<project_name>.sln):
  • Select "Master" as build configuration
  • Select "x86" as target platform
  • Select "Device" as target device
  • Plug the HoloLens through USB port
  • Select Build -> Deploy Solution from the top menu
  • Once build is completed and deploy has succeeded, unplug the HoloLens
• Put the HoloLens on your head:
  • Bloom to open the shell
  • Select "All Apps" on the right
  • Selection the application you just build
  • The OS will start the application, and Unity Splash Screen will soon appear
  • Once you see the red cube from your scene, move around, and squat
  • Notice jitter

I hope it can help you identify what differs in both our process to reproduce the issue.

 

@JasonCostanza I just tried a build while disabling Enable Depth Buffer Sharing, and Default Is Full Screen - as I noticed you uncheck it, but I still notice jitter.

Also, coud you please specify what build configuration you picked (Debug, Release, Master, MasterWithLTCG)?

Edit: if you are going to attend Unite 2019 at Copenhagen let's grab a beer and I'll bring my HoloLens to show you the issue Let's just hope that you won't break your legs by squatting in the office until September ahah

 

My Specs (only difference is OS Version)-
PC Windows OS Version (from winver.exe): 1803 (17134.915)
Latest Windows SDK version: 10.0.18362.0
HoloLens 1st Gen build version (from device portal information): 17763.615.x86fre.rs5_release_svc_prod1.190630-1301
Visual Studio Version: 15.9.14

 

I'm actively following this thread as I've also experience poor hologram stability using Unity2019.1. I've tried all types of tricks but no matter what I do I end up with a hologram swim of 1 - 1.5 cm as I gently move my head side to side. I have not tried Unity2017 but will look into that. Please update with anything interesting results!

 

@JasonCostanza
Here is the link to FogBugz case: <removed for privacy>
I just noticed I sent it from Unity 2017.4. Do I need to kill my bug report and send a new one or are the instructions in there enough?

 

This bug isn't on the issue tracker (or at least it's not showing up when I do a search for it), so I have no way to upvote it.

@JasonCostanza can you add this to the issue tracker so others can comment and upvote it?

 

Yes, it does not reproduce on 2017.4. Jitter is only noticeable starting from 2018.

Thanks for making it public!

 

@JasonCostanza Jason, I can access the bug in the issue tracker by clicking on the link you provided, but I noticed it does not get referenced in the search engine (same while browsing issues). See: https://unity3d.com/search?refinement=issues&gq=1174865

Does it take time for it to show up there or is it another privacy setting that needs to be tweaked?

EDIT: Nevermind, I missed one filter.

 

Just so you guys know, I just saw a fix for Unity 2018.4.6f1

• XR: Fix issue with Depth Based LSR on HoloLens V2 that caused significant jitter. (1169760, 1169763)

I tested just out of curiosity for HoloLens V1, but I still notice jitter :/

 

Fantastic! Thanks for the feedback.

Let me know if you need any information or testing. I definitely agree it can be a real pain to deal with this kind of issue unfortunately.. good luck!

 

Ok, so I tested the sample app, very nice and clean minimal example by the way, and found a potentially interesting thing with regards to jitter and MRC.

Our apps actually do not use the recommended render settings, instead we use Multipass and no depth buffer sharing.

When building the sample app this way, the cube becomes pretty much rock solid, except when starting MR capture! However, there is a blurriness and jitter that is clearly introduced during MR capture (I think it has to do with the hardware "hijacking" one render pipeline and lowering the resolution, one of the many tricks the HL engineers had to pull to get all amazing features in one small package such as HL1).

However, if I build using either Single Pass or depth buffer sharing, the render is jittery all the time. Didn't try the Single Pass instanced although I think that's actually the recommended one: https://docs.microsoft.com/en-us/windows/mixed-reality/performance-recommendations-for-unity

(Built using 2018.4.3f1 and VS2017, deployed to HL1 with RS5, 10.0.17763.615)

 

Hi all,

on Unity 2019.2.0b7, we had jittery holograms.
XR settings were Single Pass Instanciated, 24bits Z-buffer & Depth Buffer Sharing Enabled.
Now, with XR settings changed to Single Passe Instanciated, 16bits Z-buffer, & Depth Buffer Sharing DISABLED, holograms are back to stable holograms(our frame rate is around 50hz). We use LWRP. I don't know if it's he combination of 16bits Z-buffer AND SPI disabled or only SPI disabled which did the trick...

Cheers,
Diego
realcast.io

 

Any updates on that issue..? We're still getting "bouncy" holograms with MRTK on Unity 2018

 

Hi.

This was already improved (not sure from which Unity version).
We're currently using Unity 2018.4.8f1, and this isn't happening anymore (even using the .NET scripting backend).

For the other people that commented, please note that your holograms may "bounce" because of other issues (e.g. not enabling the depth buffer sharing in the build settings, very wrong ipd setting on Hololens, etc). That is why reporting this issue was difficult to begin with.

Also, try changing to IL2CPP scripting backend and compare results. IL2CPP should not have this issue even on Unity 2018.3.x versions (the one used when I created this thread).

Finally, when you write "bouncy" or "jitter", this is quite vague (that is the reason I took the videos). There are many effects that happen on the holograms, we need to make sure we're talking about the same effect. Try recording a video and post it and maybe I can help you further.

Good luck everyone

 

Thanks for these suggestions, in response to the issues you raised-

IPD Settings- Should not be an issue, we calibrate IPD for each user. This was in fact the 1st thing we checked when we saw this issue.

Depth Buffer Sharing- Should not be an issue. If you check the notes provided, along with steps to reproduce, the settings are set according to the documentation directions.

Scripting Backend- Doesn't matter issue repeats regardless of backend used to compile.

I have sent numerous videos to the Unity Team showcasing the difference between 2017 and 2018 builds. There are obvious differences in hologram stability despite the only change being which Editor version was used to create the build.

 

Sorry to hear that this is not fixed for you... We currently have the same degree of stability we had in 2017 but we're also not really sure how it was suddenly fixed... It was after a Unity update though.

By the way, did you try to do a factory reset to the device? Also, are you using incremental build (as in build to an already existing uwp project)? If so, can you try to delete the existent one and build a fresh one? At the time it was fixed for us, we tried to build to an existent solution by accident and the problem remained (thus my suggestion).

Good luck.

 

Hey,

Something to also look into, apart from Unity, is the version of Windows on HoloLens. There has been an inofficially confirmed degrading of spatial tracking quality in one of the updates. Namely, anchors saved into a WorldAnchorTransferBatch would be less stable when created on an old version and unpacked in the new. They'd also be less stable within the same version when more distant from the viewer or when viewed from angles dramatically different from the angle of the anchor creation (from behind). Interestingly enough, Microsoft seems to have fixed the issue for when anchors are created and unpacked under the same Windows version, but the old-to-new problem remains. This led our team to simply freeze Windows at one of the versions with proven spatial tracking stability: 10.0.14393.2248 (yes, with the Device Portal file uploading bug and a javascript fix for it) and forbid Windows updates for our installs. Degrading happened one or two versions after that, but Im not certain. And fixed a few more versions later.

Not related to your original question about Unity, but something to keep in mind.

We use Unity 2019.1.9f1 and it works great, now jitter like on your pictures.

Cheers