[Unresolved] Strange spikes on terrains, but not reproducible

[ Edit: I did run into the spikes on my computer once finally. They seem to be very rare, not at all related to distance/precision errors, and will not remain if the same area is reloaded using the exact same deterministic noise. ]

I've got 1-2+ players that get these weird spikes on my terrains... I have sold 10,000+ copies, been working on this 4 years, and never once seen them myself, and talk daily with lots of other players that have never seen these. One player that has them is very active and cooperative, I just am not sure where to start... He says sometimes he'll load, everything will look fine, he'll do a 360 and the spikes will show up. Some days he gets none, some days he gets a lot I use the FastNoise library to generate heightmaps based on world positions. One of the screenshots he showed me recently very clearly showed that the spikes max height were follow the terrain height, ie: they were all +250m or something like that... I don't get it

Could there be some kind of a compute inaccuracy on his CPU? He's out at like 500,000m, so I'm feeding the noise numbers some pretty big numbers, but again... nothing like that on my screen, and I've gone out to 10,000,000m+...?

 

https://forum.unity.com/threads/floating-point-errors-and-large-scale-worlds.526807/
Read the beginning of this thread for info. (Ignore the "put everything under one transform root"... that's nonsense).

 

I did not go through the whole thread with a fine comb if you were trying to point something specific out, but these are not floating point precision errors, perse. I do use a "world shift" method and have for over 3 years, without problems (I've had a player go out to 1,000,000,000m where the combat math/monster health math and things start to error, but the biome/world generation still/world shifting still works). I also don't have player hosts (non-dedicated servers) enable world shifting, so it's easy to "host" a game, cheat teleport out to any distance and experience the effects cause by floating point precision errors, which are primarily a jittering of camera/meshes, nothing like spikes in the terrain. Furthermore these are experienced by less than 0.1% of my players, so if it's a precision error it's a hardware error, as the vast majority of hardware does not seem to produce the problem. If it is hardware I'm not sure how I would go about detecting and intercepting it though that should be possible

 

here's a screenshot as multi-player host (with world shifting disabled) at 1,058,000m for reference

 

Ouch, I'm an idiot, and a programmer. You meant literal "spikes", on the screen. Sorry about that.
Honestly, they look like to me as if they were a half-rendered trees with the wrong LOD. I would give him a debug build, log everything happening, ask him to play normally and when it happens, send the log back to you. Maybe you have a rare error. Also if you made everything deterministic, you can log out the seed you're using to generate the landscape, so you can reproduce it locally (if you didn't, then it's not possible, probably).

 

I got ahold of his world data file, where the main biome/world seed settings etc are stored, put it in a fresh world folder and loaded into it and got the spikes first attempt... I've repeated those steps a dozen times, and made a test spawner to randomize the order in which it spawns the terrains using the same method/seed/math (there is some built-in engine math that kicks in when neighboring terrains double check to make sure their edges are meeting up precisely I was thinking) and some things... but I haven't been able to get it to happen a second time I should have taken a screenshot the first time... off to bed for today, but apparently it can happen on my machine as well, and his specs were an Intel i7 and GTX, so nothing obscure... just very rare? He said he played 8-10 hours yesterday and only saw one spike during his last restart/login for that day...

 

So the errors that I was finally able to see in the editor on my own machine yesterday were at like 2000m, just fyi.

I ran the same math/methods that generate the terrain heightmaps using 256x256 random values between 0-999999 and reran the same math 100 times to try and just brute force an error, checking each step against the first. Generated new numbers and ran then again for 100 times. Doing that whole thing for 250 iterations over about half an hour and was not able to generate any math errors.

I wrote a message to Jason Booth the author of Microsplat as that's the shader I'm using on the terrains, in case he has any idea what could be going on.

If it's not my math, and it's not the shader, Unity's internal math is the only thing left I can think of... I'm at a loss of what else to look into/test though.

The errors are not consistent, I can set up a method to log the current heightmap array in the event of someone spotting one, to try and look at if there isn't something weird with the actual numbers internally, but that's about it, and that wouldn't really provide me any sort of approach to repair =/

 

Hi @malkere, can you please show a couple of images of these spikes? The two images you have provided do not seem to show any that I can see (I assume some of the distant trees are without leaves/branches deliberately?).
You said you use a form of world shifting, what threshold do you use for the shift?

 

I use a system of floating origin with a threshhold of 8192m. I have seen the spikes, only once unfortunately, in the editor within 2000-3000m though.

 

Real floating origin does not move the player (It starts and stays at Zero), so you are using world/origin shifting with conventional player movement.

That is a pretty big threshold and is likely to produce errors. Errors from this player movement algorithm are random because they vary with position, and are magnified by the amount of calculation performed each frame.

I can make some educated guesses as to how you might reproduce your issue.

Sometimes the closer you get to objects the more likely you will see this kind of error. You may be able to reproduce the issue if you increase the threshold further and move to near the threshold distance from the origin. If you do that for both (rather than just z or z ) x and z then it will likely be easier to reproduce.

Since you are talking about vertical spikes, it may be related to z-buffer fighting from tree/terrain objects. Try travelling (larger and larger) thresholds from the origin and viewing a bunch of those trees that have steep terrain just behind them.

 

Unfortunately, as stated early in the thread, I disabled "shifting" and travelled out beyond 1,000,000km where you can see the player model is severely affected by the precision errors, but was still unable to produce any sharp terrain spikes anywhere.

Furthermore what the screenshots reveal is something along the lines of terrain heightmap data which ranges from 0.0-1.0 somehow going from for example 0.2 to 0.25, and only at certain points for some reason. If they were precision errors we would need to be making use of the precision with numbers in the millions to be generating errors at the second decimal.

I appreciate the thought cycles on the problem though, it still persists, much more frequently for one player than the rest, but I have seen it a big once ever.

 

I'm in the middle of preparing for release out of EA, so haven't really tested on what's happened recently, BUT

last night I switched from an AMD RX 590 to a GeForce GTX 1080, and almost immediately ran into a very spikey terrain. I had already implemented (after starting this thread) a console command to trigger a a heightmap check on all of the terrains in scene and print out any points it finds that are more than 120m different than any of their neighbors. The method found zero occurrences in terrain heightmap data despite the 400-500m spikes plainly visible.

This makes me think it has something to do with the GPU drivers? Or something under the hood converting numbers to C++, or?? I still don't understand why this seems to only be happening to me.

 

This is a super interesting bug!

A few questions:

1. "almost immediately ran into a very spikey terrain" - is that in Editor, or in a build? If in Editor, do you only see the spikes in Game view or also in Scene view?
2. Just to be extra sure - when you select one of those spikes in scene view, does it select the terrain?
3. Do the spikes persist if you drag the terrain around? If it doesn't happen in Editor, maybe testing a console command that just moves the terrain game object forward by increments of X could work.
4. It's hard to tell from the screenshot - but it looks like each spike is the same height, rather than reaching the maximum terrain height - is that correct? if so, can you check what exactly is that height? That might be a clue.
5. Can you move towards one of those spikes? Do they block the player/projectiles?

 

Well Steam f***** up my release, now I'm waiting for a Steam rep to fix it, so I can look into this
I saw it most recently while recording footage in the executable. Again the console command was designed to look for any changes greater than 0.03 in any neighbor of every point in the heightmaps, and came up with no results. trying to move them and click on them are good ideas though, and I have seen them very rarely in the editor in the past, so maybe I can try to spot them in the editor again with my new card.

Yes, from what I've seen the spikes do seem to be the same height. My terrains are all 4000 height, so a 0.1 would be 400m. My game also runs a second background set of terrains at 1/16th the size, to fake a long distance view of 48km. These terrains are only rendered on a second, lower depth camera though, and can stick through the main terrain/camera if they are erroneously jutting upwards. Normally they're kept like 20m lower to make sure they don't poke through, but the spikes can. I'm not sure if it's only one or the other that gets the spikes, but the fake terrains don't have colliders. I will try to test some of that out though. Thank you for the ideas.

 

Are you generating this terrain or are these pre-made (editor-made) terrains that are glitching?

Make some instrumentation so that when you see the spikes, it writes the raw heightmap data to a file.

Doing so might isolate if the heightmap data is truly faulty (what it appears to be), or if this is some weird video driver tesselation issue, or something related to LOD on the heightmaps.

Do they change with distance?

 

(Again swamped with the release of my game, and going well! Yay!)

I don't think they change with distance I've seen screenshots of people walking right up to them. They are 100% run time generated. I could try exporting a raw heightmap, my math checker didn't seem to notice anything:

Code (CSharp):

1. using System.IO;
2. using UnityEngine;
3.  
4. public class HeightmapExport : MonoBehaviour {
5.  
6.     void Start() {
7.         Terrain t = GetComponent<Terrain>();
8.  
9.         float[,] heights = t.terrainData.GetHeights(0, 0, 256, 256);
10.  
11.         string JSON = "{";
12.  
13.         JSON += "\n" + "MapSeed:" + TerrainsData.seed + ",";
14.  
15.         for (int x = 0; x < 256; x++) {
16.             for (int z = 0; z < 256; z++) {
17.                 if (x == 0 || z == 0 || x == 255 || z == 255) { continue; }
18.  
19.                 float height = heights[x, z];
20.  
21.                 float topDifference = 0;
22.  
23.                 GetDifference(height, heights[x, z + 1], ref topDifference);
24.                 GetDifference(height, heights[x + 1, z + 1], ref topDifference);
25.                 GetDifference(height, heights[x + 1, z], ref topDifference);
26.                 GetDifference(height, heights[x + 1, z - 1], ref topDifference);
27.                 GetDifference(height, heights[x, z - 1], ref topDifference);
28.                 GetDifference(height, heights[x - 1, z - 1], ref topDifference);
29.                 GetDifference(height, heights[x - 1, z], ref topDifference);
30.                 GetDifference(height, heights[x - 1, z + 1], ref topDifference);
31.  
32.                 if (topDifference > 0.03) {
33.                     JSON += "\n" + "DifferenceOf:" + topDifference + ", at " + x + ", " + z;
34.                 }
35.             }
36.         }
37.  
38.         JSON += "\n}";
39.  
40.         string fileName = Application.dataPath + "/" + t.gameObject.name + ".txt";
41.  
42.         StreamWriter writer = File.CreateText(fileName);
43.         writer.Write(JSON);
44.         Debug.Log("Exporting " + fileName);
45.         writer.Close();
46.     }
47.  
48.     void GetDifference(float centerHeight, float testHeight, ref float topDifference) {
49.         float difference = Mathf.Abs(centerHeight - testHeight);
50.         if (difference > topDifference) { topDifference = difference; }
51.     }
52.  
53. }

(some of that is copied from somewhere I believe, no actual JSON going on)