ECS with Texture Mipmap Streaming?

goonter · Apr 25, 2019

I have a setup similar to the megacity project where I load in entities using subscenes + HLOD + StaticOptimizeEntity. I'm testing out loading an object that is textured with mipmap streaming enabled and it only seems to load the lowest quality mip level and never any higher no matter how close the camera gets to the object. I have another similar object in the main scene that is working fine so I know mipmap streaming is working for normal gameobjects in my test build.

Any ideas about how to fix this? I assume it's because the Hybrid Renderer is not communicating with the mipmap streaming system, but I'm hoping there is a workaround or some modifications I could make to the Hybrid Renderer package to fix it.

goonter · Apr 26, 2019

Also fyi, the texture that is on the entity with mipmap streaming enabled shows up as blue in debugging view, which the docs say is "Textures that are not set to stream, or if there is no renderer calculating the mip levels"

goonter · May 1, 2019

If anyone is interested, I ended up writing a system using ECS that uses the Texture Streaming API to change the requested streaming mips manually for Hybrid Rendered Entities. The basic structure is this:

StreamingTexture SharedComponent

Texture2D MainTex

Texture2D NormalTex

Texture2D RoughTex

StreamingTextureInfo Component (added during conversion if the MeshRenderer mainTex is set to stream)

float MeshUVDistributionMetric

float TexelCount

StreamingTextureChunkMip Component

int Value

StreamingTextureCamera Component

float MipMapBias

float FieldOfView

float PixelHeight

A StreamingTextureSystem then queries by SCD StreamingTexture and StreamingTextureInfo, calculates the desired mip for each StreamingTextureInfo based on the info + WorldBounds + CameraPosition. Then I set the lowest requested mip for each chunk in StreamingTextureChunkMip.

Then a MipChangeSystem that gets all StreamingTexture chunk, sorts them by value using NativeArraySharedValues, then for each unique value, check for the lowest StreamingTextureChunkMip, and finally GetSharedComponentData for the unique StreamingTexture and do the Texture Streaming API calls for each Texture2D (i.e. MainTex.requestedMipmapLevel = lowestChunkMip).

With 250,000 streaming texture RenderMesh entities, the StreamingTextureSystem runs at about 0.10ms, and the MipChangeSystem at about 0.70ms.

Hope this helps someone who is wanting to do something similar. This was really the missing piece for me with the megacity demo and adding it in makes all the tech together a really nice world streaming toolset.

Also just want to say to the Unity team that the Texture Streaming system is amazing. I have six test 8k textures that would usually be 85.3MB each in memory in a build, and with Texture Streaming at the lowest mip, they only load 5.7KB into memory!!! It's like magic.

Opeth001 · May 2, 2019

Can you share some ScreenShots of Performance + your Code ?
Thanks!!

goonter · May 7, 2019

I'm still working out some weird bugs but I might share it once I get it in a better state! One hack for anyone who attempts something similar: You need to have at least 1 standard non-ecs game object in your scene that uses texture streaming for the system to become enabled. If you don't, I think the whole Texture Streaming system turns off and the ECS-based mip change requests don't do anything.

julian-moschuering · Dec 17, 2020

My current solution for this. You have to provide CameraData singleton, everything else should work out of the box.
Does ignore transform scales, as I don't need that atm.

Code (CSharp):

using System.Collections.Generic;

using Unity.Collections;

using Unity.Entities;

using Unity.Mathematics;

using Unity.Rendering;

using Unity.Transforms;

using UnityEngine;

public struct CameraData : IComponentData

{

public ushort Width;

public ushort Height;

public half FovX;

public half FovY;

}

internal struct TS_Init : IComponentData

{

public bool IsValid;

}

internal struct TS_MinDistance : IComponentData

{

public float Value;

}

// Buffer as ChunkComponent (which is possible by manipulating Chunk entity directly) leads to crashes on destruction atm

internal struct TS_TextureIndex : IComponentData

{

public byte Count;

public unsafe fixed ushort Value[8];

}

[UpdateInGroup(typeof(StructuralChangePresentationSystemGroup))]

[WorldSystemFilter(WorldSystemFilterFlags.Default)]

public class TextureStreamingInitSystem : SystemBase

{

EntityQuery missingQuery;

public NativeList<float> metric;

public NativeList<byte> currentMipLevel;

public NativeList<byte> lastMipLevel;

public NativeList<byte> maxMipLevel;

public List<Texture2D> textures;

Dictionary<Material, ushort[]> materialTextures = new Dictionary<Material, ushort[]>();

protected override void OnCreate()

{

missingQuery = GetEntityQuery(new EntityQueryDesc

{

All = new[] {ComponentType.ReadOnly<RenderMesh>()},

Any = new[] {ComponentType.ReadOnly<RenderBounds>(), ComponentType.ReadOnly<WorldRenderBounds>()},

None = new[] {ComponentType.ChunkComponentReadOnly<TS_MinDistance>(), ComponentType.ChunkComponentReadOnly<TS_TextureIndex>()},

Options = EntityQueryOptions.IncludeDisabled | EntityQueryOptions.IncludePrefab

});

metric = new NativeList<float>(Allocator.Persistent);

currentMipLevel = new NativeList<byte>(Allocator.Persistent);

lastMipLevel = new NativeList<byte>(Allocator.Persistent);

maxMipLevel = new NativeList<byte>(Allocator.Persistent);

textures = new List<Texture2D>();

}

protected override void OnDestroy()

{

metric.Dispose();

currentMipLevel.Dispose();

lastMipLevel.Dispose();

maxMipLevel.Dispose();

textures = null;

base.OnDestroy();

}

protected override unsafe void OnUpdate()

{

var initTypes = new ComponentTypes(ComponentType.ChunkComponent<TS_Init>(),

ComponentType.ChunkComponent<TS_MinDistance>(),

ComponentType.ChunkComponent<TS_TextureIndex>());

EntityManager.AddComponent(missingQuery, initTypes);

var renderMeshHandle = GetSharedComponentTypeHandle<RenderMesh>();

// todo: do a bursted parallel for to check if there is anything to do

// todo: assign by SharedComponentIndex in bursted job if known

Entities.ForEach((ref TS_TextureIndex textureIndices, ref TS_Init init, in ChunkHeader chunkHeader) =>

{

var chunk = chunkHeader.ArchetypeChunk;

if (init.IsValid && !chunk.DidChange(renderMeshHandle, LastSystemVersion))

return;

init.IsValid = true;

var renderMesh = chunk.GetSharedComponentData(renderMeshHandle, EntityManager);

var material = renderMesh.material;

if (material != null) // why?

{

var indices = GetMaterialTextures(material, renderMesh.mesh, renderMesh.subMesh);

textureIndices.Count = (byte)math.min(indices.Length, 8);

for (var i = 0; i < textureIndices.Count; i++)

textureIndices.Value[i] = indices[i];

}

}).WithoutBurst().Run();

}

// todo: yes, we ignore that each texture might be used on different meshes

// our evaluation will just always use the highest resolution

ushort[] GetMaterialTextures(Material material, Mesh mesh, int subMesh)

{

if (materialTextures.TryGetValue(material, out var indices))

return indices;

var ids = material.GetTexturePropertyNameIDs();

var indexList = new NativeList<ushort>(Allocator.Temp);

for (var i = 0; i < ids.Length; i++)

{

var texture = material.GetTexture(ids[i]) as Texture2D;

if (texture == null || !texture.streamingMipmaps)

continue;

var index = textures.IndexOf(texture);

if (index < 0)

index = AddNewTexture(texture, mesh, subMesh);

indexList.Add((ushort)index);

}

indices = indexList.ToArray();

indexList.Dispose();

materialTextures[material] = indices;

return indices;

}

ushort AddNewTexture(Texture2D texture, Mesh mesh, int subMesh)

{

var index = textures.Count;

textures.Add(texture);

metric.Add((texture.width * texture.height) / mesh.GetUVDistributionMetric(0));

currentMipLevel.Add((byte)(texture.mipmapCount - 1));

maxMipLevel.Add((byte)(texture.mipmapCount - 1));

lastMipLevel.Add(0);

return (ushort)index;

}

}

[UpdateInGroup(typeof(UpdatePresentationSystemGroup))]

[WorldSystemFilter(WorldSystemFilterFlags.Default | WorldSystemFilterFlags.EntitySceneOptimizations)]

public class TextureStreamingSystem : SystemBase

{

TextureStreamingInitSystem textureStreamingInitSystem;

protected override void OnCreate()

{

base.OnCreate();

textureStreamingInitSystem = World.GetExistingSystem<TextureStreamingInitSystem>();

// make sure everything that happened before is reset

foreach (var texture in Resources.FindObjectsOfTypeAll<Texture>())

{

if (!(texture is Texture2D t2d) || !t2d.streamingMipmaps)

continue;

t2d.ClearRequestedMipmapLevel();

}

}

protected override void OnUpdate()

{

var refPoint = new NativeArray<float3>(1, Allocator.TempJob);

Entities.WithAll<CameraData>().ForEach((in LocalToWorld localToWorld) =>

{

refPoint[0] = localToWorld.Position;

}).Schedule();

CalculateDistances(refPoint);

CalculateRequired(GetSingleton<CameraData>());

AssignRequired();

refPoint.Dispose(Dependency);

}

void CalculateDistances(NativeArray<float3> refPoint)

{

var worldRenderBoundsHandle = GetComponentTypeHandle<WorldRenderBounds>(true);

var dontCareDistance = 80.0f;

Entities.WithReadOnly(refPoint).WithReadOnly(worldRenderBoundsHandle)

.ForEach((ref TS_MinDistance minCameraDistance, in ChunkHeader chunkHeader, in ChunkWorldRenderBounds chunkWorldRenderBounds) =>

{

if (DistanceSq(chunkWorldRenderBounds.Value, refPoint[0]) > dontCareDistance * dontCareDistance)

{

minCameraDistance.Value = dontCareDistance;

}

else

{

// per instance distance

var chunk = chunkHeader.ArchetypeChunk;

var worldRenderBounds = chunk.GetNativeArray(worldRenderBoundsHandle);

var distanceSq = dontCareDistance * dontCareDistance;

for (var i = 0; i < worldRenderBounds.Length; i++)

{

distanceSq = math.min(distanceSq, DistanceSq(worldRenderBounds[i].Value, refPoint[0]));

}

minCameraDistance.Value = math.sqrt(distanceSq);

}

}).ScheduleParallel();

}

static float DistanceSq(in AABB aabb, float3 point)

{

var nearest = math.max(math.min(point, aabb.Max), aabb.Min);

return math.distancesq(point, nearest);

}

struct MipSet

{

public ushort TextureIndex;

public byte Mip;

}

unsafe void CalculateRequired(CameraData cameraData)

{

var metrics = textureStreamingInitSystem.metric.AsArray();

var currentMipLevel = textureStreamingInitSystem.currentMipLevel.AsArray();

var cameraHalfAngle = math.radians(cameraData.FovY) * 0.5f;

var screenHalfHeight = cameraData.Height * 0.5f;

var cameraEyeToScreenDistanceSq = math.pow(screenHalfHeight / math.tan(cameraHalfAngle), 2.0f);

var aspectRatio = (float)cameraData.Width / cameraData.Height;

if (aspectRatio > 1.0f)

cameraEyeToScreenDistanceSq *= aspectRatio;

var results = new NativeQueue<MipSet>(Allocator.TempJob);

var resultsWriter = results.AsParallelWriter();

Entities.WithReadOnly(metrics).WithReadOnly(currentMipLevel)

.ForEach((ref TS_MinDistance minCameraDistance, in ChunkHeader chunkHeader, in TS_TextureIndex textureIndices) =>

{

for (var i = 0; i < textureIndices.Count; i++)

{

// todo: scale not handled; we don't scale really

var textureIndex = textureIndices.Value[i];

var dSq = minCameraDistance.Value * minCameraDistance.Value;

var v = metrics[textureIndex] * dSq / cameraEyeToScreenDistanceSq;

var baseMip = 0.5f * math.log2(v);

var mip = (byte)math.clamp(baseMip, 0, byte.MaxValue);

if (mip < currentMipLevel[textureIndex])

resultsWriter.Enqueue(new MipSet {TextureIndex = textureIndex, Mip = mip});

}

}).ScheduleParallel();

// reduce

Job.WithCode(() =>

{

while (results.TryDequeue(out var item))

currentMipLevel[item.TextureIndex] = item.Mip < currentMipLevel[item.TextureIndex] ? item.Mip : currentMipLevel[item.TextureIndex];

}).Schedule();

Dependency = results.Dispose(Dependency);

}

void AssignRequired()

{

var count = textureStreamingInitSystem.textures.Count;

var currentMipLevel = textureStreamingInitSystem.currentMipLevel.AsArray();

var lastMipLevel = textureStreamingInitSystem.lastMipLevel.AsArray();

var maxMipLevel = textureStreamingInitSystem.maxMipLevel.AsArray();

var changeCounter = new NativeArray<int>(1, Allocator.TempJob);

var changed = new NativeArray<ushort>(count, Allocator.TempJob, NativeArrayOptions.UninitializedMemory);

Dependency = new CollectChanged

{

ChangeCounter = changeCounter,

Changed = changed,

CurrentMipLevel = currentMipLevel,

LastMipLevel = lastMipLevel,

}.ScheduleBatch(count, JobsUtility.CacheLineSize, Dependency);

// todo: easy fix, don't forget

Dependency.Complete();

Job.WithCode(() =>

{

lastMipLevel.CopyFrom(currentMipLevel);

currentMipLevel.CopyFrom(maxMipLevel);

}).Schedule();

var changeCount = changeCounter[0];

for (var i = 0; i < changeCount; i++)

{

var textureIndex = changed[i];

textureStreamingInitSystem.textures[textureIndex].requestedMipmapLevel = lastMipLevel[textureIndex];

}

changeCounter.Dispose();

changed.Dispose();

}

struct CollectChanged : IJobParallelForBatch

{

[NativeDisableParallelForRestriction]

public NativeArray<int> ChangeCounter;

[NativeDisableParallelForRestriction]

public NativeArray<ushort> Changed;

[ReadOnly] public NativeArray<byte> CurrentMipLevel;

[ReadOnly] public NativeArray<byte> LastMipLevel;

public unsafe void Execute(int startIndex, int count)

{

ref int counter = ref UnsafeUtility.ArrayElementAsRef<int>(ChangeCounter.GetUnsafePtr(), 0);

var end = startIndex + count;

for (var i = startIndex; i < end; i++)

{

if (CurrentMipLevel[i] != LastMipLevel[i])

Changed[Interlocked.Add(ref counter, 1) - 1] = (ushort)i;

}

}

}

}

Search Unity

Unity ID

Useful Searches

ECS with Texture Mipmap Streaming?

goonter

goonter

goonter

Opeth001

goonter

julian-moschuering