Compressing a big OpenCV Matrix into a 3D texture with Unity or not

I am a student and my project group wants to convert a very large OpenCV Matrix (about ten gigabytes) into 3D Texture, but the maximum resource capacity we have is 2 gigabytes for the 3D texture. we don't know how to optimize this matrix to be able to create the texture. we are working with UNITY tool and it is for 3d visualization project

 

Out of curiosity (this may even help):
How much bytes does it take per sample/voxel?
How do you plan to visualize it?

 

What is your question?

Maximum 3d texture size in unity is 2048x2048x2048, which is 8.5 billion texels. Depending on texture format, this will be at least 8 GiB, but the size can go up if you choose different format.
https://docs.unity3d.com/Manual/class-Texture3D.html

Additionally, you can use Compute SHaders, and with compute shaders you can use structured buffers.

If you want to compress the texture, then you can save the texture as dxt. Depending on the data type, this can seriously crunch the data.

Additionally, you can use whatever compression algorithm you want by passing data through structured buffer and decompressing it within the shader.

 

Basically, our project consists in taking as input an OpenCV matrix which will be hundreds of gigabytes, transforming it into a 3D texture whose capacity must be in line with that which Unity accepts (This is where optimization comes into play ) and then make cut planes and output a chosen 2D matrix. I don't know if it's understandable now.

 

So you have to show a 2D matrix at a time. What's the resolution (and size if you know) of said matrix? What are your hardware minimum requirements?

 

It is understandable, however, it is still unclear what your question or problem is.

If you want to compress it, then compress it.

 

The problem is how compress OpenCV Matrix easyly because 3D texture can't support hundreds of gigabytes. We are beginners in the field.
We have I7 10700 16go RAM DDR4 Nvidia T400, 2go VRAM

 

What are dimensions and type of data?

Unity can go up to 2048x2048x2048 with volumetric textures. The volumetric texture can be stored as lossy compressed format, as far as I'm aware.

You can also roughly gauge how compressible the data is by trying to bluntly 7z it. If it compresses well, you can probably create something similar. If it is highly irregular numeric array, you're probably out of luck.

In general, the ways you could compress data go like this:
* Downsample. Create lower resolution representation based on nearby values. Works for vector fields, gradients.
* Octree. Start with high level structure, subdivide, store data in leaves only when data is present in the cell. Works for sparse data with a lot of empty space.
* Quantize. Build a palette of values, use palette index in the main texture instead of color. Depending on source data precision, you can remove a lot of bytes this way. The result can be lossy.
* Quantize with compression. Same deal, except apply any compression to the indexed color.
* Reduce precision. If you're using double precision, switch to single or half precision. Or even 8bit. This is lossy too.

It basically depends on what you're trying to do with that matrix of yours.

Also please note that you can chat to free ChatGPT about it and it might give you some ideas.

 

You may slice your 3d matrix to n 2d matrix and save them to disk as a texture, then load one 2d matrix at a time and display it with Unity. I can't say much more if you aren't more specific about your project. Your hard disk type will influence greatly the performance of this kind of solution, if you have a NVMe one would be much better.

 

Hello, we cannot do this because it is up to us to clear the oblique planes as well. so if we divide into n 2d matrix, it will be complicated to save that

 

What do you mean with "clear the oblique planes"?
You surely need a way to save and compress your 3D texture to disk unless you are still using OpenCV inside of Unity.

 

Our project concerns medical data so we cannot afford to lose data. Input data are TIFF images that have been converted to OpenCV 3d Matrix. is it possible to use Octree Structure for 3D Texture?
Beside, where can we store our 3d texture which is bigger that our VRAM capacity?

 

I mean that we need to generate some planes that aren't orthogonal to the classic axes x,y,z.

 

Do you have some kind of functional MRI black and white images there?

 

Yes, we have that and others colors images.

 

That should be sparse data so you could experiment with the octree data structure to test what level of compression you can get, if the whole tree would fit in 16GB of RAM so you can transfer a slice/part of that to VRAM to show it.

 

Maybe you could try a low rank approximation of your big matrix - most images have a small amount of principal eigen vectors relative to their size. This would allow you to split your big matrix into a product of two (potentially much) smaller matrices.