Profoundly stupid question: why are all textures squares or rectangles

I’m hardly a mathematician, but I do understand that we use power of 2 images that fit neatly into ram.

But why is that the default?

Why do we not use triangle and hexagon shaped images with sizes relative to those constraints and just sacrifice some pixels in the name of disrupting pattern recognition?

Even though the display uses a square grid, when ray casting and only picking a few pixels of an image does it really matter if the images pixels are hexagon or triangle if the ray is just returning values?

Im not making an argument that this is how it should be done, just wondering why.

Certainly a hexagon repeating texture would be far less susceptible to tiling issues. And triangle images would certainly fit UV maps a lot better in a lot of cases.

The pixels don’t HAVE to be square it seems.

and even if that were the case you could easily just arrange them differently by file.

So why is this not a thing?

 

Well, there is a reason, but on the other hand, it IS a thing, we just do not see it.
For once, the reason is memory efficiency. You can imagine memory as just a singular very, very long list of values. Now the challenge is how to store your 2D Data in that 1D structure. If you store a rectangle, you only need to know the width and the total number of bytes of your texture.
If you store random shapes, you need additional information that somehow describe that shape. So you have to use space on that additional information and furthermore there will be some computation effort to map that shape later onto whatever needs to be done with it.

That said, if you are willing to take the computation cost, some things can be done. Try to save a rainbow colored triangle as a .png with transparent background and then try to save a rainbow rectangle (of full size) as a .png.
The former file will end up smaller because .png can recognize that there is a lot of transparent pixels and stores them more efficiently. Technically the file format then does not store a plain rectangle for the triangle anymore.
Nvidia GPUs (and maybe AMD now too?) have hardware compression to squeeze more into the VRAM. The Driver handles this under the hood.

However the actual computation units still need the data in rectangular\"linear" form because they work sequencially (even if massively parallelized on GPUs).

P.s. hexagonal tiles should be achievable via a special shader. Not sure they would really help with appearing repetitive though.
There are other techniques for that: https://blog.unity.com/technology/procedural-stochastic-texturing-in-unity

 

I think you misunderstood me.

I know it’s always going to come down to current graphical libraries.

Im saying why not make graphical libraries that accommodate that?

And I’m not saying take a file type that already exists and make it do the thing. I’m saying make a new file type all together.

Square pixels make perfect sense in our current approaches, but even the guy that made the first image files said assuming square pixels was a mistake.

Give me a sec and I’ll cite.

EDIT: keep in mind I’m asking not asserting.

EDIT 2:

https://www.wired.com/2010/06/smoothing-square-pixels/amp

 

Also keep in mind that common knowledge dictated that an RGB sensory array found in digital cameras should be evenly distributed until biologists pointed out that we do not have an even amount of color receptors so modern cameras now have 50% green, 25% red, and 25% blue.

It’s the same reason we went from “blue screens” to “green screens”.

Because modern cameras have more green detail, like our eyes.

 

I think it's pretty simple. You have a bunch of bytes, it's TRIVIALLY easy to arrange them in a rectangle. Any other shape would require more slightly more work/computation.

 

Hmm, think it will still come down to the execution units performing sequentially aka in lines and rows. Not sure that could be accommodated well for other shapes.

 

And as an example here are two images of the same file size using triangle and rectangle pixels:

 

I can see computation being the crux.

Im a terrible coder and have to believe that people smarter than me know what they are doing.

I’m just curious is all.

 

At nothing else, I think it’s a fun read to see that an 80-something code monkey is still at it.

 

Yeah, but how do you represent triangular pixels vs. square? I imagine you are veering dangerously close into the non-trivial.

Here's the thing- Game engines focus on performance, and one of the key principles of optimization is to "make the common case fast". If you always represent textures in the exact same format then you can razor-focus on making that fast, but if you have multiple texture data representations, then your focus is divided.

In practice, you don't change the fundamentals to accommodate ease-of-use. Rather, you write a software abstraction layer to translate ease-of-use to the fundamentals.

Whether you like square textures or not is not important because textures are for the renderer, not for you.

 

Doesn't triangle texture require some kind of rasterization?
Square texture, you just load and sample pixel from that.

 

I can't believe nobody mentioned the most obvious thing.

Because when texture size is power of two you can calculate texel address using bitwise shift, which is significantly less expensive than multiplication.

When accessing texture

Pixel address = textureBaseAddress + y * scanlineSize + x * pixelSize

, where

scanlineSize >= xSize * pixelSize

, depending on packing.

If your texture size is not power of two, you have to use multiplications, although there are still tricks to avoid multiplication if you're using rectangles only.

If it is power of two, then you can calculate offset using bitwise shift, which is very cheap and

y* scanlineSize

turns into

i << shiftOffset

. It is also worth keeping in mind that some of the very early CPUs did not even have multiplication and division instructions. For example, z80 did not have it.

This sort of optimization makes a huge difference when you're programming for something working at clorkc rate of 66 Mhz. And regarindg non-power-of-two textures, those were expensive for a long time. In the past you could kill framerate by displaying NPOT screen aligned quad on a radeon. Been some time since then, though.

 

Going all the way back we had only 2D tiling engines and consoles with dedicated hardware that draws rectangular images (sprites). Triangles only came after that. Also a rectangle image format is trivial whereas a triangle data format is not, you have to add control data to know when to skip to the next line. And then a rectangle is always a rectangle no matter the size, with power of two simply being a special case. With triangles you can have the right triangles, the even-sided triangles or simply any kind of triangle down to a single line. That makes things oddly more complicated to store and read and interchange data (imagine Unreal using even-sided triangle textures whereas Unity supported only right-angle triangle textures - we’ve been there before with competing industry standards such as VHS and Bluray). In the end it‘s simple all around easier to map a rectangle to a bunch of triangles than it is to match a triangle to a bunch of triangles that perhaps do not match the image triangle‘s shape.

So why make things more complicated when there is absolutely no benefit?

 

It's already a thing in certain areas. It really depends on many factors, physical (hardware/software not a problem anymore) and not (habits, market adoption). For example Scalable Vector Graphics format can represent non raster images and is used in website development instead of png or other raster formats (usually because filesize can be smaller). I wouldn't use SVG for game development if not for particular unusual game styles, even if you would software support is rather limited.

 

The easiest way to organize 2d, 3d, and Nd data in memory is a grid. The grid has useful properties in sense, that you can easily subdivide it, scale it up or down and easily calculate pixel address. Additionally, computers use rectangular displays so it maps onto display easily as well, plus you can easily tile it.

You cannot subdivide a hexagon. Due to the way it is arranged, some hexagons will be cut.

Spoiler

You can subdivide a triangle, however, you cannot fill space with copies of the same triangle, because in triangular grid objects have to change orientation.

Spoiler

And lastly, when we speak about vector based image formats, then grid have advantage over them, because it is finite. You know how much RAM it is going to take to store image data in a grid. You do not know how much RAM it is going to ake to store a 2d vector image, because it can potentially have infinite number of shapes.

Basically, when we build houses with bricks, why are bricks rectangular? Why we aren't using prisms, tetrahedra or rhombic dodecahedron? Or why minecraft uses cubes and not another shape? The reason is similar.

 

Getting just a point, or distance on edge of triangle, is computationally expensive
Entering trigonometry area, or needing of using roots.



Or you could approach problem with Tales Theorem




With any other shape, problem becomes only more complicated, as other shapes are just bunch of triangles.
Wheels and curves are a bit different, but not less easier / cheaper to calculate.

Where in 2D greed of square, you simply take 2 two points and subtract. Job done.
I am talking about simple case scenario of course.

 

Isn’t that like saying Henry Ford should have made faster horses instead?

 

@neginfinity I get what you’re saying about bit shifting and that makes a lot of sense, but isn’t it still relatively cheap to use A relatively simple method to Translate locations?

using that triangle pattern you showed you can easily make a triangle with 256 triangle pixels.

add five more Triangles with 256 pixels and you’ve got a hexagon.

 

You can not just pick single triangle case scenario. You need take into consideration complexity of meshes. Like characters, terrain, props etc.

These are using tons of textures. Each polygon of each model need find position on the texture. Then do anything that shader may need to do with it. Now you scale relatively small problem into millions.

There is most likely no computational gain, having different type of shape. Instead most likely cost would increase.

Computing performance will be always on priority, over arbitrary convenience.
There tools, including Shader Graph, which deals which inconveniences for developers.

Also, mathematically computing square textures (i.e shaders) is easier, than any other shape.

 

Isn't a faster horse called motorcycle?

Henry Ford was not making faster horses, he was making faster carriages. And for a carriage a horse is not a necessary element, as it turned out.

Which method though? Do you know it or do you assume that it exists?

And what am I going to do with it? Our images are rectangular and triangle doesn't tile (rectangle does)

It will still take more operations to find pixel address than using a grid. The problem with triangles is that they have orientation, it matters, and they have varying number of triangles per row.

Spoiler

It is simply inconvenient. Every second triangle is flipped and that has to be taken into account.

Or I can add one more rectangle and it will become a very convenient rect.

Spoiler

So why would I want to store texture data as a hexagon?

It is also worth keeping in mind that there's already a way to store color information as an arbitrary shape. It is called vertex colors.

 

@Not_Sure the most important thing you should consider is...
What do you gain by switching to different data format?

The interesting thing is that in most cases you'll find out that underlying data is represented as a grid. So why not use a grid in the first place?

For example.
Hexagonal grid is actually a skewed rectangle..

Spoiler

Triangular grid is also a skewed rectangle:

Spoiler

Images are rectangles, video is rectangles, computer display is a rectangle.

Why not just use a rectangle in this case?

 

This.

You can of course come up with a smarter image format which gets a better looking image stored in less data, and that's super groovy, but at the end of the day it has to be displayed as a grid of pixels anyway, because that's how our displays physically work. And with that in mind, storing and using stuff as a grid of pixels earlier on is both simpler to understand and computationally less expensive for earlier stages in the the chain, too.

Referencing the cited Wired article, it strikes me more as a cool approach to compression than really changing the shape of a pixel. It starts with a standard, square-pixel image, splits it into sub-grids of 6x6, and then analyses them in different orientations to see which axis of separation has the highest contrast, then stores just 2 colour values and the shape. When it displays them each shape is mapped to a number of pixels on the display device and filled in with the stored colour. You may note that both input and output pixels are all rectangular.

It sounds a heck of a lot like a compression algorithm to me. That's supported by the article's own description, bold emphasis mine:

That is exactly image compression, right there. The algo is cool and it seems to get pretty nice results. Its output looks a little unique, because it keeps and discards very different information than most algos. It's a cool proof of concept.

But it hasn't really eliminated "square" pixels. It's just filling them in with a different approach.

- - -

As an aside, back at university in the computer graphics course, one of the interesting things we were taught is that "pixels do not represent rectangles". They are stored in a grid and they are displayed as rectangles for purely technical reasons, but the data in them could have come from anywhere within the represented area, and thinking as if it represents the whole area is a mental shortcut which can be detrimental in some cases. They argued instead that we should think about the data stored in a pixel as one or more samples of light taken from within that rectangular area and aggregated together.

That seemed like a distinction without a difference to me at first, until they got us to write our ray tracers. Early on we'd have our grid of pixels and fire one ray through the center of each one to pick a colour value from whatever shape we intersected. A little experimentation revealed plenty of cases where this could give arbitrary results for a given pixel, though, basically making our picture "noisy" (especially if we added textures or animation). Basically, the colour which happened to be at the center of a pixel's area wasn't necessarily a good representation of the whole area.

To address that, the simple approach is to fire many rays into each pixel's area, instead of just one, and then aggregate the result. More rays = a more accurate and stable representation of the actual image. It also gives natural anti-aliasing, because if there's an edge in the pixel's area then it's likely that samples will fall on both sides of it, distributed by where in the area the edge is (i.e. if the object fills most of the pixel, most samples will fall within the object). This is also a pretty close simulation of how an actual digital camera captures images - each photon which hits the light sensor is the physical equivalent of a ray coming from somewhere within that pixel's area on the far side of the lens.

As an example of how this applies to raster-based rendering, anti-aliasing techniques often use the principle in a fundamental sense. Old-school multi-sample anti-aliasing (MSAA) renders at a higher resolution (i.e. takes multiple samples per output pixel) and then downsamples (i.e. aggregates mutiple samples together) before display. Temporal AA (TAA) is a little trickier because it only generates one new sample per frame, but re-uses samples from previous frames and "jitters" the render position to spread out multiple samples for (most) display pixels. Texture filtering often approximates spreading out samples, too.

- - -

So, in short, Kirsch is absolutely right that assuming pixels are squares can be problematic for some computer graphics uses, but that doesn't mean that storage and computation using grids is going to go away.