Render at half resolution, upsample to full resolution

I'm trying to render a camera at half resolution and then upsample the resulting rendertarget to fullresolution.
So if Screen.width = 640 and Screen.height = 480, the camera should render at 320x240 and then upsample the image to 640x480 using a custom shader (hq2x, xBR, etc).

I tried different methods, but feel like all of them are wrong and very hacky. Basically, i always have to display the resulting, upscaled image through the gui (which seems really unelegant. I'd rather use OnRenderImage / Graphics.Blit ).
Also, none of the methods i tried work on the Ouya (and on other tegra/mobile devices if i remember correctly).



Here's one of my current solutions

Code (csharp):

1.  
2. RenderTexture rtFull; //full size rendertarget
3. int upsampleFactor = 2; //render at half res
4. Camera downSampleCamera;
5. Rect orgCameraRect
6.  
7. void Start () {
8.         downSampleCamera = GetComponent<Camera>();
9.         orgCameraRect = new Rect(downSampleCamera.rect);
10.  
11.         rtFull = new RenderTexture((int)(Screen.width), (int)(Screen.height), 0, RenderTextureFormat.Default, RenderTextureReadWrite.Linear);
12.         downSampleCamera.rect = new Rect(orgCameraRect.x/upsampleFactor, orgCameraRect.y/upsampleFactor, orgCameraRect.width/upsampleFactor, orgCameraRect.height/upsampleFactor);
13.         downSampleCamera.targetTexture = rtFull;
14.        
15.         mtlUpscale = new Material( Shader.Find("Hidden/Upsampler/hq2x") );
16.     }
17.  
18. void OnRenderImage(RenderTexture src, RenderTexture dest) {
19.         src.filterMode = FilterMode.Point; //Set filtering of the source image to point for hq2x to work
20.         Graphics.Blit(src, dest, mtlUpscale); //Upscale the image
21.     }
22.  
23. void OnGUI()
24.     {
25.         //Display the image on screen...this is an ugly solution...
26.         GUI.DrawTexture(new Rect(orgCameraRect.x*Screen.width,
27.                                  orgCameraRect.y*Screen.height,
28.                                  rtFull.width,
29.                                  rtFull.height),
30.                         rtFull);
31.     }
32.  

So what's the correct way of doing this? The method from above is ugly and (as stated) doesn't work on the Ouya.

We really need this to increase performance of our game In Between on mobile devices / Ouya.

Click to enlarge





Any help would be GREATLY appreciated!

 

I can't see a reason for a separate render texture... Just clip the rendering to half res window and leave the OnRenderImage as you have it now...

 

Thanks for you reply.

If i don't use a separate render texture, then the destination rendertarget in OnRenderImage will only be half res (as the camera is half res, too). In order to upsample from half res to full res i need a full res rendertarget.

No separate rendertarget. Final Image stays at half res:


Separate rendertarget. Final Image is upsampled to full res:


I hope that helps to understand what the actual problem is

 

Another option is to write custom shaders so that you build-in the upscaling algorithm into the original rendering pass, straight to the backbuffer, removing the need for the render texture altogether.

 

Hmm.. one would think setting Camera.rect wouldn't shrink the render target to the rect's size. Have you tried overriding Camera.rect in OnPreRender and back OnPostRender? That might trick it into using the full target size.
If that doesn't work, you could try fiddling with the camera's projection matrix manually...

 

Why can't you use Screen.SetResolution?

http://docs.unity3d.com/Documentation/ScriptReference/Screen.SetResolution.html

 

Thanks for the answers! Much appreciated.

How would i do that? Replacing (or adding) the actual model shader with an upscaling filter won't help me. I need to apply the upscaling in screenspace, not per model. The idea behind all this is to reduce the overall pixelcount the camera has to render. So rendering at fullscreen, with an upscale filter on each model wouldn't help me here, except killing the fps even more.
Or did i get you wrong and you mean something completly different?

Yes, i already tried setting Camera.rect in OnPreRender and OnPostRender. This doesn't work (And it makes sense that it doesn't, but once you are desperate you try lots of things that shouldn't work. )
Changing the camera matrix might work...i'll give that a try. I'm not sure if this will lead to a performance increase...but i'll definitely give it a try. Thanks for the idea!

I want to upscale the image with an upscale shader like hq2x, xBR, SaI, etc. Using Screen.SetResolution just scales down everything and the resulting image is then scaled back to full size on the monitor, resulting in a very blurred image. You can try that for yourself: Play a game, set it's resolution to the lowest resolution available. The resulting image on your screen will be very blurry. Using an upscale shader will get rid of most of the blurriness. Also, using Screen.SetResolution will scale down the GUI, too, resulting in a blurred GUI. Using an upscale filter, the GUI is rendered at full size and only the camera view needs to be upscaled. This way you get a crisp GUI and an acceptable game view.

 

Why is your original render target full resolution? Isn't that one supposed to be half resolution?

I'd say these are the steps:
1. Render whole scene at half resolution. This could be achieved by simply adding a camera that's a copy of the main camera and putting a half resolution RenderTexture as target.
2. The main camera now only needs to upscale the half size texture. It doesn't need to render anything else. I think you can set it to render an unused layer, so it skips rendering everything. Then you can do a little hack in a post processing script. Instead of reading the given source, you read your half size texture. (This does break the intention of the post process pipeline a bit, but it keeps thing fairly simple.)

Code (csharp):

1.  
2. public RenderTexture half;
3.  
4. void OnRenderImage(RenderTexture src, RenderTexture dest) {
5.         // We are completely ignoring src
6.         half.filterMode = FilterMode.Point; //Set filtering of the source image to point for hq2x to work
7.         Graphics.Blit(half, dest, mtlUpscale); //Upscale the image
8. }
9.  

Of course you must make sure your "half" camera renders before the main camera. And any additional post process effects need to be applied after the upscale post process.

A better way would be to not use a second camera of course. In that case you have to crop the rendering into a quarter of the backbuffer and the post process that quarter back to a full screen. That probably does require some adjustments to any of the shelve hq2x shader. (The input texture coordinates will need some adjustment.) I guess that's also pretty much what Dolkar already said.

 

Unity pretty much has this built in with resolution settings. If you render a smaller res than native it'll upscale.

 

But can you provide the shader/material that does the upscaling?

 

Isn't it hardware linear upsampling that happens automatically when using a non-native screen resolution and hence works only in full screen and only on hardware that supports that resolution?

 

It's honestly not required for mobile hardware including mini consoles like Ouya. You make your game normally as you do, with resolution independence - then just change the res in Start and Unity will handle it all internally for you, for a free speed bump.

Why would you want to supply the shader for it? I thought your problem here was one of speed.

 

I don't foresee any point in using a (not-bilinear) scaling algorithm; it'll be slower than rendering at native res. It would be useful for pixel art, but if you don't want pixel art, then don't make pixel art. You aren't making pixel art.


That said, I would like to know how we should handle this kind of thing. In an older thread on the same topic , Eric advised me to put the render texture into a GUI Texture, which works, but I don't know if that is the cleanest or fastest solution. Here's my current process:

1. Set low-res point filtered texture as render target for cameras.
2. Blit those into larger, bilinearly-filtered textures, of an integral multiple size.
3. Assign those textures to the GUI Textures' texture properties.

When do I do step 2 and 3??? OnGUI is all that I can get to reliably work, but I can't believe that's what I should be using.

 

If I look at the screenshots, it does resemble pixel art a bit. So, I assume that'sbthe logic behind the hq2x upscale. In terms of performance it might not be the most logical thing to do, but it can still perform better than the full screen render. That is to say if the fill rate is a major bottle neck and the overdraw is high enough to justify another full screen upscale pass.

In general, a bilinear upscale looks like smudge, while a hq2x upscale looks like sweet retro times, so that choice I can understand. Microsoft actually made the best looking retro upscale algorithm I've seen yet. Which I find surprising, because the retro feeling is a major selling point of large competitor in the game market. The microsoft paper is located here by the way. I don't think you'll be able to get that running on a mobile device, but it looks so nice you want to start rendering everything at thumbnail size.

 

Well, you are probably right. With DirectX 10+ compute shaders and random writes, it might be possible to get it running though. It's not fully raster based like most of these algorithms, but the first steps still are. And with DirectX 10+ you can probably even run the spline optimization on the GPU. Granted it wouldn't be the easiest task. 790ms on a CPU, could still be realtime on a GPU though.

 

I must disagree...

I don't think any of those vector versions look like what the pixel artists intended. The graphics go from looking like competent people made them, when technology was limited, to looking like they were made in Illustrator, by somebody who had never used it before, but had to make a deadline in an hour.

I think pixel art upscaling algorithms are curiosities that don't actually solve problems. Pixel art rotation, however, I think is a worthy endeavor.

 

Wow, I haven't been here for a while.

Not necessarily. About 7 years ago, I looked into finding an efficient algorithm. I think it's still one of the best performing algorithms for nonlinear upscaling: http://scholar.google.dk/citations?view_op=view_citation&hl=da&user=YkM6sYoAAAAJ&citation_for_view=YkM6sYoAAAAJ:_FxGoFyzp5QC.

 

I guess it's also about personal taste. I find that running old roms on a high resolution screen doesn't fully use the potential of the screen resolution when using nearest neighbor upscaling. And these pixel art upscaling algorithms solve that problem for me. Besides that I find hqx upscaled pixel art more pleasing to the eye than nearest neighbor upscaled pixel art.

 

I gave up on creating an upsampler and red through the tegra architecture guide instead.

Turned out the tegra isn't using a tiled renderer and therefor you can make use of AlphaTesting (which is slow on PowerVR for example).
So i wrote a simple AlphaTest-Shader using clip(), applied that to big structures/sprites and got way better fps on tegra as overdraw has been reduced.
Haven't tried on other devices yet, but i have a feeling that it might also give a small perf increase on PowerVR devices as the z culling optimization is quite big now (we have lots of screen covering sprites in most of our story scenes) and should compensate the slow alphatesting on those devices.

While on it, i also turned off volumetric lights of our in-house 2D lighting system as it was making heavy use of alpha blending. Doesn't look as pretty on the Ouya now, but it is still looking good enough.


Also back to the upsampling stuff and if it's worth it:
For us, the main benefit would haven been a crisp gui. We could have lived with linear upsampling the game scene if hq2x would have been too slow (or a custom upsampler. Actually a semi-advanced sharpening filter already achieved good results in our case) .
However, changing the overall game resolution and upsampling the gui didn't work for us as it just looked horrible.

On a sidenote: I prefer point filtering when playing old games on a large screen. I can't stand most of the looks those upscale filters create. xBR isn't as bad as most of the others but it still kind of kills the look of old games.
Don't get me wrong though, those filters still have their use (bilateral upsampling is great for half-/quartersize SSAO for example) and i'm always impressed what people come up with.

In the end it all depends on the game and personal taste. Upsampling would have worked in our case as we don't have pixelart. On most pixelart games i prefer a simple point filter, though.