CustomRenderTexture with TerrainPaintUtility

I assume this is possible, but so far I've only been able to make modifications to the terrain in a rectangle, not the shape of the texture I'm assigning to my CustomRenderTexture (Maybe I'm missing something?)

Here's what I've tried so far:

1. Created a gray-scale JPG for my test brush
2. Added this script to my terrain object:

Code (CSharp):

1. public class TerrainTest : MonoBehaviour
2. {
3.     [SerializeField] ComputeShader shader;
4.     public RenderTexture terrainRenderTexture;
5.     public CustomRenderTexture brush;
6.     public Texture brushTexture;
7.     public Vector2 rectSize;
8.  
9.     private Rect rect;
10.     private Terrain terrain;
11.     private PaintContext paintContext;
12.  
13.     void Start()
14.     {
15.         terrain = GetComponent<Terrain>();
16.     }
17.  
18.     void Update()
19.     {
20.         if (Input.GetMouseButton(0))
21.         {
22.             RaycastHit rayHit;
23.             var ray = Camera.main.ScreenPointToRay(Input.mousePosition);
24.  
25.             if (Physics.Raycast(ray, out rayHit))
26.             {
27.                 Vector2 terrainPoint = new Vector2(rayHit.point.x, rayHit.point.z);
28.                 rect = new Rect(terrainPoint, rectSize);
29.  
30.                 ModifyTerrain(rect);
31.             }
32.         }
33.     }
34.  
35.     void ModifyTerrain(Rect selectionRect)
36.     {
37.         paintContext = TerrainPaintUtility.BeginPaintHeightmap(terrain, selectionRect);
38.  
39.         terrainRenderTexture = paintContext.sourceRenderTexture;
40.  
41.         brush = new CustomRenderTexture(terrainRenderTexture.width, terrainRenderTexture.height);
42.         brush.enableRandomWrite = true;
43.         brush.format = RenderTextureFormat.R16;
44.         brush.initializationTexture = brushTexture;
45.         brush.Create();
46.  
47.         Graphics.CopyTexture(terrainRenderTexture, brush);
48.  
49.         shader.SetTexture(shader.FindKernel("CSMain"), "Result", brush);
50.         shader.Dispatch(shader.FindKernel("CSMain"), 1, 1, 1);
51.  
52.         Graphics.CopyTexture(brush, paintContext.destinationRenderTexture);
53.  
54.         TerrainPaintUtility.EndPaintHeightmap(paintContext, "Terrain");
55.  
56.         terrain.terrainData.SyncHeightmap();
57.     }
58. }

3. Used this compute shader (I'm honestly not even sure what this compute shader is actually modifying, I just stumbled across someone else's code while trying to look up examples of TerrainPaintUtility in action and this is the only thing I could find):

Code (CSharp):

1.  
2. #pragma kernel CSMain
3.  
4. RWTexture2D<float4> Result;
5.  
6. [numthreads(8,8,1)]
7. void CSMain (uint3 id : SV_DispatchThreadID)
8. {
9.     Result[id.xy] += 0.0001; //Still can't quite figure out what on the object that gets passed to this is actually being modified.
10.     Result[id.xy] = clamp(Result[id.xy], 0, 1);
11. }

Is my methodology flawed here? I assumed that TerrainPaintUtility would allow you to "paint" onto the heightmap, thus allowing you to use "brushes" with different shapes, instead of doing things the terrainData.SetHeight() way where you need to find points mathematically (and thus might not let you implement selection areas in shapes other than basic circles and squares)

 

Update: Using "CopyActiveRenderTextureToHeightMap", I was able to apply my brush shape to the terrain, however it still isn't without issue:

Here's the modification shown on the terrain object itself
Here's the resultant heightmap

While I at least was able to see a modification that actually follows the shape outlined in the brush's JPG, it appears that the modification is not additive like I was hoping it would be, it's essentially the same thing as dragging the heightmap into paint and pasting the brush JPG over it, rather than modifying the alpha channel with the alpha values from the brush (As shown here). In addition, the brush itself is all jacked up, so I imagine some settings somewhere are not correct, I just can't identify which ones they might be.

Here's my code:

Code (CSharp):

1. using UnityEngine;
2. using UnityEngine.Experimental.TerrainAPI;
3.  
4. public class TerrainTest : MonoBehaviour
5. {
6.     public Texture brushTexture;
7.     public Vector2 rectSize;
8.  
9.     private Vector3 terrainPosition;
10.     private Rect rect;
11.     private Terrain terrain;
12.     private PaintContext paintContext;
13.  
14.     void Start()
15.     {
16.         terrain = GetComponent<Terrain>();
17.         terrainPosition = terrain.transform.position;
18.         Texture2D texture2D = new Texture2D(brushTexture.width, brushTexture.height, TextureFormat.RGBA32, false);
19.     }
20.  
21.     void Update()
22.     {
23.         if (Input.GetMouseButton(0))
24.         {
25.             RaycastHit rayHit;
26.             var ray = Camera.main.ScreenPointToRay(Input.mousePosition);
27.  
28.             if (Physics.Raycast(ray, out rayHit))
29.             {
30.                 int terrainCoordinateX = Mathf.FloorToInt(((rayHit.point.x - terrainPosition.x) / terrain.terrainData.size.x) * terrain.terrainData.heightmapResolution);
31.                 int terrainCoordinateY = Mathf.FloorToInt(((rayHit.point.z - terrainPosition.z) / terrain.terrainData.size.z) * terrain.terrainData.heightmapResolution);
32.                 ModifyTerrain(terrainCoordinateX, terrainCoordinateY);
33.             }
34.         }
35.     }
36.  
37.    
38.     void ModifyTerrain(int xCoordinate, int yCoordinate)
39.     {
40.         RenderTexture currentRenderTexture = RenderTexture.active;
41.         RenderTexture renderTexture = new RenderTexture(brushTexture.width, brushTexture.height, 32);
42.         Graphics.Blit(brushTexture, renderTexture);
43.         RenderTexture.active = renderTexture;
44.         RectInt sourceRect = new RectInt(new Vector2Int(0, 0), new Vector2Int(renderTexture.width, renderTexture.height));
45.         terrain.terrainData.CopyActiveRenderTextureToHeightmap(sourceRect, new Vector2Int(xCoordinate, yCoordinate), TerrainHeightmapSyncControl.None);
46.         terrain.terrainData.SyncHeightmap();
47.     }

(I realize the modification function itself is the opposite of performant in this state, but this was just to see if it would work before going any further)

 

From the picture and code provided it looks like you're directly assigning the brush texture to the heightmap instead of adding it to the current one, which is why the effect is the one you see. I'm not sure if manipulating RenderTextures is only available in compute shaders, but that's the only way i've found how. In any case, it will have to be done inside some kind of shader because we're looking to manipulate textures. In order to achieve the effect of actually painting on the heightmap you need to copy the current heightmap. This can be done via CopyTexture or maybe even by PaintContext. You then need to send both the brush and the copied heightmap texture to the compute shader and dispatch them with logic similar to this:

1. 
   
   Code (CSharp):

   1. #pragma kernel CSMain
   2.  
   3. RWTexture2D<half> heightmap; // Halfs can be used for full precision instead of float because heightmaps are int16 anyway.
   4. Texture2D<half4> brush; // Brush texture which you're using to paint on top of the heightmap
   5. half height_multiplier = 0.01; // Brush intensity
   6.  
   7. [numthreads(32,32,1)] // search a bit more about compute shaders to figure out how to use numthreads properly
   8. void CSMain (uint3 id : SV_DispatchThreadID)
   9. {
   10.     half new_heightmap = heightmap[id.xy].x + brush[id.xy].w * height_multiplier; // Use whichever channel you want from the brush. You'll also have to figure out how to sample proper uv coordinates from the brush texture instead of using "id.xy".
   11.     heightmap[id.xy].x = clamp(new_heightmap , 0, 0.5);    // 0.5, because that's the range devs decided to treat the texture internally. Bugs occur beyond 0.5.
   12. }

After you generate the new heightmap just copy it back via "CopyActiveRenderTextureToHeightmap".

 

I had sidelined this for a bit to work on some other stuff so sorry for the delayed response!

Your reply makes sense, although I think I've run into a syntax problem with Unity's API on how to actually accomplish this

Code (CSharp):

1.     void ModifyTerrain(Rect selection)
2.     {
3.         paintContext = TerrainPaintUtility.BeginPaintHeightmap(terrain, selection);
4.  
5.         RenderTexture terrainRenderTexture = paintContext.sourceRenderTexture;
6.  
7.         terrainPaintShader.SetTexture(terrainPaintShader.FindKernel("CSMain"), "originalHeightmap", terrainRenderTexture);
8.         terrainPaintShader.SetTexture(terrainPaintShader.FindKernel("CSMain"), "brush", brushTexture);
9.         terrainPaintShader.Dispatch(terrainPaintShader.FindKernel("CSMain"), 1, 1, 1);
10.  
11.         Graphics.CopyTexture(terrainRenderTexture, paintContext.destinationRenderTexture);
12.  
13.         TerrainPaintUtility.EndPaintHeightmap(paintContext, "Terrain");
14.  
15.         terrain.terrainData.SyncHeightmap();
16.  
17.     }

When I run this, however, I get: "IndexOutOfRangeException: Invalid kernelIndex (0) passed, must be non-negative less than 1.
UnityEngine.ComputeShader.SetTexture (System.Int32 kernelIndex, System.String name, UnityEngine.Texture texture) (at <63fe705fb8b344d5be62daf83244d046>:0)"

So I'm thinking that I'm not passing my parameters to the compute shader properly. I tried searching for "compute shader multiple parameters" but, for the life of me, I couldn't find any relevant threads or documentation.

 

I think that error comes from 2 situations only:
1. The kernel which you're looking for doesn't exist. Which means either the string provided in the FindKernel is incorrect or the kernel is never defined inside the compute shader with a #pragma or the kernel is defined in the compute shader, but a function with the same name is not.
2. The kernel was found, but the variable you're trying to assign a value to inside one of the kernels doesn't exist. Which either means the variable you're trying to assign was never defined in the shader or you mistyped the name of the variable you're trying to assign the value to.

Assigning multiple parameters to the compute shaders isn't the issue. As far as i know shaders support hundreads of parameters to which you can assign values to. If you'd post the shader code you're using, the mistake should be quite easy to find.

 

Sorry about that, I meant to specify that I was using the shader code you had provided:

Code (CSharp):

1. // Each #kernel tells which function to compile; you can have many kernels
2. #pragma kernel CSMain
3.  
4. // Create a RenderTexture with enableRandomWrite flag and set it
5. // with cs.SetTexture
6. RWTexture2D<half> originalHeightmap;
7. Texture2D<half4> brush;
8. half height_multiplier = 0.01;
9.  
10. [numthreads(8,8,1)]
11. void CSMain (uint3 id : SV_DispatchThreadID)
12. {
13.     half newHeightmap = originalHeightmap[id.xy].x + brush[id.xy].w * height_multiplier;
14.     originalHeightmap[id.xy].x = clamp(newHeightmap, 0, 0.5);
15. }
16.  

I don't see any typos, is there a chance this is a type issue? C# is my first statically-typed language so I'm sure it's a possibility, although I would have imagined that Unity or Visual Studio would have attempted to tell me about the type mis-match

 

Upon second viewing i noticed that the texture you're sending to the shader is a PaintContext one. Those do not have the enableRandomWrite set to true. Not sure why though, probably the devs just forgot about it. You'll have to do an additional texture copy to create a proper RenderTexture which you can send to the shader.

Code (CSharp):

1. // Replace the
2. RenderTexture terrainRenderTexture = paintContext.sourceRenderTexture;
3. // with
4. RenderTexture terrainRenderTexture = new RenderTexture(paintContext.sourceRenderTexture.width, context.sourceRenderTexture.height, 0, RenderTextureFormat.R16);
5. terrainRenderTexture.enableRandomWrite = true;
6. Graphics.CopyTexture(context.sourceRenderTexture, terrainRenderTexture);
7.  

You'll have to dispose of this texture on your own after you're done with it with terrainRenderTexture.Release().

 

hey @palten08 and @crysicle
I'm trying to get this to work, but I don't get any change in the terrain.
Did i miss something?

Code (CSharp):

1. using UnityEngine;
2. using UnityEngine.Experimental.TerrainAPI;
3.  
4. public class TerrainModifier : MonoBehaviour
5. {
6.     [SerializeField] ComputeShader terrainPaintShader;
7.     public Texture brushTexture;
8.     public Vector2 rectSize;
9.  
10.     private Rect rect;
11.     private Terrain terrain;
12.     private PaintContext paintContext;
13.  
14.     void Start()
15.     {
16.         terrain = GetComponent<Terrain>();
17.     }
18.  
19.     void Update()
20.     {
21.         if (Input.GetMouseButton(0))
22.         {
23.             RaycastHit rayHit;
24.             var ray = Camera.main.ScreenPointToRay(Input.mousePosition);
25.  
26.             if (Physics.Raycast(ray, out rayHit, Mathf.Infinity))
27.             {
28.                 Vector2 terrainPoint = new Vector2(rayHit.point.x, rayHit.point.z);
29.                 rect = new Rect(terrainPoint, rectSize);
30.                 ModifyTerrain(rect);
31.                 Debug.DrawRay(ray.origin, ray.direction * 500, Color.red);
32.             }
33.         }
34.     }
35.  
36.     void ModifyTerrain(Rect selection)
37.     {
38.         paintContext = TerrainPaintUtility.BeginPaintHeightmap(terrain, selection);
39.  
40.         RenderTexture terrainRenderTexture = new RenderTexture(paintContext.sourceRenderTexture.width, paintContext.sourceRenderTexture.height, 0, RenderTextureFormat.R16);
41.         terrainRenderTexture.enableRandomWrite = true;
42.         Graphics.CopyTexture(paintContext.sourceRenderTexture, terrainRenderTexture);
43.  
44.         terrainPaintShader.SetTexture(terrainPaintShader.FindKernel("CSMain"), "originalHeightmap", terrainRenderTexture);
45.         terrainPaintShader.SetTexture(terrainPaintShader.FindKernel("CSMain"), "brush", brushTexture);
46.         terrainPaintShader.Dispatch(terrainPaintShader.FindKernel("CSMain"), 1, 1, 1);
47.  
48.         Graphics.CopyTexture(terrainRenderTexture, paintContext.destinationRenderTexture);
49.  
50.         TerrainPaintUtility.EndPaintHeightmap(paintContext, "Terrain");
51.  
52.         terrain.terrainData.SyncHeightmap();
53.  
54.     }
55. }
56.  

ComputeShader:

Code (CSharp):

1. // Each #kernel tells which function to compile; you can have many kernels
2. #pragma kernel CSMain
3.  
4. // Create a RenderTexture with enableRandomWrite flag and set it
5. // with cs.SetTexture
6. RWTexture2D<half> originalHeightmap;
7. Texture2D<half4> brush;
8. half height_multiplier = 0.01;
9.  
10. [numthreads(8, 8, 1)]
11. void CSMain(uint3 id : SV_DispatchThreadID)
12. {
13.     half newHeightmap = originalHeightmap[id.xy].x + brush[id.xy].w * height_multiplier;
14.     originalHeightmap[id.xy].x = clamp(newHeightmap, 0, 0.5);
15. }
16.  

Terrain Inspector:


Also set the terrain tools to paint and Raise/Lower don't know if that has any effect, but i did notice that we can paint displacement during run time in the sceneview which effects the terrain in the gameview, without any lag so that's a good sign.

 

1. The amount of heightmaps you're manipulating is not 100x100, but 8x8. That's because the numthreads in the shader is [8,8,1] and you're dispatching the shader with shader.dispatch(kernel, 1, 1, 1) which simply multiplies the numthreads. To actually manipulate 100x100, you'll need to divide the size of the rect by the number of numthreads for each axis instead of just passing 1 for each of them.

Should look something like this:
shader.dispatch(kernel, Mathf.Ceilling(rect.width / numthreads.x), Mathf.Ceilling(rect.height / numthreads.y), 1)

2. The starting point of sampling from the paint texture inside the shader is in the bottom corner because we're not adding any offset. All of the pixels then get accessed by the manipulation size, in this case 8x8, because id always starts at 0 and goes from 0 to numthread amount for each axis when a shader is dispatched.

Here's an image of what's going on - https://i.imgur.com/ZMHuWgZ.png
Note that inside the shader we're not giving any offset yet, but if we did, it's how it would sample from the brush texture.

Based on your TerrainBrush in the editor it looks like the beginning 8x8 pixels of that texture are fully transparent, meaning we're adding 0 * height_multiplier to the heightmap. I'm not sure if you tried using other textures, but a fully white texture should probably change an 8x8 grid in some way. Because the w coordinate of the texture is used, its possible your textures completely lack transparency, the shader might be defaulting to 0 just because the transparency layer doesn't exist for the texture that is used. You could also try using any of the other channels for sampling.

I'd suggest to also add an inspector variable to your script for the height_multiplier which would equal 1 and send it to the shader. It will be easier to debug by having control over all shader variable values inside the inspector.

 

@crysicle Awesome, your explanation makes total sense! Thanks! it even gave me a better insight on how Compute shaders work.

 

@crysicle (Solved look at the post below) hey i got it to work back when you posted by just making my texture 32x32 and my numthreads 32x32
as far as i understand atm, that makes 1 thread responsible for 1 pixel.
now i need a bigger part (rect) to manipulate the terrain. So i need the offset you talked about. i tried adding it in the compute shader as:

Code (CSharp):

1. #pragma kernel CSMain
2.  
3. half height;
4. half2 offset;
5. RWTexture2D<half4> heightmap;
6. Texture2D<half4> brush;
7.  
8. [numthreads(32, 32, 1)]
9. void CSMain(uint2 id : SV_DispatchThreadID)
10. {
11.     //Raise - Lower Terrain
12.     //half4 new_heightmap = heightmap[id.xy] + brush[id.xy] * height;
13.     //Set terrain to height
14.     for (int i = 0; i < 2; i++)
15.     {
16.         half4 new_heightmap = lerp(heightmap[id.xy + (32 * i)], height, brush[id.xy]);
17.         heightmap[id.xy + (32 * i)] = clamp(new_heightmap, 0, .5);    // 0.5, because that's the range devs decided to treat the texture internally. Bugs occur beyond 0.5.
18.     }
19. }

off course that didn't work... I guess I should offset x and y separately?
as you can see in the picture, my brush is 64x64 but i only manipulate 32x32.

with the for loop in the shader i get this:


also i don't understand this in your picture:


and here is how i feed the shader: (selection rect is 64x64)

Code (CSharp):

1. ...
2. terrainPaintShader.SetFloat("height", height*smoothOffset);
3.         terrainPaintShader.SetVector("offset", selection.position);
4.         terrainPaintShader.SetTexture(terrainPaintShader.FindKernel("CSMain"), "heightmap", terrainRenderTexture);
5.         terrainPaintShader.SetTexture(terrainPaintShader.FindKernel("CSMain"), "brush", brushTexture);
6.  
7.         terrainPaintShader.Dispatch(terrainPaintShader.FindKernel("CSMain"), (int)Mathf.Ceil(selection.width / 32), (int)Mathf.Ceil(selection.height / 32), 1);
8.  
9.         Graphics.CopyTexture(terrainRenderTexture, paintContext.destinationRenderTexture);
10.  
11.         TerrainPaintUtility.EndPaintHeightmap(paintContext, "Terrain");
12.  
13.         terrain.terrainData.SyncHeightmap();
14. ...

 

aha ok got it to work like this:

Code (CSharp):

1. #pragma kernel CSMain
2.  
3. half height;
4. half2 offset;
5. RWTexture2D<half4> heightmap;
6. Texture2D<half4> brush;
7.  
8. [numthreads(32, 32, 1)]
9. void CSMain(uint2 id : SV_DispatchThreadID)
10. {
11.     //Raise - Lower Terrain
12.     //half4 new_heightmap = heightmap[id.xy] + brush[id.xy] * height;
13.     //Set terrain to height
14.     for (int i = 0; i < 2; i++)
15.     {
16.         for (int j = 0; j < 2; j++)
17.         {
18.             half4 new_heightmap = lerp(heightmap[int2(id.x + (32*i), id.y + (32*j))], height, brush[int2(id.x + (32 * i), id.y + (32 * j))]);
19.             heightmap[int2(id.x + (32 * i), id.y + (32 * j))] = clamp(new_heightmap, 0, .5);    // 0.5, because that's the range devs decided to treat the texture internally. Bugs occur beyond 0.5.
20.         }
21.     }
22. }

only strange thing now is i get strange artifacts, any idea where they could come from? the texture i use as brush is also 64x64


changing the shader to have 16x16 numthreads and make the for loop go over 4 softens the artifacts? no idea why, could someone explain the relation?

 

How did you manage to get TerrainPaintUtility to work correctly? I tried your code and came across two problems:

1. Height of the edited surface is zeroed in height. Adjusting the values in Height does not affect anything. It turns out as in the picture. Terrain is 100 m high

2. If the landscape consists of several terrain tiles, only the one that was placed first is edited. This is magic. No matter how much I looked for the reasons, I could not find. Why can't he see the height map of other terrains? After all, every time I send to the method exactly the Terrain that was discovered by the raycast hit.

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4. using UnityEngine.Experimental.TerrainAPI;
5.  
6. public class DigTerrainWithTerrainPainUtility2 : MonoBehaviour
7. {
8.     [SerializeField] ComputeShader terrainPaintShader;
9.     public Texture brushTexture;
10.     public Vector2 rectSize;
11.     public float height = 100f;
12.     public float smoothOffset = 100f;
13.  
14.     private Rect rect;
15.     private PaintContext paintContext;
16.  
17.     void Update()
18.     {
19.         if (Input.GetMouseButton(0))
20.         {
21.             RaycastHit rayHit;
22.             var ray = Camera.main.ScreenPointToRay(Input.mousePosition);
23.  
24.             if (Physics.Raycast(ray, out rayHit, Mathf.Infinity))
25.             {
26.                 Vector2 terrainPoint = new Vector2(rayHit.point.x, rayHit.point.z);
27.                 rect = new Rect(terrainPoint, rectSize);
28.                 Terrain ter = rayHit.collider.GetComponent<Terrain>();
29.                 ModifyTerrain(rect, ter);
30.                 Debug.DrawRay(ray.origin, ray.direction * 500, Color.red);
31.             }
32.         }
33.     }
34.  
35.  
36.     void ModifyTerrain(Rect selection, Terrain ter)
37.     {
38.         paintContext = TerrainPaintUtility.BeginPaintHeightmap(ter, selection);
39.         RenderTexture terrainRenderTexture = new RenderTexture(paintContext.sourceRenderTexture.width, paintContext.sourceRenderTexture.height, 0, RenderTextureFormat.R16);
40.         terrainRenderTexture.enableRandomWrite = true;
41.  
42.         terrainPaintShader.SetFloat("height", height * smoothOffset);
43.         terrainPaintShader.SetVector("offset", selection.position);
44.         terrainPaintShader.SetTexture(terrainPaintShader.FindKernel("CSMain"), "heightmap", terrainRenderTexture);
45.         terrainPaintShader.SetTexture(terrainPaintShader.FindKernel("CSMain"), "brush", brushTexture);
46.         terrainPaintShader.Dispatch(terrainPaintShader.FindKernel("CSMain"), (int)Mathf.Ceil(selection.width / 32), (int)Mathf.Ceil(selection.height / 32), 1);
47.         Graphics.CopyTexture(terrainRenderTexture, paintContext.destinationRenderTexture);
48.         TerrainPaintUtility.EndPaintHeightmap(paintContext, "Terrain");
49.         ter.terrainData.SyncHeightmap();
50.  
51.     }
52. }

Code (CSharp):

1. #pragma kernel CSMain
2. half height;
3. half2 offset;
4. RWTexture2D<half4> heightmap;
5. Texture2D<half4> brush;
6. [numthreads(32, 32, 1)]
7. void CSMain(uint2 id : SV_DispatchThreadID)
8. {
9.     //Raise - Lower Terrain
10.     //half4 new_heightmap = heightmap[id.xy] + brush[id.xy] * height;
11.     //Set terrain to height
12.     for (int i = 0; i < 2; i++)
13.     {
14.         for (int j = 0; j < 2; j++)
15.         {
16.             half4 new_heightmap = lerp(heightmap[int2(id.x + (32*i), id.y + (32*j))], height, brush[int2(id.x + (32 * i), id.y + (32 * j))]);
17.             heightmap[int2(id.x + (32 * i), id.y + (32 * j))] = clamp(new_heightmap, 0, .5);    // 0.5, because that's the range devs decided to treat the texture internally. Bugs occur beyond 0.5.
18.         }
19.     }
20. }

 

hey @Lesnikus5 , i ended up using this as compute shader

Code (CSharp):

1. #pragma kernel CSMain
2. half height;
3. half angle;
4. half2 rotOffset;
5. RWTexture2D<half4> heightmap;
6. Texture2D<half4> brush;
7. half2 RotateShape(half2 pos, float angle)
8. {
9.     float s, c;
10.     sincos(radians(angle), s, c);
11.     float2x2 rotationMatrix = float2x2(c, -s, s, c);
12.     return mul(pos.xy - rotOffset, rotationMatrix);
13. }
14. [numthreads(2, 2, 1)]
15. void CSMain(uint2 id : SV_DispatchThreadID)
16. {
17.     half2 uv = RotateShape(id.xy, angle) + rotOffset;
18.     half4 new_heightmap = lerp(heightmap[id.xy], height, brush[uv].w);
19.     heightmap[id.xy] = clamp(new_heightmap, 0, .5);
20. }

end this is the cs, it needs some clean up work, but the functions you're looking for are GetAreaToModify, ModifyTerrain & RestoreTerrain

Code (CSharp):

1. using System.Collections.Generic;
2. using System.Linq;
3. using UnityEngine;
4. using UnityEngine.Experimental.TerrainAPI;
5. [System.Serializable]
6. public class TerrainModifier
7. {
8.     [SerializeField] ComputeShader terrainPaintShader;
9.     //public Dictionary_Int_Texture2D brushes = new Dictionary_Int_Texture2D();
10.     public bool toCenter;
11.     private Vector2Int brushSize;
12.     private float heightOffset = .5f;
13.     private Texture2D brushTexture;
14.     private float height;
15.     public Terrain terrain;
16.     private RenderTexture prevRenderTexture;
17.     private Vector2 terrainPoint = Vector2.one;
18.     private Vector2 previousTerrainPoint;
19.     private Vector2 hitPoint;
20.     private List<TreeInstance> trees = new List<TreeInstance>();
21.     private bool baked;
22.     private uint x, y, z;
23.     private RaycastHit hit;
24.     private Matrix4x4 rotationMatrix;
25.     private float angle;
26.     private float prevAngle;
27.     public TerrainData terrainDataBU;
28.     public Vector2 VegetationArea { private get; set; }
29.     #region Terrain Modification
30.     //public TerrainModifier(ComputeShader computeShader)
31.     //{
32.     //    terrainPaintShader = computeShader;
33.     //}
34.     public void SetBrushData(Texture2D brush, float height, Vector2Int size)
35.     {
36.         brushTexture = brush;
37.         heightOffset = height;
38.         brushSize = size;
39.     }
40.     public void Execute(bool isInBounds, float angle)
41.     {
42.         if(!isInBounds)return;
43.         GetAreaToModify();
44.         this.angle = angle;
45.     }
46.     void GetAreaToModify()
47.     {
48.         hit = toCenter ? CameraBase.Ray(false, true, "Terrain") : CameraBase.Ray(false, false, "Terrain");
49.         if(hit.point == Vector3.zero) return;
50.         terrain = hit.transform.GetComponent<Terrain>();
51.         terrainPoint = new Vector2((int)((hit.point.x - (brushSize.x * .5f)) - terrain.transform.position.x), (int)((hit.point.z - (brushSize.y * .5f)) - terrain.transform.position.z));
52.         hitPoint = new Vector2(hit.point.x, hit.point.z);
53.         if (terrainPoint != previousTerrainPoint || prevAngle != angle)
54.         {
55.             Rect prevRect = new Rect(previousTerrainPoint, brushSize);
56.             if (prevRect.height != 0 && prevRect.width != 0 && prevRenderTexture)
57.                 RestoreTerrain(prevRect);
58.             Rect rect = new Rect(terrainPoint, brushSize);
59.             ModifyTerrain(rect);
60.             previousTerrainPoint = terrainPoint;
61.             prevAngle = angle;
62.         }
63.     }
64.     void ModifyTerrain(Rect selection)
65.     {
66.         Vector3 c1 = new Vector3(selection.position.x + terrain.transform.position.x, 0, selection.position.y + terrain.transform.position.z);
67.         Vector3 c2 = new Vector3(selection.position.x + terrain.transform.position.x, 0, selection.position.y + terrain.transform.position.z + selection.size.y);
68.         Vector3 c3 = new Vector3(selection.position.x + terrain.transform.position.x + selection.size.x, 0, selection.position.y + terrain.transform.position.z + selection.size.y);
69.         Vector3 c4 = new Vector3(selection.position.x + terrain.transform.position.x + selection.size.x, 0, selection.position.y + terrain.transform.position.z);
70.         //if(selection.size.x == 32)
71.         //{
72.         //    Debugger.Log("c1: "+c1);
73.         //    Debug.DrawRay(c1, Vector3.up * 100, Color.red, Mathf.Infinity);
74.         //    Debugger.Log("c2: "+c2);
75.         //    Debug.DrawRay(c2, Vector3.up * 100, Color.blue, Mathf.Infinity);
76.         //    Debugger.Log("c3: "+c3);
77.         //    Debug.DrawRay(c3, Vector3.up * 100, Color.green, Mathf.Infinity);
78.         //    Debugger.Log("c4: "+c4);
79.         //    Debug.DrawRay(c4, Vector3.up * 100, Color.yellow, Mathf.Infinity);
80.         //}
81.         //HandleTerrainTrees(hitPoint, VegetationArea, angle);
82.         PaintContext paintContext = TerrainPaintUtility.BeginPaintHeightmap(terrain, selection);
83.         RenderTexture terrainRenderTexture = new RenderTexture(paintContext.sourceRenderTexture.width, paintContext.sourceRenderTexture.height, 0, RenderTextureFormat.R16);
84.         terrainRenderTexture.enableRandomWrite = true;
85.         Graphics.CopyTexture(paintContext.sourceRenderTexture, terrainRenderTexture);
86.         prevRenderTexture = new RenderTexture(paintContext.sourceRenderTexture.width, paintContext.sourceRenderTexture.height, 0, RenderTextureFormat.R16);
87.         Graphics.CopyTexture(paintContext.sourceRenderTexture, prevRenderTexture);
88.         float h0 = terrain.SampleHeight(c1);
89.         float h1 = terrain.SampleHeight(c2);
90.         float h2 = terrain.SampleHeight(c3);
91.         float h3 = terrain.SampleHeight(c4);
92.         height = ((h0 + h1 + h2 + h3) / 4f) / terrain.terrainData.size.y;
93.         terrainPaintShader.GetKernelThreadGroupSizes(terrainPaintShader.FindKernel("CSMain"), out x, out y, out z);
94.         terrainPaintShader.SetFloat("height", height * heightOffset);
95.         terrainPaintShader.SetFloat("angle", angle);
96.         //Debug.Log(x+"/"+y+"/"+z);
97.         terrainPaintShader.SetVector("rotOffset", new Vector2(brushSize.x/2, brushSize.y/2));
98.         terrainPaintShader.SetTexture(terrainPaintShader.FindKernel("CSMain"), "heightmap", terrainRenderTexture);
99.         terrainPaintShader.SetTexture(terrainPaintShader.FindKernel("CSMain"), "brush", brushTexture);
100.         terrainPaintShader.Dispatch(terrainPaintShader.FindKernel("CSMain"), (int)(selection.size.x/x), (int)(selection.size.y/y), (int)z);
101.         Graphics.CopyTexture(terrainRenderTexture, paintContext.destinationRenderTexture);
102.         TerrainPaintUtility.EndPaintHeightmap(paintContext, "Terrain");
103.         terrain.terrainData.SyncHeightmap();
104.     }
105.     public void HandleTerrainTrees(Vector2 position, Vector2 size, float angle, bool isMultipule = false)
106.     {
107.         if (trees == null)trees = new List<TreeInstance>();
108.         if (trees.Count > 0 && !isMultipule)
109.         {
110.             trees.AddRange(terrain.terrainData.treeInstances);
111.             terrain.terrainData.treeInstances = trees.ToArray();
112.             trees.Clear();
113.         }
114.         ////Trees inside circle
115.         //foreach (var tree in terrain.terrainData.treeInstances)
116.         //{
117.         //    Vector2 p = new Vector2((tree.position.x * terrain.terrainData.size.x) + terrain.transform.position.x, (tree.position.z * terrain.terrainData.size.z) + terrain.transform.position.z);
118.         //    if(MathHelper.PointInCircle(p, position, size.x))
119.         //        trees.Add(tree);
120.         //}
121.         //Trees inside rectangle
122.         Vector2 A = Vector2.zero;
123.         Vector2 B = Vector2.zero;
124.         Vector2 C = Vector2.zero;
125.         foreach (var tree in terrain.terrainData.treeInstances)
126.         {
127.             Vector2 p = new Vector2((tree.position.x * terrain.terrainData.size.x) + terrain.transform.position.x, (tree.position.z * terrain.terrainData.size.z) + terrain.transform.position.z);
128.             angle = isMultipule ? 360 - angle : angle;
129.             A = MathHelper.GetRotatedCornerPoint(position, new Vector2((position.x + size.x / 2), (position.y + size.y / 2)), angle);
130.             B = MathHelper.GetRotatedCornerPoint(position, new Vector2((position.x + size.x / 2), (position.y - size.y / 2)), angle);
131.             C = MathHelper.GetRotatedCornerPoint(position, new Vector2((position.x - size.x / 2), (position.y - size.y / 2)), angle);
132.             if (MathHelper.PointInRectangle(p, A, B, C))
133.             {
134.                 trees.Add(tree);
135.                 //Debug.DrawRay(new Vector3(p.x, 30, p.y), Vector3.up * 100, Color.yellow, Mathf.Infinity);
136.             }
137.         }
138.         //Debug.DrawRay(new Vector3(A.x, 30, A.y), Vector3.up * 100, Color.red, Mathf.Infinity);
139.         //Debug.DrawRay(new Vector3(B.x, 30, B.y), Vector3.up * 100, Color.blue, Mathf.Infinity);
140.         //Debug.DrawRay(new Vector3(C.x, 30, C.y), Vector3.up * 100, Color.green, Mathf.Infinity);
141.         //Debug 1 set of removed trees
142.         //if (trees.Count > 0)
143.         //{
144.         //    terrain.terrainData.treeInstances = trees.ToArray();
145.         //}
146.         if (trees.Count > 0)
147.         {
148.             TreeInstance[] newTrees = terrain.terrainData.treeInstances.Except(trees).ToArray();
149.             terrain.terrainData.treeInstances = newTrees;
150.             if (isMultipule)
151.                 trees.Clear();
152.             terrain.Flush();
153.         }
154.     }
155.     public void BakeAndRefresh(Building building)
156.     {
157.         HandleTerrainTrees(hitPoint, VegetationArea, angle);
158.         baked = true;
159.     }
160.     private void RestoreTerrain(Rect selection)
161.     {
162.         if (baked)
163.         {
164.             baked = false;
165.             return;
166.         }
167.         PaintContext prevPaintContext = TerrainPaintUtility.BeginPaintHeightmap(terrain, selection);
168.         Graphics.CopyTexture(prevRenderTexture, prevPaintContext.destinationRenderTexture);
169.         TerrainPaintUtility.EndPaintHeightmap(prevPaintContext, "Terrain");
170.         terrain.terrainData.SyncHeightmap();
171.     }
172.     public void ApplyTerrainData(TerrainData terrainData)
173.     {
174.         if(!terrain)
175.             terrain = GameObject.FindGameObjectWithTag("ResortTerrain").GetComponent<Terrain>();
176.         // Terrain collider
177.         terrain.terrainData.SetHeights(0, 0, terrainData.GetHeights(0, 0, terrain.terrainData.heightmapResolution, terrain.terrainData.heightmapResolution));
178.         // Textures
179.         terrain.terrainData.SetAlphamaps(0, 0, terrainData.GetAlphamaps(0, 0, terrain.terrainData.alphamapWidth, terrain.terrainData.alphamapHeight));
180.         // Trees
181.         terrain.terrainData.treeInstances = terrainData.treeInstances;
182.         // Grasses
183.         //terrain.terrainData.SetDetailLayer(0, 0, 0, terrainData.GetDetailLayer(0, 0, terrain.terrainData.detailWidth, terrain.terrainData.detailHeight, 0));
184.         //terrain.terrainData = terrainData;
185.         terrain.Flush();
186.     }
187.     #endregion
188. }

In the end we aren't using this in our game, so i never completed optimizing it or look at how it could be done more efficient...
It did teach me a lot about compute shaders though.
As I understand it now: u (best) define a thread group holding 64 or 32 threads depending on graphics type u use. on PC mostly 64 so something like [8,8,1]. Then when u dispatch, it tells the compute shader how many of these you want.
so a dispatch like this, compute.Dispatch(0, 64,64,1); would be like a 512x512 2D array (or texture).
Hope that helps a bit, I still need to wrap my head around lots of stuff concerning compute shaders myself, so good luck, it's a rabbit hole for sure.