Roman mosaic style using OpenCV for Unity

Hello dear users. I have such a question. Interested in image processing on OpenCV for Unity. I would like to turn the image into a mosaic, as in the example. Actually a question. How to implement or where to start, perhaps someone has an example.

 

This is absolutely not easy.

The simplest thing you could do is to generate a voronoi cell texture:

Spoiler

Then color each cell with the image color it lands on.

But this will not produce you the right result, because mosaic will be chaotic.

So you'd actually need to run turn your image into a sketch, which could theoreticlaly be done by running edge detect, reducing colors then simplifying contours you generated. After that you'd need to generate cells that follow the edges of contours and cells that fill everything else.

This is doable, but this is not a straight forward process and it will require experimentation.

 

I remember implementing this before either in Processing or Max/MSP and I was definitely using code written to do this with video. I am not good at shaders but I would think it's possible as a post-processing effect by someone who knows what they are doing. Here's a link from 2012 that shows it's definitely possible and the basic concept:

https://forum.processing.org/one/to...es-using-voronoi-diagrams-willing-to-pay.html

 

Why is OpenCV a requirement? Is this an assignment or homework?
I hope not...
You can look for the paper "Simulating Decorative Mosaics". It describes an algorithm that can be implemented with OpenCV image processing functions (plus some extension for CVD) creating mosaics with oriented tiles made of custom shapes. See here for some results.

 

So I’ll say that I didn’t work with the voronoi library, some examples are very interesting for me, but I can’t find any scripts. For example, how to process a texture using the library.

 

I worked with opencv and I know that you can build a mosaic by calculating edges.

 

It offers a ton of convenient image processing functions and web camera access. Getting OpenCV to work in unity is also a fun side project.

It is not a library. The proper name is voronoi diagram.
https://en.wikipedia.org/wiki/Voronoi_diagram

This sort of image is usually supported by procedural textures in 3d software, and can be used to produce mosaical effects, among with other things.

 

First of all you need to integrate OpenCV with Unity, you can compile it yourself or you can get a ready-to-go OpenCV integration asset from the store, there is even a version for .NET but full compatibility with original library may not be guaranteed. How to process a Unity texture will depend slightly on the integration you choose.

 

Well I asked ChatGPT to do this and it did produce code without an error:

Code (CSharp):

1. using UnityEngine;
2. using System.Collections.Generic;
3.  
4. public class VoronoiDiagram : MonoBehaviour
5. {
6.     public Texture2D image;
7.     public int numberOfSites = 100;
8.     public float scale = 1;
9.     private List<Vector2> sites;
10.     private Texture2D voronoiDiagram;
11.     private Color[] colors;
12.  
13.     private void Start()
14.     {
15.         sites = GenerateSites();
16.         voronoiDiagram = new Texture2D(image.width, image.height);
17.         colors = image.GetPixels();
18.         CreateVoronoiDiagram();
19.         GetComponent<Renderer>().material.mainTexture = voronoiDiagram;
20.     }
21.  
22.     private List<Vector2> GenerateSites()
23.     {
24.         List<Vector2> sites = new List<Vector2>();
25.         for (int i = 0; i < numberOfSites; i++)
26.         {
27.             float x = Random.Range(0, image.width);
28.             float y = Random.Range(0, image.height);
29.             sites.Add(new Vector2(x, y));
30.         }
31.         return sites;
32.     }
33.  
34.     private void CreateVoronoiDiagram()
35.     {
36.         for (int x = 0; x < image.width; x++)
37.         {
38.             for (int y = 0; y < image.height; y++)
39.             {
40.                 Vector2 currentPoint = new Vector2(x, y);
41.                 int closestSite = GetClosestSite(currentPoint);
42.                 voronoiDiagram.SetPixel(x, y, colors[closestSite]);
43.             }
44.         }
45.         voronoiDiagram.Apply();
46.     }
47.  
48.     private int GetClosestSite(Vector2 currentPoint)
49.     {
50.         int closestSite = 0;
51.         float closestDistance = float.MaxValue;
52.         for (int i = 0; i < sites.Count; i++)
53.         {
54.             float currentDistance = Vector2.Distance(currentPoint, sites[i]);
55.             if (currentDistance < closestDistance)
56.             {
57.                 closestDistance = currentDistance;
58.                 closestSite = i;
59.             }
60.         }
61.         return closestSite;
62.     }
63. }
64.  

Here is the result:

Spoiler

It is recognizing the colors of the image, but when I asked ChatGPT to detect edges and it was on the right tract, using a Sobel filter, but then it just died and wouldn't finish the script. If it did, I would say it's the end of the world that I could ask it this and get a correct response!

Edit: Apparently you can just ask ChatGPT to finish writing the script and it will. But, it doesn't work at detecting edges.

 

OpenAI neutered it quite badly, so it starts losing context after around a screen of text. Given the informational density of programming code, it is highly likely that after your second request it forgot what it was doing in the first place.

 

Great script. Now I think it is necessary to use the coordinates of the points of the lines instead of the random values of the points. But is it possible to get them using Unity standard libraries. Or take opencv. In the opencv library, you can define edges using the cv2.canny function or from the opencvforunity package using the Imgproc.Canny function.
How to take these coordinates of these edges?