Simple Audio/Spectrum Visualizer With ECS - Need Feedback

Hi guys, currently I'm learning how ecs works by creating a simple audio visualizer with ecs + job system.
I don't know whether this been done before with the new job system, but might as well add more example for new people coming to this discussion

Without ECS on my mid 2015 mac book pro 2.2 GHz Intel Core i7 I could only get 2048 samples.
With ECS, I could do 4096 samples for two channels, roughly 4x speedup?

So what this thing does is first it is going to get the raw data from both the 0 and 1 channel, then passing those data to the job system. The job system would then see the value and adjust the height of the cubes.

Feedbacks are welcomed.

Next step is trying to change colors depending on how high the amplitude of the frequency is, but I'm stuck trying to make the color pass between the main loop and the job system, I tried adding NativeVector<Color> but then the performance was pretty bad.

Code (CSharp):

1. using UnityEngine;
2. using Unity.Collections;
3. using Unity.Jobs;
4. using UnityEngine.Jobs;
5.  
6. struct MoveJob : IJobParallelForTransform
7. {
8.     public float maxScale;
9.     public float dynamics;
10.     public float epsilon;
11.     public NativeArray<float> currentSpectrums;
12.     public NativeArray<float> prevSpectrums;
13.     public NativeArray<Vector3> origins;
14.     public NativeArray<Color> colors;
15.  
16.     public void Execute(int index, TransformAccess transform)
17.     {
18.         float current = currentSpectrums[index];
19.         float prev = prevSpectrums[index];
20.  
21.         float val = (dynamics*prev + (1 - dynamics)*current);
22.         prevSpectrums[index] = val;
23.  
24.         float valAdjusted = val*maxScale;
25.         float halfHeight = valAdjusted*0.5f;
26.         transform.localScale = new Vector3(1, valAdjusted, 1);
27.  
28.         Vector3 origin = origins[index];
29.         if(val >= epsilon)
30.         {
31.             transform.localPosition = new Vector3(origin.x, halfHeight, origin.z);
32.         }
33.         else
34.         {
35.             transform.localPosition = new Vector3(0, 0, 0);
36.         }
37.     }
38. }
39.  
40. public class SpectrumWrapper
41. {
42.     public int channel;
43.     public float[] spectrumBuff;
44.     public Transform[] transforms;
45.     public NativeArray<float> currentSpectrums;
46.     public NativeArray<float> prevSpectrums;
47.     public NativeArray<Vector3> origins;
48.     public NativeArray<Color> colors;
49.     public TransformAccessArray transformsAccess;
50.  
51.     MoveJob moveJob;
52.     JobHandle jobHandle;
53.  
54.     public void CreateJob(float inMaxScale, float inDynamics, float inEpsilon)
55.     {
56.         AudioListener.GetSpectrumData(spectrumBuff, channel, FFTWindow.BlackmanHarris);
57.         currentSpectrums.CopyFrom(spectrumBuff);
58.  
59.         moveJob = new MoveJob()
60.         {
61.             maxScale = inMaxScale,
62.             dynamics = inDynamics,
63.             epsilon = inEpsilon,
64.             currentSpectrums = this.currentSpectrums,
65.             prevSpectrums = this.prevSpectrums,
66.             origins = this.origins,
67.             colors = this.colors,
68.         };
69.  
70.         jobHandle = moveJob.Schedule(transformsAccess);
71.     }
72.  
73.     public void CompleteJob()
74.     {
75.         jobHandle.Complete();
76.         moveJob.prevSpectrums.CopyTo(prevSpectrums);
77.     }
78.  
79.     public void Destroy()
80.     {
81.         currentSpectrums.Dispose();
82.         prevSpectrums.Dispose();
83.         origins.Dispose();
84.         transformsAccess.Dispose();
85.     }
86. }
87.  
88. public class Bootstrap : MonoBehaviour
89. {
90.     public Transform cameraTransfrom;
91.  
92.     public GameObject prefab;
93.  
94.     public float maxScale;
95.     public float dynamics;
96.     public float epsilon;
97.     public float rotationSpeed;
98.     public int spectrumSize;
99.  
100.     public Transform parentLeft;
101.     public Transform parentRight;
102.  
103.     SpectrumWrapper left;
104.     SpectrumWrapper right;
105.  
106.     void Start()
107.     {
108.         cameraTransfrom.position = new Vector3(0, 55, -120);
109.         cameraTransfrom.eulerAngles = new Vector3(15, 0, 0);
110.  
111.         left = InitChannel(0, parentLeft);
112.         right = InitChannel(1, parentRight);
113.     }
114.  
115.     SpectrumWrapper InitChannel(int channel, Transform parent)
116.     {
117.         SpectrumWrapper result = new SpectrumWrapper();
118.         result.channel = channel;
119.         result.spectrumBuff = new float[spectrumSize];
120.         result.transforms = new Transform[spectrumSize];
121.         result.currentSpectrums = new NativeArray<float>(spectrumSize, Allocator.Persistent);
122.         result.prevSpectrums = new NativeArray<float>(spectrumSize, Allocator.Persistent);
123.         result.origins = new NativeArray<Vector3>(spectrumSize, Allocator.Persistent);
124.  
125.         float distance = 10.0f;
126.         for(int index = 0;
127.             index < spectrumSize;
128.             ++index, distance -= 0.02f)
129.         {
130.             GameObject obj = Object.Instantiate(prefab);
131.  
132.             int n = index;
133.             int x = 0, z = 0;
134.             if(--n >= 0)
135.             {
136.                 int v = (int)Mathf.Floor(Mathf.Sqrt(n + 0.25f) - 0.5f);
137.                 int spiralBaseIndex = v * (v + 1);
138.                 int flipFlop = ((v & 1) << 1) - 1;
139.                 int offset = flipFlop * ((v + 1) >> 1);
140.                 x += offset; z += offset;
141.  
142.                 int cornerIndex = spiralBaseIndex + (v + 1);
143.                 if(n < cornerIndex)
144.                 {
145.                     x -= flipFlop * (n - spiralBaseIndex + 1);
146.                 }
147.                 else
148.                 {
149.                     x -= flipFlop * (v + 1);
150.                     z -= flipFlop * (n - cornerIndex + 1);
151.                 }
152.             }
153.  
154.             obj.transform.parent = parent;
155.             obj.transform.localPosition = new Vector3(x, 0, z);
156.  
157.             result.transforms[index] = obj.transform;
158.             result.origins[index] = obj.transform.localPosition;
159.         }
160.  
161.         result.transformsAccess = new TransformAccessArray(result.transforms);
162.  
163.         return result;
164.     }
165.  
166.     void Update()
167.     {
168.         this.gameObject.transform.Rotate(Vector3.up*(rotationSpeed*Time.deltaTime));
169.  
170.         left.CreateJob(maxScale, dynamics, epsilon);
171.         right.CreateJob(maxScale, dynamics, epsilon);
172.     }
173.     void LateUpdate()
174.     {
175.         left.CompleteJob();
176.         right.CompleteJob();
177.     }
178.  
179.     void OnDestroy()
180.     {
181.         left.Destroy();
182.         right.Destroy();
183.     }
184. }

 

I think the key to add a color fast is to use DrawMeshInstanced for those blocks, then you can use those colors in MaterialPropertyBlock.

 

Update---

Hi guys, I did a lengthy three part series write-up on learning ecs, from monobehavior to unity job then to ecs.

You could see the difference on performance from the video below

MonoBehavior

Job

ECS + Job

Hope you guys could learn something from it. Cheers

 

Cool presentation and comparison. Also great music taste

Now, on technical side.
in first MonoBehavior and second example Job, you use halve of vertices as in ECS + Job.
I mean, cool that ECS + Job is running greater count of these, but was there other reason, why you have doubled them, other than check performance? I see GPU FPS counter is 3 higher, which is cool.
Also, I have noticed, that ECS + Job, is jittery when rotating. Knowing the reason?

 

Hi, thanks for replying!

The only reason that the MonoBehavior and the Job example use half cube than the ECS+Job was because the two previous version would run slowly on my machine, almost like 20fps-ish which is pretty slow if you try to record and play it.

Nice catch though, Job version theoretically would be faster than MonoBehavior, but it turns out that MonoBehavior with 8192 cube have nearly the same performance as 4086 Job with two channels, which is the same cube amount, this is probably an error that I don't really know why, but it is interesting to check why does this happen.

For the jittering probably because of the way the rotation is different between version, on the MonoBehavior and Job code, all the cubes were parented to a 'root' object, then this root object is rotated in it's place. On the ECS version, since I don't know yet how to laid the ecs object in a child parent relation, the rotation movement is basically just the camera that is rotating the cubes in circles, this would cause it to jitters.

 

It was mentioned in past few days, or even yesterday that apparently parented objects, don't jobbyfing. If I got that right. I think tertle my have mentioned it. But I assume, you got only two parents, each for each of channel. Then that shouldn't be a problem in theory.

Regarding camera jittering, I may only suspect time time syncing issue. Something like Fixed Update, which don't really execute every fixed frame, but delta time fluctuates, something similar may be in your case. Camera motion doesn't feel comfortable.

Did you managed to achieve anything,regarding colors? You try store physically Color class, or you got own blittable color struct?

There are also few discussions regarding arrays on this sub forum. They may help, if you haven't seen them yet. Or simply worth to ask specific question, as separate topic.

 

That's so cool! I explored doing something similar but inspiring to see someone combine ecs with it to beef up speed and scale.

And now for my genuinely helpful but also super shameless spot... When using GetSpectrumData initially I noticed a lot of my visualizations (and others') had really intense base values and not so much intensity in mids and highs even if notes in those ranges were loud and clear in the song.

So I made an asset that helps with this. It lets you select the sample count for getspectrumdata (like say 2048) but selects only 128 values from the returned frequency bins that match musical note frequencies (instead of arbitrary linear division of the frequency range).
Then it converts the return from voltage to decibels (20 * Log10 (V/reference value))
It has a slider for the reference value which helps you get your return data from the negative to 0 db range to a 0 to positive range.
Then it also has an equal loudness contour calibration curve so the volume changes across different frequencies are calibrated better to the way the human ear hears.

Doing all this helps your visualizations connect to music a bit better than getspectrumdata will right out of the box.

If you're interested it's called GetSpectrumDataHelper: https://assetstore.unity.com/packages/3d/getspectrumdatahelper-188353