Search Unity

Spawning objects (beats) in sync with music

Discussion in 'Scripting' started by Bogaland, Sep 9, 2019.

  1. Bogaland


    Feb 19, 2017

    I'm attempting to create a game where the player tries to hit the beats of a song. I have the beats sorted in a list where the first item in the list is the first beat in the song (time-wise), and then the second and so on. This has been precalculated before the song plays, so the beats themselves are not created at run-time.

    Now for my problem. I'm creating objects in my game on a track, very similar to AudioSurf, a specific amount of seconds (usually 5 seconds) before they will reach the player. So say for example a beat occurs 10 seconds into the song. The object will then spawn 5 seconds into the song, 5 seconds before it should hit, and then lerp towards the player for 5 seconds.

    This works well for any beat that occurs after the initial 5 seconds, but not for the beats that occur before. So if I would have a beat that should hit 2 seconds into the song, it would have to spawn at -3 seconds, before the song starts, which of course is impossible.

    I'm having trouble with trying to figure out how I could fix this. I tried adding the beat time to the objects spawning before 5 seconds as well as the initial 5 seconds. So again let's say I have a beat at 2 seconds. That would mean it should hit within 7 seconds, because the audio will play for 2 seconds and then the beat has another 5 seconds, just as it should, to reach the player. It does not seem to work properly, however. The beats in the beginning are way off in time and seem to create some odd behaviour overall, adding beats when in-between correctly hitting beats. I've checked the list and it is sorted correctly.

    I've added the relevant code below (note that some variables are not used even though they are declared):

    Code (CSharp):
    1. using System.Collections;
    2. using System.Collections.Generic;
    3. using UnityEngine;
    5. public class NewPlotController : MonoBehaviour
    6. {
    7.     SongController songController;
    8.     SpectralFluxAnalyzer spectralFluxes;
    9.     public List<System.Numerics.Vector2> peaks;
    10.     AudioSource audio;
    11.     public float timeAhead = 5.0f; //time until a beat should hit
    12.     public GameObject peakPoint;
    13.     public GameObject spawnPoint;
    14.     public float pointSpeed = 5.0f;
    15.     private double bpm = 160.0;
    16.     private int currBeat = 0;
    17.     private double secondsPerBeat = 60 / 160.0;
    18.     private bool initiate;
    19.     double currTime;
    20.     private double inverse = 1 / 44100f;
    21.     System.Numerics.Vector2 currPeak;
    22.     private double timeToReach;
    24.     bool setup;
    25.     // Start is called before the first frame update
    26.     void Start()
    27.     {
    28.         setup = true;
    29.         initiate = false;
    30.         peaks = new List<System.Numerics.Vector2>();
    31.         songController = GameObject.Find("SongController").GetComponent<SongController>();
    32.         audio = GameObject.Find("SongController").GetComponent<AudioSource>();
    33.     }
    35.     // Update is called once per frame
    36.     void Update()
    37.     {
    38.         //Start calculating peaks when done with analyzis
    39.         if (songController.completed && setup)
    40.         {
    41.             GetPeaks();
    42.             setup = false;
    43.         }
    45.         if (initiate)
    46.         {
    47.             if (!audio.isPlaying)
    48.             {
    49.                 //find all peaks before timeAhead and spawn them before the music starts
    50.                 for (int i = 0; i < peaks.Count; i++)
    51.                 {
    52.                     if (peaks[i] != null)
    53.                     {
    54.                         if (peaks[i].X < timeAhead)
    55.                         {
    56.                             timeToReach = timeAhead + peaks[i].X;
    57.                             AddPoint(peaks[i].X);
    58.                             peaks.RemoveAt(i);
    59.                         }
    60.                     }
    61.                 }
    62.                 timeToReach = timeAhead;
    63.                 Debug.Log("Playing audio");
    64.                 audio.Play(); //all right we are done, start the music!
    65.             }
    67.             else
    68.             {
    69.                 StartCoroutine("AddPeaks");            
    70.             }
    72.         }
    73.     }
    75.     IEnumerator AddPeaks()
    76.     {
    77.         for (int i = 0; i < peaks.Count; i++)
    78.         {
    79.             if (peaks[i] != null)
    80.             {
    81.                 if (peaks[i].X - timeAhead < audio.time)
    82.                 {
    83.                     AddPoint(peaks[i].X);
    84.                     peaks.RemoveAt(i);
    85.                 }
    86.             }
    87.         }
    88.         yield return null;
    89.     }
    92.     void GetPeaks()
    93.     {
    94.         //Gather the peaks in a list
    95.         for (int i = 0; i < songController.spectralFluxSamples.Count; i++)
    96.         {
    97.             if (songController.spectralFluxSamples[i].isPeak)
    98.             {
    99.                 System.Numerics.Vector2 peak = new System.Numerics.Vector2(songController.spectralFluxSamples[i].time, songController.spectralFluxSamples[i].spectralFlux);
    100.                 peaks.Add(peak);
    101.             }
    102.         }
    103.         initiate = true;
    104.     }
    106.     void AddPoint(double time)
    107.     {
    108.         Object.Instantiate(peakPoint, spawnPoint.transform);
    109.         peakPoint.GetComponent<PointController>().time = time;
    110.         peakPoint.GetComponent<PointController>().speed = pointSpeed;
    111.         peakPoint.GetComponent<PointController>().timeToReach = timeToReach;
    112.     }
    113. }
    NewPlotController is what controls the adding of objects. As you can see, it has two for-loops. One which runs before the audio starts (adding my "pre-peaks") and the other running when those have spawned and the music has started. timeAhead is the time before the beat should hit. timeToReach is used to modify timeAhead for the pre-peaks.

    Code (CSharp):
    1. using System;
    2. using System.Collections;
    3. using System.Collections.Generic;
    4. using UnityEngine;
    6. public class PointController : MonoBehaviour
    7. {
    8.     public double time;
    9.     public double speed;
    10.     AudioSource audio;
    11.     private Transform target;
    12.     public double timeToReach;
    13.     private double inverse = 1 / 44100f;
    14.     public double timeLeft;
    15.     private double distance;
    16.     private float distanceToMove;
    17.     private Vector3 direction;
    18.     private Vector3 newPos;
    19.     double t;
    20.     Vector3 startPosition;
    22.     // Start is called before the first frame update
    23.     void Start()
    24.     {
    25.         audio = GameObject.Find("SongController").GetComponent<AudioSource>();
    26.         target = GameObject.Find("Goal").transform;
    27.         //Set initial position
    28.         startPosition = transform.position = new Vector3((float)(speed * timeToReach), (float)-0.1596, (float)1.038119);
    29.     }
    31.     // Update is called once per frame
    32.     void Update()
    33.     {
    34.         t += Time.deltaTime / timeToReach;
    35.         transform.position = Vector3.Lerp(startPosition, new Vector3(target.position.x, (float)-0.1596, (float)1.038119), (float)t);
    36.         timeLeft = timeToReach - Time.deltaTime;
    37.     }
    39.     private void OnTriggerEnter(Collider other)
    40.     {
    41.         string s = "Hit at: " + time.ToString() + " should be at: " + audio.time.ToString();
    42.         Debug.Log(s);
    43.         Destroy(gameObject);
    44.     }
    45. }
    This code controls the spawned objects and is attached to each object. It shows the lerping. The target is the position of the player, which can only move sideways and not in the direction the objects are moving. It also displays the time the beat hits and when it should hit. I've attached an image showing the logs, which hopefully makes sense.

    I hope I explained it somewhat coherently. Otherwise just ask!

    Many thanks!

    Attached Files:

  2. YetAnotherKen


    Jun 17, 2019
    You can use Object Pooling to make it possible to put objects in the right place much more quickly than trying to create a new GameObject or instantiate a Prefab. You do this by making these objects before the game starts. You can also place those objects out in front of the player so they are in the right positions for the first 5 seconds, then just switch their active state to true to make them visible when needed.
  3. Antistone


    Feb 22, 2014
    On line 56, you have:
    timeToReach = timeAhead + peaks[i].X;

    If "timeAhead" is how far ahead of the music you normally spawn peaks, and peak.X is how long from the start of the song until that peak, then this calculation is wrong. If a peak occurs 3 seconds into the song, and you spawn it when the song starts, then you want it to reach the player after 3 seconds, not after 8 seconds. It should just be:
    timeToReach = peaks[i].X;

    Or more generically,
    timeToReach = peaks[i].X - audio.time;

    i.e. the time from now until the peak should be the time from the start of the song to the peak minus how much of the song has already been played. (This last version could be used both for setting up the beginning of the song and for adding new stuff in the middle of the song.)

    Some other notes about your code that you might want to look into:
    • It looks like NewPlotController is using the variable "timeToReach" as if it were a sort of "hidden parameter" on the AddPoint function, setting its value before you call the function in each iteration of the loop in Update, which makes it harder to see your dependencies. You should consider passing this as an explicit parameter, OR giving AddPoint() enough smarts that it can calculate the correct value for itself.
    • AddPeaks() is a coroutine, but it doesn't need to be--it isn't doing anything after the first yield, so it could just run synchronously and then stop.
    • Removing elements from your "peaks" list while you are iterating over it probably means you are inadvertently skipping some elements in your loop. You need to be really careful if you are modifying the contents of a collection while iterating over it.
      If you can put your peaks in chronological order, then you'd probably be better off just remembering what index you are up to and starting the iteration from there next time, rather than removing the elements as you spawn them. (Then you could also stop your iteration as soon as you find a single peak that isn't ready to spawn yet, which would save iterating over the whole list every time.)
      If you can't do that, I'd probably search the list for all the peaks meeting your condition and copy those peaks into a second list. When you've finished your iteration and found them all, then remove them from the original list, spawn them from the second list, and finally clear the second list when you're done.
    • In PointController, the "timeLeft" variable doesn't seem to be used for anything...
    • ...which is good, because you're calculating it wrongly, always setting it equal to the total time minus the time of the most recent frame (ignoring any time that elapsed in previous frames)
  4. Antistone


    Feb 22, 2014
    Just to clarify: object pooling is a way of improving the efficiency of spawning new objects under certain conditions. It won't help you get the functionality correct (and in fact may make it a lot harder to get the functionality correct, since it will make your code more complicated).

    I wouldn't worry about this until you have the basics working correctly. And even then, if your game isn't having any performance issues, it might not be worth it.
  5. YetAnotherKen


    Jun 17, 2019
    some Object Pooling strategies are complicated, others are simple. So you can use a simple one tailored to your needs. I'd use it in integrating advanced technologies into my game, so it's not likely you'd notice the complexity increase when you see the savings in runtime overhead.
  6. Antistone


    Feb 22, 2014
    My overall point is that when a newbie is having trouble getting their basic functionality correct, suggesting that they employ advanced and unfamiliar techniques targeted at improving runtime efficiency is kinda counterproductive for their immediate problem.
    aaro4130 likes this.
  7. YetAnotherKen


    Jun 17, 2019
    Well, pre-instantiation of objects for the first 5 seconds or even for the whole song, then activating objects as needed is very much a part of object pooling. If the song was long enough and the number of objects high enough, then pooling would be the next logical step, so I don't think I have given useless extra information in this context.
  8. palex-nx


    Jul 23, 2018
    You may add 3 sec intro without beat to all songs or you may spawn object not in the initial position and state, but rather pretending to be alive for 3 secs already. For example, if it starts at 0,0 and moves with velocity of 0,1 then spawn it to 0,3 and continue like it was this.
  9. Bogaland


    Feb 19, 2017
    Thanks! This solved it and it works now! Also thanks for your general help, I will certainly look into it when I make the code more efficient. The code now is just proof of concept-ish.

    Thanks for your input! I agree with Antistone on this one, I don't really want to add more complex methods right now before I get the rest to work properly. However, I will definitely look into it in the future!

    I've thought about this as well, but I don't really want to wait for the music in case I decide to increase the time for the beats to a longer period of time. Thanks for your help anyway though!