10

You may be familiar with ryhtm games as Dancing Line, Geometry Dash, or even Guitar Hero. They have something in common : every single action in the game is synchronized with the music. My purpose is to make a game like Geometry Dash where the player has to do some actions in rythm while the character scrolls automatically in the level. I have a few questions about this type of game :

  • Do I have to keep my FPS constant as it has to be always synchronized with the music ?
  • How can I build my levels with a music in mind to make sure that it will be synchronized ?

I am working on Unity using C#.

Shashimee
  • 1,240
  • 3
  • 10
  • 26

4 Answers4

19

I don't recommend using Time.deltaTime to track progress through playing the song as described in another answer.

It's great for smoothing & interpolating effects to play out in realtime, but when you want to match the song's time you should ask the song.

In Unity, you can measure an Audio Source's progress through playing a track directly using AudioSource.time or AudioSource.timeSamples to get the precise playhead position. (Note that the latter measures in samples, which may vary in rate per second depending on how you've compressed your track)

This way, you don't need to run your own separate timer and hope they stay in sync. You can wire up game triggers of beat matching effects to run off of the song's own timeline.

For layering gameplay sounds onto the music, you can use AudioSource.PlayScheduled to ensure your sounds land exactly on the beat, as even a small mismatch can be noticed

DMGregory
  • 134,153
  • 22
  • 242
  • 357
  • Any ideas how to sync graphics to the AudioSource? F.e. I want to show an explosion or collision connected to the music. I want to show it exactly when sound played. – syabro Aug 26 '18 at 18:48
  • That's what all of the answers here are about. Did you find any trouble putting the suggestions in these answers into practice, @syabro? – DMGregory Aug 26 '18 at 20:38
2

As an alternative if you are fine with not working from scratch.

There are ready made assets already (under Editor Extensions/ Audio like https://www.assetstore.unity3d.com/en/#!/content/54639 (paid) or https://www.assetstore.unity3d.com/en/#!/content/39835 (free)) where you can sync music tracks to event. In that example the weapon is shooting on the beat of the music.

If you want to learn how they work, try 1-2 of the free ones and look at the source.

Zibelas
  • 4,281
  • 2
  • 13
  • 22
  • I saw a presentation from the Koreographer devs a little while back - looked like a great piece of tech! At the time its audio analysis capabilities were limited, but if you knew in advance the track you're working with, you could use the tech to very precisely schedule game events to specific beats/moments in the music. I don't know whether they've added the ability to do more analysis automatically since that demo. – DMGregory Aug 01 '17 at 16:03
1

Unity already decouples gameplay from framerate. If you remember to always use Time.deltaTime in your Update-functions, the actual graphics framerate should not affect gameplay speed. So when you start the game and the audio track at the same time, they should stay synchronized.

Regarding designing levels around music, there are two approaches:

  • The manual approach. Meticulously hand-craft levels around specific audio tracks. Having the original sheets music at hand can help, but it's also possible to do it just by ear.

  • The procedural approach. Use audio analysis algorithms to auto-generate levels around audio tracks. The most simple is to just look for volume spikes. That should already allow you to identify drum beats in many songs and spawn obstacles accordingly. A step further is detecting volume shifts separately in different frequency spectrums. This makes beat detection more reliable and allows you to spawn different obstacles for different pitches. But that's still just scratching the surface. Audio analysis is a very wide and interesting field. People wrote a lot of scientific literature about it.

    How deep you need to dive into this field depends on how ambitioned you are. Some games are satisfied with just detecting the beats and randomly-generate all the rest of the level design(Crypt of the Necrodancer). Others try to put as many aspects of level design under the control of the audio as they can (Audiosurf).

Many rhythm games use a hybrid approach. You generate a first draft of the level using a generator and then tweak it by hand to make it more playable.

Community
  • 1
Philipp
  • 119,250
  • 27
  • 256
  • 336
  • Thanks for your answer, I'll look into it ! Is Time.deltaTime really precise and there won't be any time lag created with long musics ? – Shashimee Aug 01 '17 at 09:15
  • @Shashimee It should be precise enough. If you want to be really sure to not lose anything to rounding differences, use Time.time instead. – Philipp Aug 01 '17 at 09:18
  • I would recommend procedural approach because it opens many more possibilites for game to develop. Players can add their own tracks, you can change mesh and all the other stuff to the beat, like particles. Also, you won't spend a loooooot of time on designing levels. – Candid Moon _Max_ Aug 01 '17 at 12:51
  • 1
    @Philipp unfortunately Time.time can be even more susceptible to imprecision if the game has been running a while. Since it measures game time since startup, rather than since the start of the track or the start of the level, after playing a song or two it will have lost as many low bits of precision as a naive sum of deltas, and the error can continue to increase if the game is left running / slept-restored repeatedly. – DMGregory Aug 01 '17 at 13:52
0

Very common is the approach to have a "beat map". Any kind of data that represents the beats in the song. It can be a texture, datafile or anything else that holds that information. Most of the time it's created manually to create different beats and combinations, versions of speeds for an audio map.

It can be generated procedurally from audio file data, though.

How to get the speed to match the "beat point"? When to spawn an object?

Speed = Distance / Time.

Distance = Distance between points of creation of GameObject and of "beat point".

Time = Audio length and constructed beat map should be enough. With this information you can analyze the data of audio/map to predict when an object must be spawned to reach the exact point. Now it all depends on when you want to spawn that object, if you want it to be very fast after spawn, then you spawn it later with great speed. If you want it to be at a normal speed - then it will need time to reach the player and audio needs time to play to the right moment.

You can even generate a new map with time stamps for spawns of beat objects for particular audio map. You have to make some predictions at some point to be able to match the beat, for most popular style I would say, to predict when to spawn the object for it to move with certain speed. Those can be made in runtime, or read from some file calculated previously, or saved in RAM...

I assume games like Osu don't have this Distance, they just have time of circles shrinking, bpm... Maps are generally custom made. So approach to calculate visuals depends on the style of the game you are making as well.

So to sum up everything, the values are interdependent. Depends on how many beats you have, on the distance between spawn point and player, and sometimes depends on data provided [some games have the speed of every beat and its appearance encoded in some data file which is created manually]. If you are doing it procedural way, then you can calculate everything basically by using audio data.

Candid Moon _Max_
  • 2,856
  • 1
  • 18
  • 38