Adaptive music in video games: What it is and how it works

What is adaptive music?

The term adaptive music is used to refer to music that is designed to adapt/change based on predefined rules. It is often used in video games, virtual reality (VR) and other interactive experiences.

In the context of video games, it means that the music changes/mutates based on the player's actions or game events.

Isn't all video game music adaptive?

Before jumping into the answer let's first compare video game scores to film scores due to the similarity of the media (audiovisual content).

In film, composers typically work with fixed-length edited scenes, and the music is composed to match their linear progression precisely. These scenes are then combined to create the final product.

When we watch the final result on cinema or TV, the music follows the film's events from the beginning to the end in a synchronized way. It won't ever change at its determined point in time. This kind of composition is often referred to as linear music.

In contrast to film though, events in a video game won't often unfold in a completely linear fashion. Depending on the type of game and just to give an example, the player can move back and forth in the game world, change locations, enter or avoid combat scenarios, discover treasures and so on...

The game can also trigger events based on whatever the player's doing at any given time. Due to this the player's progression throughout the game is not linear and as such linear music compositions are often not the best approach for the most immersive experiences.

This does not mean that games can't be scored linearly. Many games being released today still use linear music composed to aesthetically match specific sections or the overall mood of the game.

In summary, and to answer the question above: No, not all video game music is considered to be adaptive.

Adaptive, interactive or dynamic?

These terms are sometimes used interchangeably to mean the same thing but they can actually have different meanings. Let's go over them:

What is interactive music?

Often used to refer to adaptive music in the past, the term interactive music implies an interaction of some sort which is not a requirement when we're talking about adaptive music.

Because of this, nowadays interactive music is considered to be a better term when referring to music that is designed to respond to user input in a direct and immediate kind of way.

This is the case of rhythm video games such as Guitar Hero where to progress within the game the player has to actively "interact" with the songs.

While the songs in the game may not change themselves according to the user input, they could. In such a case "Interactive music" would still be a better term to describe the game's soundtrack instead of adaptive music.

What is dynamic music in video games?

Dynamic music in games can have different meanings depending on the context or individual interpretation. While it can be used as an alternative term to adaptive or interactive music, it can also refer to a piece of music with a high level of dynamic range.

Dynamics are variations in volume or intensity that occur within a musical composition. Orchestral compositions often present a high degree of variation in dynamics for example. As such, these compositions can also be considered to be dynamic music.

When did adaptive music first appear?

As explored in the article The History of Video Game Music, one of the first examples of adaptive music in games was in the title Space Invaders released in 1978.

The soundtrack of Space Invaders was rudimentary by today's standards as it only consisted of a few repeating notes with a very basic electronically generated sound. The developers programmed the musical elements to increase pace as the enemies descended on to the player. This resulted in increased tension among players, making for a more immersive experience.

As simple as this technique may sound it was actually very effective in eliciting the desired emotional response, helping to grab the player's attention and in turn leaving them wanting more, extending playing sessions.

Interestingly this game is also considered to be the first to implement music in general. It's then easy to conclude that adaptive music is not a new concept. It is closely tied to this form of media and its interactive nature.

After the very first uses, major games that used adaptive soundtracks were Wing Commander, Monkey Island 2: LeChuck's Revenge and Ultima Underworld: The Stygian Abyss.

How does adaptive music work?

Adaptive soundtracks are adjusted in real-time based on certain conditions. There are multiple techniques that can be employed by composers and audio professionals to make these adjustments happen, such as vertical re-orchestration, horizontal re-sequencing, dynamic mixing and algorithmic composition.

Besides the music related techniques, an adaptive music system is also needed to apply them. These systems can be built by programmers along with the rest of the game in the game's engine/framework.

Another and easier way to build adaptive music systems is by using what's commonly referred to as audio middleware solutions. I'll go into more detail about the musical techniques and implementation details in the following sections.

Games with adaptive music

There is no shortage of games that use adaptive music, and the tendency is for this usage to keep increasing as developers and game studios discover its benefits over linear musical compositions. It is important to note though, that in addition to the technical aspects, non-linear music also presents a significant creative side.

When considering an adaptive video game soundtrack many questions may need to be asked. Here are a few possible ones:

• Are there actions or events that would benefit from added musical feedback?
• Which and how many actions or events will make the composition change?
• In which way the musical composition will change?
• Does the change improve gameplay?

Every game is different and most will present unique opportunities in which adaptive music can be used in an effective way.

What games use adaptive music?

Some games that use adaptive music include Untitled Goose Game, Age of Empires IV and Assassin’s Creed Valhalla. Below I'll go over some clever ways in which the developers of these games made the soundtracks "adapt" to the gameplay for an enhanced and more immersive experience.

Untitled Goose Game

Untitled Goose Game is a game full of humor where you play as a mischievous goose. In this game you complete various silly objectives and annoy as many people as possible.

The adaptive music in this game was actually inspired by people misunderstanding the original trailer. The trailer had its visuals cut around a solo piano piece from the Debussy’s Préludes that had a "stop-start" nature.

Probably due to this, people thought the background music was reacting to the gameplay. The developers then decided to try and create what people thought they were already doing.

They figured they could use a bunch of different préludes, but have them play in a way that made it seem like a live pianist was watching the player's actions and performing along.

Age of Empires IV

Age of Empires IV is the fouth entry in the popular real-time strategy video game series where players take control of a civilization. In order to conquer their opponents and achieve victory players must build up their economy, infrastructure and military.

This title was inspired by Age of Empires II, one of the most successful RTS (Real-time strategy) games of all time and a fan-favourite entry in the series. Age of Empires II featured a soundtrack that was much-loved by its audience, but it was played "jukebox style" in a linear fashion.

For Age of Empires IV, the soundtrack was reimagined to both better represent the player's progression during gameplay and to evoke the player's emotions.

In Age of Empires players go through different "ages" that reflects the time period and the culture. In this game each age has its own unique pieces of music. The age's musical timeline consists of a short intro segment, an outro segment, and several interchangeable one minute-long main segments that play randomly.

In addition, each of these segments has four vertical layers (vertical re-orchestration) that are used for game states:

• Exploration layer: Features calm ambient background music that plays as a backdrop when building towns and units.
• Tension layer: When engaging in a small skirmish, the composition moves to a layer containing elements of tension, such as low drums or tremolo strings. This layer can be combined with the exploration, combat and rare layers.
• Combat layer: This one plays when the scale of the combat increases. Bigger drums and horns, larger string sections and combat melodies. This layer can be combined with the tension layer or combined with the tension and rare layers together.
• Rare (Variety) layer: This layer presents even bigger hits for added punctuation, big horn blasts, and occasional counterpoint melodies to interplay with other layers. This layer can play along with the tension layer, or with the tension and combat layers combined.

Another feature of the soundtrack in Age of Empires IV is how it reacts when player's age up in the game (move to a more modern era). There are special "age up" transition segments of music that contain the key, tempo, and instrument changes needed to move from one age to the next.

These transitional elements also contain musical "flourishes" worthy of this notable moment in the game. The transitional elements are composed in four distinct layers so that the music will continue to play the appropriate intensity layer as player's progress.

Due to all the variety and complexity of the game's non-linear music system, Age of Empires IV ended up with approximately 170 unique music files. This represents a huge amount of work, detail, and iteration from the composers.

Assassin’s Creed Valhalla

Assassin's Creed is a popular action-adventure video game franchise revolving around an order of assassins who fight against various historical and fictional enemies to preserve peace and order in the world.

Assassin’s Creed Valhalla presents an open-world style gameplay and features one of the largest maps ever made in the series. Due to this Assassin’s Creed Valhalla needed many pieces of music to cover all aspects of the game.

Besides the common challenges open-world style games present regarding their soundtracks, this title presented a particular one: The need to enhance the many quick fights that happen during gameplay with musical passages.

An internal article revealed that in previous entries 80% of the fights lasted less than a minute, which didn't leave enough time to fully engage the combat music. This created a challenge to transition from the intro to the outro musical cue within such a short time frame. To solve this problem the audio team split fights into two categories: normal and intense.

A "normal" fight in the game is defined by the total threat points, which must remain below a certain threshold. If the exploration background music is already playing when the fight starts, its fight layer will simply kick in. If no exploration music is playing, no music will play.

Exploration musical tracks are separated by regions. Each region has many variations for daytime. It also has one variation for nighttime and a special one for subdued territories.

An intense fight will initiate the appropriate pieces of music whether exploration tracks are playing or not. Intense fight compositions are the same across all territories. When this music comes in, players instantly know they'll be facing a tough fight.

This approach helped ensure that the heavier fight musical cues are triggered only in situations where they have enough time to play.

Adaptive music techniques

Adaptive music techniques are the methods used to dynamically change the soundtrack based on the player's actions or game events. These methods allow for the musical compositions to adapt to the gameplay and create a more immersive and engaging experience for the player.

Horizontal re-sequencing

Horizontal re-sequencing is an arrangement technique where tracks of audio are placed horizontally on a timeline. If an event is triggered while a track is playing the playhead will "jump" from that track to another one.

This jump can be made instantaneously or after some musical time to ease the transition. Horizontal re-sequencing is essentially just a way to switch between tracks like we would on a regular music player by pressing the "skip" button.

After this explanation you may ask: Why not just stop the current track and play the next one? Well, It's a similar method indeed but with horizontal re-sequencing the change can be smoother. Tracks can change on the beat (based on the track’s tempo), they can be crossfaded or both.

These methods will be familiar to DJs as they also need to change between tracks in a smooth way. There are also other things that a simple "skip" wouldn't allow, like non-sequential skipping, or jumping to a specific location on a track and not just its beginning.

Horizontal re-sequencing can be useful for a variety of events such as, changing a game scene, entering a new zone, or when changing a game state from exploration to a combat scenario, for example.

Vertical re-orchestration

Vertical re-orchestration is a composition technique where a musical composition is divided into a number of different layers. Every layer is just a portion of the complete track. For example, one layer can contain the rhythm section, other the melody and another a counter-melody.

Generally speaking these layers can be used by themselves as a regular musical composition or they can be combined in different ways. When all the layers are active at the same time you end up hearing the full composition.

Just like horizontal re-sequencing, the vertical re-orchestration technique can also be useful in multiple ways. One possible use is to regularly change layer combinations in order to provide musical variety. This helps to avoid musically fatiguing players while they listen to the same looping composition. This can be especially useful if players can become stuck in a place for a long period of time, for example.

Other possible use case is to use layers for game states like detailed before in the Age of Empires IV example. As a different example, the first layer could contain an ambience for a relaxed state. The second could be activated for added tension, making it suitable for a danger state, and a third layer could be activated to elevate the energy of the track to suit a combat scenario.

Other uses are possible as well for the vertical re-orchestration method, such as making the track more emotionally charged as layers are gradually activated or to signal the player's progression.

Algorithmic composition

The algorithmic composition method refers to music that is generated by algorithms instead of relying on individual tracks of audio as is the case with the horizontal re-sequencing and the vertical re-orchestration techniques.

The algorithms can be designed to generate content on the fly and to respond to different aspects of the gameplay, such as the player's actions, the state of the game world, or the progress of the story.

Dynamic mixing

Dynamic mixing refers to the technique of adjusting audio properties in real-time, in response to changes in the game's state or the player's actions.

When applying this technique, volume, EQ, and other audio parameters can be adjusted on different tracks. For example, compositions might become more intense (louder) during a battle or quieter during a cutscene. This allows for greater flexibility and control over the soundtrack, making it easier to customize it to match the specific needs of the game.

Adaptive music implementation

Implementing adaptive music involves integrating the above techniques into the game by designing a system that responds to the player's actions and game events.

Dynamic music systems can be designed along with the other game systems in the project's game engine or game framework. Nonetheless, developing the necessary tools to enable the creation of these systems from scratch can be both challenging and time-consuming. It can require a significant investment of resources and expertise. Due to this it is often easier to use specialized software tools and plugins that can help build them.

If you're considering using adaptive music within your project, you may find this article useful: How to choose the right audio engine for your game

Adaptive music software

The most important software tool to build adaptive music systems is called an audio middleware. Popular ones today include Audiokinetic's Wwise, Firelight's FMOD, and Criware CRI-ADX.

Audio middleware solutions provide a focused audio environment and advanced audio processing tools. They also often support real-time mixing and provide tools to make building dynamic music systems easier. Another advantage of these tools is that they can facilitate asset management and collaboration between the audio and development teams.

In addition to audio middleware, plugins can also be useful for building adaptive music systems. Engine plugins such as Fabric extend the audio capabilities of the Unity game engine, for example. Fabric provides an extensive set of high-level audio components that allows for the creation of complex and rich behaviours.

Audio plugins are also commonly available for game engines and audio middleware solutions providing additional audio generation and processing options. These include instruments such as synthesizers and sound effects such as convolution reverbs.

For a deeper dive into the software options available, you may find this article useful: Adaptive music software: A roundup of the best options for video games.

Blips adaptive music packs

The horizontal re-sequencing and vertical re-orchestration video examples above were created by making use of Blips.fm adaptive music packs. With these packs, video game developers are able to affordably improve the sound of their games by making use of these industry standard techniques.

Blips packs are created by professional composers to enhance gameplay and to provide the maximum flexibility for developers. Besides the ability to use the vertical re-orchestration technique with each and every pack, the “M” and “L” size packs enable horizontal re-sequencing by providing two or three musical related tracks.

These packs can also be combined with algorithmic compositions or extended with dynamic mixing techniques for endless possibilities.

Blips packs provide the best music for games and allow developers to unleash the power of adaptive music in order to craft more immersive experiences for players. Besides the affordable prices, Blips also provides free packs. These can be used in both non-profit and commercial projects. For more information visit Blips.fm.