Making an animated data visualisation / sonification

Over the past few years, I've typically been making visualisations in R, and then touching them up in Adobe Illustrator. More recently, I've been exploring animated data visualisations, with R, and gganimate. While these are really good, I've found them a bit limiting for what I was aiming at, so I started to look into using Adobe After Effects - an application for motion graphics and digital effects. I had a bit of a play, and figured there's potential to do some really good stuff.

I've also recently become interested in data sonification - interpreting data with sound, and it seemed that there was a real opportunity to make something really interesting, so that's what I set out to do.

I started to work on a visualisation and sonification using Covid case rates per capita for countries in the G20, with extra focus on the US. And I'm well-chuffed with how it came out:

Quite a few people have asked how I made this, so I figured it'd be good to quickly rattle through the steps - not an exhaustive step-by-step tutorial, because that'd be too long, but enough information about the tools steps to hopefully give others a leg up to doing similar.

High level overview

This is the order of doing stuff, and the software involved:

  1. Get the data from Our World in Data
  2. Decide on a visualisation approach and work out how to make it data driven in After Effects
  3. Process the data in R (transform the data, output the data in the right format for After Effects, make the sound files)
  4. Make the animiation in After Effects
  5. Edit the soundfiles in GarageBand
  6. Put it all together in Adobe Premiere

This all sounds pretty straightforward, but there's some complexity in there, so let's pick it all apart.

1. Get the data from Our World in Data

Probably the easiest step. Our World In Data is a big repository of data, and they provide access to cleaned and aggregated data on Covid from around the world (and loads of other stuff). I used this file from their github repo - i just downloaded the csv and was good to go.

2. Decide on a visualisation approach and work out how to make it data driven in After Effects

I'd already been messing about with After Effects, so I knew the rough style that I wanted to go for. I'd made something previously with the Saber plugin, and wanted to use it again. This is a mask that gives a customisable light saber effect (or neon, or electricity, or even the patronus from Harry Potter). After some trial and error, and some passing ideas back and forth with my partner, I settled on a series of rotating hexagons, with the rotation speed, colour, and glow intensity all defined by the case rate per capita.

Now the idea was in the bank, I had to think about getting the data in the right shape.

This required a lot of Duck Duck Going, and a lot of trial and error. One thing about the After Effects community that's different from the R community is that they love to make a video to show you how to do stuff. If you've got a problem in R, you can search it up, and read what you need, but with After Effects, you got to watch a load of videos.

I knew that I had a set of objects, and to animate them, I needed add keyframes to each, defining the colour, rotation speed and glow intensity for each day. With over a year's worth of data, it's just not practical to do that by hand, for multiple objects. I managed to work out that you can paste text data onto each object's property, and I knew I could make that text data by scripting it in R! Not completely satisfactory, because it still needs manual intervention, but it's better than nothing.

Example of the format required for pasting into After Effects

This is an example of the sort of thing I was aiming for. So over to R I went to create it.

3. Process the data in R

I'm a huge fan of R - scripting the data processing this way means it is repeatable, and shareable very easily.

The full script is available in this gist:

Hopefully this is fairly self explanatory to those that know R, but the basic premise is:

  1. get and filter the data
  2. do some work to fill in blanks
  3. filter the data a bit more, and calculate the fields needed for After Effects (rotation amount, colour, glow)
  4. make the text files for each country to copy into After Effects
  5. make the sonification for each country

(Don't judge me for the state of the script - it's well-hacky)

4. Make the animation in After Effects

After Effects

Here's what my After Effects environment looked like after I'd put it all together. There's a load of objects in the bottom-left pane, which is where you add new things. Each object is a solid layer, and the hexagons are shape mask. Each hexagon has an effect added, from the excellent (and free!) Saber plugin.

The basic principle in After Effects is that the bottom-right panel is where the keyframes are set. You set a property value in the bottom right panel (eg scale 100%), and add a keyframe at 1 second. You can then add another keyframe at 2 seconds, changing the scale property to 200%. When you play the animation, After Effects will animate starting with a shape that doubles in size between 1 second and 2 seconds of your animation.

This is great, but as mentioned previously, manually creating these things would be a massive time-sink, so some degree of automation is required. I created a couple of keyframes in the property that I wanted to animate, then copied the property from the bottom left panel, and pasted into a text editor (which give something similar to the file above). We can edit this text file, and then paste it back on to the right property with more, or different keyframe information, and this is reflected in the animation. But as also mentioned before, we can create that text file programmatically, and this is what makes this animation much less time-consuming.

So once all the text files were generated, I had to remove all the double-quote marks from them. I did this by doing a find and replace across all the contents of the folder in Sublime (Cmd + Shift + F), then one-by-one, I pasted each text file onto the right property - a total of 57 files - 3 for each country.

There's a heap of other things going on in the animation - the timeline along the bottom, which is another mask that simply shrinks over time, labels for the months whose opacity changes from 0% to 100% in the right frames, and a smoke effect to give a bit more atmosphere.

Once all this was assembled, I then rendered the animation as .mov file. I decided to go with 60 frames per second, because lower frame-rates seemed to make the wagon-wheel effect worse, where some of the hexagons looked like they weren't moving because their rotation speed cause them to align with the frame movement. This gives a really good, clear animation, but it increases the render time. This animation, of 1 minute and 45 seconds, took 14 hours to render on my MacBook Pro with 8GB RAM (aside - if I'm going to do more of this, I'm going to look at a more suitable machine).

Once rendered I moved onto the sound bit...

5. Do the sound files in GarageBand

There's some processing done in R to generate the wav files for the sonification. I've written another blog post about the detail of this for a previous project, so don't want to go into too much detail, but the basic principle is to bin the data according to 88 notes (like a piano). Then generate a wav file for each country at the right tempo to match the animation.

Once these were made and saved, I dragged them into GarageBand as 19 audio files, and this is what that looked like:


The default sound for this files was a bit plain, and I knew I wanted some sound that was a bit reminiscent of 2001:A Space Odyssey, so I had a play with the options, and eventually settled on the Condensator sound under Experimental Guitars, which gave the sound I was after.

Once this was saved as an .mp3 file , the final step was to put it all together.

6. Put it all together in Adobe Premiere

Adobe Premiere Rush

This bit is fairly straightforward - it's simply a case of dragging the .mov file of the animation in, and the .mp3 audio file, making sure they matched up (which they did because all the heavy-lifting was done in R), and adding a dissolve effect to the video so that it ends gracefully. I then outputted this to a few mp4 files, in various quality bands, and also pushed straight to YouTube.

Wrap Up

Hopefully this gives enough of an idea as to my approach to this. It's hard to document all the steps, because there are so many of them. If you have specific questions, please do feel free to contact me and I'll do my best to help. You can find me on Twitter: @northernjamie