Music Canvas

The music canvas project, developed together with Sharif Hadidi, was my first step in music analysis in JavaScript.

The goal was to create a paint tool, that its brush changes according to the properties of a track that is being played in the background.
In order to ‘translate’ audio properties to visual properties, we set up the following guidelines:

  1. The size of the brush will be set according to the beat of the song.
  2. The color of the brush will be set by the tone, and the amplitude of the song.

While searching for a way to run beat detection on the played track, i stumbled upon this GitHub gist by b2renger. I ended up now using this solution, but it lead me to believe that P5.js sound library might do what I need.

music canvas: J. Pollock Mode
music canvas: J. Pollock Mode

P5.js sound library

The P5.js sound library is relatively new and basic, but it covers more than I actually needed.

These are the main functions that I used to get the data I needed:


function setup(){
.
. // some unimportant stuff here
..
amp = new p5.Amplitude(); // creating a new Aplitude object
fft = new p5.FFT(); // creating a new FFT object
.
.. // more unimportant stuff here
}

function draw(){
.
. // some unimportant stuff here
..
amplitude = amp.getLevel(); // getting amplitude level
fft.analyze();
beat = fft.getEnergy('lowMid'); // getting amplitude level at lowMid frequencies
color = fft.getCentroid(); // getting the brightness/tone level
.
.. // more unimportant stuff here

}

Data normalization

In order to use the data we audio analysis data for P5’s fill(); or ellipse(); we had to normalize the numbers. In order to do that, we the a linear scale function from d3.js.


// setting up the scales
var colorScale = d3.scaleLinear()
.domain([0, 8000]) // expected input
.range([0,255]); // normalized output

var ampScale = d3.scaleLinear()
.domain([0, 1])
.range([0,255]);

var beatScale = d3.scaleLinear()
.domain([0, 300])
.range([0,30]);

// using the scales
normColor = colorScale(color);
normAmp = ampScale(amplitude);
normBeat = beatScale(beat);

I must say that these scale functions are among my all time favorites. They are very usable for many different scenarios. On this project, for example, I used these functions to generate a scale of colors.

Building debugging tools

Since I am relatively new to the audio subject, I decided to spend some time building a debug function. This function will print all the data I need to the console in runtime.

This was one of the best decisions I made while working on this project. It allowed me to learn the ranges of my data, and to understand what are these numbers stand for.

You can try to use the debug function yourself. Just open the console and type: printData(‘help’); .
You should see something like that –

open console > printData('help');
open console > printData(‘help’);

 

Live

I quickly setup a node.js server, so currently the music canvas is live 🙂

Check it out:

Automatic drawing mode (aka ‘J. Pollock mode’) – http://music-canvas.herokuapp.com

Manual drawing mode (aka ‘the paint’) – http://music-canvas.herokuapp.com/draw

music canvas: Paint Mode
music canvas: Paint Mode

Published by

Dror Ayalon

@drorayalon

Leave a Reply

Your email address will not be published. Required fields are marked *