Generative Notation and Score with Tone.js

The concept: Notation System for Generative Music

For a long time, I try to find the bridge between generative music, which usually rely on synthesized music, and acoustic (or amplified) music playing and composition.

It felt that starting with creating a notation system for generative music could be the right approach.

MIDI

The effect that the invention of the MIDI protocol had on the evolution of generative (synthesized) music is well known and far beyond the scope of this blog post.

Since the MIDI protocol includes information about the notes that should be played and the way these notes should be played, it served as an inspiration behind my notation system for generative music.

My Generative Music Notation System

The notation system includes two main components:

Legend

The legend is sheet, that defines all possible notes (aka “objects”) that could be played during the generative score. Each note includes the following properties:

  • Color – A unique hex color code that will be used for visualization purposes to represent the note.
  • Frequency – The frequency of the note. Can be presented in Hz (e.g. 440) or letters (e.g. A#3).
  • Amplitude – Volume / velocity in range between 1 (loudest) and 0.
  • Duration – The duration of the note, including an envelope if needed.
  • Loops – The number of time the note should be repeated in a row on the score.
  • Connected_notes – This is the main difference from the MIDI protocol. The connected_notes property will hold a list of notes that should be played with or after this note. Each item on the list, which refer to a connected note, should include the following properties:
    • Color/index number of the connected note according to the legend.
    • The time on which the connected note should be initiated, including maximum and minimum values for silence after the initial timestamp (e.g. if the connected note should be played after the original note, the time will be <the_original_note_duration>+<silence_if_any>).
    • Probability value that will represent the chances that the connected note will be played. All connected notes probability values together should not exceed the value of 1 (== 100% chances).
Generative Music Notation: Legend
Generative Music Notation: Legend
Generative Music Notation: Potential Score
Generative Music Notation: Potential Score

What’s Missing?

Two major properties are missing from the note objects:

  • Instrument (or timbre) – The note object is a set of instructions that could be applied by any instrument. Since I believe that the process of generating music will include the usage of computers (or digital devices), the score can be played with a variety of instruments. The decision about the sound of the piece will be left by the hands of the performer.
  • Timing – Again, since the note object is a set of instructions, these instructions can be initiated and applied anytime during the score, by the performer or by the score itself. The decisions about the timing will also remain by the hands of the performer. The only timed notes are the connected notes which hold instruction that should specify if the note will be initiated with the original note, after the original note, during the original note, etc.

Example

For example, if we will use the legend above and will start the score with the first two notes (7D82B8 & B7E3CC), we will get the following result –

Demo

Using Tone.js, I was able to experiment with generating music based on the legend and score shown above.

The project can be seen here – http://www.projects.drorayalon.com/flickering/.

The current limitations of this demo are:

  • No instrumentation: All notes are being played using the same instrument
  • No dynamics: One of the most likable elements of a music performance is the dynamics and tensions the performer creates while playing the piece. The current implementation doesn’t support any dynamics :\
  • No probability: The current implementation presents a linear and predictable score. Notes have only 1 connected note, and no code was written to support the probability factor that will utilize the notation system to its maximum potential and will make this generative music more interesting (in my opinion).
  • Low-tech visualization: The notation system I described above set up the foundation for a readable visual representation of the score. This visual representation has not been implemented yet.

Some Code. Why Not

This is the code I’m using to run the demo shown above –

//-----------------------------
// play / stop procedures
//-----------------------------
var playState = false;

$("body").click(function() {
  if (playState === false) {
    play();
  } else {
    stop();
  }
});

function play(){
  playState = true;
  $("#click").html("i told you. it is now flickering really badclick anywhere to stop");
  console.log('playing...');
  Tone.Transport.schedule(function(time){
  	noteArray[0].trigger(time);
  }, 0.1);
  Tone.Transport.schedule(function(time){
  	noteArray[1].trigger(time);
  }, 0.4);

  // Tone.Transport.loopEnd = '1m';
  // Tone.Transport.loop = true;

  Tone.Transport.start('+0.1');
  setTimeout(backColorSignal, 100);
}

function stop(){
  playState = false;
  $("#click").html("it is probably still flicker really bad, but it will stop eventuallyclick anywhere to keep it going");
  console.log('stopping...!');
  console.log(Tone.Transport.seconds);
  Tone.Transport.stop();
  Tone.Transport.cancel(0);
}

//-----------------------------
// creating an array of note objects (noteArray)
//-----------------------------

// array of manually added notes
var noteArray = [];

// note constructor
function noteObject(index, color, frequency, amplitude, duration, loops, connected_notes_arry) {
  this.index = index;
  this.color = color;
  this.frequency = frequency;
  this.amplitude = amplitude;
  this.duration = duration;
  this.loops = loops;
  this.connected_notes = connected_notes_arry;
  this.trigger = function(time, index=this.index, frequency=this.frequency, duration=this.duration, connected=this.connected_notes){
    // console.log('time: ' + time);
    // console.log('index: ' + index);
    console.log('');
    console.log('------------');
    console.log('it is ' + Tone.Transport.seconds);
    console.log('playing: ' + index);
    console.log('frequency: ' + frequency);
    console.log('duration: ' + duration);

  	synthArray[index].triggerAttackRelease(frequency, duration, time);

    if (connected !== null) {
      var nextIndex = connected[0];
      var nextTime = 0.01 + Tone.Transport.seconds + connected[1] + parseFloat((Math.random() * (connected[2] - connected[3]) + connected[3]).toFixed(4));
      console.log('generated: ' + nextIndex);
      console.log('at: ' + nextTime);
      Tone.Transport.schedule(function(time){
        noteArray[nextIndex].trigger(time);
      }, nextTime);
    }
  };
}

// starting notes
noteArray.push(new noteObject(0, '7D82B8', 'c3', 1, 1.520*5, 0, [2,1.520*5,0.020*5,0.020*5,0.9]));
noteArray.push(new noteObject(1, 'B7E3CC', 'e2', 1, 6.880*5, 0, null));

// the rest of the notes
noteArray.push(new noteObject(2, 'C4FFB2', 'b2', 1, 1.680*5, 0, [3,1.520*5,0.40,0.80,1]));
noteArray.push(new noteObject(3, 'D6F7A3', 'c#2', 1, 3.640*5, 0, [4,0,0.8,1,1]));
noteArray.push(new noteObject(4, 'ADD66D', 'b2', 1, 0.650*10, 0, [5,0.650*10,0.2,0.2,1]));
noteArray.push(new noteObject(5, 'A4FF7B', 'a2', 1, 1.800*5, 0, [6,0,0,0,1]));
noteArray.push(new noteObject(6, '7BFFD2', 'f#2', 0.2, 1.800*5, 0, [0, 1.800*5, 1, 2, 1]));


//-----------------------------
// creating an array of synth objects (synthArray), based on note objects (noteArray)
//-----------------------------

var synthArray = [];

for (var i=0;i<noteArray.length;i++){
  options = {
    vibratoAmount:1,
    vibratoRate:5,
    harmonicity:4,
    voice0:{
      volume:-30,
      portamento:0,
      oscillator:{
        type:"sine"
      },
      filterEnvelope:{
        attack:0.01,
        decay:0,
        sustain:0.5,
        release:1,
      },
      envelope:{
        attack:0.1,
        decay:0,
        sustain:0.5,
        release:1,
      },
    },
  voice1:{
    volume:-30,
    portamento:0,
    oscillator:{
      type:"sine"
    },
    filterEnvelope:{
      attack:0.01,
      decay:0,
      sustain:1,
      release:0.5,
    },
    envelope:{
      attack:0.01,
      decay:0,
      sustain:0.5,
      release:1,
    }
  }
  };
  synthArray.push(new Tone.DuoSynth(options).toMaster());
}

//-----------------------------
// low-tech visualization
//-----------------------------
b = new Tone.Meter ("signal");
synthArray[1].connect(b);
// synthArray[2].connect(b);

function backColorSignal(){
  if (b.value === 0){
    setTimeout(backColorBlue, 100);
  } else {
    var color = "rgba(0, 0, 255," + b.value + ")";
    $("html").css("background-color", color);
    setTimeout(backColorSignal, 100);
    // console.log('b.value: ' + b.value + " " + color);
  }
}

function backColorBlue(){
  var color = "rgba(0, 0, 255,1)";
  $("html").css("background-color", color);
  setTimeout(backColorSignal, 100);
}

 

Composition as a window to an artistic process

As a long time Wes Anderson fan, I always pay close attention to the graphic design in his films. To me, Anderson’s wonderfully crafted frames are not only a very good reference for aesthetics, symmetry, and story telling, but also, work that seem both enjoyable to watch and to make.


Wes Anderson // Centered from kogonada on Vimeo.

To emphasize these ideas, I will analyze a movie poster designed for one of Anderson’s latest films – Moonrise Kingdom (2012).

Moonrise Kingdom: Movie Poster // Wes Anderson
Moonrise Kingdom: Movie Poster // Design: P+A, Director: Wes Anderson

Composition

A movie poster is graphic design piece that its major objective is to promote a movie. And therefore, by analyzing a movie poster we can get a unique perspective about the way the movie creators (and promoters) looked at the cinematographic outcome.

The Moonrise Kingdom poster composition really tells a story. To me, it’s not only the story of the movie, but also the story of the making the movie and promoting it.

On first sight, it seems like the Moonrise Kingdom poster cherish the regular Wes Anderson symmetry, that can be seen in any one of his movies. In fact, this poster composition shows the growth process Anderson went through as a director, and a story teller, and the confidence he (and his talented crew of designers) felt about this movie before its release.

The Moonrise Kingdom poster: A grid that puts the story in the front row
The Moonrise Kingdom poster: A grid that puts the story in the front row

When we look at this poster, first, our eyes are being attracted to the title of the movie, floating in the middle of the frame in big letters. This alone tells us that, for the makers of this film, and the designers of this poster, the movie or the story, is the most important thing. As if they believe that the story is so good, that it will sell itself.

As you can see, in then center of the piece the designer places the two main characters of the film – Sam & Suzy – played by Jared Gilman & Kara Hayward, two relatively unknown young actors. Anderson (or the designer of this poster) doesn’t promote the movie using his big stars. Instead, he puts all characters at the same level, in terms of graphical hierarchy, and the main character of the story are right there in the middle — this story is not about big movie stars, it’s about two young people and their fantastic adventure.

In fact, we can say that Anderson was so confident about this movie, that on the promotional poster, that was published before the movie premier, the big star do not appear visually at all. Again, it’s all about the story this time.

The Moonrise Kingdom 'coming soon' poster: The big starts are not even there. It's all about the story.
The Moonrise Kingdom ‘coming soon’ poster: The big starts are not even there. It’s all about the story.

Between personal development and poster composition

We can clearly use Anderson’s movie poster designs as a reference for his growth has as a director. Let’s have a look a few movie posters from Anderson’s earlier works – From the star-heavy The Royal Tenenbaums (2001), to the latest The Grand Budapest Hotel (2014).

On The Royal Tenebaums (2001), we can see the Anderson uses his league of stars as the main selling point for the movie. The actors appear in the middle of the frame, and their name is highlighted at the top of the poster. The movie title appears at the lower part of the frame, makes it easily ignored.

The Royal Tenenbaums movie poster: The names of the actors is the focus. The movie title comes second.
The Royal Tenenbaums movie poster: The names of the actors is the focus. The movie title comes second.

On his next movie, The Life Aquatic with Steve Zissou (2004), we can see that Anderson (or his promoters) is still somewhat  not confident about the story itself as a good selling point for the movie. The poster composition is more focused on star of the show, Bill Murray, than on anything else. The character is placed in the middle of the composition, and our eyes our being attracted to the title of the poster – “Bill Murray”. The title of the movie is blended into the yellow submarine.

The Life Aquatic with Steve Zissou movie poster: Focuses more on Bill Murray than anything else.
The Life Aquatic with Steve Zissou movie poster: Focuses more on Bill Murray than anything else.

Anderson gains a lot of confidence towards his next full feature movie, The Darjeeling Limited (2007). Here, the movie poster composition puts the three fabulous actors as equal, in terms of hierarchy, to the movie title itself. Both the actors and the title of the movie are shown in the middle of the frame, and do not compete over the viewers’ attention. It feels like one cannot stand without the other, which in some weird way, tells the story of this movie, which would not be the same with different characters or different actors.

The Darjeeling Limited movie poster: The story and the title are as important as the famous actors.
The Darjeeling Limited movie poster: The story and the title are as important as the famous actors.

Let’s jump to Anderson’s latest feature – the Golden Globe and Academy Awards winner – The Grand Budapest Hotel (2014). As usual on Anderson movies, The Grand Budapest Hotel features a few of the most desired actors in the business. But even though the movie is rich with talented personnel, Anderson shows full confident in his new movie, and world he created. On the movie poster, Anderson puts only the movie title, and place on which the story happens. That’s it. Anderson is fully focused on storytelling and the fantastic world he created. Not only that we don’t see the fancy cast, their names do not even appear on the poster. As viewers, we are totally focused on the amazingly designed hotel, and on the stories that happen inside it.

The Grand Budapest Hotel movie poster: Anderson focuses on the story and the world he created.
The Grand Budapest Hotel movie poster: Anderson focuses on the story and the world he created.

Color palette

Let’s get back to the Moonrise Kingdom. This movie is all about teen spirit and adventures. To emphesize that, the movie poster is using  a color pallete full of greens and browns, colors that we usually see in the nature.

Moonrise Kingdom movie poster: Color palette
Moonrise Kingdom movie poster: Color palette

Also, the poster tones of the blue and yellow, give the entire image its aesthetic youth spirit. The yellow color of the titles is similar to the scounts’ yellow neckerchiefs. This similarity makes the two connected, as if the designer is trying to say that “this movie is about these scouts”.

Typography

The handwriting font of the titles on the Moonrise Kingdom poster is usually associated with note taking or a writing of a journal. The usage of this font gives the impression that the movie will tell us about an adventure, as it was written by one of the characters.