Generative Notation and Score with Tone.js

The concept: Notation System for Generative Music

For a long time, I try to find the bridge between generative music, which usually rely on synthesized music, and acoustic (or amplified) music playing and composition.

It felt that starting with creating a notation system for generative music could be the right approach.

MIDI

The effect that the invention of the MIDI protocol had on the evolution of generative (synthesized) music is well known and far beyond the scope of this blog post.

Since the MIDI protocol includes information about the notes that should be played and the way these notes should be played, it served as an inspiration behind my notation system for generative music.

My Generative Music Notation System

The notation system includes two main components:

Legend

The legend is sheet, that defines all possible notes (aka “objects”) that could be played during the generative score. Each note includes the following properties:

  • Color – A unique hex color code that will be used for visualization purposes to represent the note.
  • Frequency – The frequency of the note. Can be presented in Hz (e.g. 440) or letters (e.g. A#3).
  • Amplitude – Volume / velocity in range between 1 (loudest) and 0.
  • Duration – The duration of the note, including an envelope if needed.
  • Loops – The number of time the note should be repeated in a row on the score.
  • Connected_notes – This is the main difference from the MIDI protocol. The connected_notes property will hold a list of notes that should be played with or after this note. Each item on the list, which refer to a connected note, should include the following properties:
    • Color/index number of the connected note according to the legend.
    • The time on which the connected note should be initiated, including maximum and minimum values for silence after the initial timestamp (e.g. if the connected note should be played after the original note, the time will be <the_original_note_duration>+<silence_if_any>).
    • Probability value that will represent the chances that the connected note will be played. All connected notes probability values together should not exceed the value of 1 (== 100% chances).
Generative Music Notation: Legend
Generative Music Notation: Legend
Generative Music Notation: Potential Score
Generative Music Notation: Potential Score

What’s Missing?

Two major properties are missing from the note objects:

  • Instrument (or timbre) – The note object is a set of instructions that could be applied by any instrument. Since I believe that the process of generating music will include the usage of computers (or digital devices), the score can be played with a variety of instruments. The decision about the sound of the piece will be left by the hands of the performer.
  • Timing – Again, since the note object is a set of instructions, these instructions can be initiated and applied anytime during the score, by the performer or by the score itself. The decisions about the timing will also remain by the hands of the performer. The only timed notes are the connected notes which hold instruction that should specify if the note will be initiated with the original note, after the original note, during the original note, etc.

Example

For example, if we will use the legend above and will start the score with the first two notes (7D82B8 & B7E3CC), we will get the following result –

Demo

Using Tone.js, I was able to experiment with generating music based on the legend and score shown above.

The project can be seen here – http://www.projects.drorayalon.com/flickering/.

The current limitations of this demo are:

  • No instrumentation: All notes are being played using the same instrument
  • No dynamics: One of the most likable elements of a music performance is the dynamics and tensions the performer creates while playing the piece. The current implementation doesn’t support any dynamics :\
  • No probability: The current implementation presents a linear and predictable score. Notes have only 1 connected note, and no code was written to support the probability factor that will utilize the notation system to its maximum potential and will make this generative music more interesting (in my opinion).
  • Low-tech visualization: The notation system I described above set up the foundation for a readable visual representation of the score. This visual representation has not been implemented yet.

Some Code. Why Not

This is the code I’m using to run the demo shown above –

//-----------------------------
// play / stop procedures
//-----------------------------
var playState = false;

$("body").click(function() {
  if (playState === false) {
    play();
  } else {
    stop();
  }
});

function play(){
  playState = true;
  $("#click").html("i told you. it is now flickering really badclick anywhere to stop");
  console.log('playing...');
  Tone.Transport.schedule(function(time){
  	noteArray[0].trigger(time);
  }, 0.1);
  Tone.Transport.schedule(function(time){
  	noteArray[1].trigger(time);
  }, 0.4);

  // Tone.Transport.loopEnd = '1m';
  // Tone.Transport.loop = true;

  Tone.Transport.start('+0.1');
  setTimeout(backColorSignal, 100);
}

function stop(){
  playState = false;
  $("#click").html("it is probably still flicker really bad, but it will stop eventuallyclick anywhere to keep it going");
  console.log('stopping...!');
  console.log(Tone.Transport.seconds);
  Tone.Transport.stop();
  Tone.Transport.cancel(0);
}

//-----------------------------
// creating an array of note objects (noteArray)
//-----------------------------

// array of manually added notes
var noteArray = [];

// note constructor
function noteObject(index, color, frequency, amplitude, duration, loops, connected_notes_arry) {
  this.index = index;
  this.color = color;
  this.frequency = frequency;
  this.amplitude = amplitude;
  this.duration = duration;
  this.loops = loops;
  this.connected_notes = connected_notes_arry;
  this.trigger = function(time, index=this.index, frequency=this.frequency, duration=this.duration, connected=this.connected_notes){
    // console.log('time: ' + time);
    // console.log('index: ' + index);
    console.log('');
    console.log('------------');
    console.log('it is ' + Tone.Transport.seconds);
    console.log('playing: ' + index);
    console.log('frequency: ' + frequency);
    console.log('duration: ' + duration);

  	synthArray[index].triggerAttackRelease(frequency, duration, time);

    if (connected !== null) {
      var nextIndex = connected[0];
      var nextTime = 0.01 + Tone.Transport.seconds + connected[1] + parseFloat((Math.random() * (connected[2] - connected[3]) + connected[3]).toFixed(4));
      console.log('generated: ' + nextIndex);
      console.log('at: ' + nextTime);
      Tone.Transport.schedule(function(time){
        noteArray[nextIndex].trigger(time);
      }, nextTime);
    }
  };
}

// starting notes
noteArray.push(new noteObject(0, '7D82B8', 'c3', 1, 1.520*5, 0, [2,1.520*5,0.020*5,0.020*5,0.9]));
noteArray.push(new noteObject(1, 'B7E3CC', 'e2', 1, 6.880*5, 0, null));

// the rest of the notes
noteArray.push(new noteObject(2, 'C4FFB2', 'b2', 1, 1.680*5, 0, [3,1.520*5,0.40,0.80,1]));
noteArray.push(new noteObject(3, 'D6F7A3', 'c#2', 1, 3.640*5, 0, [4,0,0.8,1,1]));
noteArray.push(new noteObject(4, 'ADD66D', 'b2', 1, 0.650*10, 0, [5,0.650*10,0.2,0.2,1]));
noteArray.push(new noteObject(5, 'A4FF7B', 'a2', 1, 1.800*5, 0, [6,0,0,0,1]));
noteArray.push(new noteObject(6, '7BFFD2', 'f#2', 0.2, 1.800*5, 0, [0, 1.800*5, 1, 2, 1]));


//-----------------------------
// creating an array of synth objects (synthArray), based on note objects (noteArray)
//-----------------------------

var synthArray = [];

for (var i=0;i<noteArray.length;i++){
  options = {
    vibratoAmount:1,
    vibratoRate:5,
    harmonicity:4,
    voice0:{
      volume:-30,
      portamento:0,
      oscillator:{
        type:"sine"
      },
      filterEnvelope:{
        attack:0.01,
        decay:0,
        sustain:0.5,
        release:1,
      },
      envelope:{
        attack:0.1,
        decay:0,
        sustain:0.5,
        release:1,
      },
    },
  voice1:{
    volume:-30,
    portamento:0,
    oscillator:{
      type:"sine"
    },
    filterEnvelope:{
      attack:0.01,
      decay:0,
      sustain:1,
      release:0.5,
    },
    envelope:{
      attack:0.01,
      decay:0,
      sustain:0.5,
      release:1,
    }
  }
  };
  synthArray.push(new Tone.DuoSynth(options).toMaster());
}

//-----------------------------
// low-tech visualization
//-----------------------------
b = new Tone.Meter ("signal");
synthArray[1].connect(b);
// synthArray[2].connect(b);

function backColorSignal(){
  if (b.value === 0){
    setTimeout(backColorBlue, 100);
  } else {
    var color = "rgba(0, 0, 255," + b.value + ")";
    $("html").css("background-color", color);
    setTimeout(backColorSignal, 100);
    // console.log('b.value: ' + b.value + " " + color);
  }
}

function backColorBlue(){
  var color = "rgba(0, 0, 255,1)";
  $("html").css("background-color", color);
  setTimeout(backColorSignal, 100);
}

 

NOMNOM 2: The Video Machine – The Physical Computing Aspects of the Project

 

NOMNOM: The Video Machine
NOMNOM: The Video Machine

Intent

The purpose of this project was to allow users to play music (or a DJ set) using videos from YouTube.

NOMNOM is an advanced version of The Video Machine presented for the mid-term. It controls the playback of videos presented on a web browser.
By pressing a button on the controller, the correlated video is being played on the screen and heard through the speakers. The videos are being played in sync with one another. Only the videos that are being played, are being heard.

On the new version, The Video Machine controller offers four functions that allow making changes to the way the videos are being played:

  • Repetition – Affects The number of times a video is being played during a single loop (1-4 times).
  • Volume – Affects the volume of the selected video.
  • Speed – Changes the speed of the selected video.
  • Trim – Trims the length of the selected video.

The first prototype of the new version in action –

NomNom: The Video Machine

Main Objectives

The goal was to gain a few critical improvements from the previous versions of the product. After brainstorming for possible improvements, and reviewing the feedback we had received, the following objectives were chosen:

  • To keep it simple, while introducing more functionality – One of the major strength of the original version was its simplicity. We were able to achieve a design that allowed simple and self-explanatory interaction, that was enjoyable for both experienced DJs and users with zero experience.
    For the new version, new features, such as a consistent and predictable playback sequence, an automatic beat-sync between the played videos, the ability to change the number of times a video will be played over a single loop, and the ability to change the playback properties for each one of the videos while it is being played.
    The new features of the new version allow the user the achieve great results more easily, by using the same simple controls of the old version. A total of 6 new features and improvements were added to the product while adding only a single rotary switch to the previous layout.

    The new features of the new version allow the user the achieve great results more easily, by using the same simple controls of the old version.

  • To make it feel solid – The first impression the user has on a product comes from looking at it. NOMNOM was built from solid materials in order to allow the user to feel free to physically interact with it. The solidity of the controls freed up users from thinking about the physical interaction and concentrating on the content (the video and the sound).

NOMNOM: The new version

  • To smoothen the controls – Enjoyable interaction cannot be achieved only by providing a fast and easy way to complete a task. The time the user spends using the product should be enjoyable as well.
    Is order to build a smooth and fun tangible interaction, a research was done around different potentiometers, buttons, and switches. Eventually, the controls that provide the best ‘feel’, and that were the most accurate, were chosen.

  • To take further development in current considerations – In most cases, the ability to innovate comes from deep understanding of the way a certain system works. To allow further development, there was a need to build the product in a way that will make it be easy to learn and to understand, to both for us and for other future contributors. Therefore, an effort was done to design and build the inner parts of the box in a way that will be very understandable for anyone who reveals it.
    The design of the structure of the internal electronic parts, not only allowed clarity on the debugging stages, but also fast analysis and understanding of the implications of any change or addition.
NOMNOM: Designing the inner structure
NOMNOM: Designing the inner structure

There was a need to build the product in a way that will make it be easy to learn and to understand, to both for us and for other future contributors. Therefore, an effort was done to design and build the inner parts of the box in a way that will be very understandable for anyone who reveals it.

NOMNOM: In the making of
NOMNOM: In the making of

Decision-Making and challenges

Design Overview

Leaning on the design of the previous version, we made a few improvements to our electric circuits, and a few major improvements to our physical interface design.

NOMNOM: Schematic
NOMNOM: Schematic

Doing More With the Same Buttons

On of the major limitations of the first version was that in order change the playback mode (properties / attributes) of a video, the user had to stop the playback, make the changes using the knobs, and start the playback again. Therefore, one of the most important features of the new version, was the ability to change the playback mode (properties / attributes) of a single video while the video is being played.

To avoid adding a series of knobs for each on of the videos, the existing buttons are being used for two functions:

NOMNOM: A single press to start / stop
A single press to start / stop
NOMNOM: Press & hold to make changes to the video playback
Press & hold to make changes to the video playback

 

 

 

 

 

 

The component that was used for the buttons is the Adafruit Trellis, a single PCB that connects 16 press buttons.

The Trellis PCB and its Arduino library support two modes:

MOMENTARY – A mode on which button press event is detected only a buttons is being held down.
LATCHING – A mode on which button press event changes the state of the button (e.g. from ON to OFF).

NOMNOM: One of the challenges was to make the Trellis PCB support both of its different modes at the same time
NOMNOM: One of the challenges was to make the Trellis PCB support both of its different modes at the same time

One problem was that by default, the Trellis can operate on only one of these modes at the time.
Another challenge was to find an efficient way (in terms of performance) to read the button states, so the controller will be very responsive to the user actions — the changes on the screen, and on the LEDs on the controller should be immediate.

After 3-4 weeks of research on the way the Trellis PCB works and coding different experiments, the following Arduino code allowed the support of the two modes simultaneously.


#include 
#include "Adafruit_Trellis.h"

Adafruit_Trellis matrix0 = Adafruit_Trellis();
Adafruit_TrellisSet trellis =  Adafruit_TrellisSet(&matrix0);

#define NUMTRELLIS 1
#define numKeys (NUMTRELLIS * 16)
#define INTPIN A2

int LEDstatus[16] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
int blinkStatus = 1;
int blinkTime = 0;
int buttonPress[16] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
int oldStatus[16] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};

void setup() {
  Serial.begin(9600);
  pinMode(INTPIN, INPUT);
  pinMode(5, INPUT);
  pinMode(6, INPUT);
  pinMode(7, INPUT);
  pinMode(8, INPUT);
  digitalWrite(INTPIN, HIGH);

  trellis.begin(0x70);  // only one trellis is connected

  // light up all the LEDs in order
   for (uint8_t i = 0; i < numKeys; i++) {
     trellis.setLED(i);
     trellis.writeDisplay();
     delay(50);
   }

  // then turn them off
  for (uint8_t i = 0; i < numKeys; i++) {
    trellis.clrLED(i);
    trellis.writeDisplay();
    delay(50);
  }
  while (Serial.available() <= 0) {
    Serial.println("hello"); // send a starting message
    delay(300);              // wait 1/3 second
  }
}

void loop() {
  delay(80); // 30ms delay is required, don't remove me!


  /*************************************
  // SENDING DATA TO P5.JS
  *************************************/
  if (Serial.available() > 0) {

      // reading serial from p5.js
      int incoming = Serial.read();

      // print current status
      for (int i = 0; i < 16; i++) {
        Serial.print(LEDstatus[i]);
        Serial.print(",");
      }

      // step knob
      int pot1Value = 0;
      if (digitalRead(5) == HIGH) {
        pot1Value = 4;
      } else if (digitalRead(6) == HIGH) {
        pot1Value = 3;
      } else if (digitalRead(7) == HIGH) {
        pot1Value = 2;
      } else if (digitalRead(8) == HIGH) {
        pot1Value = 1;
      }
      Serial.print(pot1Value);
      Serial.print(",");

      // volume knob
      int pot2Value = analogRead(A1);
      int pot2ValueMapped = map(pot2Value, 0, 1020, 0, 100);
      Serial.print(pot2ValueMapped);
      Serial.print(",");

      // speed knob
      int pot3Value = analogRead(A0);
      int pot3ValueMapped = map(pot3Value, 0, 1020, 0, 100);
      Serial.print(pot3ValueMapped);
      Serial.print(",");

      // cut knob
      int pot4Value = analogRead(A3);
      int pot4ValueMapped = map(pot4Value, 0, 1020, 0, 100);
      Serial.print(pot4ValueMapped);
      Serial.print(",");

      // blink data
      Serial.print(blinkTime);

      Serial.println("");
  }

  /*************************************************
  // CHANGING BUTTON STATES BASED ON BUTTON PRESSES
  **************************************************/
  blinkTime = blinkTime + 1;
  if (blinkTime == 5) {
    blinkTime = 0;
  }

  trellis.readSwitches();
  for (uint8_t n = 0; n < numKeys; n++) {
    if (trellis.justPressed(n)) {
      LEDstatus[n] = 3;

      continue;
    }

      if (LEDstatus[n] == 3) {
        buttonPress[n]++;
        if (blinkTime >= 4) {
          if (trellis.isLED(n)) {
            trellis.clrLED(n);
            trellis.writeDisplay();
            } else {
              trellis.setLED(n);
              trellis.writeDisplay();
            }
        }
      }

    if (trellis.justReleased(n)) {
      if (buttonPress[n] > 8) {
        LEDstatus[n] = 1;
        oldStatus[n] = 1;
        buttonPress[n] = 0;
        trellis.setLED(n);
        trellis.writeDisplay();
      } else {
        buttonPress[n] = 0;
        if (oldStatus[n] == 1) {
          LEDstatus[n] = 0;
          oldStatus[n] = 0;
          trellis.clrLED(n);
          trellis.writeDisplay();
        } else {
          LEDstatus[n] = 1;
          oldStatus[n] = 1;
          trellis.setLED(n);
          trellis.writeDisplay();
        }
      }
    }
  }
}

This code includes a fast and efficient protocol to read the different states from the Trellis board using a single read command, and to communicate them to the web browser using ‘handshaking’.

At a first glance, this code looks simple, but it includes a fast and efficient protocol to read the different states (“ON”, “OFF”, and “Being pressed”, a state that was used to make changes to the video playback) from the Trellis board using a single read command (trellis.readSwitches()), and to communicate them to the web browser using ‘handshaking’.

More about the programming behind NOMNOM can be found on this blog post, and on the project’s GitHub repository.

Finding the Right Potentiometers

As much as the Trellis board was satisfying as our press buttons, the movement of the potentiometers needed an upgrade. A long research and multiple experiments with different types of potentiometers and knobs (mostly from Adafruit, DigiKey) were made. It appeared that the knobs and potentiometers offered by Mammoth Electronics were the smoothest to turn, most built using high-quality materials, and fit best with our design vision.

Fabrications

One of the major objective for the new version was to make the physical interface feel as stable as the software that supports it. The desire was to build the box from more solid materials, which do not feel breakable like wood or delicate like thin acrylic. Therefore, a solid metal enclosure was used to add sense of strength and stability to the overall interaction.

To avoid any ‘shaky’ feeling when interacting with the product, the design of the drilled holes on the enclosure had to be very accurate and tight to the size of the electronic components.

NOMNOM: Design sketch before the drilling process
NOMNOM: Design sketch before the drilling process

User Testing

After building the first fully functional prototype, a user testing phase some light on the strength and weaknesses of the product.

Luckily, the physical interaction worked well and was largely understood by the users. A few changes were done to the terminology – The term “Steps”, which described the number of times a video will be played within a single loop, was changed to “Repetitions”, and the term “Cut”, which described the ability to trim the video, was changed to the term “Trim”.

The rest of the changes, based on the users’ feedback, were done on the graphical user interface, which now includes a much simpler and straight forward indications for each and every video status.

Presenting the Project to New Audience

As part of the process, I presented the product in front of a new audience, outside of the ITP community. This experience allowed us to get feedback from an audience that is closer to our target audience, and helped us to be more prepared for the (intense) presentation at the ITP Winter Show.

\

The ITP Winter Show

NOMNOM: The Video Machine was presented at the ITP Winter Show 2016.

NOMNOM: The Video Machine @ ITP Winter Show 2016
NOMNOM: The Video Machine @ ITP Winter Show 2016

Final Project Proposal – The SoundSystem

Overview

Ever since popular music has been broadcasted by radio stations (somewhere between 1920’s and 1930’s), and consumed by listeners all over the world, artists were recording most of their music as 3-5 minutes songs.

This convention was born out of a technical limitation – The Phonograph, an early version of the record players we use today, could only play 12” vinyl records. Moreover, when an artist recorded a new album, or a new single, the only way to ship it to the local or national radio station was by sending it using the US Post Office services. The biggest box one could send at that time, for a reasonable price, was a box that could only hold only a 12” record. As you can probably guess, a 12” vinyl record can hold a tune no longer than 5 minutes.

A century ago, music production, consumption, and distribution processes have gone completely digital. Even though most of the music we listen to today is basically bits of data that can be manipulated, we still consume it in the 3-5 minutes linear format. Unlike other mediums, such as text or video, which in many cases are being consumed in a non-linear form, audio is still being consumed in short linear sprints.

I believe that in the age of data, we can do more than that.

Inspirations

The inspiration for the problem, and for the first steps of the solution, can to me from watching and interacting with The Infinite Jukebox project, build by Paul Lamere. Lamere posted a blog post, that tell about the process of making this project.

The Infinite Jukebox - user interface
The Infinite Jukebox – user interface

snapshot-111212-1004-am snapshot-111212-1005-am

 

Project proposal – The SoundSystem

I would want to build a system that will liberate music creators from composing their musical ideas into 3-5 minute songs.
Instead, artists will be able to focus and record their musical idea, and the system will generate an infinite, interactive, and dynamic piece of music, “conducted” by the artist.

Since I won’t be able to build the entire project for the ICM course final, I plan to build the first part of this project. The specifications of this part are highlighted in the text.

This how I would imagine the interaction (at least of the prototype)

Recording and analysing the recorded sound:

  • Artist will record a short snippet of audio.
  • The system will identify the tempo of the recorded snippet (beat detection).
  • The system will analyse the recorded snippet to get frequency data, timbre, etc. (and maybe in order to identify notes and / or chords?).
  • The system will suggest a rhythmic tempo to go along with the snippet.
  • The system will play the recorded snippet as in infinite loop, along with the rhythmic tempo.
  • The system will try to find new ‘loop opportunities’ within the snippet, in order to play the loop in a none linear way.
  • The artist will be able to record more musical snippets.
  • The artist will be able to choose which parts will be played constantly (background sounds), and which parts will be played periodically.
  • The system will suggest new and interesting combinations of the recording snippets, and play these combinations infinitely.

The listener interacts with the played tune:

  • Since the tune can be played infinitely, some controls will be given to listener. Each and every artist will be able to configure these controls differently. For example, one can decide that the controls will include 2 knobs, one of them changes the tune from ‘dark’ to ‘bright’, and the other changes the tune from ‘calm’ to ‘noisy’. The artist will decide what will happen when each one of these knobs is being turned.
  • For the ICM final, a generic user interface will be provided to the listener. The interface will include a visual representation of the played tune, and will allow the listener to change the rhythmic tempo.

Applying machine learning algorithms:

  • The system will try to generate new music, based on the recorded snippets, and earlier decisions by the same user. This new music will stretch the length of the recorded tune.

Modifying the system’s decisions:

  • The artist will be able to effect the system’s decisions about the looped tune, and about the new music it generates. For example, the user will be able to decide when a specific part enters, or which algorithmic rules won’t generate new music.

Applying sensors and automations

  • The artist will be able to set rules based on 3rd party data or sensors. For example, the tune can be played differently if it is rainy on the first day of the month, if it is currently Christmas, if it is exactly 5:55am, or if the light in the room was dimmed to certain level. These rules will apply to each tune separately.

Formatting

  • There should be a new music format that could hold the tune (or the snippets) and the data necessary for playing it correctly. In the same way, a new player should be introduced in order to read the data and to play the tune correctly.
  • This format should allow the artist to update the tune configuration or the musical snippets at any time, after the tune was distributed to the listeners.
  • For the ICM final (and probably for the end product as well), the tune will be played in the web browser.

 

Controlling video playback features

Overview

For the first time in my ITP history, I was able to combine home assignment for ICM with home assignment for Pcomp.

I created a video manipulation interface, that could be controlled by a physical controller.
The entire project was build with Mint for our Physical Computing class mid-term.

The Video Machine - Web Interface
The Video Machine – Web Interface

 

The Video Machine – Web Interface from Dror Ayalon on Vimeo.

Functionality

I used the JavaScript video API to do the following manipulations on the video playback:

  • Loop – Playing the video in an infinite loop.
  • Volume – Changing the volume of the sound.
  • Cut – Trimming the length of the video.
  • Speed – Changing the speed of the video playback.
The Video Machine
The Video Machine

Code

Here is the JavaScript code I used for this project –

The Video Machine

Overview

The Video Machine is a video controller, powered by an Arduino, that controls the playback of videos presented on a web browser. By pressing a button on the controller, the correlated video is being played on the screen and heard through the speakers.
Videos are being played in an infinite loop.
Only the videos that are being played, are being heard.

I was lucky enough to work on this project with the super talented Mint for our Physical Computing class mid-term.
Working with Mint not only was a great learning experience, but also a lot of fun! I hope I’ll be able to work with her again on our next project (more on that below).

The Video Machine from Dror Ayalon on Vimeo.

Many thanks for Joe Mango, our beloved resident, who assisted a lot with finding the right technologies for the project, and helped us on one critical moment, when suddenly nothing worked.

The Video Machine – Split Screen from Dror Ayalon on Vimeo.

The building process

The process of building The Video Machine went through the following stages:

  • Prototyping – Once we had a broad idea about what we want to make, we wanted to test how hard would it be to build such interaction, and if the interaction feels ‘right’ to us.
  • Understanding the complications The prototyping stage helped us understand what could be the possible complications of this product, and what might be the limitation. We analysed what could be the limitations of the serial communication between the Arduino board and our laptop computer, and what types of video manipulations could be easily achieved using JavaScript.
    Understanding what’s possible helped us shape our final design, and the different features
  • Designing the architecture – Before we started to build the final product, we talked about the technical design of the product under the hood. These decisions basically defined the way the end product would operate, and the way users would interact with it.
  • Picking up the technologies – To apply our technical design, we needed to find the right tools.
    For the video manipulations, we decided to use vanilla JavaScript, because its easy to use video API. The biggest discussion was around the implantation of the buttons, on which the user needs to press in order to play the videos. After some research, and brainstorming with Joe Mango, we decided to use the Adafruit Trellis. That was probably the most important decision we took, and one that made this project possible to make, given the short amount of time we had at that point (four days).
  • Building, and making changes – We started to assemble the project and the write the needed code. While doing that, we changed our technical design a few times, in order to overcome some limitations we learned about along the way. And then came then moment where everything worked smoothly.
The Video Machine - Final product
The Video Machine – Final product

Some code

The entire code can be viewed on our GitHub repository.

Reactions

The reactions to The Video Machine were amazing. The signals started to arrive on the prototyping stage, when people constantly wanted to check it out.

When we showed the final project to people on the ITP floor, it appeared that everyone wants to put a hand on our box.

The Video Machine
The Video Machine

People were experimenting, listening, looking, clicking, laughing, some of them even lined up to use our product.

The Video Machine
The Video Machine

Further work

I hope that Mint and I will be able to continue to work on this project for our final term.
I cannot wait to see the second version of The Video Machine.
I believe that the goals for the next version would be:

  • To add more functionality, that will allow better and easier video/sound manipulation.
  • To make playing very easy for people with no knowledge of music or playing live music. Beat sync could be a good start. The product should allow anyone to create nice tunes.
  • To find a new way to interact with the content, using the controller. This new interaction needs to be somethings that allows some kind of manipulation to the video or the sound, that is not possible (or less convenient) using the current and typical controller interface.
  • To improve the content so all videos will be very useful for as many types of music as possible.
  • To improve the web interface to make it adjustable for different screen sizes.
The Video Machine - Controller
The Video Machine – Controller

The ITP GitHub Hall of Fame

To practice my web APIs skills, I got some data from the GitHub API, and used it to create a list of the most starred (== popular) ITP related repositories.

See it here.

The ITP GitHub Hall of Fame
The ITP GitHub Hall of Fame

The expected (unpleasant) surprise

The process of making this page wasn’t smooth at all. In fact, this project cannot be further away from my original intentions. Here’s the truth:

  • I decided to let users enter a word (as an input), and to use this word to form a Haiku poem from three different poems.
  • I found poetry.db, which at first glance looked like the perfect source for my project.
  • I started to work on the logic that will form new Haiku poems out of the text I’ll get from poetry.db.
  • The logic was partially ready, and I was eager to test it on real data.
  • Oh no! My browser is shouting at me that I have a cross origin request error, and it is blocking my request to the API. I’m starting an endless search to find a solution, while going over countless Stackverflow threads, that appeared to be somewhat useless in my poor situation.
  • Aha! Apparently, JSONP should be JS’s workaround for this problem. My project could be saved!
  • Oh no! poetry.db’s server does not support JSONP. My options are:
    • To setup a web server and to hope for the best.
    • To try to find a new project that has better chances of success. — Chosen.
  • A new project! This time I’ll go for an API that was build by great developers, for the average developer, something that is a ‘one size fits all’, and I can assume that is stable and well maintain – GitHub’s API!

Some conclusions

GitHub’s API is well structured, fast, and reliable. I will surely recommend it to everyone.

Eventually, this process taught me (again), that when I deal with web APIs, the hustle is just part of the game. I’m sure that if I had some more time (a few extra days) I could have found a solution for the poetry.db API, and I could have completed my original project.

The cross-origin problem is a problem I face often, and I hope that we will be able to talk about it in class. I really feel that mastering the logic behind the cross-origin workaround is critical for web development.

(if you got to this point, feel free to scroll up and click that ‘See it here’ link again).

Domino Blueprint

For this week’s assignment, I created a sequential domino scene, as a homage for a game I used to play a lot when was younger. I still think that it is a brilliant game.

Take a look – click here

Or watch the video –

Domino Blueprint from Dror Ayalon on Vimeo.

This piece required many lines of code, but it was somewhat expected :\

btw – I’m officially in ♥ with object constructors.

First step towards the mid-term project

For the mid-term project, Mint and I decided to work on a game.
Without exposing too much about the the project – we are working on a game that will combine physical cannon, and a digital targets.

The cannon is sort of a shaker. User needs to shake to cannon to a certain point in order to shoot it. Needless to say, shaking and aiming at the same time will make the game challenging, and hopefully interesting and enjoyable.

We used the built-in accelerometer on Arduino 101, and we wrote Arduino code that will identify when user shakes the cannon, and when the cannon was shakened  long enough, and it is ready to fire.

It sounds a little abstract, but I’m sure that it will be very clear when the game will be ready.

This is what we have so far:

The making of ‘the pcomp shooter’: part 1 from Dror Ayalon on Vimeo.

 

This is the Arduino code we used (the examples on www.arduino.cc were very helpful):

#include "CurieIMU.h"
#include "pitches.h"

float oldx, oldy, oldz, newx, newy, newz = 0;   //scaled accelerometer values
float axp, ayp, azp;
int bullet = 0;

int bang[] = {
 NOTE_B3, NOTE_C4
};

int noteDurations[] = {
 2, 2
};

void setup() {
 Serial.begin(9600); // initialize Serial communication
 while (!Serial);    // wait for the serial port to open

 // initialize device
 Serial.println("Initializing IMU device...");
 CurieIMU.begin();

 // Set the accelerometer range to 2G    -2 to 2
 CurieIMU.setAccelerometerRange(2);
}

void loop() {

 // read accelerometer measurements from device, scaled to the configured range
 CurieIMU.readAccelerometerScaled(newx, newy, newz);

 // percentage
 axp = 100 - (oldx / newx) * 100;
 ayp = 100 - (oldy / newy) * 100;
 azp = 100 - (oldz / newz) * 100;

 // display tab-separated accelerometer x/y/z values
 Serial.print("axp: ");
 Serial.print(axp);
 Serial.print("%");
 Serial.print("\t");

 Serial.print("ayp: ");
 Serial.print(ayp);
 Serial.print("%");
 Serial.print("\t");

 Serial.print("azp: ");
 Serial.print(azp);
 Serial.print("%");
 Serial.print("\t");

 Serial.print("bullet: ");
 Serial.print(bullet);
 Serial.print("%");
 Serial.print("\t");

 if (( axp > -100) && ( axp < 100 )) { bullet = bullet - 0.5; } else { bullet = bullet + 4; Serial.print("SHAKING!!!"); Serial.print("\t"); } if (bullet >= 100) {
   Serial.print("BANG!!!!!");
   Serial.print("\t");

   int noteDuration = 1000 / noteDurations[0];
   tone(8, bang[0], noteDuration);
   int pauseBetweenNotes = noteDuration * 0.5;
   Serial.print(bang[0]);
   delay(pauseBetweenNotes);
   noTone(8);

   noteDuration = 1000 / noteDurations[1];
   tone(8, bang[1], noteDuration);
   pauseBetweenNotes = noteDuration * 0.5;
   Serial.print(bang[1]);
   delay(pauseBetweenNotes);
   noTone(8);

   bullet = 0;
 }

 Serial.println();

 oldx = newx;
 oldy = newy;
 oldz = newz;

 delay(5);
}

What’s next?

  • To build the graphical part of the game in p5.js.
  • To solve the sound problem: We couldn’t find a way to make the sound louder from the speaker connected to the Arduino.
  • To do some fabrication work and to make sure that the cannon has the ‘right’ feeling in the users’ hands.
  • To play the game over and over, make some changes, and make sure that it’s fun.

The ‘Loopster’ failure

Planning

This week I tried to to build a project that originally simple to me. I wanted to build Loopster – a web app that will allow users to record short samples of music, save them, and use loop them to create a musical piece. I wanted Loopster to allow users to share their loops publicly, and to use loops by other users. Sort of the GitHub for music making.

So the goals were:

  • To build a web application that will allow users to record live stream (basically an analog stream).The recording could be done using the laptop microphone or the laptop input jack.
  • To allow users to trim their loops (or to try to detect the beats and to do it automatically).
  • To allow users to put some loops, which were created by them or by others, on a sequencer, and to enjoy them all together.

Bottom line:  Almost none of than happened.

Failing

I decided to use Web Audio API to implement the main functionality of the app. The Web Audio APIs are a little complicated, and it took my about 1-2 hours to be able to record sound through the laptop microphone.

Anything beyond that, just failed to work.
I spent another 8-10 hours trying to gain some progress on this project, but nothing seems to work anywhere near my expectations.

Eventually, the only thing that this app can do, is to open the laptop microphone, and play what it goes through it (which cause an immediate feedback..).

#fail
#fail

This is the (complicated) code for this (simple) interaction:
For some reason, WordPress took my html code too literaly

var AudioContext = window.AudioContext || window.webkitAudioContext; //cross browser compatibility
var myAudio = new AudioContext();
var record;

// get user's audio as an inputPoint
navigator.getUserMedia = (navigator.getUserMedia ||
                          navigator.webkitGetUserMedia ||
                          navigator.mozGetUserMedia ||
                          navigator.msGetUserMedia);

// audio nodes
var gain = myAudio.createGain();
gain.gain.value = 0;

// connecting the nodes
if (navigator.getUserMedia) {
   console.log('getUserMedia supported.');
   navigator.getUserMedia (
      {
         audio: true,
      },

      // Success callback
      function(stream){
        showButtons();
        play(stream);
        recordInput(stream);
      },

      // Error callback
      function(err) {
         console.log('The following gUM error occured: ' + err);
      }
   );
} else {
   console.log('getUserMedia not supported on your browser!');
}

function showButtons(){
  $("body").append('');
  $("body").append('');
  buttonEvents();
}

function play(stream) {
  var source = myAudio.createMediaStreamSource(stream);
  source.connect(gain);
  gain.connect(myAudio.destination);
}

// start recording
function recordInput(stream){
  record = new MediaRecorder(stream);
  record.start();
  console.log(record.state);
}

function buttonEvents() {
  console.log('callback worked');
  $("#play").click(function(){
    gain.gain.value = 0.5;
    console.log('click on play');
  });

  $("#stop").click(function(){
    gain.gain.value = 0;
    console.log('click on stop');
  });
}

// stop recording
$(window).keypress(function(key) {
  if ((key.keyCode === 32) && (record.state == 'recording')){
    console.log('Space pressed -- stopping recording process');
    record.pause();
    // record.exportWAV( doneEncoding );
    record.stop();
    console.log(record.state);
  }
});

You can try to use it using this link, but Chrome (and I’m sure that this is relevant to any other browser) doesn’t allow any web app to open the microphone if the app’s files are not being served using a secured connection.

#DoubleFail
#DoubleFail

If you still want to see it happening, you can go to the project page on GitHub, download the files, and open index.html on your beloved browser.

Learning (at least trying to..)

Ok, some lessons from this unpleasant experience:

  • I chose the wrong tools. Web Audio API is definitely a wonderful API, but learning it was not the purpose of this project. P5.js, for example, could do most of the things I wanted to do. Also, using P5.js could get be pretty fast to the stage where I can experiment with the app and test the overall experience, the design of the interaction, and so on, which to me, are way more interesting.
  • I must find ideas that I’m more passionate about. The idea behind Loopster is nice, but not nice enough. A ‘nice enough’ idea would have got me to get rid of all the complications and to build it as fast as possible.
    On the Loopster case, the only passion I found was in learning new API or to experiment with new tools. But to me, the tools are not the story.

I hope I’ll do better next time :\

Device motion and orientation data

In continue to my previous post, I decided to take the first step in building my next project (more about it soon).

I  followed the Web API instructions about how to detect device’s movement (using the device accelerometer sensor) and orientation (using the device orientation sensor), and experimented with getting and presenting the this data.

You can see the end result here.

I wrote the following code for the task (This code gets the device movement and orientation data, and appends it to the DOM’s body) –

And this is the result (the changes in the blue numbers occur because I rotated my phone, while the most of the changes in the red numbers cause because the Web API library is still in beta, and doesn’t deal well enough with the device accelerometer) –

Apparently, my Macbook Pro also has an accelerometer, to protect its hard-drive while moving the laptop –

Try it yourself.

Here’s the entire code on GitHub.