Final Project Proposal – The SoundSystem

Overview

Ever since popular music has been broadcasted by radio stations (somewhere between 1920’s and 1930’s), and consumed by listeners all over the world, artists were recording most of their music as 3-5 minutes songs.

This convention was born out of a technical limitation – The Phonograph, an early version of the record players we use today, could only play 12” vinyl records. Moreover, when an artist recorded a new album, or a new single, the only way to ship it to the local or national radio station was by sending it using the US Post Office services. The biggest box one could send at that time, for a reasonable price, was a box that could only hold only a 12” record. As you can probably guess, a 12” vinyl record can hold a tune no longer than 5 minutes.

A century ago, music production, consumption, and distribution processes have gone completely digital. Even though most of the music we listen to today is basically bits of data that can be manipulated, we still consume it in the 3-5 minutes linear format. Unlike other mediums, such as text or video, which in many cases are being consumed in a non-linear form, audio is still being consumed in short linear sprints.

I believe that in the age of data, we can do more than that.

Inspirations

The inspiration for the problem, and for the first steps of the solution, can to me from watching and interacting with The Infinite Jukebox project, build by Paul Lamere. Lamere posted a blog post, that tell about the process of making this project.

The Infinite Jukebox - user interface
The Infinite Jukebox – user interface

snapshot-111212-1004-am snapshot-111212-1005-am

 

Project proposal – The SoundSystem

I would want to build a system that will liberate music creators from composing their musical ideas into 3-5 minute songs.
Instead, artists will be able to focus and record their musical idea, and the system will generate an infinite, interactive, and dynamic piece of music, “conducted” by the artist.

Since I won’t be able to build the entire project for the ICM course final, I plan to build the first part of this project. The specifications of this part are highlighted in the text.

This how I would imagine the interaction (at least of the prototype)

Recording and analysing the recorded sound:

  • Artist will record a short snippet of audio.
  • The system will identify the tempo of the recorded snippet (beat detection).
  • The system will analyse the recorded snippet to get frequency data, timbre, etc. (and maybe in order to identify notes and / or chords?).
  • The system will suggest a rhythmic tempo to go along with the snippet.
  • The system will play the recorded snippet as in infinite loop, along with the rhythmic tempo.
  • The system will try to find new ‘loop opportunities’ within the snippet, in order to play the loop in a none linear way.
  • The artist will be able to record more musical snippets.
  • The artist will be able to choose which parts will be played constantly (background sounds), and which parts will be played periodically.
  • The system will suggest new and interesting combinations of the recording snippets, and play these combinations infinitely.

The listener interacts with the played tune:

  • Since the tune can be played infinitely, some controls will be given to listener. Each and every artist will be able to configure these controls differently. For example, one can decide that the controls will include 2 knobs, one of them changes the tune from ‘dark’ to ‘bright’, and the other changes the tune from ‘calm’ to ‘noisy’. The artist will decide what will happen when each one of these knobs is being turned.
  • For the ICM final, a generic user interface will be provided to the listener. The interface will include a visual representation of the played tune, and will allow the listener to change the rhythmic tempo.

Applying machine learning algorithms:

  • The system will try to generate new music, based on the recorded snippets, and earlier decisions by the same user. This new music will stretch the length of the recorded tune.

Modifying the system’s decisions:

  • The artist will be able to effect the system’s decisions about the looped tune, and about the new music it generates. For example, the user will be able to decide when a specific part enters, or which algorithmic rules won’t generate new music.

Applying sensors and automations

  • The artist will be able to set rules based on 3rd party data or sensors. For example, the tune can be played differently if it is rainy on the first day of the month, if it is currently Christmas, if it is exactly 5:55am, or if the light in the room was dimmed to certain level. These rules will apply to each tune separately.

Formatting

  • There should be a new music format that could hold the tune (or the snippets) and the data necessary for playing it correctly. In the same way, a new player should be introduced in order to read the data and to play the tune correctly.
  • This format should allow the artist to update the tune configuration or the musical snippets at any time, after the tune was distributed to the listeners.
  • For the ICM final (and probably for the end product as well), the tune will be played in the web browser.

 

Usability case study: The NYC subway turnstiles

This week I went out for an observational study about the interaction between people and the NYC subway turnstiles.

NYU subway turnstile
NYU subway turnstile

The study was made on two busy subway stations: Bedford Avenue station in Brooklyn, and Union Square station in Manhattan. The observations, and the conclusions, were similar on both stations. Therefore, I will specify my findings in an unanimous form.

Findings

  1. A few meters before they get to the turnstiles, people reach to their pockets, wallets, or bags to get a paper card, shaped as a credit card. One face of the card is painted in yellow, and the other one is painted white.
    It is very unusual to see people get to the machine empty handed.

    Getting the cards in advance
    Getting the cards in advance
  2. People swipe the card on the right side of the machine.

    Swiping the metro cards
    Swiping the metro cards
  3. Usually, a ‘click’ sound is heard after the swipe, and the ‘swiper’ moves across the gate to enter the subway platform.
  4. Occasionally, a ‘beep’ sound was heard after people swiped their cards. After the ‘beep’, people were seen swiping their cards again. This scenario repeats itself until the ‘click’ sound was heard, after which, the swiper entered the platform.
    A closer look shows an ‘error’ message on the small display, placed just above the card swiping deck.

    Errors
    Errors
  5. On rare occasions, the swiper was swiping the card, a ‘beep’ sound was heard, and the swiper seemed angry.
    This situation made other people, who were lined-up behind the swiper, angry as well. A few of the waiting people were trying to move to another turnstile line.
    The swiper turned back, and went to the ‘refill machine’. After doing so, the swiper repeated the steps from step 2.
  6. The turnstile was used, by different people, both for entering the platform, and exiting from it.
    On many occasions, people were trying to exit the platform from a turnstile that was occupied by other people, who where trying to enter the platform.
    Ofter 1-2 second of confusion, the ‘exitter’ (it was never the one who enters) was turning to another turnstile to exit the platform. Some of the exitters took the liberty of exitting the platform from the emergency exit, which was open at all times.

    Commuters go in and out the platform through the turnstile, or through the emergency exit door
    Commuters go in and out the platform through the turnstile, or through the emergency exit door

Major insights

  1. The NYC subway turnstile is a very understandable device. Whether it is because people tend to use it often, of because its design is very simple, people rarely get confused while using it.
  2. At first sight, it felt like the the fact that people use the same turnstile to enter and to exit the platform indicates a design flaw, that might become a major one on rush hours. A longer observation on the turnstiles reveal that people tend to work-around this limitation pretty easily, while spending about a second or two doing so.
  3. Since there is no indication on the card for the amount of funds attached to it, people tend to block the turnstile on cases of ‘insufficient funds’, and to repeat previous steps in the interaction.

Conclusions

Be careful of over-engineering – Intuitively, I was surprised that the engineers, who designed the turnstile, were not concerned by the fact that people would have to cross the turnstile from both of its sides at the same time (to enter and to exit the platform). I assumed that a better design was one that allowed entering and exiting the platform from different locations.
But engineering and design are all about decision making, and compromises. The design I thought about is one that requires more space, and eventually less effective. Apparently, people do not care that much about this ‘limitation’, and address it very easily.
There might be an optimal design that could solve this situation, but sometimes, looking for complex solutions for simple situation is just over-engineering.

An indication for the amount of funds on the cards could be helpful – Since people tend to hold their subway (metro) cards a few steps before they reach the turnstile, having an indication for the amount of funds on the card could save time and frustration when addressing a situation of ‘insufficient funds’.
Also, it could have been useful if people were able to fill their cards with ‘rides’, instead of plain money. For example, one would be able to fill a card with 1-10 rides, instead of $20-$60. This situation could make it easier for people to track and remember the number of rides they still have in their credit (currently, the cost per ride is $2.75, which make it relatively hard to calculate the number of rides in $20, and supposably leads to ignorance).
Here are few suggestions for such indications on the NYC metro card:
Suggestion for NYC metro card design

Music Canvas

The music canvas project, developed together with Sharif Hadidi, was my first step in music analysis in JavaScript.

The goal was to create a paint tool, that its brush changes according to the properties of a track that is being played in the background.
In order to ‘translate’ audio properties to visual properties, we set up the following guidelines:

  1. The size of the brush will be set according to the beat of the song.
  2. The color of the brush will be set by the tone, and the amplitude of the song.

While searching for a way to run beat detection on the played track, i stumbled upon this GitHub gist by b2renger. I ended up now using this solution, but it lead me to believe that P5.js sound library might do what I need.

music canvas: J. Pollock Mode
music canvas: J. Pollock Mode

P5.js sound library

The P5.js sound library is relatively new and basic, but it covers more than I actually needed.

These are the main functions that I used to get the data I needed:


function setup(){
.
. // some unimportant stuff here
..
amp = new p5.Amplitude(); // creating a new Aplitude object
fft = new p5.FFT(); // creating a new FFT object
.
.. // more unimportant stuff here
}

function draw(){
.
. // some unimportant stuff here
..
amplitude = amp.getLevel(); // getting amplitude level
fft.analyze();
beat = fft.getEnergy('lowMid'); // getting amplitude level at lowMid frequencies
color = fft.getCentroid(); // getting the brightness/tone level
.
.. // more unimportant stuff here

}

Data normalization

In order to use the data we audio analysis data for P5’s fill(); or ellipse(); we had to normalize the numbers. In order to do that, we the a linear scale function from d3.js.


// setting up the scales
var colorScale = d3.scaleLinear()
.domain([0, 8000]) // expected input
.range([0,255]); // normalized output

var ampScale = d3.scaleLinear()
.domain([0, 1])
.range([0,255]);

var beatScale = d3.scaleLinear()
.domain([0, 300])
.range([0,30]);

// using the scales
normColor = colorScale(color);
normAmp = ampScale(amplitude);
normBeat = beatScale(beat);

I must say that these scale functions are among my all time favorites. They are very usable for many different scenarios. On this project, for example, I used these functions to generate a scale of colors.

Building debugging tools

Since I am relatively new to the audio subject, I decided to spend some time building a debug function. This function will print all the data I need to the console in runtime.

This was one of the best decisions I made while working on this project. It allowed me to learn the ranges of my data, and to understand what are these numbers stand for.

You can try to use the debug function yourself. Just open the console and type: printData(‘help’); .
You should see something like that –

open console > printData('help');
open console > printData(‘help’);

 

Live

I quickly setup a node.js server, so currently the music canvas is live 🙂

Check it out:

Automatic drawing mode (aka ‘J. Pollock mode’) – http://music-canvas.herokuapp.com

Manual drawing mode (aka ‘the paint’) – http://music-canvas.herokuapp.com/draw

music canvas: Paint Mode
music canvas: Paint Mode

Composition as a window to an artistic process

As a long time Wes Anderson fan, I always pay close attention to the graphic design in his films. To me, Anderson’s wonderfully crafted frames are not only a very good reference for aesthetics, symmetry, and story telling, but also, work that seem both enjoyable to watch and to make.


Wes Anderson // Centered from kogonada on Vimeo.

To emphasize these ideas, I will analyze a movie poster designed for one of Anderson’s latest films – Moonrise Kingdom (2012).

Moonrise Kingdom: Movie Poster // Wes Anderson
Moonrise Kingdom: Movie Poster // Design: P+A, Director: Wes Anderson

Composition

A movie poster is graphic design piece that its major objective is to promote a movie. And therefore, by analyzing a movie poster we can get a unique perspective about the way the movie creators (and promoters) looked at the cinematographic outcome.

The Moonrise Kingdom poster composition really tells a story. To me, it’s not only the story of the movie, but also the story of the making the movie and promoting it.

On first sight, it seems like the Moonrise Kingdom poster cherish the regular Wes Anderson symmetry, that can be seen in any one of his movies. In fact, this poster composition shows the growth process Anderson went through as a director, and a story teller, and the confidence he (and his talented crew of designers) felt about this movie before its release.

The Moonrise Kingdom poster: A grid that puts the story in the front row
The Moonrise Kingdom poster: A grid that puts the story in the front row

When we look at this poster, first, our eyes are being attracted to the title of the movie, floating in the middle of the frame in big letters. This alone tells us that, for the makers of this film, and the designers of this poster, the movie or the story, is the most important thing. As if they believe that the story is so good, that it will sell itself.

As you can see, in then center of the piece the designer places the two main characters of the film – Sam & Suzy – played by Jared Gilman & Kara Hayward, two relatively unknown young actors. Anderson (or the designer of this poster) doesn’t promote the movie using his big stars. Instead, he puts all characters at the same level, in terms of graphical hierarchy, and the main character of the story are right there in the middle — this story is not about big movie stars, it’s about two young people and their fantastic adventure.

In fact, we can say that Anderson was so confident about this movie, that on the promotional poster, that was published before the movie premier, the big star do not appear visually at all. Again, it’s all about the story this time.

The Moonrise Kingdom 'coming soon' poster: The big starts are not even there. It's all about the story.
The Moonrise Kingdom ‘coming soon’ poster: The big starts are not even there. It’s all about the story.

Between personal development and poster composition

We can clearly use Anderson’s movie poster designs as a reference for his growth has as a director. Let’s have a look a few movie posters from Anderson’s earlier works – From the star-heavy The Royal Tenenbaums (2001), to the latest The Grand Budapest Hotel (2014).

On The Royal Tenebaums (2001), we can see the Anderson uses his league of stars as the main selling point for the movie. The actors appear in the middle of the frame, and their name is highlighted at the top of the poster. The movie title appears at the lower part of the frame, makes it easily ignored.

The Royal Tenenbaums movie poster: The names of the actors is the focus. The movie title comes second.
The Royal Tenenbaums movie poster: The names of the actors is the focus. The movie title comes second.

On his next movie, The Life Aquatic with Steve Zissou (2004), we can see that Anderson (or his promoters) is still somewhat  not confident about the story itself as a good selling point for the movie. The poster composition is more focused on star of the show, Bill Murray, than on anything else. The character is placed in the middle of the composition, and our eyes our being attracted to the title of the poster – “Bill Murray”. The title of the movie is blended into the yellow submarine.

The Life Aquatic with Steve Zissou movie poster: Focuses more on Bill Murray than anything else.
The Life Aquatic with Steve Zissou movie poster: Focuses more on Bill Murray than anything else.

Anderson gains a lot of confidence towards his next full feature movie, The Darjeeling Limited (2007). Here, the movie poster composition puts the three fabulous actors as equal, in terms of hierarchy, to the movie title itself. Both the actors and the title of the movie are shown in the middle of the frame, and do not compete over the viewers’ attention. It feels like one cannot stand without the other, which in some weird way, tells the story of this movie, which would not be the same with different characters or different actors.

The Darjeeling Limited movie poster: The story and the title are as important as the famous actors.
The Darjeeling Limited movie poster: The story and the title are as important as the famous actors.

Let’s jump to Anderson’s latest feature – the Golden Globe and Academy Awards winner – The Grand Budapest Hotel (2014). As usual on Anderson movies, The Grand Budapest Hotel features a few of the most desired actors in the business. But even though the movie is rich with talented personnel, Anderson shows full confident in his new movie, and world he created. On the movie poster, Anderson puts only the movie title, and place on which the story happens. That’s it. Anderson is fully focused on storytelling and the fantastic world he created. Not only that we don’t see the fancy cast, their names do not even appear on the poster. As viewers, we are totally focused on the amazingly designed hotel, and on the stories that happen inside it.

The Grand Budapest Hotel movie poster: Anderson focuses on the story and the world he created.
The Grand Budapest Hotel movie poster: Anderson focuses on the story and the world he created.

Color palette

Let’s get back to the Moonrise Kingdom. This movie is all about teen spirit and adventures. To emphesize that, the movie poster is using  a color pallete full of greens and browns, colors that we usually see in the nature.

Moonrise Kingdom movie poster: Color palette
Moonrise Kingdom movie poster: Color palette

Also, the poster tones of the blue and yellow, give the entire image its aesthetic youth spirit. The yellow color of the titles is similar to the scounts’ yellow neckerchiefs. This similarity makes the two connected, as if the designer is trying to say that “this movie is about these scouts”.

Typography

The handwriting font of the titles on the Moonrise Kingdom poster is usually associated with note taking or a writing of a journal. The usage of this font gives the impression that the movie will tell us about an adventure, as it was written by one of the characters.

Physical Interaction: Interpretation and Thoughts

What is physical interaction?

Interaction: a cyclic process

In order to answer this question, I believe that I should first describe what is an interaction? According to Chris Crawford (“The Art of Interactive Design: A Euphonious and Illuminating Guide to Building Successful Software”, 2003), an interaction is:

“a cyclic process in which two actors alternately listen, think, and speak.”

Crawford claims that interactivity must be cyclic, and that it must includes a process of listening, thinking, and speaking. But what if an interaction includes only a single cycle of a question and an answer? What if the one actor doesn’t listen, and instead says something that is unrelated? What is the unrelated reply makes the first actor go for another cycle (to process what he/she heard, and to say another thing in reply)? Would that be considered an interaction?

Number of cycles: an interactivity strength scale

Crawford mentions that, to his perspective, there are degrees of interactivity. Therefore, interactions could be strong or weak, based on the level of listeningthinking, and speaking by those who participate in the interaction.

To add to Crawford’s degrees of interactivity model, I would say that since an interaction is cyclic process, the number of cycles could be used as a measurement for the strength of an interaction, and that minimum strength of an interaction is a single cycle.

I agree with Crawford’s determination that in cases where the occurrence does not lead to (at least) a single full cycle of interactivity, this occurrence would be considered as a case of cause and reaction.

For example, when two people engage in a long conversation about the theory of Repeated Games With Incomplete Information, they experience an interaction that requires many cycles of questions and answers, that lead to many other repeated cycles. This conversation would be considered as a very strong interaction.
In contrast, when two people stand in an elevator and greet each other with ‘good morning’, I would say that, even though their interaction went through a single cycle (or a single exchange), this single cycle satisfies the minimum requirement for an interaction. Obviously, the strength of this interaction is very weak (some would say that this is the weakest interaction possible, but I always consider an interaction between people as one that includes an exchange of information the surpass the the words that are being said vocally).

In case where one of the actors greets the second actor with ‘good morning’, and the second actor embraces the greeting, but do not reply, I would consider that a case of cause and reaction.

Interaction: an exchange of information

I would also add that an interaction must be a cyclic process in which actors listen, think, speak, and exchange valuable information. Information is valuable for an interaction in case where it was related to the previous cycle, and had an effect on the next on.

For example, let’s have a look at the following conversation:

John: “What is the time now?”
Dana: “The leaves are falling because winter is coming.”

If John considers Dana’s answer as unrelated to his questions, he might stop the conversation, and won’t say anything else. John might as well assume that Dana did not reply to his question, and did not even talk directly to him. In this case, I would argue that Dana did not interact with John, and therefore, there is not even a single cycle of interaction, meaning that there was no interactivity at all.

This situation is very similar to the highly frustrated experience of operating a computer, and getting a repeated error message that doesn’t seem to have any connection to the operation, and that doesn’t lead the user towards his next step.

Error code. [taken from https://xkcd.com/1024/]
Error code.
[taken from https://xkcd.com/1024/]

What if John asks his question again, assuming that Dana miss heard him? What if after Dana’s reply John forgets about the time and starts to gaze at the falling leaves? Similarly to the previous case, In this one I would say that both actors are reacting to one another, but do not perform any real interaction between them.

But what if, somehow, John understands what is the time (or at least he thinks he understands) from Dana’s answer?  In this case, even though the content of the conversation doesn’t seem to be of any value, I would consider it an interaction, since Dana’s reply was valuable to complete a single cycle. In other words, regardless to Dana’s reply content, if this reply pushed John to try to re-interact with Dana, Dana’s reply effected the next step in the interaction. And therefore, it was valuable.

To summarize it all, I would argue that an interaction is:

at least a single exchange of valuable information that leads to a cyclic process in which two actors alternately listen, think, and speak.

Physical interaction

In my opinion, a physical interaction is one that includes a physical object, and that the physical object plays a major role in it. I would consider a physical object anything that has a tangible attribute. Therefore, humans, for example, are physical objects. This conclusion led me to a disagreement with Bret Victor’s article about the future of interaction design. Even though the future, as presented on Microsoft’s concept video, does not fulfil the potential of physical interaction, humans are playing a major role in it.

Physical interaction with digital interfaces. [taken from http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesign]
Physical interaction with digital interfaces.
[taken from http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesign]

For example, an exchange of information between two servers would not be considered to be a physical interaction. But in my opinion, any interaction between human and a machine is a physical interaction.

We can say that a physical interaction is

a cyclic process, on which a physical object plays a major role, and includes at least a single exchange of valuable information.

What makes for good physical interaction?

By now, it is clear to me that a good physical interaction must include a good use of physical object. By saying ‘good use’ I mean that it should be clear that the interaction could not be as strong, or even could not occur at all, without the physical object.

Having said that, and without quoting Don Norman’s book The Design of Everyday Things, I would say that a good physical interaction would also be one that fulfils its potential in the following ways:

It is sensitive

Remember the times when you got a new shirt, or a new haircut, and no one told you anything about it, as if it wasn’t happening? Have you ever experienced a situation where someone important to you forgot about an important day for you?

Computers have the potential to impact our lives with just a minimal effort. Think about a scenario where your desktop lamp would change its tone of light according to your mood, if it is saying that it feels you. Wouldn’t that be nice? What if your home stereo would lower its volume, or just lower its treble sound when you have a headache? What if your refrigerators would pour a glass of water when its sensors hear you coughing?

It is surprisingly smart, but not arrogant

In continue to the previous topic, a good (physical) interaction is one that has a sense of surprise in it. Repeating the same interaction over and over would probably kill this effect, and therefore, a good interaction should be one that evolves, and gets smarter in time.

The best interactions would be those that does not leave the user with a feeling of “that was great, but I have no idea what happened.”. I cannot stand looking at my father as he admires new features on his phone, while secretly admits that he is no way near the understanding of this features, and how they actually work. This interaction leaves my father (and many others) with the feeling that he cannot play a major part in the new world of computer based interactions. In other words, although my father appreciate these interactions, they tend to leave him with some frustration, which leads me to the next topic –

It is satisfying

Rolling back to the earlier parts of this post, I can say that a strong interaction, one that includes many cycles, is surely satisfying. But the feeling of satisfaction could be achieved even on weak interactions, and should not be overlooked. A satisfying interaction would lead a user to be more engaged in it, and would push the user to explore new ones.