Final Project Proposal – The SoundSystem

Overview

Ever since popular music has been broadcasted by radio stations (somewhere between 1920’s and 1930’s), and consumed by listeners all over the world, artists were recording most of their music as 3-5 minutes songs.

This convention was born out of a technical limitation – The Phonograph, an early version of the record players we use today, could only play 12” vinyl records. Moreover, when an artist recorded a new album, or a new single, the only way to ship it to the local or national radio station was by sending it using the US Post Office services. The biggest box one could send at that time, for a reasonable price, was a box that could only hold only a 12” record. As you can probably guess, a 12” vinyl record can hold a tune no longer than 5 minutes.

A century ago, music production, consumption, and distribution processes have gone completely digital. Even though most of the music we listen to today is basically bits of data that can be manipulated, we still consume it in the 3-5 minutes linear format. Unlike other mediums, such as text or video, which in many cases are being consumed in a non-linear form, audio is still being consumed in short linear sprints.

I believe that in the age of data, we can do more than that.

Inspirations

The inspiration for the problem, and for the first steps of the solution, can to me from watching and interacting with The Infinite Jukebox project, build by Paul Lamere. Lamere posted a blog post, that tell about the process of making this project.

The Infinite Jukebox - user interface
The Infinite Jukebox – user interface

snapshot-111212-1004-am snapshot-111212-1005-am

 

Project proposal – The SoundSystem

I would want to build a system that will liberate music creators from composing their musical ideas into 3-5 minute songs.
Instead, artists will be able to focus and record their musical idea, and the system will generate an infinite, interactive, and dynamic piece of music, “conducted” by the artist.

Since I won’t be able to build the entire project for the ICM course final, I plan to build the first part of this project. The specifications of this part are highlighted in the text.

This how I would imagine the interaction (at least of the prototype)

Recording and analysing the recorded sound:

  • Artist will record a short snippet of audio.
  • The system will identify the tempo of the recorded snippet (beat detection).
  • The system will analyse the recorded snippet to get frequency data, timbre, etc. (and maybe in order to identify notes and / or chords?).
  • The system will suggest a rhythmic tempo to go along with the snippet.
  • The system will play the recorded snippet as in infinite loop, along with the rhythmic tempo.
  • The system will try to find new ‘loop opportunities’ within the snippet, in order to play the loop in a none linear way.
  • The artist will be able to record more musical snippets.
  • The artist will be able to choose which parts will be played constantly (background sounds), and which parts will be played periodically.
  • The system will suggest new and interesting combinations of the recording snippets, and play these combinations infinitely.

The listener interacts with the played tune:

  • Since the tune can be played infinitely, some controls will be given to listener. Each and every artist will be able to configure these controls differently. For example, one can decide that the controls will include 2 knobs, one of them changes the tune from ‘dark’ to ‘bright’, and the other changes the tune from ‘calm’ to ‘noisy’. The artist will decide what will happen when each one of these knobs is being turned.
  • For the ICM final, a generic user interface will be provided to the listener. The interface will include a visual representation of the played tune, and will allow the listener to change the rhythmic tempo.

Applying machine learning algorithms:

  • The system will try to generate new music, based on the recorded snippets, and earlier decisions by the same user. This new music will stretch the length of the recorded tune.

Modifying the system’s decisions:

  • The artist will be able to effect the system’s decisions about the looped tune, and about the new music it generates. For example, the user will be able to decide when a specific part enters, or which algorithmic rules won’t generate new music.

Applying sensors and automations

  • The artist will be able to set rules based on 3rd party data or sensors. For example, the tune can be played differently if it is rainy on the first day of the month, if it is currently Christmas, if it is exactly 5:55am, or if the light in the room was dimmed to certain level. These rules will apply to each tune separately.

Formatting

  • There should be a new music format that could hold the tune (or the snippets) and the data necessary for playing it correctly. In the same way, a new player should be introduced in order to read the data and to play the tune correctly.
  • This format should allow the artist to update the tune configuration or the musical snippets at any time, after the tune was distributed to the listeners.
  • For the ICM final (and probably for the end product as well), the tune will be played in the web browser.

 

The ITP GitHub Hall of Fame

To practice my web APIs skills, I got some data from the GitHub API, and used it to create a list of the most starred (== popular) ITP related repositories.

See it here.

The ITP GitHub Hall of Fame
The ITP GitHub Hall of Fame

The expected (unpleasant) surprise

The process of making this page wasn’t smooth at all. In fact, this project cannot be further away from my original intentions. Here’s the truth:

  • I decided to let users enter a word (as an input), and to use this word to form a Haiku poem from three different poems.
  • I found poetry.db, which at first glance looked like the perfect source for my project.
  • I started to work on the logic that will form new Haiku poems out of the text I’ll get from poetry.db.
  • The logic was partially ready, and I was eager to test it on real data.
  • Oh no! My browser is shouting at me that I have a cross origin request error, and it is blocking my request to the API. I’m starting an endless search to find a solution, while going over countless Stackverflow threads, that appeared to be somewhat useless in my poor situation.
  • Aha! Apparently, JSONP should be JS’s workaround for this problem. My project could be saved!
  • Oh no! poetry.db’s server does not support JSONP. My options are:
    • To setup a web server and to hope for the best.
    • To try to find a new project that has better chances of success. — Chosen.
  • A new project! This time I’ll go for an API that was build by great developers, for the average developer, something that is a ‘one size fits all’, and I can assume that is stable and well maintain – GitHub’s API!

Some conclusions

GitHub’s API is well structured, fast, and reliable. I will surely recommend it to everyone.

Eventually, this process taught me (again), that when I deal with web APIs, the hustle is just part of the game. I’m sure that if I had some more time (a few extra days) I could have found a solution for the poetry.db API, and I could have completed my original project.

The cross-origin problem is a problem I face often, and I hope that we will be able to talk about it in class. I really feel that mastering the logic behind the cross-origin workaround is critical for web development.

(if you got to this point, feel free to scroll up and click that ‘See it here’ link again).

Physical Interaction: Interpretation and Thoughts

What is physical interaction?

Interaction: a cyclic process

In order to answer this question, I believe that I should first describe what is an interaction? According to Chris Crawford (“The Art of Interactive Design: A Euphonious and Illuminating Guide to Building Successful Software”, 2003), an interaction is:

“a cyclic process in which two actors alternately listen, think, and speak.”

Crawford claims that interactivity must be cyclic, and that it must includes a process of listening, thinking, and speaking. But what if an interaction includes only a single cycle of a question and an answer? What if the one actor doesn’t listen, and instead says something that is unrelated? What is the unrelated reply makes the first actor go for another cycle (to process what he/she heard, and to say another thing in reply)? Would that be considered an interaction?

Number of cycles: an interactivity strength scale

Crawford mentions that, to his perspective, there are degrees of interactivity. Therefore, interactions could be strong or weak, based on the level of listeningthinking, and speaking by those who participate in the interaction.

To add to Crawford’s degrees of interactivity model, I would say that since an interaction is cyclic process, the number of cycles could be used as a measurement for the strength of an interaction, and that minimum strength of an interaction is a single cycle.

I agree with Crawford’s determination that in cases where the occurrence does not lead to (at least) a single full cycle of interactivity, this occurrence would be considered as a case of cause and reaction.

For example, when two people engage in a long conversation about the theory of Repeated Games With Incomplete Information, they experience an interaction that requires many cycles of questions and answers, that lead to many other repeated cycles. This conversation would be considered as a very strong interaction.
In contrast, when two people stand in an elevator and greet each other with ‘good morning’, I would say that, even though their interaction went through a single cycle (or a single exchange), this single cycle satisfies the minimum requirement for an interaction. Obviously, the strength of this interaction is very weak (some would say that this is the weakest interaction possible, but I always consider an interaction between people as one that includes an exchange of information the surpass the the words that are being said vocally).

In case where one of the actors greets the second actor with ‘good morning’, and the second actor embraces the greeting, but do not reply, I would consider that a case of cause and reaction.

Interaction: an exchange of information

I would also add that an interaction must be a cyclic process in which actors listen, think, speak, and exchange valuable information. Information is valuable for an interaction in case where it was related to the previous cycle, and had an effect on the next on.

For example, let’s have a look at the following conversation:

John: “What is the time now?”
Dana: “The leaves are falling because winter is coming.”

If John considers Dana’s answer as unrelated to his questions, he might stop the conversation, and won’t say anything else. John might as well assume that Dana did not reply to his question, and did not even talk directly to him. In this case, I would argue that Dana did not interact with John, and therefore, there is not even a single cycle of interaction, meaning that there was no interactivity at all.

This situation is very similar to the highly frustrated experience of operating a computer, and getting a repeated error message that doesn’t seem to have any connection to the operation, and that doesn’t lead the user towards his next step.

Error code. [taken from https://xkcd.com/1024/]
Error code.
[taken from https://xkcd.com/1024/]

What if John asks his question again, assuming that Dana miss heard him? What if after Dana’s reply John forgets about the time and starts to gaze at the falling leaves? Similarly to the previous case, In this one I would say that both actors are reacting to one another, but do not perform any real interaction between them.

But what if, somehow, John understands what is the time (or at least he thinks he understands) from Dana’s answer?  In this case, even though the content of the conversation doesn’t seem to be of any value, I would consider it an interaction, since Dana’s reply was valuable to complete a single cycle. In other words, regardless to Dana’s reply content, if this reply pushed John to try to re-interact with Dana, Dana’s reply effected the next step in the interaction. And therefore, it was valuable.

To summarize it all, I would argue that an interaction is:

at least a single exchange of valuable information that leads to a cyclic process in which two actors alternately listen, think, and speak.

Physical interaction

In my opinion, a physical interaction is one that includes a physical object, and that the physical object plays a major role in it. I would consider a physical object anything that has a tangible attribute. Therefore, humans, for example, are physical objects. This conclusion led me to a disagreement with Bret Victor’s article about the future of interaction design. Even though the future, as presented on Microsoft’s concept video, does not fulfil the potential of physical interaction, humans are playing a major role in it.

Physical interaction with digital interfaces. [taken from http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesign]
Physical interaction with digital interfaces.
[taken from http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesign]

For example, an exchange of information between two servers would not be considered to be a physical interaction. But in my opinion, any interaction between human and a machine is a physical interaction.

We can say that a physical interaction is

a cyclic process, on which a physical object plays a major role, and includes at least a single exchange of valuable information.

What makes for good physical interaction?

By now, it is clear to me that a good physical interaction must include a good use of physical object. By saying ‘good use’ I mean that it should be clear that the interaction could not be as strong, or even could not occur at all, without the physical object.

Having said that, and without quoting Don Norman’s book The Design of Everyday Things, I would say that a good physical interaction would also be one that fulfils its potential in the following ways:

It is sensitive

Remember the times when you got a new shirt, or a new haircut, and no one told you anything about it, as if it wasn’t happening? Have you ever experienced a situation where someone important to you forgot about an important day for you?

Computers have the potential to impact our lives with just a minimal effort. Think about a scenario where your desktop lamp would change its tone of light according to your mood, if it is saying that it feels you. Wouldn’t that be nice? What if your home stereo would lower its volume, or just lower its treble sound when you have a headache? What if your refrigerators would pour a glass of water when its sensors hear you coughing?

It is surprisingly smart, but not arrogant

In continue to the previous topic, a good (physical) interaction is one that has a sense of surprise in it. Repeating the same interaction over and over would probably kill this effect, and therefore, a good interaction should be one that evolves, and gets smarter in time.

The best interactions would be those that does not leave the user with a feeling of “that was great, but I have no idea what happened.”. I cannot stand looking at my father as he admires new features on his phone, while secretly admits that he is no way near the understanding of this features, and how they actually work. This interaction leaves my father (and many others) with the feeling that he cannot play a major part in the new world of computer based interactions. In other words, although my father appreciate these interactions, they tend to leave him with some frustration, which leads me to the next topic –

It is satisfying

Rolling back to the earlier parts of this post, I can say that a strong interaction, one that includes many cycles, is surely satisfying. But the feeling of satisfaction could be achieved even on weak interactions, and should not be overlooked. A satisfying interaction would lead a user to be more engaged in it, and would push the user to explore new ones.