interactive sound design studio

Virtual DJ

2008-10-06T09:05:00.000-07:00

by Rong Ma

Overview in brief

The project is named as Virtual DJ, namely, our group intends to bring fresh experience to the people for trying out DJ activities. Typical DJ actions include "mixing" and "scratching", which generally manipulates the music track that is been played. In our example, we use color recognition as interaction method. The program been created is to capture color movements within a given space then generates various types of music sample loops. In this case a web cam is been used as a capture tool for the movement. We then introduced some specially marked gloves, with different colors on each fingertip. These colors are the key to the manipulation of the music. With a combination of these elements, users can easily perform actions as if they are real DJs.

Design Motivation

Our initial goal is to design a simple interface with easy to execute operations to recieve satisfying feedback. We first came across with the idea of using Wii controller as an input device, but as same decisions has been applied with many other project ideas, we abandoned this thought and stick with hand gesture and movements, as it is indeed very simple to initiate. Not only because it is directly percieved through the senses, but also resembles the actions of a real DJ. We want to stick true to the concept, so the decision had risen to the surface at the end: pure and simple hand actions only. Despite limitations and other issues, we decided to include "scratching" and "mixing" in this particular installation.

Related and Influential works

During the brainstorming process, we came across with some cool ideas related to manipulation of music, such as video game controller and DJ style music, we later decided to combine these together to form a more understandable and unique object. We discovered that the idea has already been created by someone else which uses the Nintendo Wii controller to achieve DJ actions. Although it is quite similar to some of the existing video game ideas such as Dance Dance Revolution and Guitar Hero, both use simulations of real life tools to perform actions of the actual activity. As for the Wii controller, it does not resemble anything related to the actual DJ movements, so we didn't choose this idea and approached from a more intuitive view.

Technical Method

We used Max/MSP as the programming tool, together with some jitter patches for performing various actions. The program is been run on a Vista installed PC, rather than a typical Mac, but the actions are all there as for the compatibility is no different.

Web Cam is using to capture the movement of the color markers within the given space. With different positions input different data, the program needs to recognize all the changes and initialize actions at the same time. In the final stage we decided to use 4 types of colors, and thus for the operator to see through the changes clearly, 4 visual screens were embeded using jitter patchs, each screen recognize one specific color.

As for the scratching part, another jitter patch is been applied for manipulating the basic looping sample. By apply the x and y axis into a radial style marker, the program controlls the playback speed from slow to fast. The camera captures the movement of the red color markeron the right hand, the radial marker analyse the angles and send the data though a bucket in order to manipulate the playback speed.

The other patches are used to play additional sample loops, as for the three different colors on the other hand. The jitter patch used is to detect the color appearance on the camera, once the color appeared, a sample is playing once, while if color has appeared multiple times, the sample also will play multiple times by the appearance. The groove patch is used to controll the playback length of each track.

User Experience/Usability Considerations

We intended to let the installation speak for itself, as it is simple from inside out. The interaction interface is very simple, we used a wooden box painted black with a clear perspex stick onto the top, where the users can perform actions on. Inside there's the camera for capturing the hand movements. Although sometimes the appearance did affect people's desire, we somehow managed to deliver the installation in an artistic way, that is, to include a poster. As I am taking care with the production of the poster, the result come out quite satisfactory.

At first we didn't quite have an clear outline for what we need to achieve and accomplish, we decided to take it the simple way: think of the basics, make sure everything is working and then adding extra features. We wishes to include also disc panning and volumne controll, but those can not be executed due to time concerns, we also have to simplify the remaining possibilities to the minimum.

As for the final result, it recieved plenty of possitive feedbacks. Most comment on the creativity and the ideas, we were very happy that our initial goal has achieved. But some users seem to be concerned about the operation process of the installation, we however need to instruct people each time they play for getting the best result.

Final Thoughts

The installation had some problems, such as the color recognition and scratching speed issues. But the exhibition was successful. I am greatly appreciated. I've learned a lot during this course such as intensive teamwork and time management. It is a great experience.

I wish thank Sam and Kirsty for putting up the entire course and the final exhibition. Also my group mates Serge and Rhys, I am very happy to been working with you guys and it is been a wonderful experience throughout the past few days. I want to thank all the other participants and people who took part in this course and exhibition. This success wouldn't be possible without you. : )

References

Audio

http://www.audiojungle.net

http://www.freesounds.org

WiiJ

http://www.djwiij.com/

Max

http://www.cycling74.com/

Virtual DJ – Documentation by Rhys Griffiths

2008-10-06T05:44:00.000-07:00

Brief overview description

The Interactive Sound Studio project created by Serge Viranian, Rong Ma and Myself is a DJ simulation installation titled ‘Virtual DJ’. ‘Virtual DJ’ attempts to create a ‘DJ-ing’ experience for users through its ability to modify and manipulate techno music through actions such as ‘Mixing’ and ‘Scratching’.

Gloves worn by the user have various colours on the fingertips which are tracked by a webcam encased inside a three sided box with a clear Perspex top.

The data collected from the spatial movement of these colour tipped fingers is then used in the installations program, created in Max/MSP, to carry out various operations.

Here is a picture of the installation.

Design motivation

In the early stages of the course, our team initially had similar ideas to create an installation which dealt with the control of music - ‘DJ-ing’ came to mind.

In real life, DJs use their hands to control and modify music. DJ’s use their hands to manipulate vinyl records to create ‘scratching’ noises. They also press buttons and turn knobs to create various sound effects and modify the music. We figured that the hands would be the best type of controller.

Related and influential works

Whilst researching and looking for inspiration, we found numerous related compositions which utilized the Nintendo Wii’s ‘Wii remote’ - ‘Wii J DJ’, ‘Wii Conductor Hero’ and the ‘Wii Loop Machine’ to name a few.

With the success of Video Game titles such as Guitar Hero, in which users use a controller with the appearance similar to that of a real guitar, it could be assumed that users feel immersed and truly stimulated when using controllers resembling their real counterpart.

After discovering the abovementioned compositions, and with the knowledge that many of our peers would be using ‘Wii remotes’ in their projects, we decided that we would rather try to use something else as our major controller.

As we decided on the concept of ‘DJ-ing’ and because our research showed us that people liked to use a simulation of the real thing, we decided that the ‘Wii remote’ was not suitable for this purpose and that the use of spatial movements of the hand on would be better.

Technical method

The encased webcam was used to detect movement data of the coloured markers on the white gloves. Once detected, the data was used by the program created in Max/MSP.

This picture is a snapshot of the program used in the installation which was created in Max/MSP. The webcam tracks the colours on the fingertips of the gloves worn by the user and sends the data to the appropriate box - one box deals with red colours, another with blue, another with green and another with yellow.

These boxes are made up of more codes and operations. The output of these codes and operations is sent to the audio output which is in the bottom left of the picture.

This snapshot shows the process of the ‘scratching’. The data of the spatial movement of the red colours on the right hand glove is taken, and turned into an angle, as can be seen on the left hand side of the picture. This angle data then determines the playback speed of the main looping track.

This was slightly problematic because at some point the angle inverts its values making very high or low playback speeds very difficult to perform.

This snapshot is of the process of playing additional sound loops, as a result of movements of the blue, yellow and green colours on the left hand glove. If colour is detected, the trigger will change its value to true and play the sound. At the top of the picture, the sound files used in the program are being loaded.

User experience / Usability considerations

We wanted to create an experience for the user in which they felt they were being a DJ. We also wanted to create a unique and interesting interface which all sorts of people could use and appreciate.

As the installation intentionally did not have clearly outlined instructions, our goal was to create an experience where the user would be able to take the information they were receiving (the sound output) and then conclude what their movements and actions were affecting.

This worked quite well in most cases, but some users which spent little time using the installation did not completely understand what they had just experienced.

DJ’s mix and manipulate music and try to keep rhythm. Throughout the course, it became apparent that our initial proposal would be too difficult to achieve within the given time frame. We had to leave out panning, volume and tone controls and simplify the other operations.

Therefore, the main challenge and focus of the installation became trying to mix the sounds together in rhythm and for the music to continually flow – as a true DJ would attempt to do.

Resources & References

Computer Hardware

ú Laptop

ú High quality headphones

ú Webcam

ú Computer speakers

Computer Software

ú Max/MSP with CG Jitter Patch

ú Sound Track Pro

ú Audacity

The Box

ú Wood: roughly 3 50cm x 30cm pieces

ú 4mm thick Perspex top

ú Portable fluorescent light

ú Semi-gloss black spray paint

ú Screws

ú Silicone glue

ú Tools and machines from University Workshop

www.audiojungle.com
www.freesounds.org
www.soundsnap.com
‘DJ Wii J’
‘Wii Conductor Hero’
‘Wii Loop Machine’
‘Guitar Hero’

Conclusion

Our installation was well received at the Interactive Sound Studio Exhibition. Users commented that the installation was fun and that they had not used anything like it before. We learn valuable skills in programming, time management and presentation and also learned a lot about sound and how to use it effectively.

FINAL REPORT ON VIRTUAL CONDUCTOR: M. Khalil

2008-10-06T05:16:00.000-07:00

PROJECT OVERVIEW

Virtual Conductor was established to control a musical arrangement in real time much the same way as a conductor controls an orchestra. A musical piece is heard and the conductor has control over overall tempo, overall dynamics, as well as individual dynamic enhancement of the four instrument sections (horns, woodwinds and strings, percussion and the solo violin. This was done using a Wii remote was programmed through max/msp.

All the members of our group were involved in all the processes in order to perfect this project. We decided that we should all be in charge of one major aspect of then assignment.
Stephen: Head programmer, Nick: Head of documentation and research, Michael: Head of Audio and pictures.
We collated all our parts together to create an immersive and enjoyable experience for the user of our system.

RELATED AND INFLUENTIAL WORKS

Our idea was conceived because all members of the team had
musical backgrounds. The main works that influenced this idea was
Guitar hero @ guitarhero.com/, and Wii Music @
http://en.wikipedia.org/wiki/Wii_Music and on
e3.nintendo.com/wii/wiimusic/index.html.
Our project turned out quite similar to the Wii Music work, and this
Is exactly what we hoped for.

DESIGN MOTIVATION

Our intent was to provide the user with an immersive but enjoyable experience, they could use over and over again. The main motivation for this project is that we all have musical back grounds and wanted to control music using gestures in a similar way to which a conductor controls an orchestra.
I personally felt that we successfully achieved this, given the positive reaction of users on the presentation night.

TECHNICAL METHOD

We wanted the user to simulate what a conductor of an orchestra does instinctually. For this we decided that the use of the wiimote and Nun-chuk expansion would be a suitable interface for these purposes. It works by the wiimote controlling the playback speed of pre-recorded orchestral tracks and the Nun-chuk attachment controlling the volume.
The aka.wiimote patch was used for mapping of the wiiremote
To execute this we used the IRCAM software supervp.play~ patch to control the playback speed changes, with limiting the minimum and maximum speeds reachable, while pitch and envelope correction maintained the original pitch and envelope. The nun-chuk joystick is used to increase the dynamics of one of four musical sections. The C and Z button controls the master volume of playback, up and down.

The simplicity in the interface is designed that a person with no musical back ground can come wave their hand to get a varying tempo and use there other hand to adjust volume levels. This I believe is achieved efficiently for the user as there is not many button presses involved for the user.

The interface is also defined that the two parameters that the user will control are laid out one in each hand.

The music was composed with usability of the interface in mind. Knowing that tempo and dynamics were our main controls, I composed the piece so that it builds in intensity and power from beginning to end. This enables the operator to instinctually feel, when something is to be louder or softer, or speed up. The song goes for three minutes and has 4 major sections to it. The 16 or 32 bar sections enable the user to easily manipulate the song, as it is easy to perceive an oncoming change even if you are not familiar with the song.

The music was composed in my studio in Pro Tools. 90% of the sounds came from an orchestral sampler program called EWQLSO GOLD. This was used because of its lifelike and “Hollywood movie” typed sound qualities. This was this taken to be mixed using Izotope Ozone 2.
I also recorded an electric violin in the final dramatic section. I felt like the piece was perfect for our Virtual Conductor example due to its intensity and room for dynamic and tempo articulation. I received very positive feedback on this from almost all the people who came to use our system on the night.

USER EXPERIENCE AND USABILITY CONSIDERATIONS

The user will be a conductor in charge of directing the flow of a piece of music using two Wiimotes as batons.
One hand will be triggering musical sections (strings, horns, woodwind, brass), and the other hand will be manipulating variables such as sustain, dynamics and expression.

It was easy for users to understand and use our system but like anything else it was difficult to master at first attempt. This would make it a great game ( Wii music).
In our earlier reports we made it clear that an instinctual and easy system of mapping must be implemented for ease of use and practicality, I believe we achieved this.
Every person enjoyed using this device especially us, who mastered conducting the song. It was quite addictive and was not too embarrassing waving your hands around.

The user really got the feel of interacting as a conductor in our system. It should not be too embarrassing for a user to move their arms around, as it will be enjoyable to hear the variations in the audio samples in sympathy to the users motions.

One of the main difficulties the user faced is co-ordinating both hands simultaneously for the best possible result. This was exactly what we predicted, but this is another challenge that would make this a good starting basis for a computer game.
Our goal would be for a user to have an immersive and fun experience through manipulating a musical arrangement.

FINAL THOUGHTS

After seeing our final project on show at the presentation night we were quite pleased with the final result. All but one of our major ideas were implemented.
We initially wanted to have additional controls over the 4 sections, especially sustain and reverb. This would have greatly enhanced the experience but was unable to be implemented due to the system crashing. It was something to do with supervp patch lagging the system. In fact I even had to bounce down the individual musical sections as mono, because having 4 stereo tracks was overloading the system too.
To make this a commercial computer style game some sort of graphical representation of what was going on would be seen on the screen, this was definitely out of our league within the time given.
One criticism of our project is that there was only one song. In a game there would be at least a dozen or so songs. Once people played and mastered the song some asked if there was another they could play. Even though this was the case, for this instance it would have been impossible to produce many quality songs in the time given.

Resources

Mac Book Pro- Running of applications

Max 5 demo version – programming

Wiimote – Conductors baton Nun-

chuk attachment – dynamic adjustment

Khalil Sound Production Studio – Various equipment for music production

References

www.cycling74.com

www.guitarhero.com

http://en.wikipedia.org/wiki/Wii_Music

www.e3.nintendo.com/wii/wiimusic/index.html.

www.iamas.ac.jp/~aka/maxwww.ircam.fr

sonic hockey

2008-10-06T05:15:00.000-07:00

PROJECT OVERVIEW
This project was based on the idea of air hockey game. Basically, we wanted to create sound during play, and have the movement of the puck to control the sound. The real air hockey table uses air to minimise the friction, and our table not. Therefore, we somewhat used this as part of the creation; we tried to reflect the roughness of the table in the sound created. We also wanted to have another sound played when the puck hit the goal area, and for this to happen, we used contact microphone that picked up the vibration of the puck. We selected a few sound files and have it played randomly.

DESIGN MOTIVATION
Our motivation was to create an audio interface or installation that was suitable for everyone. We had a few different ideas before we decided to pursue the idea of an audio game installation. We thought that by having a game based installation, it would be easier for people to interact, and perhaps a little less intimidating for those who have no knowledge in music. We believed that sonic hockey was a good idea because there’s no special knowledge required, exciting, and allowed you to have an interaction with not only the interface, but also with another person and of course, sound.

INFLUENTIAL WORKS
The main influence was the real air hockey table. We also looked at a game called sonic pong, a game that required the user to make a noise into the microphone to move the bat. So this is basically the opposite of what we’re trying to do. This game has sound as the input and whilst ours have sound as output.

INTENDED USER EXPERIENCE
The sonic hockey is a two-player game. The game play is similar with the air hockey, you will need to gain control of the hockey puck and try to beat your opponent by trying to score goal. The difference with sonic hockey is that it will also produce sound during play. The ideal is for the players to have long rallies and discover the variety of sound changes during play and to experiment and simply have fun with it. It’s a fun and light game for everyone to enjoy and have a play with.

USABILITY CONSIDERATION
The process of building the sonic hockey is quite tricky. In general people will start comparing it with the functionality of the air hockey table. This, for example, raised a few critical questions like how to reduce the friction and the smoothness of the surface. We had the option of getting a fairly priced second hand air hockey table, which could have made things a lot easier. However, we decided not to because in a way we’re trying to create something that is quite different and thought that it would be challenging for us to actually make something from scratch. One of the things that we had to consider was, would this custom made table interfere with the way people play, and we understood that it will not have the same effect as the air hockey game, and our solution was to somewhat use the table “flaw” as part of the creation. Using a “scratchy” or “heavy” sound effect to match the way the puck’s moving around the table. Technically, having the puck not moving so quickly, it helped the camera to track it a little better.

DESIGN PROCESS AND DEVELOPMENT
Firstly, we started off with some rough sketches, mapping out the best way to build the table, which includes the position of the goal area, camera stand or holder, and the border of the table. Basically, we are using a pre-existing table and we added a few additional attributes to make this table functional.

Before building the actual table, we did a few material tests to see which one works best. We decided to use timber as our main material because it was available for us and it’s inexpensive. For the boundary or the barrier we’re using guitar strap because it has the best bounce. The strap was stretched across an additional wooden frame, which was screwed to the bottom of the table.

The goal area was a basic square shaped box attached to each end of the table. The box had holes on each side. One small hole on the upper area of the box for the puck to goes in, and the other was on the lower part (on the other side) for the player to pick it up. We’re leaving the other half side of the box covered so that we had a place to attach the contact mic onto. And also to minimise the possibility of the puck from going all over the place when it entered the goal area.

The camera box was made out of four pieces of timber “sewed” together with plastic string. This box was used to cover the actual stand, which is a circle shaped timber with two holes on it. One was for the cable and the other one for the camera lens to see through. At first we wanted to have the camera dangling from the ceiling but assured that it was not very safe for the camera. Therefore, we decided to sit the camera on Perspex that was placed on top of two pillars on the ceiling, forming a bridge-like holder.

With the puck itself, we’re using a standard air hockey puck and attached a laminex sheet on the bottom part for smoothing and a colour label on the top part for the colour tracking. After we did a few tests we could actually hear a very loud noise caused by the collision of the puck and the bat. Therefore, we decided to wrap some rubber bands around the bat to minimise it.

TECHNICAL METHOD

Originally we wanted to use a bluetooth mouse inside a custom-made hockey puck and wii remotes for the goal area. We didn’t go with this idea because we thought it would be difficult to play this game when you have a quite heavy puck and the mouse wouldn’t work that well. We were also having a few difficulties finding a way to trigger the button of the wii remote so that the puck will always hit the button when it enters the goal area.

The two key methods we ended up using were colour tracking and contact Mic. The Colour tracking was used to track the location of the puck and the Mic was used to detect when a goal was scored.

With the colour tracking, we placed a mini DV camera on top of the table. The camera was connected to a laptop running Max/MSP. Basically, the camera picked up the colour of the puck and run it through the program. For this we’re using the “jit.qt.grab”. After the colour was locked in, the”jit.findbounds” object was used to track the x and y position of the puck. We applied a sound file to the x coordinate. The sound then played in different speed depending on the movement of the puck.

The contact microphone was secretly placed inside the goal area. It basically picked up the vibration of the object surrounding (as the input) and Max/msp played a sound as the output.
The problem was, it was very difficult to find the right value. So it was either too sensitive or the opposite. Despite of the unpredictable result, the random sound played added extra fun to the game.

CONCLUSION

The process of making this project was incredibly interesting and enjoyable. Not only I learned a lot about audio but also about how to build something from scratch. So it was such a great experience considering I was not really familiar with the area.
I personally feel satisfied with the result and it was great to see people were having fun with it.

RESOURCES
http://www.milmoe.com/artprojects/sonicpong/sonicpong.htm
http://cycling74.com/twiki/bin/view/ResourceGuide/InterestingWork

2008-10-06T04:57:00.000-07:00

The Cranky Bear - Robin Mukerjee

Overview

The Cranky Bear changed a lot from its initial conception. At first we wanted the bear to really be just an object used to plainly just manipulate a sound or a song. We decided that we wanted the teddy bear to have a character that would respond to you when you picked up the teddy bear and held it in different ways.

We set out to create a series of different dialogues and sounds that would be triggered when used. When the bear would be lying down flat on its back it would snore over a lullaby inspired song. When the bear was sitting still the bear would rant and speak randomly as if nobody was listen. When the bear was picked he would speak to you but as the dialogue was created, not take any notice of what you were saying. If you were to shake the bear it would make the sound of the of being moved back and fourth. Eventually if you shook it hard enough the bear would get angry and ask to be put down on the table. When the user was holding the bear it could tell it to “relax” or use the word “sleep” as to make the bear ask to be put back down so it could go to sleep.

The Bears character was intended to be cute yet very rude and irritable if it was annoyed.

Related and Influential Works
The Cranky Bear was influenced by other interactive toys that are on the market today. Fundamentally of these toys is the Tickle Me Elmo doll made by Tyco that was introduced onto the American market in 1996. The doll when squeezed (or “tickled”) would laugh hysterically and also shake. The interactive elements of these toys were very successful and subsequently made these toys some of the most successful of their kind.

Tickle Me Elmo

Design Motivation
We wanted to make something which was intended to be used by adults. The bear was initially going to be “The Haunted Bear” in which we wanted the bear to give the user a scary and creepy experience but it was changed to add a more comical side to the toy. We wanted to compromise the child-like nature of the toy by making the bear aggressive and angry. The bear also was to represent a character that would be amusing and relatable to the user.

Technical Method
The audio was recorded by using a Pro Tools Mbox 2 setup. To give the bear a childlike voice we pitch-shifted the original voice up by 8 semitones. Altogether about 50 minutes of dialogue was recorded, the majority of the being for when the bear is sitting and “ranting” to itself, yet the sheer size of these audio files proved a problem when putting them into MaxMSP. The song that was created for the piece was done by simply looping a mini children’s guitar with a keyboard playing a melody. To read the bears motion we used the motion-sensors of a Nintendo Wii remote that was put inside the bear which was then read by the akawii remote MaxMSP patch. We created a patch that would read the X, Y and Z axis of the movement of the bear, each time a specific movement of the Wii remote would change this would trigger another patch which would play a sound. We used the aka listen MaxMSP patch for voice recognition. When a certain word was said this would trigger the another sound within the bear.

User Experience/Usability Considerations
We wanted the user in the end to have a playful and enjoyable use of the bear, and in general the majority of people who used it were impressed and amused by the bear angry talking. Unfortunately bear did not work as well as we had planned and specific instructions were made in order to tell the user how to use it. The loose positioning of the Wii remote inside the bear made the beat unable to read the proper signals that it needed to trigger the sounds we wanted. The bear when picked would talk a little bit but not much. People need to be told to shake the bear in order to make the bear make the shaking noise and eventually get angry. The voice recognition did not work at the exhibition because of background noise that was in the room which was really unfortunate being what I felt, one of the most impressive elements of the bear. With more time and fine tuning and would have been good really get the logistics of the movement of the bear right, with the amount of audio that we had in MaxMSP only a small percentile was actually triggered at the exhibition.

MaxMSP which would detect the movement of bear when shaken

MaxMSP patch which would detect the voice

Conclusion
The few hours before the start of the exhibition I was not really happy with the final product. The way we had designed the bear to work was simply not working. But people in general (if they weren’t offended) were amused and impressed with bear which in change gave me piece of mind that we had made a successful exhibit. With more time it would have been really great to fine-tune the motion sensor and voice recognition parts of the bear as to make how we intended.

Resources
Interactive-sound.blogspot.com
http://www.tyco.com/
www.cycling74.com

2008-10-06T03:57:00.000-07:00

Interactive Narrative Final Report

version written by Michael Moretti

Overview

Our plan form the beginning was to prove our game design skills, by addressing and overcoming the constraints to make a working, functional, and immersive game. Then the idea came to make a sound only game experience using the Wii remote. So we designed a game, and a story that could be played without any visual aids. This game was title Interactive Narrative. The initial idea for the story was also to make it non-linear. Meaning the story does not just simply play through in a straight line, it can branch off depending on how the player reacts to it. The game needed to include instructions on how to use the Wii remote to do certain actions. The player of this game was also blindfolded, this was intended to add to the experience by taking away all visual distractions, and aiding them in the use of their imagination to create their own environment. In the end the completed game included the tutorial, and first mission, which when played lasted between ten and twenty minutes.

Related and Influential Works

Our influence came from any game on the Wii. The use of the Wii remote to control a character and/or any other aspects of the game. Some well known favourite Wii games include; Zelda: Twilight Princess, Wi Sports, Super Smash Bros, Mario and Sonic at the Olympic Games, and many more. Outside of the mainstream influences was the Virtual Barbershop. An experience made in the form of sound design, that recreated a scene in a barbershop getting a haircut, the way it was designed makes it feel almost real. Which showed us that sound can be extremely immersive when used correctly.

Design Motivation

Our primary design motivation came from Myles and myself being aspiring game designers. We both agreed that a true game designer can use what they are given, and work within constraints to produce a potentially successful game. So we took this on board, and sat down, went through all the constraints, and eventually came up with this game. The initial idea had us both very excited, and the challenge of pulling it off kept us motivated. We also knew that if this was successful that this game would make a great addition to our portfolios, and aid our quest to become game designers. What also motivated the design was the use of the Wii remote. Since we had access to the Wii remote, which is already a game controller, we found more inspiration. Since the wii remote is based off movement it can be used without being looked at, which supported our idea.

Technical Method

Myles was the lead programmer, but i was always close by to lend a hand. We used primarily Max/MSP to put the game together, but with the help of javascript. The javascript was used to do all the nitty gritty decisions, such as take the wii remote inputs, and determine an outcome depending on actions, which was achieved using multiple 'if' statements. Then in Max/MSP we would playback the sounds, and layout the linear function of the story and its scenes, or chapters. The use of the aka.wiiremote helped a lot, otherwise the wii input values would not have been attainable. Each chapter was very similar in set up, the basic set up of each chapter was:

The wii input was taken from a subpatcher, which received the wii values from aka.wiiremote

A switch was used to turn on or off certain wii input subpatches. If all of them were left on, the story looped backwards.

The wii input values were sent into a javascript file which determined what output to send for each action, or wii value.

Then out from the javascript were the sound files that played in response the player actions.

Then the last patch was just to play sound or dialogue that continued the story.

User Experience

When playing the Interactive Narrative, a player becomes immersed in the created world. The use of sounds creates an environment that the player, since blindfolded, and forced to sue their imagination, become fully immersed in. This world becomes their world, at least while they are playing. Also adding to this experience is the Wii remote, since the player actually has to make the actions themselves, rather than just press buttons like on an ordinary game control, they again feel more immersed. And since we were using the Wii remote, which can be used without needing to know the whereabouts of many buttons, it allowed us to blindfold the player, so they didn't have any distractions.

Recognitions

I'd like to personally thank the two members of my team. Myles, who i have always worked well with, we make a great team. He has great design ideas, and awesome programming skills with which this project would not have been possible. And James, who joined our group after the initial designs were made, and contributed some incredible sound design, which would have been a tremendous down fall of our game otherwise. his great work really made the game immersive. Also thanks to Kirsty and Sam for their help and teaching. When it came to programming issues Sam was always happy to help.

Recources

-Max/MSP

-Macbook Pro

-Macintosh Desktop

-Wii remote

Virtual DJ - Serge Viranian

2008-10-06T03:43:00.000-07:00

Brief overview

Virtual DJ is an interactive colour recognition interface where the user has control over the speed of a looped track enabling them to "scratch" the track and play various samples with the use of specially coloured gloves.

Virtual DJ is designed with the intent of being a simple and playful music interface with little to no learning curve. The user wearing white gloves with 4 different coloured fingertips (red, green, blue and yellow) waves their hands over the cam placed under a clear top box.

Influential Works

The main influence was derived from WiiJaying which is an emerging trend in performance DJing it uses MAX and Wii controllers and other professional mixing software to both DJ with live and pre-programmed tracks. This interface is more involved and by no means meant for non-professionals which lead us to develop our simple interface which did not rely on any particular controllers.

Design Motivation

We all wanted to design a musical interface that was easy to manipulate on the fly and therefore opted for a virtual DJ set we decided to move away from Wii controllers or any controllers for that matter and to stick to basic hand gestures for a more seamless and intuitive interface.

The design of the physical interface was meant to be as simple as possible so we developed an open ended box which would house the colour tracking web cam all users pretty much knew to just wave their fingertips over the web cam without being instructed.

The MAX interface was designed to be as clear as possible and included a visual screen for each colour. All patched were neatly aligned into sub patches which helped immensely when the program crashed or reset as it helped in booting then patch as quickly as possible.

Technical Method

Using Max and Jitter patches we modified and programmed the following:

jit.dx.grab to initialise the web cam we also used the web cams settings to modify the brightness, white balance and colour intensity this proved to take the longest to calibrate as lighting differed at different hours of the day.
four jit.findbounds patches in which we entered the 4 colours(yellow, red, blue and green) the exact RGB colours were determined using a sukha on the main web cam viewer. red was used for scratching the other three colours were used to initialise the sample loops.
The jit.findbounds for the samples had a togedge each for when the- colours appear only.
jit.matrix was used to display the colours on 4 different web cam previewers.
the red jit.findbounds was then connected to a cartopol and poltocar to coverts x and y coordinates into radial angles
the angles where then plugged through a bucket and multiplied by a constant so as to keep the speed in between -2 and 2 with 1 being the normal speed.
The data being transferred was also gated as to go back to normal speed when the coulours weren't appearing
All the colours including the processed red were then patched through a groove to output the selected track and samples as well as their lengths.

All that was needed as hardware was a USB headset for audio output and a simple web cam for visual input as well as back light and laptop to run the whole system.

User Experience/Usability Considerations

By trying to make it as intuitive as possible using hand gestures that DJs would use on a normal turntable we reduced the learning curve considerably. The setup was pretty much self explanatory : put headsets and gloves on and wave fingertips or "scratch" away on top of camera to get various sound effects. Nothing to it...

We tried to make the interface as plain and simple as possible but still ended up needing to instruct people in terms of positioning and interaction. The people who spent more than a minute or two on the program started getting the hang of it. When we later pasted instructions next to the DJ box things became a lot clearer for the users. So no matter how simple we may think the interaction is a little instructions never hurt.

Conclusion

All in all it was a great learning experience and given more time I'm sure the project would have been more elaborate and richer in terms of both music quality and interactive elements. Big thanks to Kirsty, Sam, Rhys and Rong was great working with you all.

Resources/References

software:
Max 5.0
Audacity
Reason
hardware:
dell xps M1530 laptop
plantronics dsp 500 headset
logitech webcam
Fluoro light

websites:
http://www.audiojungle.net
http://www.freesounds.org
http://www.djwiij.com/
http://www.cycling74.com/

Stephen Jack Report

2008-10-06T01:20:00.000-07:00

Virtual conductor Report - Stephen Jack

Overview

This project is a conductor simulator to give a person with no conducting experience the feel of conducting an orchestra. It allows the user to control tempo and volume by using the wiimote and nun-chuk attachment.

The wiimote controls the tempo by detecting the movements of the users hand, similar in the way a conductor would move there arm to control an orchestra or a band, while the nun-chuk controls the volume with the C and Z buttons controlling the master volume of playback and the joystick controlling the different “spotlighted” instruments.

Related Influential Works

Some influential works include games such as guitar hero, rock band and the upcoming wii music, and works such as wii conductor hero. When coming up with the idea we wanted to do a very basic instrument to start then started going towards the ‘music controller’ games such as those previously mentioned.

While coming up with the idea we also found similar ideas like the conductor hero kiosk, where users control the video playback of an orchestra in a similar to what our project does with audio. A team from the college of New Jersey, led by Teresa Nakra and Chris Ault, designed the kiosk.

Design Motivations

The main motivation for the design of Virtual Conductor was to have a user with no conducting experience before be capable to effectively change tempo and volume of playback. This is achieved by giving the user a simple easy to use system with not too many variables to over complicate the basics of the program. For instance the wiimote is waved like a conductors baton for the tempo control, and the nun-chuk controls all the volume variables, so the main hand controls one variable and the off hand controls the other.

At conception of the idea we wanted to also add other variables such as reverb, Eq, flangers and other effects but to keep it simple for the user and processing power we purposely left these features out.

Technical Methods of Execution

To execute our Idea we decided on the wiimote with the nun-chuk attachment would be a suitable user interface for our goals. We used the aka.wiiremote Max/MSP patch to connect to the wiimote, then decided what information from the inputs we where going to use as well, such as only needing one axis of the accelerometer information coming from the wiimote instead of all three (determining the beat).

This shows the information that we use, coming from the wiimote.

Once we determined how to recognise the beats we choose the IRCAM software supervp.play~ patch to control all of the real-time pitch and envelope correction when the tempo is changed. This patch worked successfully but is rather processor intensive.

When the supervp.play~ patch was introduced it worked well on up to three tracks but upon the fourth track it started to test the DSP power of the computer. This problem was overcome by making the patch run in mono instead of looking at a stereo buffer. Now with four simultaneous tracks running there is no signal distortion due to processor power.

The mapping of the nun-chuks joystick information was as easy as determining which locations to select then created a decision tree that allowed us to define the parameters of the values coming from the joystick.

There was also use of smoothing on the volume buttons so the changes seemed not so sharp and obvious but smoother. This was used on both the ‘spotlight’ joystick and the master volume buttons.

User Experience/ User Considerations

We wanted the user to experience a fun interactive event where they will be able to determine their success themselves by the change in the output. As stated earlier we wanted it to remain simple to keep the users experience fun and enjoyable and not laboured and boring.
Some user considerations would be that we kept the controls fairly simple to use and not being counter intuitive.

Here is the current GUI interface for the program it is very simple and is mainly for connecting the wiimote to the computer.

Possible Changes/Additions

There are some possible changes or additions to the project that would be possible for future builds. One is the ability to record a take of the ‘conductors’ performance in the way they manipulated the play back. Another would be the ability to load your own tracks into the software for playback instead of pre-programmed tracks to possibly create a mixer. Also the uses of a better GUI interface to make the program easier to use for the ‘conductor’ without having an operator there.

One idea that was a possibility at the conception of this idea was that of intergrading effects, such as reverb, delay and sweeping EQ, into the program. This wasn’t done as we wanted it to remain simple and the addition of these extra features would detract from the simplicity of the two parameters and two hands ethos we where using when creating. It is however possible in future builds to include this feature for the user to choose whether they want this feature on or off.

A possible change suggested to us was to change the way an instrument is ‘spotlighted’ by the nun-chuks joystick. To change it would be a variable gain by how far away from the centre of the axis the joystick was, to have a half emphasised value. Also was suggested to have a hold function to allow better “mixing” of the sounds together, working more like a mixer.
Some last minute additions where the inclusion of the play, stop and tempo reset buttons so the ‘conductor’ could control their performance entirely from the wiimote.

Resources

The resources we used where:
• Mac Book Pro- for both designing and running the application
• Max 5 demo version – for building the program
• Wiimote – get ‘conductors baton’ information via accelerometers
• Nun- chuk attachment – gain control
• Logitech computer speakers – audio output from the computer

References

www.iamas.ac.jp/~aka/max
www.ircam.fr
www.cycling74.com
www.tcnj.edu/~pa/video/conducting08

Druidic Rituals - Matthew Shriffer

2008-10-06T00:51:00.000-07:00

Brief Overview/ Design Motivation

Druidic Rituals is a multi-user fantasy installation modelled on a modern-day mythical or religious ceremonial experience. The idea behind the project was to create an ambient environment that encouraged user activity and place them in a setting not normally encountered in society to enforce a purposefully detached and immersive experience.

The installation consisted of three separate pillars, on top of each was contained a glass bowl half-filled with water. The pillars were arranged in a triangular pattern with three loudspeakers opposite each one. The idea was to get users to trace patterns in the water with their fingers, or wave their hands above the waters’ surface in order to take part in the “ritual”. Drone/noise based audio loops were then generated according to the motion above/in the bowls, and subsequently played in the surrounding loudspeakers.

The setup of the installation allowed for a tiered approach to interactivity between the users – if one person was taking part, they would hear the noise specific to their pillar all around them. But if two or three people were taking part, a layered wall of noise would surround the area.

Technical Method

The core of the installation was a PC laptop, running a program created by myself in Max/MSP/Jitter 5.0. Three USB cameras were contained inside the pillars (made of PVC tubing) and were connected to the laptop, as well as a Digidesign 002 rack mount unit (via FireWire). Three active loudspeakers were connected to the 002 rack’s audio outputs.

The Max/MSP/Jitter program was designed to detect hand movement, using the cv.hit.HSflow object, and use it to trigger a pre-recorded drone/noise loop. I created these loops in Reason/Pro Tools using a combination of digital samples and my own analogue recordings of guitar feedback and effects. The XY pixel displacement measured by the optical flow tracking method was also programmed to control the overdrive~ object level and pitch of the loops respectively.

Influential Works

The project was influenced by ambient works from Brian Eno (Music for Airports, Discreet Music, and Ambient 2 with Harold Budd), Peter Wright (At Last a New Dawn) and William Basinski (Disintegration Loops series).

More noise-based drone works also contributed major influence to the final aesthetic and atmosphere of the installation, such as works by bands like Sunn O))) (Black One, White One/Two, Altar), Earth (The Bees Made Honey in the Lions Skull, Sunn Amps and Smashed Guitars, Hibernaculum), Growing (Colour Wheel), Magic Lantern (At The Mountains of Madness) and Sonic Youth.

User Experience/Useability Issues

The user experience was loosely documented by me through monitoring and surveying of the users and analysis of comments/criticism received verbally through conversation.

The main concern that arose after this documentation was the issue of indecisiveness and confusion as to what the user is actually supposed to do. I noticed a few people looking at the glass bowls, not knowing whether to touch the water, or wait and see what happens if they do nothing. It was only after I sidled over and encouraged them to dip their hands in the water that they comprehended what was meant to happen. I think this is quite an obvious useability issue which could have been solved with some simple instructions – although I feel this would have taken away from the surreal and detached atmosphere I was trying to create.

Apart from this issue, most users seemed to enjoy creating music using a different and tactile method, and were quite intrigued as to the different sounds created by each pillar, moving from one to another as they viewed the exhibition. I also feel that the simplicity of the design encouraged it to be used by anyone with a childish desire to get their hands wet – and considering the weather at the time, this ended up being most people!

Thanks to Sam & Kirsty for their help in realising my project idea and helping to execute it in the exact way that I had envisioned it.

Virtual Conductor-Nicholas Verdos

2008-10-05T23:44:00.000-07:00

Virtual Conductor- Nicholas Verdos

Brief overview description

Virtual conductor was designed to be an interactive Wii remote controlled experience where a user was using the Wii remotes to control music and be a conductor of a piece of music. The idea of how we would implement this changed during the designing process. We eventually came to a conclusion that instead of it being an advanced conductor simulation we thought it would be more user friendly if it was a piece of music that the user had simple controls over such as volume of each instrument and tempo of the piece. It was broken up into four pieces of controllable sections of the musical piece and that was woodwinds, beat, strings and brass. A nun chuck on the left hand controlled the volume of the piece and of each instrument and the right hand which held the Wii remote controlled the tempo of the piece so as you swung the remote up and down, like a conductor, it would speed up the tempo according to how fast it was being swung. The music itself was conducted by use and we specifically wanted a composed original orchestral piece that had build up and feeling in the song to give the user a chance to put emphasis with the volume control on certain instruments in the song, which in my opinion turned out exactly how we desired in the final product.

Related and influential works

Some specific works that gave us ideas and influenced our basis for our design were games such as Wii conductor hero, guitar hero, rock band, singstar and the new upcoming Wii music game.
We initially wanted to make something musical and just wanted to make a visual conductor where you didn't need a wii mote but we thought the wii mote would act as more of a conductors stick and make it easier and more responsive other then video capturing.

Design motivation

Our motivation for our end design originated from each of our musical backgrounds we wanted control music using gestures in a similar way to which a conductor controls an orchestra. Our motivations are those that are mentioned in the influential works such as wii games where the interactivity from user the triggers music. Our idea changes a lot during the process of designing the system and we decided to take out the gaming experience from our original plans and this was because it was hard to implement in such a short time a way to keep points or some sort of scoring system.

Technical method

Using the IRCAM software supervp play patch that gave us control on the playback speed changes, with limiting the minimum and maximum speeds reachable, while pitch and envelope correction maintained the original patch envelope. The nun-chuck joystick is used to select between the four different streams over audio and puts emphasis on them and the C and Z button controls the master volume of playback. The wii mote controlled the tempo when was swung up and down and would send a signal to maxmsp and there was a max tempo that could be slow down when swung slower.

Above is the maxmsp patches of the programming for the wii mote and nun chuck. The nun chuck when moved the wheel stick would trigger the box on the right and trigger the instruments which were assigned to that quadrant and give it max volume.

User experience / usability considerations

The simplicity in the interface is designed that a person with no musical back ground can come wave their hand to get a varying tempo and use there other hand to adjust volume levels. This I believe is achieved efficiently for the user as there is not many button presses involved for the user. The interface is also defined that the two parameters that the user will control are laid out one in each hand. We wanted the user to not have to practice this to get a hang over how to use this particular software, but more natural in how it works. We wanted to simulate what non musical people believe a conductor does for an orchestra using simple controls. This influenced us to use a simple method of the wiimote and nun-chuck. Our goal was for a user to have an immersed experience through manipulating a musical arrangement. This would require that there are appropriate parameters that may be altered by the user. I believe we achieved our goal quite successfully and gave us a good platform to build on if we wanted to develop the idea more to accommodate a gaming experience or possibly something that a user could use for longer then just the 3 minutes length of one song, possibly by making more songs.

Resources/references

http://uk.videogames.games.yahoo.com/wii/previews/wii-music-orchestra-7d8da2.html
http://www.guitarhero.com/
http://au.wii.ign.com/objects/827/827335.html

Here a few sites of examples that gave us ideas to our final design.

Conclusion

To conclude some of the different things if we had more time, in my opinion, I would change would be I would of made a few more songs so the user has more choice of songs because playing the song it gets a bit repetitive playing the same song. Also more control on reverberation and sustain would have been nice but this was in our original plans but due to technical problems in supervp we could not implement this into the experience cause it would cause it made our system lag to do the processing needed to run. More choice of music would of gave a user a bit more incentive to stay and play with the virtual conductor. Using it myself I noticed once you figured out the song and played it a few times you wanted a new song.

Virtual conductor turned out the way we wanted with all our planning and designing shown when a user grabbed the wii mote and nun chuck they could control the song and conduct how they wanted too.

Donimo Music

2008-10-05T23:43:00.000-07:00

INTERACTIVE SOUND STUDIO
DOMINO MUSIC

Students: Bochen Wei (307132668)
Yurong Xia (308144880)

1. Introduction
The project is about creating a musical instrument that combines with playing dominos game and creating interactive sound together. The project is called “domino music”, that is an interactive sound game project. The sound or music is being created when player plays the dominos. Player can achieve pleasure and satisfaction by variety forms of setting up the dominos. Although the domino music is a simple sound instrument, it is possible to create melody with understanding of regular system. The purpose of the project is to give some ideas to improve the creativity and interactivity of toys or physical objects. The intended user can improve their imagination with digital techniques.

2. Influential works
Domino Music was inspired by playing domino game itself. However, we would like to combine playing dominos and creating interactive sound together. Making playing dominos is more interesting, creative and interactive.

3. Intended user experience
The intended users of the domino music are to face to masses and public from different ages and different countries. The domino music could be initiatory game of music for children. It also could be a simple instrument for adults to be relaxed in their leisure time. Different cultures and different countries’ people also can enjoy with it because it only involves domino game and sound or music. In addition, if the users who have some sounds knowledge, they are also able to create varies short melodies when they are playing this game. The domino music project also provides the feature of interesting for the people who have not musical knowledge, that they still achieve the satisfaction by the musical dominos.

4. Design motivation
The original idea comes from my working experience. I work in a toy warehouse as a packer. I pack more than one hundred different kinds of toys such as trucks, planes, and sailboats etc every week, and some planes, which are more than 1.5 meters length, are really big and look very cool. However, most of those toys are just remote control toys. From my point of view, they are lack of interactivity and creativity to players. So, I would like to combine creating interactive sound and playing toys together and make toys to be more interactive and creative to players. Initially, I had two ideas one is playing a helicopter and another one is playing dominos. Finally, after discussing with my partner, we decide to choose dominos to fulfill our idea. This is because playing a remote control helicopter is not very easy and need large space.

5. Technical method

Hardware requirement
For this project, the hardware requirement is not hard to be achieved. We just need a rectangle table and bind a “T” shape frame, which is used for hanging webcam, on one table leg. The webcam should be above the table face vertically and look at the table face. We also need a small mat, this is used for reduce noise when players knock down dominos. This is also why we cannot put webcam under the table. If we put webcam under the table, we need a glass table face. However, knocking down dominos on glass surface is very noise. We use 48 dominos in our project, and we divide them into 8 groups. Each group has an unique color and function.

Software requirement
We use MAX and reacTIVision to create this project. Firstly, “Max is a graphical music programming environment for people who have hit the limits of the usual sequencer and voicing programs for MIDI equipment.” (Miller Puckette, Max reference manual, 1988). So, we do all the programming work in MAX. Secondly, reacTIVision is an open source, which can be used in cross-platform, and tracks of fiducial markers attached onto physical objects, as well as for multi-touch finger tracking. In our project, we just use the tracking fiducial markers function. So, we attach 48 markers (from ID 0 to ID 47) on 48 dominos. Each domino matches one unique ID number.

Programming
We build two patchers for our project. One is used for getting the ID, which we want only from camera, and another one is used for generating sound by different ID. In the first patcher (See figure TuioClient), we want to get four kinds of ID, namely, general ID, ID=0 only, ID=45 only, and ID from 22-28 only. The ID 0 and ID 45 are also tracked position X and position Y, which can be further used.

(Figure: TuioClient)

The second patcher includes two parts namely playing MIDI notes and playing sound (See figure Playingnotes, Playingsound). From ID 1 to ID 21 are used for generating single MIDI notes, which means that camera sees one ID once then generate one MIDI note once. From ID 22 TO ID 28 are used for generating continuously MIDI notes, which means if players shake a domino a little bit, it will continuously generate the same MIDI notes. So, generating single MIDI notes is based on adding a domino under camera view, and generating continuously MIDI notes is based on update the info of a domino even if just lightly shaking. The position X of domino 0 is used for controlling the different timbres of sound. We give players 20 timbres of sound. So, players can choose the timbre they want by changing the position X of domino 0. The position Y of domino 0 is used for controlling the velocity of MIDI note. So, players can adjust the velocity by changing the position Y of domino 0.

(Figure: Playingnotes)

In the playing sound part, we use 15 buffers to store 15 different sounds, namely 5 drum sounds, 5 guitar sounds and 5 mix sounds, and then matching an ID to one sound. Furthermore, we use the position X of domino 45 to control the speed of all the sound at the same time and position Y of domino 45 to control the volume of all the sound at the same time. This means players cannot change speed and volume individually. If they change speed and volume, they change the speed and volume of all the sound. The domino 46 is used for changing all the sound back to normal speed. This is because sometime changing speed of sounds could make noise. Finally, domino 47 is used for stop all the sounds. So, when the players do something wrong or make too much noise, they can use domino 47 to stop all the sound quickly.

(Figure: Playingsound)

6. Resources/references
http://reactable.iua.upf.edu/?software
http://cycling74.com/twiki/bin/view/ResourceGuide/InterestingWork
http://cycling74.com/products/max5

Sonic Hockey - Paul Montgomery

2008-10-05T21:03:00.001-07:00

Description

The beginning of our project began with brainstorming amongst our group. After several alterations to the original concept, a refined and interesting interactive sound project was realised. Sonic hockey was based on the concept of air hockey. However, the most interesting difference was the increased friction of the sliding puck to the table, allowing the sounds being generated through colour tracking to mimic or allude to the idea of friction. Sounds were generated by using a camera to track the location of the puck. Depending on the location of the puck, a certain section of a designated sound file is played back at a certain speed and direction. The results, which were dependent on the original sound file, were varied but had a grainy/noisiness to them which helped reinforce the idea of friction. There were also goals at each end of the table. When one user scored a goal he/she would hear a celebratory sound clip. The technical method of these processes are mentioned later in this paper.

Influential Works

Obviously the most influential work on our game was the original air hockey game. This was the basis for our idea. A work that is interactive and influential to our project is the tangible user interface of reacTable. Reactable works by moving and rotating physical objects on a multi-touch tabletop. Each object represents a part of a modular synthesizer and users can therefore create complex soundscapes with the instrument.

A game called sonic pong was also investigated by our group. Sonic pong is basically the retro videogame “Pong” with one twist. Instead of the input medium being a keyboard, joystick or game controller, the user simply makes a noise into a microphone and therefore moves the bat to defend his/her goal. We found this to be an interesting parallel with our idea as it was basically the opposite of what we wanted to do (i.e. sound as input vs. sound as output). Sonic pong also gave us the idea to treat “Sonic Pong” more like a videogame in the sound design ideas.

Design Motivation

The motivation for our design began with a very simple concept. We wanted to make an interactive sound project where a user of any level of experience or knowledge would be able to interact with an interface and create sound without necessarily understanding how it was created. This idea lead us to the idea of drawing. We initially wanted to use a stylus and pad to input information and somehow generate sound from there. We soon realised this didn’t have the excitement factor that we were looking for so eventually we decided that we wanted the project to be a game which sound output would be a byproduct of. The idea of air hockey was born. We would also be capable of using some of the drawing technical concepts we had been considering on the new hockey design.

Another big factor that largely determined what was desirable and practical was the time constraint. Interactive sound studio students only had one week to complete the project, therefore works of a considerable scale were not possible

Technical Method

There were two main areas to the technical design of our hockey table. These were colour tracking for the location of the puck and the generating of the main sounds and the second was the contact microphone used to register when a goal was scored.

These methods were not our first ideas though. We originally wanted to use a Bluetooth mouse inside a specially made puck to track it’s movement, however this would have been too big and bulky and technically problematic. We also thought about using a wii remote to register a goal but we eventually decided that it would be too difficult to ensure and positive result every time for a goal.

The way the colour tracking worked was by placing a mini DV camera above the table, framing it as well as possible, and placing a coloured disc within the frame. The DV camera was connected to a laptop running Max/Msp. “jit.qt.grab” was used as a base to received the camera input. In Max/Msp the disc colour was defined and tracked with an object called “Suckah”. The “jit.findbounds” object was finally used to track the location of the puck. We used this information to track the x and y locations of puck. We simply applied a sound file to the x coordinate. The sound would then play back at various speeds and directions depending on what the users were doing to he puck and how hard they hit it, etc. We considered using the y axis as a means of controlling a filter to further manipulate the sounds, however, this proved to be too aurally complex and unpleasing.

The contact microphone concept was fairly straight forward. We placed piezoelectric microphones inside the goal boxes (see photo). This means that any vibrations caused by the puck entering the goal would trigger a sound to play. The basic concept was a simple and straightforward solution to a complex problem, however finding correct levels and thresholds was a tedious task as we had to distinguish between vibrations within the table boundaries and vibrations that actually were goals. This fine balance was eventually attained. Several “sfplay~” objects were created in the Max/Msp patch, and a “random” object allowed us to play back different victory sounds for each goal.

User Experience/Usability Considerations

The primary goal of our design was to entertain the user. We wanted the user to have fun and also be interested and curious about the workings of the system. By making the project a game that was familiar to most people, but different enough to be exciting our group ensured that anyone who used or watched the system being used would be entertained. We also wanted to limit the users understanding of how the system worked, therefore forcing them to consider solutions. We did this by concealing the camera in a box hidden above the table. This resulted in many users questioning is a sensor was under the table or how it worked.

There were several problems in terms of usability that were involved in this project. The most crucial of which was the side boundaries of the table providing enough “bounce” to successfully replicate traditional air hockey. We solved this problem by stretching guitar straps along the sides, creating an elastic edge to the table.

Conclusion

The development of this interactive system has been most rewarding and enjoyable. The end results have been more impressive than I had originally imagined. However, all works were working prototypes and therefore could all have been expanded upon.

Resources/References

http://cxycling74.com/twiki/bin/view/ResourceGuide/InterestingWork http://reactable.iua.upf.edu/?software/

http://www.milmoe.com/artprojects/sonicpong/sonicpong.htm

http://www.jocoolmultimedia.ca/joseph/bord-du-fleuve/programmation.htm

Wii TAIKO by Deon Rowe, Camilo Castillo and Dani Awad

2008-10-05T20:57:00.000-07:00

Overview/Description

Our project was the wii Taiko a somewhat virtual drum interface using the motion sensing and push button capabilities of the wii remote. Our original idea was an interface of five drums with a built in Infra-red sensor or sensors on the skin of each drum, the interface would be based on a Japanese five drum Shime-Daiko setup from lightest to heaviest. The drums would be laid out around the user and using 2 wii-remotes as bachi or drumsticks the user would swing in the direction of each drum, the accelerometer in the wii alludes to how hard the drum is struck and the infra-reds on the skins determining directionality and which drum is selected. Half way through preparation we realized that the sensors weren’t working properly, the wii remote couldn’t pick them up at a sufficient rate this was a significant blow but we focused on the motion sensing capabilities of the wii and the ability to assign the different buttons such as the “A” button of the wii to select different drum sounds and the trigger button to select a shell shot. This lead to the interface being reduced to 4 drums by necessity of a physical nature and also made the physical representations of the drums technically redundant, but they would still play a major role in presentation and user interaction.

Related and influential works

Wii TAIKO was inspired by the upcoming Nintendo game “Wii music”. In this game the user can play a wide variety of instruments by interacting with the Wii remote, the game features a drumkit, where the user can play “air drums” and the different drum sounds are triggered when the remote is being swung.

We wanted to create a drum simulator using the Wii remote technology but we didn’t want to use the idea mentioned above, that’s why we came up with the implementation of TAIKO drum samples instead of a conventional drumkit.

Different percussive virtual instruments like a standard air drum simulator which has LED’s on the tips of the drum sticks, which is being picked up by cameras were explored for our project. Another related video on which can be found on youtube involves a single virtual Taiko drum being sounded using wii controllers. There are also taiko drum arcade games that use a drum pad type interface, but we didn’t really want to head into the direction of an interactive game because we thought that this would be way beyond our capabilities and the fact that we only had a limited amount of time. Videos of modern Taiko ensemble drumming also played a part in the sound and look of the drums. The main influence of the choice of Taiko came from some of the group’s great interest in Japanese culture and wii games. The initial idea was to have air drums mimicking a standard kit; this was then changed to taiko as we found it more interesting, different and would be more accessible to people who really can’t play a standard kit but would still like to bang away and make noise.

Design motivation

The main motivation for the drum interface came about after looking through lots of photos of taiko drums, skins and Shime-Daiko setups. We arranged the drums from lightest to heaviest like a Shime-Daiko but the actual look of the drums is more of a mongrel hybrid of different elements of taiko, you see traditionally a Shime-Daiko drum is tethered with ropes and straps and pulled back at the hem, while a Byou-uchi daiko has the head nailed unto the body, we found that although it might probably be time consuming we opted for the pinned down skin as it was straighforward and visually striking. The size of the drums was still based on the Shime snare size. Taiko drum heads are mostly made out of leather, we couldn’t find any without wasting a lot of time so we used sheets of canvas.

A Taiko drum is a simple construct, timber, leather and steel. We used this philosophy to build ours. Chip wood was used for the base cut down into squares then octagons just for a diferent shape, then sanded down. Standard steel drum heads were screwed into the wooden base creating a circular head and giving the drum a decent shape, after this the canvas was cut out and arranged, and then pulled tightly over the base and pinned down using golden pins that we purchased from the news agency (at a price where you’d actually think they were gold). Black cloth was then used to line the back of the drum and hide any visual scarring the same that wa used to hide the tables. We then used black ink a brush and the steady hand of a friend for the caligraphy on the drum skins reading from left to right Byou-uchi daiko, as we used steel pins to nail the head down. Using a left over bamboo stalk it was cut and turned into a pedestal with a top and bottom base where the user can place the wii controllers after playing, one on the left and the other on the right so they do not get mixed up.

From an audio design stand point the samples we used were sourced from Sean Beeson’s Tsaiko Drum library, we thank him for his charity and free samples, we picked the most suitable ones where we felt they had the right tone and intensity and used them with respective shell shots for each where they were slightly sped up in max to achive a tighter sound. A customised reverb patch was used in max to give the drums atmosphere, air and help them translate better over the exhibition noise. A Japanese taiko instrumental interlude that coud be activated by using the home button on the wii controller was also added this allows the user to play along to something that is suitable ambient and easy to follow, but at the exhibition it was a shame that it was drowned out abit by by both our drums and the overall noise.

Technical method:

There were 2 aspects to our final design, one was the aesthetics and the second one was the actual programming.

We wanted to bring the Taiko experience to people who don’t have too much knowledge about this art. The aesthetic side of the project was based on the following idea:

The 4 drum plates were made from a wooden base, a real drum kit rim was drilled into the wooden plates, and the whole structure was then covered by canvas and stretched by pinning the sides with thumbtacks.

Japanese Characters were drawn in each drum. Byou-uchi daiko (鋲撃ち太鼓), which is the name of the type of Taiko drum, which has its drum skin nailed to the body of the drum.

The main idea was to integrate infrared LEDs inside each drum, this will provide us with accurate spatial positioning so the wii remote would know which drums was being hit. Unfortunately after 2 days of design and testing, the infrared LEDs didn’t work as we spected so the idea had to be discarded.

All of the programming was written in Max/MSP based of the patch AKA.Wiiremote. Within the programming we tried to make the experience as real as possible and as achievable as possible within the time limits.

The Human interface we used for this project was the Nintendo wii remote

Using the patch created by Masayuki Akamatsu called aka.wiiremote we were able to obtain the information stream from the wii remote, X,Y axis, button triggers and motion information.

The original aka wiiremote patch was modified to work with 2 Wii remotes simultaneously and specific button information was set as outputs.

Scaled versions of the X axis values and acceleration data were used to trigger samples. The samples would trigger when the value was past a certain threshold sending a bang to start the sample playback.

This signal also triggered a vibration feedback in the controller when ever a sample was played, this was a nice feel for the user to help connect them to the sounds being created.

The playback volume was also calculated on how hard the user swung the remote, in an attempt of making it more realistic and adding dynamics to the system. Converting the maximum g-force value from x to DB to then modify the amplitude of the triggered sample using line.

To create the 4 different drums triggers a button combination system was devised. This opened switches and changed the path of the original triggers.

For each of the four drums there are two samples available a normal hit and a shell hit (the out side of the drum). Controlled by the ButtonA routed from aka.wiiremote input you can select the other drum. Because we are using two different wii remotes we could get a combination of 2 drums and 2 shell sounds per remote.

From this picture you can see the samples Hit3 and hit4 (loaded in the buffer hit3 and hit4) are triggered from the switch under r ButtonB.

All of this was then sent to a multiplier and through a basic reverb and send to the output audio card.

The reverb was a patch built in to Max/MSP called reverberator.

Deon was in charge of the overall programming, Dani and Camilo helped with the testing of the program and gave suggestions and advice on how to overcome certain aspects that weren’t working

Since many of the features of the first prototype didn’t work and we didn’t have enough time to test more infrared LEDs we helped Deon to come up with ways around it, we suggested the implementation of the button combination and we gave Deon a hand on the programming as much as we could. Dani and Camilo were in charge of constructing the drum plates and the general interface.

Equipment:

Mac Book Pro with MAX/MSP

Motu amplifier

2 x YAMAHA active speakers

1 x Beyer Dynamic Active speaker

2 x Nintendo Wii remotes

User Experience, usability considerations and future development:

We intended the interface to be pleasant to look at, as well as, to have a use for the programming. It works as a spatial reference for people, so they know what and where to swing the Wii remote. We thought of the idea of having pressure sensors in each drum so they can be triggered by a hit, but we discarded the idea as it would defeat the purpose of our virtual Taiko experience.

We tried to make our project as user friendly as possible, we thought about how to make it ergonomic, that’s why we chose the angles we used on the drum plates. We put a lot of effort on the aesthetics of the project as we wanted people to feel comfortable with it and also we wanted to enjoy it more.

Due to the fact that not many people have heard about Taiko and also not knowing what age group was going to interact with our interface, we were nervous about the success of our project.

We provided easy to read instructions and also explained the interface to every person who tried it, we decided to provide a brief overview of the history of Taiko so that the users could learn about what they were going to experience.

We were excited about our project when we saw that the users learnt the different button combinations quickly and played for long periods of time, also some of the users came back to play more with our interface. We also saw people from different age groups enjoying the experience and we got positive feedback from them.

These made us realize the potential of our interface, we could expand it to more than one player as some of the users decided to join and play in groups of 2 people, also we could add more drums and even create a small tutorial which will engage the user a lot more.

Since we succeeded in our effort to bring a new experience to the user, we can think of many ways to develop more features for our interface, these may include:

- Infrared sensing, for more accurate direction and it will allow for more drums and more players
- Video interface, for the tutorial
- LED Lights
- Individual speakers for each drums

Special Thanks

We would like to thank Sam Ferguson and Kirsty Komuso who helped us all the way, from programming to the gear we used.

Thanks to Sean Beeson from tsaiko drums who provided us with the taiko samples we used

Thanks to Rita Tse who helped us with the character writing

And thanks to all the people who participated and gave us feedback

Resources/References:

1)[http://www.maxobjects.com/?v=authors&id_auteur=16&PHPSESSID=27e365826c1a079e071da8569ede8926]

2) www.wikipedia.com

3) http://www.enhancentertainment.com.au/taikoz.htm

4) http://au.wii.ign.com/objects/827/827335.html

5) Max/MSP programming advice by Sam Ferguson

6) www.tsaikodrums.com

2008-10-05T18:36:00.000-07:00

Interactive Narrative
Myles Blasonato, Michael Moretti, James Brown

Interactive Narrative Project Overview

Interactive Narrative is a story driven game experience where the player is the main character in a story in which they can interact with via gestures and voice to change the experience. The whole game is to be experienced with a blindfold. It is a sound only game. This allows people to create their own image of what the environment looks like using the sound as their only clue. This also allows for a seamless transition from cut-scene to game-play because of no visual cues for the player to notice that the game has come out of a cut-scene and into game-play. The player uses the Wiimote to do gestures to advance the story. These gestures range from opening doors, answering a telephone to shooting a mobster boss.

We used sound in a way that created atmosphere and that paid attention to small details to give a sense of being there in the environment.

Related and Influential Works

Interactive Narrative was influenced by a few different works. Virtual Barber Shop was probably the biggest influence. The creator used binaural recordings for the piece. It worked really great when I tried it and I thought to myself that something like this with interaction could be one of the most immersive experiences ever. I have linked the piece for everyone to check out, it’s on YouTube. Use headphones for it.

Other influences included point and click adventure games such as Monkey Island and Day of the Tentacle. There were purely narrative based and used puzzle solving as the core game-play mechanic. Audio stories and stories that are read to you were also another influence and probably the experience that I wanted to enhance the most. We all remember out parents reading us a story, how immersive was it. I wanted to re-create that and enhance the experience by adding interaction that felt natural. Using any other control would not have felt natural; I felt the Wiimote was the best input device for the experience. Of course the Wiimote and nearly all of the Wii games were influences in some way. I wanted to create an experience using the Wiimote just like the creators of the Wii games wanted to. Thanks Nintendo.

Design Motivation

My motivation for the design was the single goal of providing and immersive experience that allowed the user to be the actor in a story and change story by interacting with it. This is the goal we Team Bloodshed set at the start of the project and every decision we made was to help achieve that goal the best we could in a week. Of course there were constraints but dealing with those constraints in a way that helped achieve the goal was also a motivation in its self. Problem solving was another main motivation for me personally.

Technical Method

Our team used the aka.wiiremote object for Max/MSP to provide support for the Wiimote input. This was great but given the problem that Max doesn’t handle game related programming so well, I opted to use JavaScript which Max/MSP supports right from the get go to be used for all in game/interaction based logic, the JavaScript is used as a plug-in basically. All you have to do is pass-in the input which is great because we can actually go back into our game and use a different input device other than the Wiimote if wanted to. The flow was the most important part for our game, once we worked out the game-play flow we needed to find a way to combine Max/MSP programming with JavaScript due to Max/MSP’s sound object/class which JavaScript doesn’t have. We worked out the flow to be the following: Play sound, get interaction, play sound, get interaction. Keeping this in mind, I made sure I created a programming module that could be re-used and modified. The programming is so modulated that in fact, we could replace all the sound files and change the interactions and have another game of the same type up and running in 3 days max. Each sequence included a Wii input patch, a play sound patch and a JavaScript file for interaction. Each sequence also had a gate that was used to turn segments off which was great for debugging because we didn’t have to sit through the whole game to test the end of the game.

As you can see from the above diagram, each scene/sequence is a separate module. You will also notice how the Wii Input is passed into the JavaScript object and depending on what happens in there one of 3 sounds will play or maybe the whole 2 together. The JavaScript contains tests, functions a timer all of which was programmed just for this game.
The sound patcher was a little different. Within the sound patcher a bank of sounds was present were if a certain number was received the sound with that corresponding number would be played. We then pass that sound information to a loud speaker with a volume control so we are able to turn the volume up or down.

Sound production was all done in Pro-Tools using sound libraries and original sound that was recorded and created especially for the game, these original sounds included music, sound effects and vocals (voice acting). Little details were also included within the sound to give the players a better understand of what the environment was around them. Panning was also used to give direction to certain sounds. The music created was also great and was made to the style of 1940’s Mafia with a futuristic twist to it.

The story was also something we wanted to tailor to most people, that’s why we kept with the drama theme and we made sure that the gender of the character was never given away. We also didn’t want to detract from the immersion of the experience when teaching the user how to play which is why we opted to include the reason for the input device in the story and also why you have to learn how to use it and how to use it as well.

User experience / usability considerations

The intended user experience and core experience of the game was to make the player feel like the actor in a story. We designed everything around that core experience and made sure that playing this game felt immersive and engaging. Every aspect of the game from the interactions, to the story and to the sound were all designed around making the player feel like the main character in a narrative that they can alter. That’s where the name Interactive Narrative was deprived. Of course some usability considerations we had was to have full 5.1 surround sound incorporated into the game, more interactions, a longer story, a user friendly way of starting the game and voice recognition. The Wii Nun-Chuck was also something that we had considered to have for navigation but stayed away from it because there is just something that is not immersive about pushing a joystick to move.

Special Thanks

Firstly I would love to thank Michael for his contributions to design and creating a great story that fit the design of game extremely well. There’s nothing like a story about a guy losing his hand and having the Wiimote as the main user input device.

Huge raps to James Brown for creating the best sounds and music ever, thank your friends for me to James because there voice acting was fantastic. All your sound design skills, and your ability to create a space using only sound is inspirational and is something that can enhance any experience.

Thanks to Sam Ferguson for putting up with all my questions about Max/MSP and for helping me solve some problems I had with the Max/MSP. Also thanks for letting us borrow your laptop for the event.

Thanks to Kirsty Komuso for setting up the exhibition and for providing great feedback on the game.

Lastly thanks to all the other students who all created fantastic pieces. You should all be extremely proud of what you have done in such a small timeframe. All the best for the future.

Resources / References

Interactive-Sound Blog
http://interactive-sound.blogspot.com/

Virtual Hair Cut
http://au.youtube.com/watch?v=IUDTlvagjJA&feature=related

Wii Official Website
http://wii.com/

Sonic Hockey - Ka Ho Chan

2008-10-05T12:03:00.000-07:00

Project Description:

The “Sonic Hockey” is an interactive game with audio output. The main interactive element is the hockey puck. The hockey puck is associated with frictional sound so when it is moving on the hockey table sound is generated. And also, sound is also associated for the scoring so when each time one of the players get score, cheerful sound can be heard from the winner’s side speaker.

Figure 1.1 - Sonic Hockey

Figure 1.2 - Sonic Hockey game play

Inspiration and Design Motivation:

Sonic Hockey was inspired and developed based on the real Air Hockey game. The motivation for designing Sonic Hockey was very simple, we just wanted to create a simple and easy to play game that suitable for everyone. For playing Sonic Hockey, players don’t need to learn before playing and they don’t need any musical knowledge, the only concern for playing the Sonic Hockey is to catch the puck and shoot for score, and at the same time the players are also manipulating sound. Influential work also includes “Sonic Pong” (http://www.milmoe.com/artprojects/sonicpong/sonicpong.htm).

User Experience:

The user experience was one of the most important factors before we decided to create Sonic Hockey. Sonic Hockey is a very simple game; we want everyone can walk to the hockey table and then enjoying playing.

Usability Considerations:
Basically people would compare Sonic Hockey with the real Air Hockey, so we avoid using the word Air Hockey for our project and emphasized that this is “Sonic Hockey”, an interactive sound generation game. From the begging of building our hockey table, problems we faced included resistance between the hockey puck and the table surface, bouncing as well as the design of the goal areas. We tested different materials before building the hockey table, finally we found that attached guitar strips along the sides of the table provide best bouncing result, and stuck laminate underneath the hockey puck can reduce much of the resistance between the hockey puck and the table surface. For the goal areas, we built two little boxes to catch the falling hockey puck for both players’ sides.

Figure 1.3 - Guitar Strips along table sides for bouncing effect

Figure 1.4 -Goal-Box

Technical Method:

The main interactive element for Sonic Hockey is the hockey puck and the goal-boxes. We had though about to build a custom-made hockey puck by using a Bluetooth mouse. However Bluetooth mouse is in fact too big and heavy, make it hard to slide on our DIY hockey table. Therefore finally we decided to use camera-tracking method.

For the goal areas, originally we decided to put wii remotes inside the goal-boxes for each side, but after a few tests we found that it is difficult to let the hockey puck to trigger the buttons of the wii remotes to produce sound. We also though about using the motion sensors of the wii remotes to produce sound, but it was unsuccessful because the wii remotes is quite heavy and hard to fix into a particular location inside the goal-box. Finally we tried to put contact microphones inside the goal-boxes.

Contact microphone is not as sensitive as normal microphone; in order let the contact microphone to capture the vibration in air, it has to be attached to some sort of surface. We run several test with the contact microphones and made adjustments for their location inside the goal-boxes and also make adjustments for the programming in Max/Msp, finally we were succeed so when each time the hockey puck fall into the goal-box the contact microphone would detect the vibration of the wooden goal box and then the associated cheerful sound for scoring of the game would be produced.

Figure 1.5 - Contact Mic. hidden in Goal-Box

Programming:

The programming of Sonic Hockey was done in Max/Msp, we tangled a mini DV camera on the ceiling to film the whole hockey table, by defining the color of the hockey puck to track its motion. We could also track the motion of the hockey bats as well, however under consideration we decided just to use the hocky pucks to produce sound because if we track the hockey puck and the bats at the same time, sound would never stop and the sonic space would become complex as well. If just tracking the hockey puck, sound is associated with it only. At a result when the hockey puck is sliding on the table (inside the camera range) sound is produced, whereas the hockey puck fall to the goal-box or fall out from the hockey table (outside the camera range) sound couldn’t be heard.

Figure 1.6 - Color Tracking Patcher Max/Msp

Figure 1.7 - Audio Playback Sub-patcher for the Hockey Puck's sound

We use the “jit.qt.grab” object to capture the video form the mini DV camera in Max/Msp, using the “suckah” object to detect the colour value of the hockey puck, then use the “jit.findbounds” object to track the motion of the hockey puck. By routing the data of the hockey puck movement to x-y position, we could associate sound and other audio effect for the hockey puck for both horizontal and vertical position. Although we were able to assign different things for both x and y positions for the hockey puck, in this situation we just simply assigned sound for the horizontal position of the hockey puck by using the “play~” object in Max/Msp and leave the vertical position to do nothing. Because we tried add different audio effect for the hockey puck’s y position as well (such as reverb and filtering) but the sonic space became complex and the fractional sound for the hockey puck was not as good as expected if there are too much things going on at the same time, so finally we decided to associate sound for the horizontal position of the hockey puck only.

Figure 1.8 - Contact Microphones input signal patcher Max/Msp

Figure 1.9 - Random value of audio effects playback control sub-patcher

Figure 2.0 - Sound effects playback sub-patcher for Goal Areas

For the contact microphones, we use six “sfplay~” objects in the Max/Msp patcher, and a “random” object in the patcher for random value creation so each time when the hockey puck fall into one of the goal-boxes, one of the six pre-designed audio files is triggered randomly so the sound is unpredictable for the players when they get score. We run many tests for the contact microphones to make sure they only detect vibration at a certain level to avoid sound generation by accidents (such as players touch the goal-boxes).

Conclusion:

We are happy with the final outcome of Sonic Hockey because people were so curious in the exhibition. We hide the camera on the ceiling so people didn’t know why there was sound with the hockey puck movement. The same question they asked is why there is sound when the hockey puck is moving and why there is no sound when the hockey is not moving. This was our expectation because we wanted people excited by the fractional sound as well when they were enjoying the game. Moreover, people were happy and shocked by the cheerful sound effect when they got score because they couldn’t see the hidden contact microphones inside the goal-boxes as well.

Reference: http://www.milmoe.com/artprojects/sonicpong/sonicpong.htm
http://www.jocoolmultimedia.ca/joseph/bord-du-fleuve/programmation.htm

Rubik's Studio

2008-10-05T08:34:00.000-07:00

Piers Gilbertson & Dan Gallard

Brief overview & description

The aim of The Rubik’s Studio was to create an original audio interface from an existing product. The Rubik’s Cube seemed a perfect candidate, as there is a lot of scope for complexity within a relatively simple idea.

Each coloured face of the Cube represents an original audio composition in its entirety. Each tile of the cube represents a segment, or sample, from one of these audio files. As the cube is mixed up, these samples align to produce new and relatively limitless variations, (somewhere in the region of 48,000,000), of the original samples. These can be mixed and reordered ‘on-the –fly’ for live performance or to generate unique sounds/ combinations to be added to different instruments/media etc as a compositional element.

The real strength of the Rubik’s Studio comes from its versatility. It is very easy to make each colour represent different audio samples, giving it limitless applications in terms of composition, performance, sound effects, and the creation of completely original sounds based on the interaction of its tiles.

To hear the original samples the user must solve each face of the cube – giving the user an intellectual challenge and an element of mystery to further engage the user.

Design motivation

The idea behind the Rubik’s Studio – to make a Rubik’s Cube into an interactive sample generator with the help of reacTIVision fiducial markers and software – remained relatively unchanged throughout the project. The methodology and intended approach to the project was modified however – based on advice from Sam and Kirsty - in terms of simplification and specification, following our initial presentation.

Due to time considerations, we decided to focus on only one cube as opposed to the two in our original brief. The second cube was to be an effect cube, adjusting the parameters of the first cube in terms of reverberation, EQ, delay etc.

The second modification was again due to time constraints. Our original idea was to work collaboratively on each part of the process together. Upon beginning the project we decided that Dan (due to his stellar marks in Digital Audio last semester) would be in charge of programming the necessary code, whilst Piers (having seen Home Improvement once), would be in responsible for designing/building the interface and composing/recording the audio material.

Technical method

The Cube:

There were several considerations and goals in making the Rubik’s Studio Cube. We wanted each face to be coloured as per the original Rubik’s Cube for user familiarity; to maintain the original idea behind the Rubik’s Cube; and to trigger the audio sample each colour represents in its original form.

Furthermore, we then assigned a ReacTIVision fiducial marker to each individual tile of the Rubik’s cube. These tiles represent a 1/8^th section (sample) of the audio file that colour represents.

These individual tiles are read and passed to the ReacTIVision software via a live video feed. The software turns the incoming video signal into monochrome picture and from there identifies and tracks each marker.

Because of this, we had to determine, by trial and error, what colours we could use so that the software would still recognise the fiducials. We created a template in Adobe Illustrator based on the tile parameters and tested numerous color combinations in terms of aesthetics, opacity and contrast to black and white. Figure 1 shows the final colours we used. We then printed the tiles onto stickers, which were cut out and used to replace the originals.

Figure 1: Fiducial marker stickers for the Rubik’s cube.

Picture 1: Modified Rubik's cube with Reactivision fiducial markers. [6]

Picture 2: Fiducial markers being read by thr ReactiVision software. [6]

The Podium:

There were numerous factors to be considered in designing and building the podium:

Structural integrity: As there was expensive equipment to be contained within the podium it had to be sturdy. It also reduces your credibility if your interface collapses mid-exhibition. To this end the base of the podium was constructed out of very dense, heavy, though rather unattractive wood.

The Tardis: The podium had to be big enough to house a camera and tripod, 3 lights, an audio interface and potentially a laptop whilst still being small enough to be easily portable.

Picture 3: The Podium open showing DV camera and lighting. [6]

Internal accessibility and ventilation: We had to be able to easily access the lighting and camera equipment within the podium to fine-tune them. To this end the sides were not boarded up. This feature also solved the problem of ventilation. The lighting created quite a lot of heat, and hence a potential fire hazard, so we needed to have good consistent airflow.

Aesthetically pleasing: We wanted the interface to appear very simple, concealing the complexity of its internal workings.

The table-top frame: was made from Australian Cedar Pine, a very attractive and sturdy Native tree. We wanted it to look great in order to attract users.

Picture 4: The Podium table-top from above with the cube. [7]

The table-top: was made from 2 layers of plastic. The first layer was clear plastic so that the camera could read the fiducials through it. We tested with frosted plastic but settled on clear plastic to avoid any technical problems. On top of the clear pane was a translucent blue pane with a rectangle laser cut into the centre to define the cameras viewing parameter. This blue caused the whole interface to glow when lit from beneath making it look ‘futuristic’ and concealing the camera beneath. We used tracing paper over between the panes to further conceal the inner workings of the podium.

The curtain: Matt Storey kindly donated some dense studio curtaining which we used to surround the podium, hiding its inner workings whilst still maintaining accessibility and ventilation.

Inexpensive: the whole interface (excluding the equipment below) cost less than $60.

We would like to thank the wonderful USYD Architecture workshop team for their generosity with their time and expertise, and Matt Storey for donating the curtain.

Equipment:

The following is a list of the technical elements used to create this interface:

MacBook laptop

Digital video camera (640x480, 25fps) and tripod (connected via firewire)

3 lamps (a 12" mini-fluroescent, a 10W compact fluroescent and a 75W incandescent)

USB audio / midi interface – in this case an Edirol UA-25

2 x Yamaha MSP active studio monitors

Audio:

Having decided that the Rubik’s Studio would function initially as a sequencer – i.e. that it would play portions of audio samples in a given sequence – we set our compositional parameters to 120bpm 4/4 timing. This was merely to improve the chances of creating audio that would ‘fit’ into a sequencer and still sound good.

We layered 6 different audio tracks (1 for each colour face), which combined to create a complex rhythm or melody. These tracks are only heard in their original form when a whole colour side is presented.

We ended up with 6 different cube ‘versions’ (made up of 6 audio tracks each – 36 in total), which we could switch between with the click of a button:

Drunk Percussionist – Mainly syncopated percussion elements

Hippy bells – as above

Jungle Drums – as above

“Thank you for coming and have a pleasant evening” – vocoder messages that can only be heard when a colour is deciphered

“The Amazing Rubik’s Studio, insert advertisement here”

SFX cube – 6 different sound effects

Audio was produced in Logic Pro using Stylus RMX and Sampletank 2.2.

Max/MSP Programming:

Figure 2 shows several audio sample presets we had which loaded a different 4-second loop into a buffer for each side of the cube. This was done through the loadSamples subpatch shown in Figure 3.

Figure 2: Various Sample Presets.

Figure 3: loadSamples subpatch.

Figure 4 shows the patch used to take incoming messages from Reactivision. The TuioClient object is supplied with the Reactivision software for integration into Max/MSP. This object outputs the following TUIO messages:

addObject session_id fiducial_id

updateObject session_id fiducial_id xpos ypos angle xspeed yspeed rspeed maccel raccel

removeObject session_id fiducial_id

addCursor session_id

updateCursor session_id xpos ypos xspeed yspeed maccel

removeCursor session_id

Figure 4: Reactivision TuioClient.

Figure 5: Rubik's cube mapping

The only TUIO messages we required were the addObject fiducial_id and the removeObject fiducial_id. The route object was used to split the addObject and removeObject messages, and then the zl slice object was used to remove the session_id message, as this was not required. The remove message was then fed through a pipe object to delay it by 500ms so that the cube could be rotated and the audio would overlap. The add messages were then funneled into the Rubik’s cube mapping patch as shown in Figure 5. The select object then chose according to fiducial_id messages and sent a bang to a toggle, then a 1 to the associated send object. When the fiducial markers were removed, another bang would be sent to the relevant toggle (following the 500ms delay) and a 0 to the associated send. The send’s are patched via identically named receives located in the playBuffer patch (Figure 9).

Figure 6: Sequencer.

The sequencer uses the metro object to regulate the playback speed and then uses a counter object to step through. This information is sent into the play_samples subpatch (Figure 7) and then to the six playFace objects.

Figure 7: play_samples subpatch. [5]

The playFace objects control which part of the four second buffer is played and when. If all red squares are placed down on the podium, the whole four second loop stored in the red buffer would be played back. When the cube has been mixed up, the corresponding parts of the audio in the different buffers gets played allowing a mixing of the six audio samples. The play~ object plays the buffer contents according to the information received from the playFace patch. The minimum object automatically mixes the audio signals to prevent any digital distortion and clipping.

Figure 8: playFace patch. [5]

Figure 9: playBuffer patch. [5]

The playFace patch (Figure 8) takes one input argument to set which face of the cube it is associated to.

The playBuffer patch (Figure 9) has four input arguments (seen in Figure 8). The first two arguments set the receive object to the correct corresponding part of the Rubik’s cube which opens the gate. The second argument also sets what sequencer message that the patch responds to. Argument three sets the start time (in ms) of the buffer playback and argument four sets how long it is to be played. PlayBuffer also sets the adsr~ object which fades the volume at the beginning and end of the playback to eliminate any clicks in the audio.

User experience, usability considerations and possible future development:

Picture 5: The Exhibition. [6]

Having not been able to extensively test the interface before its unveiling, it was very interesting to observe how users reacted to the Rubik's Studio.

Everyone seemed initially fascinated and intrigued by the interface, and quickly began playing with the Cube and modifying the sounds.

However initially users weren’t spending long playing with the Cube, though all were fascinated to hear about its design. We began to realize that the audio tracks, once mixed, were sounding fragmented and ‘ bitsy’, so we sped up the playback rate and added a smoothing patch to the programme. Suddenly the Cube began to sound far more sonically interesting. It became possible to modify the Cube in real time making it possible for users to ‘jam’ with the Cube with far more satisfying results.

Users after this point became far more engaged with the interface. This was very encouraging, and with the addition of further audio we believe this could become a powerful and entertaining user interface.

Video 1: User interaction. [7]

Picture 6: User interaction. [7]

Picture 7: User interaction (trying to solve the Rubik's cube). [7]

Picture 8: Rubik's performance. [7]

Picture 9: Psuedo DJ Battle. [7]

Picture 10: Already a virtuoso [6]

Video 2: Virtuoso. [7]

It was interesting to observe also that after experimenting with the cubes sounds, they wanted to hear the original samples so began trying to solve the Cube, adding another level of engagement and satisfaction for the user.

This Cube represents a prototype of an idea. Future expansion on the interface could include:

Effects cubes (reverb etc)

Sampler cubes

Grain synth cubes

Audio sample banks for cube ‘personalities’.

Related and influential works

The Orchestra Explorer

BYORC (Bring Your Own Rubik's Cube) Music

Resources and references

http://reactable.iua.upf.edu/?software/

http://www.abstractmachine.net/blog/3/

http://cxycling74.com/twiki/bin/view/ResourceGuide/InterestingWork

Antonio Camurri, Corrado Canepa, Gualtiero Volpe, Active listening to a virtual orchestra through an expressive gestural interface: the orchestra explorer . New Interfaces For Musical Expression: Proceedings of the 7th international conference on New interfaces for musical expression, New York, New York. SESSION: Controllers and physical models. Pages: 56 – 61. ACM Publishing.(2007)

Max/MSP programming assistance by Sam Ferguson.

Photo's by Kirsty Komuso (Kirsty Beilharz).

Photo's and video's by Piers Gilbertson.

The Cranky Bear - Yosafat Gunawan

2008-10-05T06:43:00.000-07:00

Overview Description

The Cranky Bear is an interactive sound installation that let participant to tangibly interact with a toy teddy bear that could give different responses under different circumstances it experienced. When the bear put down (lying), the system will play lullaby song and snoring sound. When the bear is lifted up, the system will trigger sound that makes the bear apparently talking or ranting. When the bear is being shaken, the system will trigger shaking sound as well. If it is shaken vigorously, the system will trigger the angry sound of the bear. There is also a voice recognition system that can calm the bear and make him sleeping again. All of these actions create an interesting interactivity out of simple interaction.

All of these actions achieved by measuring movement and orientation from the motion sensor tucked inside the bear that communicates wirelessly with the computer system. The computer system determines actions should be triggered under different kind of specific circumstances.

At initial stage, we would like to create a horror-themed interaction by using the same teddy bear object. Later on, we revised our plan by characterized the bear and creating a unique interaction around the character of the bear itself. All we would like to achieve from this installation is creating a unique user-experience from interacting with toy teddy bear that has a mild character that quite different from ordinary children toys. We believed that if unique character can be displayed by an ordinary teddy bear, the results would be interesting.

Related or Influential Works

We were inspired from several talking toy dolls available on the market. These dolls could play some pre-recorded sounds when being triggered, such as pushed, squeezed, etc. For example, there is talking Barney doll, Bob the Builder doll, These kinds of interactions are very stereotyped with kids-friendly voices/sounds. Therefore, we would like to slightly alter this stereotype in this installation by creating a non kids-friendly sound/voices out of popular kids toy object.

Technical Method

For this installation, we take advantage from sophisticated motion sensor of Nintendo Wii remote. By using custom Max/MSP patch, the three-dimensional axis (X, Y and Z-axis) data from the remote can be read by the computer and used from useful purposes.
From several experiments, we figured out a set of axis ranges of the bear (with the remote tucked inside) that used to define various positions of the bear (i.e., lying position or seating position, etc.) and action being experienced by the bear (i.e., being shaken). Out of these specific axis readings, the patch will trigger other patch that initiate specific sound/voice playback. All of the music and voices used in this installation project is custom-made by one of our group members.

Max patch for detecting lying position of the teddy bear:

Max patch for detecting amount of movement of the teddy bear:

Max patch for triggering sound:

We implemented a voice recognition system as well. It is achieved by planted a wireless lapel microphone in the teddy bear’s right ear. There is other independent Max/MSP patch that recognizes human voices. When the system received a voice input, it will analyse it and recognized pre-determined words, such as “sleep”, “relax”, “quiet” and “calm”. If the system recognizes any of these words, it will trigger other sound.

Max patch of voice recognition:

User Experience/Usability Considerations

Most of the audiences enjoyed playing with the teddy bear. They enjoyed the shaking voice of the bear when it is being shaken and some quite surprised by the rather impolite response of the bear when being shaken vigorously. However, the cuteness of the teddy bear itself is a visual appeal that attracts audience’s attention at the first place. Sometimes the system experienced few latency problems while switching between positions/sounds. There is no transitional sound programmed while switching between sounds, therefore it is very rough when transitioning between each sound. There was a problem in voice recognition system as well. The system does not work properly, thus cannot constantly detect voice input.

One major problem of this installation is fixing the Wii remote inside the bear, so it will stay at the same place even being shaken extensively. We experienced a coincidence that the system cannot trigger specific sound because of the remote has displaced inside the bear.

Resources/References

aka.wiiremote - http://www.iamas.ac.jp/~aka/max/
aka.listen - http://www.iamas.ac.jp/~aka/max/

few examples of media documentation published on the blog

Approaching the Buddha.

2008-10-04T08:01:00.000-07:00

Approaching the Buddha: Interactive Sonic Space

Overview and Design Approach

In general terms, I am interested in the sound environment and the degree to which we are able to choose and shape the sounds around us in every day life. So the the concept for this installation is of an interactive 3D space for immersion in sound which works effectively with one user or a handful of users exploring and molding a sound space in interactive collaboration.

The work uses audio material from a work I produced earlier this year called touching earth, touching ground, which looked at the relationship between Tibet and China leading up to the Beijing Olympic Games.

The quality and form of the audio outcomes were of high priority. The two audio examples provided above are excerpts recorded from the presentation. The work is designed to be a delight for the ear and to be fun to move around in.

Usability considerations & actual user experience

Upon entering the space there are a number of interactive elements;

a statue of the Buddha which modifies different sonic parameters according to the users' proximity and motion
a gong which is miked and has its sounds processed according to people moving in the space
a virtual gong the output of which is processed according to movement tracking

Tibetan lanterns which are used to trigger sound files in combination or singly, and also to modify some of the sound parameters

As an installation, it is not intended that users need to interact actively with it in a continuous manner. As such the work may function as a generator of ambient sounds or music with users choosing their degree of interaction.

In a spirit of exploration and play, users may interact actively with the gongs and with the lanterns. A more passive interaction may be achieved simply by moving around in the space; how active or passive an experience depends on the awareness of the users of the relationship between their movements and the sounds they hear. The nature of the interaction is such that the users shape sound through gesture and proximity to objects after having chosen a path through a matrix of sounds. Users create their own listening experience or ambiance by triggering combinations of underlying audio files either singly or in groups.

It is intended that users explore the various audio paths available to them and enjoy shaping the sounds produced by means of gestural interaction. Relying on people's movements to sculpt the sound works best in a gallery environment where the movement is incidental to the installation. I had considered suggesting a type of sonic Tai Chi or other choreography to be performed in front of the Buddha if users wished to increase their interaction with the sounds. Users may be intrigued to try and figure out the inter-relationships between the various interactive components and the audio.

During the exhibition, the work functioned best when approached by no more than three people. Users appreciated the functionality of the work and enjoyed the connection between interaction and sound propagation. Others thought it had a psychedelic quality which they enjoyed. Some came back for a second time and brought their friends with them. Many people took delight in the sensuality of being able to shape sound by gesture and movement.

Related and Influential works

The work is influenced generally by interactive museum and gallery exhibits.

Brian Eno's concepts of ambient music are an influence.

The underlying approach to sound organisation is based on a mobile musical form developed by the founder of IRCAM, Pierre Boulez (Boulez 1996, p81). In this form, the user chooses a path through matrices of inter-related material; in this case a set of audio files.

The idea of an interactive audio matrix is also found in the web based Interactive Soundscapes project by Zach Poff & NB Aldrich (Poff & Aldrich 2008)

One influence on this piece was Forester by Leafcutter John (Burton 2008). In this software, users set off a cursor's random movement through underlying audio material which, amongst other things, is subjected to processing according to cursor position. I adapted this idea to real space, so that the users take on the role of the cursor and the sound is sculpted according to spatial position.

Technical considerations

Max/MSP forms the backbone of the system:

The original idea was to make a statue of the Buddha into an antenna which would respond to proximity by means of dielectric interference with an electromagnetic field. Users would be encouraged to pat the Buddha's head or tummy since, in some cultures, this is said to bring good luck, while such an action would disturb the field and in doing so, apply modulation to the audio. This idea of a reactive Buddha is feasible but technical resources were not available at the time of exhibition (they have since come to light). The idea was realised satisfactorily for the exhibition by using an IR proximity sensor which was placed under the table upon which the Buddha was seated while additional functionality came from a webcam which used optical flow tracking to modulate the sounds in various ways.

The i-CubeX sensors have an interface which is used as an object in Max:
One sensor was attached to the virtual gong and triggered off an audio file (MIDI note on).

The i-CubeX proximity sensor provided continuous controller data effecting a filter sweep through all audio other than live audio in the horizontal plane:

Optical flow tracking (using cv.jit. HSflow) was mapped to the following functions:

The audio files are stored in Ableton Live and referenced by means of MIDI messages from Audio Cubes. The Audio Cubes occur as an external object in Max

and require a MIDI Bridge which is an interface application developed in Max by the Audio Cube designers:

Signal processing (DSP) occurred in Max/MSP and a third audio cube functioned as an infra red sensor sending four separate and simultaneous continuous controller messages which had effect on some but not all of the signal mix by means of bussing techniques. Live audio was subject to flanging based on depth and rate messages from the webcam.

Audio signals were routed by means of Soundflower, a system extension that allows applications to pass audio to other applications.

Summary

I was pleased with the aural results of the installation. Some of the sounds were more voluptuous or sensuous than I had anticipated and quite abstract; these aspects were a pleasant result of user engagement. A future development of this project would involve spatial audio with spatialisation also to be determined in some form by user interaction.

Design and realisation of this work provided a good introduction to interactive sound design techniques and the work could function as a model for a larger installation along the same lines. The techniques learned can easily be transferred or modified for future interactive sound projects.

References

Boulez, Pierre, 1996: Par Volonté et par Hasard: Entretiens avec Célestin Deliège (Paris)

Burton, J. aka Leafcutter John, 2008: Forester
accessed August 5th 2008 at: http://leafcutterjohn.com/?page_id=14

Poff, Z & Aldrich, N.B, 2008: Interactive Soundscapes
accessed on September 5th 2008 at: http://www.interactivesoundscapes.org/index.html

Interactive Sound Studio exhibition photos online

2008-10-03T19:57:00.001-07:00

The full set and larger images available here on Flickr in the Interactive Sound Studio 2008 set
Special thanks to Sam Ferguson who facilitated the technical success of the event. Well done!

Final Documentation :: OLLY VALE "The Not So Haunted Bear"

2008-10-02T23:48:00.000-07:00

:: FINAL REFLECTION //

:: Overview -

Ultimately we felt as a group that our "bear" project was essentially successful and also a great introduction to the possibilities that abound in the interactive sound/digital media production realms. Although our original expectations as to what we were hoping to achieve were significantly altered, we felt that we developed a deep understanding of what is involved and appropriate in creating an interesting and intriguing object that is stimulating to interact with.

We feel that there is considerable potential to expand on our current model and adapt this type of approach to many different ideas and elements of interactivity. We encountered various difficulties, frustrations and general functionality issues along the way, however as a collective we believe that we exercised a high degree of co-operation and time management that allowed us to complete our project to plan within the small time frame.

:: Related/Influential Works & Design Motivation -

As mentioned in our initial proposal, some of the works that we took inspiration from included "Tickle Me Elmo" and some of the products from the "Dora The Explorer" range. We analysed numerous other examples of children's toys that we felt were groundbreaking in the development of interactive technology directed at children.

We were particularly interested in any sort of tactile usability that was essentially attractive. We wanted to develop a toy that would catch somebody's eye: something that would capture the curiosity of children and adults alike. As a result, we felt that our creation would have to be both intriguing and appealing to an audience that would not be likely to expect such a project. We explicitly decided that our design would be developed as an interesting interactive experience for adults, however we adapted this idea so that our project could potentially appeal to all ages on different mental levels as vvvrfwe believed that it was important to look at the roots of this interaction – a relationship that begins in very early childhood. Almost all children are exposed to toys and tactile objects such as teddy bears for occupation and stimulation of their young minds. Each child will develop their own experiences of their toys and react differently depending on what capabilities the design of the object has to interest, occupy and interact with the child.

:: Technical Method -

We began with a simple child's teddy bear that we purchased from K-mart. As our integral piece of technology was to be the Nintendo Wii remote controller, we first planned the size of bear that we would need to fit the remote inside and also how we would fix it inside in such a way that we would be able to easily remove it. We discussed alternative pieces of technology and also differing methods of approach and basically brainstormed until we had a loose plan of what we hoped to achieve.

Our second stage of production involved really mapping out the chronological order of events so that we would be able to complete all the necessary tasks in the time frame given. We also planned exactly how the bear was going to interact with the user, what its reactions would be and how we would essentially develop a "character" for the toy.

We decided that our bear would have numerous states in which its behaviour would differ. The user would first encounter our bear sitting upright and talking to itself. The bear would encourage the user to pick it up, and in this scenario the dialogue would change slightly and the bear would seem to interact with the user on a personal level, making comments, telling jokes etc. The "cranky" bear would then be revealed when the user begins to shake the toy - with a soft shake the bear would make a funny, trembling, shaking sound and as shaking intensified the bear would become angry and a audio clip would be triggered in which the bear demands the user to put it back down. Once placed lying down, a "sleeping" audio clip would be triggered and the bear would appear to be worn out and asleep. As an extension of this plan, Sam alerted us to the fact that we may gain access to a wireless lapel microphone from the university. Combining this idea with other aspects of our project, we decided to use a voice recognition system in Max/MSP that would allow us to "tell" the bear to sleep, which would then trigger an audio clip that had a yawning recording and the bear asking to be put down so it can sleep.

Once our approach had been decided on, we began programming. Using the custom aka.wiiremote patch developed by Masayuki Akamatsu, we began to experiment with various movements of the wii remote controller and how much they differed in the axis ranges. The patch displayed numerical values for the X, Y and Z axes so we analysed these and compared them in relation to the position of the wii remote. We set the ranges for the various positions that the bear would be in, and then the amounts of amplitude of movement experienced by the wii remote when being shaken. Once these values had been set through our programming and the majority of the bugs had been removed, we began to record our audio clips which would be triggered by the various positions and movements of the bear.

Using an AKG microphone we recorded all of our clips into ProTools and did some simple audiosuite processing including a light reverb, heavy compression/limiting and a 6 semitone pitch-shift rise to emulate a "bear voice". We combined these clips with the "groove" object in Max/MSP so that they would be triggered when certain positions were experienced by the wii remote. We then tightened up the shaking effect - to ensure that only vigorous shaking would anger the bear, not just a sharp movement etc.

To complete our programming we attached the aka.listen patch that is a voice recognition system for Max/MSP. We programmed it to trigger an audio clip whenever it recognised the words "sleep","relax", or "tired". We inserted the wireless gemini microphone inside the bear through its ear where we made a small incision and concealed the microphone piece. While using the mic in the mac labs we experienced it working to its full potential however we encountered severe voice recognition problems once used with a macbook pro, continuing to receive the "???" messages in the max print. After some experimentation, we reached a point where we felt our initial plans had been met and most of the bugs in our programming had been ironed out.

:: Usability/Interaction -

We received various positive feedback regarding our "cranky bear" during and after the exhibition from both fellow students and visitors alike. Although our project was the one that appeared the most simple, we felt that it was one of the most repeatedly visited exhibits! Some people commented that our project was "adorable", "charming", and "very intriguing and interactive", feedback that we considered to be very positive. Unfortunately there were some last minute configuration problems with the voice recognition system and our microphone was having difficulty picking up even the simplest of words. Some improvements we could have made to our project would have been more defined movements with less overlap and cutting out of dialogue clips, a more developed means of presentation and generally tighter programming to iron out the last few bugs in the system.

:: References -

www.mattel.com
www.cycling74.com
http://www.iamas.ac.jp/~aka/max/
http://en.wikipedia.org/wiki/Max/MSP
http://www.fisher-price.com/fp.aspx?st=10&e=doralanding
http://www.cycling74.com/forums/index.php?rid=0&S=c7c84e90ec35aed43cd9c86a758bbe10&t=msg&th=4153

Interactive Sound Studio final exhibition

2008-09-29T23:52:00.001-07:00

Friday 3 October 6-9pm
Hearth & Atrium, Wilkinson Building
Faculty of Architecture, Design & Planning
Access level 2 - 148 City Road (Darlington)
University of Sydney
FREE ADMISSION

our blog: http://interactive-sound.blogspot.com/
studio teachers: Kirsty Beilharz & Sam Ferguson

Wii Taiko
Interactive Narrative
Sonic Hockey
Rubick's Studio
Virtual DJ
Haunted Bear
Druidic Rituals
Virtual Conductor
Awakening the Buddha
Domino Music

Click on image for larger view

Next weekend: Electrofringe festival

2008-09-26T15:18:00.001-07:00

Just in case you're not exhausted by next weekend, and would like more immersion in electronic arts performance: note that the annual Electrofringe festival is on in Newcastle.

Electrofringe presents an annual festival and year round program of experimental electronic arts and culture dedicated to skills development and artistic exchange. The program focuses on uncovering emergent forms and has an emphasis on encouraging exchange between emerging and established artists.

Electrofringe Festival 2008 is the festival’s eleventh year and will be held over five days from 2nd - 6th of October 2008. The program presents workshops, panel discussions, masterclasses, presentations, exhibitions, screenings and interventions.

http://www.electrofringe.net/

Little Bird Electronics Arduino workshop Sydney

2008-09-25T17:55:00.000-07:00

passing on message received from Marcus Schappi - Little Bird Electronics

ARDUINO WORKSHOP IN SYDNEY TOMORROW! —WORKSHOP
We have only 5 places left for our Arduino Workshop in Sydney tomorrow!

In these one day intensive workshop you will learn:

- The layout of the Arduino Board
- Practical Electronics Theory
- Basic & Advanced Arduino Programming
- Get 1:1 consultation time on your own Arduino Project

All participants design and start their own Arduino project, we encourage you to come with your own project in mind.

You will receive (and get to keep):

- One Arduino Diecimilia Microcontroler Board
- Battery and Battery Connector
- Full-sized Breadboard
- Access to additional resources on the Workshop Wiki
... and many other components used during the workshop!

Target Group:
Art students, Computer scientists, hardware hackers, fabric experts, product developers, nerds, dancers – everyone is welcome.

GET A TICKET NOW!

DID WE MENTION WE HAVE MORE WORKSHOPS!?! —WORKSHOPS

We have 3 one day intensive workshops in Brisbane, Sydney and Melbourne. And a new Advanced Arduino Workshop in Sydney (check it out below).

In these one day intensive workshop you will learn:

- The layout of the Arduino Board
- Practical Electronics Theory
- Basic & Advanced Arduino Programming
- Get 1:1 consultation time on your own Arduino Project

All participants design and start their own Arduino project, we encourage you to come with your own project in mind.

You will receive (and get to keep):

- One Arduino Diecimilia Microcontroler Board
- Battery and Battery Connector
- Full-sized Breadboard
- Access to additional resources on the Workshop Wiki
... and many other components used during the workshop!

Target Group:
Art students, Computer scientists, hardware hackers, fabric experts, product developers, nerds, dancers – everyone is welcome.

(Every workshop has sold out so book your ticket ASAP.)

Brisbane Intensive Workshop Ticket.
Sydney Intensive Workshop Ticket, Sydney Advanced Workshop Ticket.
Melbourne Intensive Workshop Ticket.
NEW ADVANCED WORKSHOP! —WORKSHOPS

In this one day Advanced workshop you will learn:

- How to get your own PCBs manufactured
- Advanced Sensors & Servos
- Getting your Arduino Diecimila to talk online
- Get 1:1 consultation time on your own Arduino Project

SAVE $25 ON ARDUINO WORKSHOP! —COUPON
As a special offer to you we'd like to offer the first 15 people $25 off your Arduino Workshop Ticket. You can redeem your coupon by entering the code EARLY-BIRD-SYDNEY or EARLY-BIRD-MELBOURNE or EARLY-BIRD-BRISIE during the checkout process.

Thank you for your continued support whilst we have been away sick,

The crew at Little Bird Electronics.

Little Bird Electronics and the Little Bird Electronics Logo are trademarks of Little Bird Electronics. Content and design is copyright (C)2008 Little Bird Electronics.
All rights reserved.

This newsletter was sent with Campaign Monitor.

2008-09-25T17:37:00.000-07:00

Mick, Myles and James
Assessment 1: Interactive Narrative Report

Design Basis

For this project we decided to make an interactive story/game experience based on sound. The player will use the Wii remote to play the game, however they will be blindfolded so the whole environment will be created by their own imagination. The game will progress by having the player react physically to audio cues, or voice commands.. The game is sound based, so it will start off with an introduction which will be constructed with sounds to set a scenario. We are combining sound design, game design, and storytelling to produce and interactive experience involving the player into the experience and feel as thought they themselves are the protagonist in the story.

Story
The basic story is the player was in a motorcycle accident and has been taken to an underground science lab for surgery. This underground criminal corporation have decided to use him in “Project Cyborg”. They replace his lost right arm with a robotic arm, making him a cyborg.
They have chosen this guy because he is a rough and tough person, he was once in the military, has no family or loved ones, and has nothing to lose.
The scientist takes the player through a tutorial on how to use this new robot arm. This arm has features such as a phone, a gun, and of course a hand, all of which will be used in the game.
Then the player is sent on an assassination mission where he or she will use the phone, their hand, and their gun.

How the Wii remote will represent the robot arm
There are 3 main functions of the ‘robot arm’, the phone, the gun and the hand. Each will be used in the game. The game progresses as certain movement cues are met, there will be subtle hints for the player to make these movements. These reactions will be measured using the Wii remote by taking the input values of the motion sensors and microphone from the Wii remote and mapping them to specific audio cues using branching if statements and time. Time will be measured in frames, as we know there are 25 frames to a second.

Influential Approaches

There are a few influences that contributed to the idea. Being game designers is one thing, this is the reason why we chose a game, a good game designer is able to work within constraints and produce something good. The constraints in our case were; only 8 days to create the experience, no visuals are to be present, and the use of sound as the primary element in the design. Peter Moleneux a great game designer and creator of Fable showed the world how important story, characters and interaction are and what you can produce when the three are molded together well.
One major influence was the concept of making a game that blind people would also be able to play. Since we could not use visuals it aided this idea, and we then decided to blindfold any non blind players. This way the whole environment is created inside the mid of the player.
Another influence was being a kid. When we were young the stories were read to us by our parents as we lay in bed eyes closed. This use of vocals and sound to stimulate the imagination was an influence and an experience we would like to tap into again, adding interaction and sound design elements to heighten the experience and immersion.

Intended User Experience

The intended user experience has been mentioned above, but lets clearly piece it together and outline the core experience. The intended core experience is to allow the player to use the Wii remote to interact with the story and environment and thus feel immersed into the experience and feel as though they are the protagonist and main actor within this story. We want to stimulate their imagination by blindfolding them and having them create their own unique environment using the sounds as their only influence. The player’s actions can also change how the experience unravels, which means each player will potentially have a different experience.

Construction

Interactive Elements: The sound or dialogue can be changed depending on the player’s reactions. The motion and position of the player which is measured by the Wii remote will be used to interact with the story, also making the story an interactive element.
Data Types Used: Boolean: A simple test of true or false will be used to determine whether certain actions have been taken. Integer: The values of the Wii remote, such as position rotation etc will be measured as integers. Time will also be an integer which will be determined by frames. The sound response will also be measured in decibels, which will be converted to integers. Objects: Objects are used to store and call on the sound files that will be played during the game.
Mapping Methods: Rather than mapping sounds to movement we simply play the sound, when the values returned from the Wii remote are a specific value. E.G. If the Wii remote y value is under 70, the player has lowered his or her position, and the sound file will be played that says “Thank you for sitting down”.
Integration of Technologies: Combining the motion sensor and microphone from the Wii remote returning values to a computer, we allow the user to interact with their environment given the sound outputted from the computer to the speakers or headphones.
Audio Output Elements: In a theoretical sense: Dialogue, sound effects, music. In a practical sense: Headphones.
Audio Production Tasks: Recording sound effects, composing music, getting samples from a library. Recording voice acting, using binaural microphones. Equalising the audio, and applying pseudo surround effects using convolution techniques.

Major Technologies

Wii Remote: The motion sensor, microphone, and speakers inside the wii remote will be used for interation. The Wii remote has an in built accelerometer which measures acceleration, or speed over time, and measures position in regards to the three dimensional values, x, y and z, and also rotation.
Speakers: We are hoping to use some wireless headphones, so the experience is a personal one, wireless will be more convenient. Using headphones we will imitate surround sound to create a 3D environment.

2008-09-25T06:13:00.000-07:00

Design Reviews - Robin Mukerjee

Design Review 1 – Robin Mukerjee

Gesteral Speech Synth by Matt Gilbert.

The Gestural Speech Synth

Design Review

The Gesteral Speech Synth (GSS) was developed by Matt Gilbert a Georgia Tech student and also a graduate of The Art Center College of Design. The idea and fundamental concept behind the GSS is to imitate the shape made by the throat when speaking which will subsequently create the throats relative sounds. At the projects current conception it can only create the sounds of vowels (ee, ah, oo, uh). The GSS controller is constructed of a series of strings strung across a wooden arch in which the user uses his wrist and hand to create the shapes of the throat. The strings are connected to an electronic receiver which measures the tension placed upon them which then is sent to a computer in which it is run through a MaxMSP patch. When the GSS is played by continuous movement and changing of shapes these vowels sounds will be produced. The user will use his forearm and hand to generate the sound. When placed on the strings the user will apply pressure to form the relevant vocal shapes which consequently will produce the sound. The sound will be playing continuously as the user changes shapes.

Code Review

The output of the GSS controller is determined by a PIC microchip (Programmable Interface Controller or Programmable Intelligent Computer) which is then received by Max.MSP via the serial port of the computer. The PIC microchip will receive the information regarding the tension that is placed on specific strings which will establish the vocal shape that is produced. Max/MSP will receive this information and using a filter patch which determines the sound that will eventuate from a simulated vocal tract.

PIC microchip and strings

Approach

Matt Gilberts main influence to undertake the construction of the GSS was based on his studies of Brad Story’s and Ingo Titze’s “A preliminary study of voice quality transformation based on modifications to the neural vocal tract area function” from the Journal of Phonetics (2002) and also “The relationship of vocal shape to three voice qualities” from the Journal of the Acoustical Society of America (2001). By learning the physical mechanisms of the human vocal chords Matt Gilbert intended to represent the relevant sounds in a physical instrument

Shapes to create the different sounds

Reference
http://www.mattgilbert.net/article/17/gestural-speech-synth

Design Review 2 – Robin Mukerjee

Talking Popcorn by Nina Katchadourian

The Talking Popcorn Machine

Design Review

The Talking Popcorn Machine was made by Nina Katchdouriun, an artist from California, USA. The idea of the Talking Popcorn Machine (TPM) was that the sounds created by the process of corn kernels popping when heated would produce a sound that was picked up by a microphone and generated into Morse code. The Morse code would then be generated into a language(“Popcornese” as stated) that would be spoken by ac computer generated voice that was at the bottom of the machine. Words that were produced by the TPM words like “mvinli” but it was also noted to say words distinguishable by the English language like “hi”; “lob” and “ma” This design was very success full because of the fluency and conversion of the sounds in language to the computer generated voice, Martha Schwendener from Time Out New York reported that “ Katchadouian also makes you pay attention to language – to the intersection between spoken and written words:to language’s rhythm and pacing , from the short and long sounds of Morse code to the blips and silences of radio communication: and the silent but highly communicative vocabulary of visual images”. The idea of a language being spoken by a machine is very interesting, the process gives a life-like quality to a machine which we see explored in many fictional avenues everyday from talking cars and toasters in television and advertising.

Code Review

The code that was used to produce the sounds from the popcorn into Morse Code then further into a spoken language was call the “Popcorn Patch” for Max/MSP and was designed by Josh Goldberg, a prominent sound artist and programmer from New York City. The patch was designed to take a running average of the amount of silence in-between each pop of the kernels which then, by giving a measurement of the actual noise that the pop makes will assign it as either a dot or a dash in Morse code, but it is unclear by the explanation of weather each sound of the popping was measured or alternatively in groups because of the change in the speed of the popping. The “Popcorn Patch” was designed to attribute each 5 different aspects of Morse code, the dot (.) , dash (-), short gap between each letter, a medium length gap between each word and a long gap between each sentence.

Approach

Nina Katchdourians main influence in putting together the TPM was her interest in language, translation and Morse code. Morse code was developed for transmitting telegraphic information that used short and long sounds that would represent letters, numerals and punctuation, the sounds are commonly referred to as dots and dashes, its sonic association to that of the sound popcorn makes when cooking helped the overall success of the piece.

The Talking Popcorn Machine at exhibition

Reference

http://www.ninakatchadourian.com/press/ra_timeout_2001.pdf
http://www.ninakatchadourian.com/languagetranslation/talkingpopcorntxt.php
http://www.ninakatchadourian.com/languagetranslation/talkingpopcorn.php

~

2008-09-25T00:38:00.000-07:00

:: INTERACTIVE SOUND STUDIO BLOG POST // ASSESSMENT I

:: (OLLY, ROBIN & YOSA) // DESC 9187 //

“SOUNDS FROM A HAUNTED BEAR”

:: Major Project Prototype Design –

As our major project we will be developing an interactive child’s teddy bear that, when moved in specified angles and directions, will apply a series of synthesizer

parameters and audio processing to a typical music box recording. This will ultimately allow the user to experience a normally solely tactile object in a different environment with a new sonic aspect. We feel that some of the controls of a synthesizer can add an element of eeriness to an otherwise simple and innocent sound – as a result, one of our main objectives is to discover a darker, more sinister tone to a human’s tactile experience with a toy – something that we hope to achieve by manipulating the sound through the movement of the object.

:: Design Basis & Influences –

Although we have explicitly decided that our design will be developed as an interesting interactive experience for adults, we believe that it is important to examine the roots of this interaction – a relationship that begins in very early childhood. Almost all children are exposed to toys and tactile objects such as teddy bears for occupation and stimulation of their young minds. Each child will develop their own experiences of their toys and react differently depending on what capabilities the design of the object has to interest, occupy and interact with the child.

There are many examples of interactive children’s toys that employ sound as interactive element, which showed us how important to both manufacturers and people raising children it is to develop and use tactile objects that stimulate multiple senses: in this case sight, sound and touch.

One of the more famous examples is “Tickle Me Elmo” – when a child applies pressure in certain ways to the doll, it triggers a series of “Elmo laugh” recordings. This simple approach to interactive sound for children was incredibly effective and “Tickle Me Elmo” became one of the most popular children’s

toys ever manufactured.

Another example is the “Dora the Explorer” range from Fisher Price – The “Buenas Noches Dora” typifies the design approach we wish to undertake quite precisely, as there is a motion sensor within the doll that can sense when the doll is being laid down and when it is upright. When the doll is laid down, Dora says “Good Night” and sings her “Buenas Noches” song. We are fascinated by this type of interaction, and it has been one of our main influences in deciding to adopt this type of design approach – the only difference being that we will examine the possibilities of creating a more disturbing atmosphere by altering the sound through movement.

:: Intended user experience –

We hope that as an individual interacts with our “haunted teddy bear”, they will experiment with movement as a tactile manipulator of sound. They will consciously analyse how they can be in control of the output of the speakers and also discover what parameters are affected by different types of movements. As we previously mentioned, this will NOT be intended as an interactive sound experience for children, but rather an amusing and stimulating experience for an adult interested in how the teddy bear defies the typical conventions of a children’s toy.

:: Outline of interactive elements, data types used, mapping

methods, integration of technologies, audio output elements and audio production tasks -

The interactive elements of our project will involve the physical movement of the bear by the user. We hope that the initial sound will be triggered by picking up the bear from the table. We would like to design the setup so that as movement commences (i.e when the bear is “lifted” from its platform), the soundtrack will begin in loop. By moving the bear up and down the volume of the sound will be affected within a certain range and the higher you hold the bear the louder the output will be. Moving the bear backwards or forwards will affect the tempo of the soundtrack - by moving the bear forward the tempo

will increase and subsequently moving the bear backwards will decrease the tempo. By moving the bear from left to right we hope to trigger a panning effect between left and right speakers. By tilting and rotating the bear at different angles we wish to affect the sonic output with modulation and distortion effects, modulation triggered by tilting left to right and back and forward, and left to right rotation effecting distortion levels.

We plan for the audio output of the bear to be generated by a motion sensor inside the bear. At this preliminary point we intend for it to be a Nintendo Wii controller. The motion sensor will be detected by the computer and it will interpret the signal and movement messages internally through Max/MSP where all the synthesized elements of the sound will be controlled. Max/MSP will manipulate a sound file that will be on a loop of approximately 5 minutes. The output of Max/MSP will run to a stereo set up of a left and right speaker.

Artisitically speaking, the audio soundtrack will be created in ProTools as a mono audio track. Using many layers of recorded sound we will make utilize various “child” inspired musical elements including music boxes, children’s voices, ukulele and mandolin recordings and whatever else we feel will add to the overall expression of the piece.

:: Major technologies required –

We plan to use a Wii remote that we will plant inside the bear. We have decided to use this because of the functionality that the Wii remote possesses. It has an ability to sense all the physical movements that we require by use of the internal ADXL330 accelerometer and a sensor. By use of infra-red receiver we will receive the motion signals from the bear. In addition, we will simply require a laptop computer with Max/MSP installed and some form of audio output to a stereo pair of monitor speakers. We may utilize a basic DJ mixer for this requirement – to not only control the gain and basic EQ settings of the output but this will also allow us to simultaneously record the audio output of the monitors into ProTools for later use.

Assignment 1 Domino Music

2008-09-24T22:37:00.000-07:00

Design Proposal

DOMINO MUSIC

Students: Bochen Wei
Yurong Xia

1. Introduction
The project we intend to do is a simple musical instrument that combines with elements of the domino game. The project called “domino music”, that is an interactive sound game project. The sound or music are being created when player knock down the dominos. Player can achieve pleasure and satisfaction by variety forms of setting up the dominos. Although the domino music is a simple sound instrument, it is possible to create melody with understanding of regular system. The purpose of the project is to give some ideas to improve the creativity and interactivity between digital system and human interaction. The intended user can improve their imagination with digital techniques.

2. Design basis and influential approaches
As we know, dominos is a famous game about setting them up and then knocking them down. And numerous methods and multivariate forms are the typical character of the domino game. By reason of this feature, the project is designed into two parts. The first part is setting up dominos with multivariate forms. The second part is the sound or short music is being created while the intended user is knocking down dominos. In the setting up phase, users set up dominos on a square table which is one meter times one meter. The table is divided into forty-nine sections that are 7 rows multiply 7 columns. Each section stands for one output of musical note. Column 1 and column 7 are pre-record different short sound tracks or musical note. From column 2 to column 6 are different scale such as C4, C5, C6 ect. (As figure 3 shows below). The intended users are able to forecast the expected sound output and create different short music by setting different forms of dominos. In the knocking down phase, each domino will collide with each other. Each collision will generate a single sound. Then, a set of dominos fall down and collide each other will generate a short length of sound. (For, the technologies of making this project works please see part five major technologies)

3. Intended user experience
The intended users of the domino music are to face to masses and public from different ages and different countries. The domino music could be initiatory game of music for children. It also could be a simple instrument for adults to be relaxed in their leisure time. Different cultures and different countries’ people also can enjoy with it because it only involves domino game and sound or music. In addition, if the users who have some sounds knowledge, they are also able to create varies short melodies when they are playing this game. The domino music project also provides the feature of interesting for the people who have not musical knowledge, that they still achieve the satisfaction by the musical dominos.

4. All hardware + Interactive elements
Playing this project is just like playing a game. In fact, it is a special domino game.
For the hardwares, we have four main hardwares. (See figure 1)
a): Camera: The camera is used for capturing data from the masks which are pasted on each domino.
b): Mask: Each mask is pasted on each domino front face. So, when the players knock down dominos, the camera will see the mask.
c): Domino: There are approximately 28 dominos for playing this project.
d): Table: The table includes two parts, namely, plane and bevel. The plane is used for setting up dominos and the bevel is used for sliding dominos.

Figure 1: All hardware

5. Data types used, Mapping methods, Audio output elements, Audio production tasks
In this project, we use camera tracking as the main technology. Therefore, firstly, the input is video data from camera tracking. Secondly, the video data is transmitted into software MAX/MSP. Then, in MAX.MSP, the video data is changed into digital numbers. We use position X and Y to decide the sound output. In our project, we have three types sound. The first two types sound is generated when dominos on table plane and the third type sound is generated when dominos slide down on table bevel. The first type is single sounds which are from C2 (36) to B6 (95). The second type is pre-record sound track (playback). We store 14 different sound tracks in computer. The sound tracks can be any sound tracks, but most are guitar, violin, and piano and Chinese flute sound tracks. The third type is continuous sound. When dominos slide on the table bevel, the positions of X and Y are continuously changed. So, the continuous sound is generated. The final sound output can be several single sounds, or can be music, or can be single sounds and sound tracks. It depends on the position of each domino.

Figure 2: Data types used, Mapping methods, Audio output elements and Audio production tasks

Figure 3: Structure map of table plane
The table plane is divided into 49 areas, which are 7 rows multiply 7 columns. Each area stands for one output. Column 1 and column 7 are pre-record different sound tracks, column 2 to column 6 are different single sounds such as C4, C5, and C6. For example, if a domino falls down in area C4, then, the camera records the position of the mask and sends the video signal to MAX/MSP. In MAX/MSP, the video signal will be transmitted into number 60. The number 60 is the MIDI number. So, the output sound is single sound C4 (60).

6. Major technologies
There are two phases during playing this game, namely, setting up phase and knocking down phase. Generally, setting up is to put dominos on a large table, and knocking down is to knock dominos down and generates continuous sounds.

In the setting up phase, the intended users need to set up dominos on a large square table. The table includes 47 areas, 7 rows multiply 7 columns. Users can think about what sounds they want to create then put dominos on purposed areas.

In the knocking down phase, the users need to knock down the dominos. The major technology is camera tracking. The position of the camera should be putted in the centre and above the table and the camera view is proposed to in the black frame (See figure 4). The mask should be pasted at the upside of each domino front face. So when dominos fall down, each domino will not cover the mask on the next domino.

Figure 4: The position of camera and camera view.

7. Conclusion
To conclude the proposal, the music domino is design to improve the interaction between people and music by digital technologies. The design basis and approach, processes and system of design and technologies are explained precisely. The domino music project can be widely played by masses and integrated musical elements within the games.

ASSESSMENT 2 – Yurong Xia

2008-09-24T21:31:00.001-07:00

Review #1. – Ravezooka

Design Review
Ravezooka is a musical instrument that design as a weapon. The shape of the Ravezooka is inspired from bazooka. There is a guitar strap to let user put it on shoulder. The user pulled trigger handle, the hardcore sound is being played. There is a light shooting from the Ravezooka to determine the person or object in front of the instrument. The distance from the Ravezooka is a key to control the sound that is being played. The closer the distance achieves lower frequency range of sound. The trigger handle can give a distortion of effects as back as it is pulled towards to the user.

Ravezooka, musical weapon, created by Benedetta Piantella Simeonidis and Lesley Flanigan

Code Review
The sound of the Ravezooka is played though the computer with MIDI data. The sounds are recorded from a guitar or bass amplifier. The MIDI data are collected by the sensors. The sound seems like a machine gun by resembling an analog synthesizer manipulating sine wave frequencies.

MAXMSP is the software of the Ravezooka. And hardware are Arduino, Ultrasonic Range Sensor, Infrared Proximity Sensor, Luxeon LED, amplifier and laptop computer. Arduino is a physical board that can be used to connect to MAXMSP running on the computer. It applies Wiring language.

Arduino

The user can control the range of frequencies, distortion effects, volume and on or off.
The range of frequencies is based on distance of the target. The distortion effects is based on the position of slider handle. The volume is controlled by the potentiometer knob. The trigger handle control it on or off.

The protocols are from the Ultrasonic Range Sensor and Infrared proximity Sensor though MIDI. These two groups of data are processing in MAXMSP. Finally, the output sound are being played through a guitar amplifier from the computer.

Physical Digram of ravezooka

Approach
Ravezooka can shoot different hardcore sound that depends on the different distances from the targets. The closer the target, the lower the frequency ranges. The user pulled the trigger handle, the hardcore sound is being played. At the same time, an initiate light is shot from the Ravezooka. When the user is moving around with the Ravezooka, the frequency range of the hardcore sound are changed by the distance of the person or objects in front of the Ravezooka. The closer distance gets lower frequency range of hardcore sound. The light shot from the Luxeon LED gives a visual clue of range of sound that the user wants to achieve. The trigger handle also achieve the similar purpose as the light. To pull the trigger handle towards forth or back provides extra change of the distortion effects of the sound. The trigger handle towards more to the user, there are more distortion of effects. Volume is able to be controlled by turning the potentiometer knob on the side of the Ravezooka.

Reference
http://www.seseyann.com/ravezooka/ravezooka.html
http://en.wikipedia.org/wiki/Arduino

Review #2. – Talking Popcorn

2001, Popcorn machine, laptop computer with custom-written program, speakers, paper popcorn bags with Morse Code, rubber stamp, salt shaker, plinth, and floor circle, 90 x 90 x 72 inches overall, Courtesy of the artist, Brooklyn, New York

Design Review
Talking Popcorn is a sound sculpture that interprets popcorn popping as Morse Code. Talking Popcorn looks very similar as the normal popcorn machine. There is a microphone to pick up the sounds of popping popcorn in the cabinet underneath the popcorn machine. The popping sounds are recorded by the microphone. The popping sounds are translated to Morse Code. MAXMSP is used as the computer program for Talking Popcorn to translate the popping sounds according to the patterns and dictates of Morse Code. The computer program will read the words to you. A text-to-speech program provides simultaneous translation.

Code Review
Morse Code and MAXMSP are used for the Talking Popcorn.
“Morse Code is a character encoding for transmitting telegraphic information, using standardized sequences of short and long elements to represent the letters, numerals, punctuation and special characters of a given message. The short and long elements can be formed by sounds, marks, or pulses, in on off keying and are commonly known as "dots" and "dashes".” The Talking Popcorn only can generate short beeps of Morse Code with a timing system. That is the problem of it.
A series of random letters are created by translation from the popping sounds with Morse Code. And the time system turns the random letters to a word. Or it is translated to the closest matching word.

The Max code might to organize the sequences of the words to make sense or in a correct grammar.

Approach
Talking popcorn is interesting project to interact. It expands the idea of interaction with machines. In this perspective of the idea, many projects are able to be created as sound or language system.

Reference
http://www.ninakatchadourian.com/languagetranslation/talkingpopcorntxt.php
http://www.ninakatchadourian.com/languagetranslation/talkingpopcorn.php
http://tang.skidmore.edu/2/calendar/show/660/doc/1577/
http://en.wikipedia.org/wiki/Morse_code
http://upload.wikimedia.org/wikipedia/commons/1/1f/International_Morse_Code.PNG

Bochen Wei Assignment 2

2008-09-24T21:29:00.000-07:00

Review One: Block H

Design Review
Block H is an interactive installation which includes two areas, namely, exterior area and interior area, and the name of Block H originally comes from The Maze/ Long Kesh prison and H-Blocks that held paramilitary prisoners up until recent times. The exterior area (game projections) is called Counter-Strike Block H which is a multiplayer virtual outdoor environment. This means that the exterior is also a game by modifying counter strike source and half life source. The outdoor environment is simulated as the Northern Ireland which includes Northern Irish housing estates, their peace lines, Long Kesh Prison (and its H block),the spiritual labyrinth/confusing maze and so on. The two communities, Loyalist and Nationalist, are located in a housing estate split by a peace line and watchtower, and each community has its own sectarian symbols such as murals, bonfires, flags and graffiti etc. The interior (television) is a physical world (tent) where one table with a television to be placed at the back, with another placed in the centre in front a chair. A participator is placed in a tent and sits on the chair, face to the television. The game is projected on the transparent canvas at the left and right sides of the participator. Furthermore, the participator is given a two button remote controller to change the contents on the television, at the same time, changing the contents of the game.

The layout of inside and outside elements

Exterior elements
There are two accompanying player interfaces outside the tent, one for each team.

Interior elements
One participator, the television and remote control are situated inside the tent.

Code Review
1. Game development: the exterior game is the modification of Counter-Strike. Therefore, the game Counter-Strike Block H is developed by Source Engine and Hammer & CS console.
2. Video (on television): Video works comprise the most of frame-by-frame retouching and compositing techniques by using Photoshop, AfterEffects, and proprietary morphing softwares. Flash and Director are used for video sequencers.
3. Audio (on television not in game): MAX/MSP is good at synchronizing audio sequencing and offers the great dynamic sound for block H interior area. It uses footages of cultural icons which are the sound of victims and perpetrators of violence during “the troubles (late1960’s to 1998)” which over 3,000 people were killed and many more seriously injured. The participator sits in the tent can change the audio and video channels by a two button remote controller. The images/video on television is shaked, and the sounds stutter themselves in real time to synchronise with any one of the soundtracks.

Approach
Block H combines the virtual game world and real physical world together to re-build the atmosphere and the environment of “the troubles” in Northern Ireland. The exterior of block H is a game world by modifying Counter-Strike, and the interior of block H is the simulated environment of “the trouble” by images and sound from television. The connection between exterior and interior is by projecting the exterior game on the transparent tent’s walls, and the participator sits in the tent and faces to the television. So, the participator not only can see the on-going game on the walls but also can watch the images/video and hear the sound from the television. Furthermore, participators can change the contents on the television, at the same time, changing the contents of the game.

Review Two: Sound Lathe

Design Review
The Owl Project explores of the sonic properties of woodwork, and Sound Lathe is one of its new works. The Sound Lathe is very different from many electronic instruments that Sound Lathe produces a unique wooden object at the end of each performance and also produces audio data, saw dust, noise and wood chippings. So, The Sound Lathe combines electronic music and traditional wood craft together. The Sound Lather machine is a traditional pole lathe which a set of eight sensors are putted on spindles. The turned spindles shape wood into different wave shape profiles, and then the different wave shape profiles are transmitted into different digital signals to MAX/MSP to create complex sounds such as tones, glitches and beats. The rhythm of the pole lathe decides the rhythm of the music.

Code Review
The audio is created in software MAX/MSP. A video input and a munio interface are used for converting the shape carved into the wood into digital signals, and then the digital signals works in MAX/MSP to generate complex sounds. The shapes of the wood chippings directly decide the shape of the audio waves. “The pulse of Lathe being used is sensed as the wood-turner pumps the drive pedal, this is converted to a midi clock pulse which is passed to custom written sequencing software written in SuperCollider.”

Approach
The creator combines traditional wood cutting and creating sounds together ingeniously. Each performance of the Sound Lathe is not only a process of creating a wood object, but also is a process of creating sounds. The eight sensors are used for changing the wave shape profiles of wood chippings into different digital signals. The software MAX/MSP generates different sounds according different digital signals. Furthermore, the rhythm of the final sounds can be controlled by the rhythm of the pole lathe, so the final sounds can be some complex sounds. The project gives us some ideas of how to create sound while doing some traditional manual works. For example, the traditional carving and sewing also can do a similar job just like the Sound Lathe project.

Bibliography:
http://www.blockh.net/
http://www.blockh.net/Block_H_documentation_1.pdf
http://www.blockh.net/Block_H_documentation_2.pdf
http://www.blockh.net/Block_H_dissertation.pdf
http://www.owlproject.com/

Matthew Shriffer - Assessment 1 - Druidic Rituals

2008-09-24T20:30:00.000-07:00

Design Basis & Influential Approaches

My project design is based on the idea of a multi-user virtual fantasy ritual. It involves three separate “stations” at each of which will be a glass/clear plastic “orb”, placed on a pillar or pole at about waist height. These stations will be arranged in a triangular shape (diagram below), placing the users with their backs to each other, facing a speaker(s). The idea is to create a tiered social interaction, which changes in degrees of complexity depending on how many users there are.

The project combines the premise behind a previous project of mine from Spatial Audio (generative music in surround) and another project from Music Technologies (noise/drone based music) with an interactive user-controlled installation - to create a surreal, noise-based druidic summoning ritual.

Intended User Experience

When the users move their hands over the orbs, pre-recorded ambient/noise based loops will play and be modified using DSP in Max/MSP, depending on where their hands are placed on and around the orb. My previous project for Spatial Audio made use of delayed buses on each speaker to create a layered surround sound effect - one channel was triggered in the left speaker, then 2 seconds later it was heard in the centre speaker, and so on, creating a complex layering of sound that spun around the room, changing as new sounds were introduced into the mix.

Applying this concept to my current idea, when one user has placed their hand over the orb, they will hear a triggered sound that will start in the speaker(s) in front of them, and then proceed to delay to the other stations. This has the effect of creating a curious social interaction - if only one person is using a station, they will be able to hear sounds all around them, which they will control using hand positioning around the orb. However, if two or three people are using all the stations, a complex cacophony of noise will consume the area as the sounds from each orb delay from speaker to speaker. All the users will feel as if they are creating an atmosphere together, as they hear all their “creations” combine in the surrounding speakers.

I will also be applying generative principles to the Max/MSP coding – taking one property of the signal at one station (such as pitch) and using it to augment the property of the signal at another station (such as distortion) using DSP. For example, if one of the XY controls on the first orb controls the pitch of its sound, I will use that property to trigger a distortion object on the second sound – if one user raises the pitch of the first orbs sound using their hand, they will also inadvertently raise the distortion level of the sound at the second orb. The users will not know the outcome of their actions on the total soundscape (at least not instantly!)

In this way the user experience is designed to change dramatically, depending on how many people use the stations, and the specific actions of those users. The overall effect is intended to be a surreal, eerie wall of noise, indicative of a mysterious religious/spiritual ritual.

Audio Output Elements, Processing & Production Tasks

There will be 3 separate audio outputs, one for each speaker located in front of the stations. They will be linked in such a way that each sound not only plays in the speaker directly in front of the orb, but also delays to the other two speakers in a clockwise direction (or anti-clockwise, as long as it is a continuous pattern). For this to be achieved, delay buses will need to be set up on each speaker entirely separate from the sfplay~ objects used to trigger the pre-recorded sounds.

The pre-recorded sounds will be composed by me, using Reason & analogue recordings of musical instruments that I own. In terms of processing/DPS, they will be heavily processed (distortion/reverb/delay, pretty much anything I can get my hands on) in order to create a surreal, other-worldly effect. I plan on making good use of the overdrive~, delay~ and analogue~ (external oscillator) Max/MSP objects.

Data Type, Mapping Methods

My project will contain XY data information as its main source of sound manipulation and playback. The two axis’ will be linked to two properties of the sfplay~ DSP chain (for example, reverb & volume), enabling the user to control (in this example) the loudness and amount of reverb present in the signal chain depending on where their hand(s) is placed in the XY tracking area. In order for the generative applications of the project to work, the XY tracking data also needs to be side-chained into properties of the other “stations” DSP chains.

Structure Chart

Major Technologies Required

The major technology required for my project is camera tracking, and I will be using the cv.jit external package for Jitter in order to realise my project. There are a number of different forms of camera tracking offered by the cv.jit package, including optical flow (HS and LK methods), image area tracking (which tracks bright spots on greyscale images), feature tracking (which tracks individual pixel movement).

I obviously need to experiment with each type of tracking to see which works best for my project, but after discussion and recommendation from Sam, it is most likely that I will use a type of optical flow tracking. Using this method, the values of the output matrix are estimations of that pixels displacement compared to the previous frame. Negative values indicate leftward and upward movements. Because this tracks movement and not features, the users’ hands will have to be constantly moving in order to create noise – which will keep them active whilst they are using the “orbs”. The way I plan to have the delayed buses set up will eliminate any stop-start jerkiness of the sound generation – once the sound stops (i.e. movement of the hands stops) it will be caught in a wash of reverb and delay until the optical flow tracking detects movement again, creating a constant wash of noise and feedback.

The Musical Gestures Project

2008-09-24T20:03:00.000-07:00

Assignment 2.2
By Piers Gilbertson

Approach
The Musical Gestures Project (hereafter referred to as The MGP) is based at Oslo University, and aims to create sound by simulating the gestures made when playing an actual musical instrument like a drum.

For more information on the people involved please see: http://www.hf.uio.no/imv/forskning/forskningsprosjekter/musicalgestures/participants/index.html

The MGP state their primary objective as being:

“… to work towards a coherent theory of the relationship between musical sound, human gestures and musical concepts”
(www.hf.uio.no)

Given ‘gestures’ is a rather broad category; The MGP came up with three sub categories:

1. “Sound-producing gestures”, such as hitting, stroking, bowing, blowing, singing, kicking, etc”
2. “Sound Accompanying Gestures,” like dance.
3. “Amodal, affective or emotive gestures”. These are naturalistic gestures we are not really aware consciously that we are making.
( www.hf.uio.no )

They realize that there is overlap between these categories; that the gestures used for, say, playing a banjo could correlate with amodal gestures, and also sound-accompanying gestures.

Their first step was to thoroughly research the gestures involved in playing certain instruments including the piano, drums and clarinet. They employed several techniques to do this, based on traditional scientific models:

• Review of existing material
• Interviews of musicians
• Observation of musicians
• Experimentation with musicians
• Simulation by musicians

Having analyzed the gestures the task becomes to mimic those gestures in a way that Max/MSP can register.

Code
The MGP uses bundled software they refer to as the “Musical Gestures Toolbox”. It is built and run in Max/MSP. It is not stand alone software and is run in Max. There are several input parameters and functions that the software recognizes/performs:

• Video capture
• Hand tracking
• Cursor positioning
• Hand path smoothing
• Computer vision analysis
• Etc.

Having analyzed the gestures The MGP then had to analyze the gestures effect on the audio produced. For example: what happens when a pianist uses bigger movements? – Louder noise.
To this end The MGP use “Realtime audio analysis” software, again built in Max/MSP/Jitter. This analyzes:

• Spectrogram
• Brightness
• Loudness
• Noisiness
• Pitch detection
• Onset detection
• BARK values

By comparing these 2 analysis techniques correlation ans causative relationships can be established.

Software is described in more detail at: (http://www.hf.uio.no/imv/forskning/forskningsprosjekter/musicalgestures/resources/soundanalysis/index.html)

Review
I am very interested in the nature of the relationship between body and instrument. Also whether that experience can be recreated via the interaction of man and technology.
I know that when I go from playing an acoustic drum-kit to a midi one, (even of very good quality), I have to completely change my technique. The way I produce dynamic control changes, the speed at which I can trigger sound changes, and the expression of subtleties is completely different. I personally find MIDI drumming a very useful compositional tool, but as a performance instrument it is both musically, physically, and emotionally unsatisfying.

I think that this problem is inherent in the technology rather than the concept. Ten years ago pressure sensitive midi pads were (from my experience) terrible. Now however the Roland V-20 for example is a lot closer to the real thing. So this project excites me. I would love to be able to play expressively, dynamically and subtly with only gestures. My neighbours would probably appreciate it too.

References

http://www.hf.uio.no/imv/forskning/forskningsprosjekter/musicalgestures/

http://www.hf.uio.no/imv/forskning/forskningsprosjekter/musicalgestures/publications/index.html

Matthew Shriffer - Assessment 2

2008-09-24T18:58:00.000-07:00

Assessment 2: Review 1

Graffonic by 3kta

Design Review

Graffonic is a reactive work of art by designer 3kta that utilises video tracking & a laser pointer to generate audio and RGB images on a projector screen. The design uses an iSight video camera, an analog to digital video converter, a video projector and a custom "can" that includes two laser pointers, four batteries, one electric switch and one button. The installation is connected as follows:

The designer has presented the installation as a virtual graffiti wall, in which users can “spray” RGB graphics (including custom created lines, textures, fonts etc) onto the projection screen. Sound is also generated, according to the positioning and activation of the custom laser-pointer “spray can”. The result is a complex texturing of sound and image that utilises generative principles to create an immersive audiovisual experience.

The first version of this installation was presented at the University of Barcelona in Jully 2006 and at Politecnico de Milano University in December 2006 The current RGB version was presented at oPorto and Estremoz, Portugal in July and August 2007, and San Francisco in April 2008.

Code Review

“This system allows the machine to give the users control upon the application with reference to the positioning of their hand on the plane” [1]

“Graffonic uses custom software programmed in the graphical programming environment Max/MSP and includes some of the brilliant Jitter objects.” [1]

Extrapolating from the information given about the design and the demonstration videos, it uses video tracking to pick up the infrared signal given out by the laser pointer on the projection screen, and converts the information to numerical XY data in Max/MSP. The data is then used to trigger both RBG video information using Jitter objects as well as audio using Max/MSP objects. The video information is then fed back into the projector, displaying it on the screen, and the audio information is output to the audio amplifier/loudspeakers.

“Accordingly, using its relative cognitive ability (supported by algorythmic processes), it will provide an audiovisual answer. The computer's answer is grounded on a process of generative design in order to create an esthetical reflex on a real time scale.” [1]

Generative processes appear to have been used on both the video and sound outputs. This creates a unique creation each time the installation is used, depending on who uses it and how it is used. The soundscapes created effectively complement the random nature of the graphic art being presented.

Approach

The result of the above is presented as an installation, in which the audience can see themselves and the artist create a musical and visual canvas via a projection screen. The artist can then demonstrate fully the generative capabilities & aesthetic creativity of the system on a wide screen, which can be viewed as a virtual “wall” on which “graffiti” is then painted on.

The designer 3kta describes his objective as an installation that “pretends to be a space open to participation, leaving always the last decisions to the users before the final result of this work… The content users/generators will be able to experience and explore the encounter with the machine in a human's scale physical place. In this way, real and virtual confront themselves completing the meaning of my installation.
I intend that it should be used as a piece of urban furniture with entertaining purposes” [1]

“…it is convenient to remember David Rockeby saying according to which if someone reacts intuitively to an audio or visual event, a closed circuit is created in which music and movement slowly unify themselves.” [1]

This approach involves the audience in all stages of the presentation, which creates an interesting and intriguing installation.

Bibliography

[1] 3kta - http://indicio.net/graffonic/

Assessment 2: Review 2

Audio Cubes by Percussa [1][2]

Design Review

Audio Cubes are 3D Perspex cubes capable of creating unique sounds and musical patterns without twiddling knobs. They are battery powered, run off USB 2.0 and are also MIDI compatible. They do not need any special detection system or surface, and are basically an unconventional DSP/MIDI control unit that displays beautiful colours that synchronise to their inputs. Percussa have attempted to combine musical creation and manipulation with an aesthetic that is both intuitive and artistic – and in my opinion, they have succeeded.

Code Review

The Audio Cubes contain two 1/4 inch jacks for input/output, as well as 4 faces. Each face contains a sensor that can be used for physical manipulation of MIDI data. Percussa recommend using AbletonLive for MIDI data manipulation, as the cubes have been optimised to work with its looping/timing functions.

The cubes do not contain any DSP processing in themselves – rather they are a MIDI controller that can be used to control effects inside Ableton. In this way, an infinite number of effect chains can be chained inside Ableton, and wired to the different faces of the cubes. The cubes can then be chained together, creating a visual representation of DSP inside Ableton. When it is broken down, the coding/programming is very simple, using MIDI In/Thru/Out functions – it is the way in which the Audio Cubes have been aesthetically presented which makes them so interesting and entertaining.

Approach

Percussa have created a tactile sound interaction device that is intuitive and fun to use. By removing all labels on controls, inputs, outputs etc. and using an unconventional shape for a DSP control unit, they have approached the idea of sound generation/manipulation from a different perspective and created a versatile “instrument” that is at home in a professional studio, art installation and everyday situations. The colourful synchronisation also creates a unique visual aspect for live audio performance.

Bibliography

[1] Audio Cubes Online Manual - http://manuals.percussa.com/index.php?title=AudioCubes_Getting_Started_Manual

[2] Percussa - http://www.percussa.com/

Assignment 1 - Sonic Hockey - Paul Montgomery

2008-09-24T16:50:00.000-07:00

Sonic Hockey

Design Basis & Influential Approaches
Our original idea was based on the idea of using everyday objects and activities to create sound. We considered various methods but landed on a simple drawing concept. We intended to use a stylus and pad as an input device and either create sounds and/or manipulate existing sounds through this unique medium. After further consideration we concluded the topic was not only too broad but would also result in an a fairly dull user experience.

With these ideas in mind we decided that we need a more dynamic and exciting input medium. Air hockey is a game that is fun and also provides us with the chance to monitor the position of an input device using new technologies. We decided that an interactive audio installation based on the idea of the air hockey game with a “space, cyber, synth” soundscape will be the compromise we have been seeking.

One of the main concepts with drawing and air hockey was to create something that is fun and playable by anyone, regardless of whether they understand digital audio and interactive technologies. And to collaborate both a simple game idea with audio which involves unique user interaction. Air hockey is basically based on the same principle as drawing, both involve moving objects and x / y interaction.

One of the major influences for our air hockey idea is Music Technology Group’s “reactable” design which involves placing objects that represent modular synthesiser elements on an advanced multi-touch table thus generating and manipulating sound tangibly.

Intended user experience
Obviously air hockey is a two player game thus dictating the requirements for play. However, air hockey is quite an exciting spectator game and passers-by will be interested especially with the interesting sound elements added.
During play sound will be produced based on the speed, direction, orientation and position of the puck (disc). So while the participants are having a rally (playing a game) several different audio events will be occurring. Existing synthesiser and “spacey” sounds will be manipulated along with generated waves/sound based on the above criteria. Each time the disk collides with the player’s bat (handle) or the wall (edge of play) a sound will be triggered. In this case, when the player scores or wins the game, a suitable victory sound will be played. So the two players are not only interacting with one another but also interacting with sound. The experience is somewhat imitating a “video game” feel.

As Air hockey is a fun and simple game to play there are not any age or knowledge requirements for the user. This is optimal because they can play and enjoy the game without realising that they are actually producing and manipulating sound. This basically means, our proposed project is quite suitable for everyone to use.

Interactive Elements
The single most important interactive element in our project is undoubtedly the disc. It will generate different sound depending on its position on the table. So basically, with every movement of the disk, sound is produced. This will provide a unique sonic experience with ever game played. It is interesting because the user doesn’t have direct control over the sound, but he/she can direct the sound by using different strengths, directions and spins of the disc.

We plan to use either a small Bluetooth mouse that is hidden inside the disc or camera tracking in order to track the puck’s location. We will have to experiment and see the most efficient way to achieve what we want. We also consider using a wii remote but conclude it was too large and offered more features than we really needed.
We will use the “MouseState” object in Max/Msp to communicate with the mouse, and define the x/y position for the mouse. The software will read the sound sample file by using the “Buffer” object. By defining the mouse’s position, Max/Msp read those data (x/y position) as just number, then by the “scale” object, Max/Msp scaling the number send by the mouse’s position. When the mouse moves around a certain defined point of the table, the speed and pitch of the sound file will be changed. The data is scaled to restrict the amount of change and keep the computations reasonable.

If we decide to use camera tracking we have to use different colours for each different element on the table to give it a unique identifier.

We intend to use the wii remote buttons to trigger goal sounds. The remote will be positioned in the goal so that is the disc enters the goal it will physically strike the button and trigger the goal sound which can hopefully be randomised so the celebration isn’t predictable.

Possibly, another option to trigger the goal sounds is by using a contact microphone placed at the goal mouth which will detect the sound of the puck hitting the goal and hence trigger the sound. This will limit the risk of not physically triggering a wii remote button and completely missing the sound. It also introduces the complication of determining which sound is a goal and which sound is just game play.

Data Mapping
We have found and intend to use an existing patch called “aka.wiimote” and we have already modified it to work with our own wii remotes. Orignally this patch was designed for communication between the wii remote and computer only, it doesn’t involve any sound elements in the original design.

We have modified the patch and assigned different sound files to different buttons by using the “Buffer” object in Max/Msp to read the sound file. By defining the Boolean of the buttons, we used “select” object, we tested each button for its value when they are pressed, we used “Select (Button’s value)” for on, and “Select 0” for off, so when the button is pressed, it turned on the sound, when the button was released, the sound was turned off.

We decided not to use the axis (x, y, z) of the wii remotes to interact with Max/Msp because we are going to place the wii remotes in a fixed position, the force of the disk will only trigger the buttons, therefore the motion sensor of the wii remote is not useful in this case. The reason for choosing wii remotes were obvious. It can communicate with the computer via Bluetooth and it is relatively accessible, affordable and small.

Audio Output & Audio Types:
We will use two speakers on each side of the air hockey table. The sound produced by the disk’s movement/position will output sound from those two speakers. The sound will come out from the left / right speaker depending on the x position of the disk (spatial panning).

We will have to process and create the “spacey synth sounds” which will be created in ProTools or Reason environment using various digital synthesisers. These will then be catalogued and fed to the buffer in Max/Msp.

For the Wii remote, we are looking at using “crowd” like sound effect to simulate a victory celebration or perhaps something comically space realted. There will be two types of “goal sounds”, one is for the winner and the other is for the loser. Each being played out of the respective player’s speaker(s).

Structure:
We are taking the “Output/ Movement” of the disk to produce sound, the disk is associated with sound when moved. The wii remotes are associated with sound also when struck by the disk.

Materials & Construction:
The way a real air hockey table works is by using air that pushed upwards through little holes in the surface of the table, keeping a continuous layer of air between the air hockey puck and table surface. Since we are not using the real air hockey table, we need to find another solution to be able to simulate this. In order to achieve this we need a very smooth surface and puck in order for the puck to be able to move smoothly. If a good feel for the game can’t be created, the entire integrity of the project will be in jeopardy.

We intend to use an existing simple table as a base to add upon. Lamenex sheeting or Perspex are possible materials to achieve a smooth surface. Lamenex is a material that is often used on bench tops and doors. Lamenex comes in different varieties of smoothness, which should help our cause. The possible problem that could arise from using Perspex is if it becomes scratched, the performance of the puck sliding will be compromised.

Assignment 1 - Serge, Rong & Rhys

2008-09-24T16:43:00.000-07:00

Virtual DJ

We are looking to put together a virtual DJ installation controlled by the
userís hand and/or finger gestures which then trigger samples, scratches,
speed changes, etc.just like a real DJ.

The motivation is to create a fun, interactive and user-friendly DJ
application with no visual interface. Others have developed a similar concept
of DJing with a Wii controller (WiiJing) using buttons and an actual visual interface. We want a more intuitive installation using simple hand/finger gestures and independent of any controller (whether this is feasible is still to be determined).

Initially we were looking at putting together a virtual conductor/maestro installation but this idea has been developed extensively and there is a game for the Nintendo wii called Virtual Maestro.

It is important to engage the user and as this is a sound installation lacking visual elements, we decided to use music - especially drum & bass or techno beats to keep the user interested and immersed. We want to make it easy for the user to play around with the music by using an intuitive and easy method.

The user will have to stand over an encased motion detection camera and put on gloves with special markers located on the fingers, palm and the back of the gloves.

The user will initialize the music through a start/stop marker on the
back of the hand. On the enclosed case there will be a certain number of coloured circles that represent different sample beats or loops. These are initialized every time the user runs their hand over them. The user can only play a certain loop again after it ends but can play multiple loops at the same time.

The main input will be from the motion of the markers located on the gloves captured by the motion tracking camera. This will enable the user to control the soundtrack, volume, play additional samples, perform scratching, alter speed and panning of the stereo speakers. We will be using two speakers (right and left) and the user will also wear headphones to increase immersion and concentration.

By moving a finger on the left hand up and down they control the volume.

Movements of the fingers on the left hand will control things like volume and pitch.

Circular motion of the right hand will cause the music to speed up or to slow down depending on direction. The user will be able to 'scratch the disc' by moving the fingers on the right hand in the appropriate motion.

We may consider enabling panning between speakers as part of the users control.
All of the aforementioned motions and control techniques will need reviewing once we commence testing.
Whether Wii controllers will be better suited to the installation will become clearer after testing.

The markers and motion will either be captured as Boolean data types. These will affect things such as the on/off of the music piece, scratching loop or numbers such as the speed of the track, the volume control, the different samples to be
played and the panning of the stereo speakers. The volume will range from 0 to 10 with 0 being muted. The speed will range from -1 to 1 with 0 being the normal speed and 1 being double the speed. The samples will be numbered from 1 to 5 and can be played together. The panning of the stereo sound will be 1 for right 2 for left and 0 for both.

The audio loops/ tracks/samples will be obtained either from libraries or mixed using Reason software. Prerecorded loops are preferred as this will give us more
time to concentrate on the installation and programming and to sort out all the problems we may come across with input data.

The software we will be using is Max/MSP applets for motion tracking and playing our loops and samples.

Motion tracking involves locating moving objects through a camera. It is very hard to track the object’s location due to the changes of the position relative to the video frame rate. Thus we
may have to define the field that the camera will capture into ‘Macroblocks’.

We may use Reason or other mixing programs to generate our own samples. The hardware we require includes: a motion tracking camera, PC, stereo speakers and headphones.
We believe that calibrating and setting up the camera will be the hardest part of this installation. This will determine whether motion tracking is feasible
or whether using Wii controllers with a Wii remote Applet through Max is going to be more suitable.

Our goal is to make an enjoyable user experience for those who have never experienced 'DJing' before as well as experienced DJ's. We'll also try to make the samples as catchy as possible in order to draw people’s attention.

The Aurora

2008-09-24T14:58:00.000-07:00

Assignment 2 Part 1. By Piers Gilbertson

Approach
The Aurora is described by its makers, (Matt Aldritch, Mike Garbus and Maro Sciacchitano), as being a MIDI DJ mixer.

It came in kitset form, and once assembled The Aurora is powered by and connects to a computer, via serial USB.

Once assembled and connected The Aurora was designed to interface with common music production software, for example Logic or Protools. But using Max/MSP it could be applied to whatever you desired.

Image from http://createdigitalmusic.com

The user interface is divided into 2 halves, or tracks, with a cross -fader between them - very similar to a standard DJ mixer. Each half has 12 dials divided into 4 groups. These groups represent different effect parameters, i.e. filters, and the dials represent variables within that effect i.e. high, mid and low pass filters.

So as a practical example, a DJ could lay out his set tracks in Protools (on 2 separate stereo tracks) allowing the DJ, as he/she played live, to concentrate on the performance effects, such as filters, phasors, modulators, tempo, cross-fade etc.

The other feature The Aurora has is the LED lighting under the main interface, which allows you to control its lighting. The lights can also respond to ambient lighting (see Code below).
The knobs are also illuminated and color-coded by effect to make them easy to see in a darkened performance environment, they also make the interface look cool.

The interface, being open source, and easy to take apart, could be modified and used in many other applications.

Code Review
The Aurora came packaged with some pre-programmed software designed in Max/MSP but bundled for Windows or Mac. It was all fully open source, designed to be modified and manipulated to suit the users needs.
Below is an outline of the bundled patches and their function taken from http://www.auroramixer.com/files.html

Below are example patches from http://createdigitalmusic.com .

1. Aurora synth patch in MAX

2. A tempo controller patch:

3. A pulse wave

Design Review
The Aurora, with some innovation could be a far more useful interface. I personally would love it to use it with Stylus RMX through Logic as there are midi learnable parameters within that plugin far to numerous for your average keyboard to control. The addition of The Aurora would allow the user to take far more advantage of these parameters in live performance.

The major advantage The Aurora has over say a standard DJ mixer is that you can use it to interface with whatever you please! If you are a fan of sourcing and programming patches in Max/MSP then this would provide a fantastic multi purpose controller.

User reviews for The Aurora are somewhat hard to come by as it is sadly out of production as they only received 20 orders.
However some comments on the system taken from http://createdigitalmusic.com include:
• Fiddly knobs
• Lack of distinction between other MIDI controllers
• You need a midi interface (hardware or software) between the Aurora and your computer.

The same site praises The Aurora’s customize-ability and potential applications in terms of both the software and hardware.
Also the projects ethos is also commendable, the developers made all the patches open source and freely available for people to download, modify and use.

Video of the Aurora is available on
http://createdigitalmusic.com/2008/08/27/hands-on-with-aurora-open-source-dj-control-surface-shipping-now/

References

• http://www.auroramixer.com
• http://www.auroramixer.com
• http://createdigitalmusic.com/2008/08/27/hands-on-with-aurora-open-source-dj-control-surface-shipping-now/
• http://cycling74.com/twiki/bin/view/ResourceGuide/InterestingWork
• http://farm4.static.flickr.com/3095/2622267524_2aa7610256.jpg
• http://72.219.216.102:8000/mike/ecs/aurora/files/aurora_224_opensource_dj_mixer_whitepaper.pdf