# Camera controlled musical instrument

Posted 11 years ago
15125 Views
|
3 Replies
|
17 Total Likes
|
 Here is a very quick and dirty Mathematica implementation of the idea discussed in this blog. May require tinkering with the value of "threshold". With[{keySize = 40, threshold = 0.05},   DynamicModule[{image, images, old = Table[True, {320/keySize}], new,     notes = {1, 3, 6, 8, 10, 13, 15, 18},     overlayImage = Image[Table[       If[i === keySize || i < keySize && Mod[j, keySize] === 0, 1, 0], {i, 1, 240}, {j, 1, 320}]]},    Dynamic[image = ImageDifference @@ (images = CurrentImage[2]);    new = (Mean[Flatten[ImageData[#]]] > threshold) & /@       Flatten[ImagePartition[ImageTake[image, keySize], keySize]];    EmitSound[Sound[SoundNote[notes[[First /@ Position[Transpose[{old, new}], {False, True}]]]]]];   old = new;   ImageAdd[images[[2]], overlayImage],    SynchronousUpdating -> False]]]
3 Replies
Sort By:
Posted 11 years ago
 Its very neat. Because the camera is for the users to perceive real image  it does not work as a mirror. Meaning if I move to the right my camera image will move to the left. This is a bit inconvenient for playing the instrument though. I'd like to see my screen hand moving in the same direction as my real hand. We could replace this line images = CurrentImage[2]with something like thisimages = ImageReflect[#, Left -> Right] & /@ CurrentImage[2]This is fun. Though one probably cant play Toccata and Fugue in D minor, Id challenge anyone for Der Flohwalzer. Thanks Chris!
Posted 11 years ago
 Cool.  But one thing I noticed is that because you writeimages = CurrentImage[2]the framerate is divided by two.  A better way to do it might beimage=CurrentImage[];oldimage=CurrentImage[];Dynamic[oldimage=image;image=CurrentImage[]]Here is a rewritten faster version With[{keySize = 40, threshold = 0.05},  With[{overlay =      Image[Table[       If[i === keySize || i < keySize && Mod[j, keySize] === 0, 1,         0], {i, 1, 240}, {j, 1, 320}]]},   DynamicModule[{image = ImageReflect[CurrentImage[], Left -> Right],      oldImage = ImageReflect[CurrentImage[], Left -> Right], imageKeys,      diff, imageKeysDiff, keysPressed},    Dynamic[    oldImage = image;    image = ImageReflect[CurrentImage[], Left -> Right];    imageKeys =      Flatten[ImagePartition[ImageTake[image, keySize], keySize]];    diff = ImageDifference[oldImage, image];    imageKeysDiff =      Flatten[ImagePartition[ImageTake[diff, keySize], keySize]];    keysPressed =      Flatten[Position[       Mean[Flatten[ImageData[#]]] > threshold & /@ imageKeysDiff,        True]];    EmitSound[Sound[SoundNote[#, .2, "Xylophone"]]] & /@ keysPressed;    Show[ImageAdd[image, overlay], ImageSize -> 1000]    ]]]]
Posted 11 years ago
 Nice! Maybe incorporate it into a Manipulate app for selection of instrument, octave, etc.