Camera controlled musical instrument

Ideas
, , ,
1

Here is a very quick and dirty Mathematica implementation of the idea discussed in this blog. May require tinkering with the value of “threshold”.

With[{keySize = 40, threshold = 0.05}, 
 DynamicModule[{image, images, old = Table[True, {320/keySize}], new, 
   notes = {1, 3, 6, 8, 10, 13, 15, 18}, 
   overlayImage = Image[Table[
      If[i === keySize || i < keySize && Mod[j, keySize] === 0, 1, 0], {i, 1, 240}, {j, 1, 320}]]}, 
  Dynamic[image = ImageDifference @@ (images = CurrentImage[2]);
   new = (Mean[Flatten[ImageData[#]]] > threshold) & /@ 
     Flatten[ImagePartition[ImageTake[image, keySize], keySize]];
   EmitSound[Sound[SoundNote[notes[[First /@ Position[Transpose[{old, new}], {False, True}]]]]]];
   old = new;
   ImageAdd[images[[2]], overlayImage], 
   SynchronousUpdating -> False]]]
11 Likes
2

It’s very neat. Because the camera is for the users to perceive real image – it does not work as a mirror. Meaning if I move to the right my camera image will move to the left. This is a bit inconvenient for playing the instrument though. I’d like to see my screen hand moving in the same direction as my real hand. We could replace this line

images = CurrentImage[2]

with something like this

images = ImageReflect[#, Left -> Right] & /@ CurrentImage[2]

This is fun. Though one probably can’t play Toccata and Fugue in D minor, I’d challenge anyone for Der Flohwalzer. Thanks Chris!

3 Likes
3

Cool. But one thing I noticed is that because you write

images = CurrentImage[2]

the framerate is divided by two. A better way to do it might be

image=CurrentImage[];
oldimage=CurrentImage[];
Dynamic[oldimage=image;
image=CurrentImage[]]

Here is a rewritten faster version

With[{keySize = 40, threshold = 0.05},
 With[{overlay = 
    Image[Table[
      If[i === keySize || i < keySize && Mod[j, keySize] === 0, 1, 
       0], {i, 1, 240}, {j, 1, 320}]]},
  DynamicModule[{image = ImageReflect[CurrentImage[], Left -> Right], 
    oldImage = ImageReflect[CurrentImage[], Left -> Right], imageKeys,
     diff, imageKeysDiff, keysPressed},
   Dynamic[
    oldImage = image;
    image = ImageReflect[CurrentImage[], Left -> Right];
    imageKeys = 
     Flatten[ImagePartition[ImageTake[image, keySize], keySize]];
    diff = ImageDifference[oldImage, image];
    imageKeysDiff = 
     Flatten[ImagePartition[ImageTake[diff, keySize], keySize]];
    keysPressed = 
     Flatten[Position[
       Mean[Flatten[ImageData[#]]] > threshold & /@ imageKeysDiff, 
       True]];
    EmitSound[Sound[SoundNote[#, .2, "Xylophone"]]] & /@ keysPressed;
    Show[ImageAdd[image, overlay], ImageSize -> 1000]
    ]]]]
3 Likes
4

Nice! Maybe incorporate it into a Manipulate app for selection of instrument, octave, etc.