Message Boards Message Boards

Camera controlled musical instrument

GROUPS:
Here is a very quick and dirty Mathematica implementation of the idea discussed in this blog. May require tinkering with the value of "threshold".
 With[{keySize = 40, threshold = 0.05},
  DynamicModule[{image, images, old = Table[True, {320/keySize}], new,
    notes = {1, 3, 6, 8, 10, 13, 15, 18},
    overlayImage = Image[Table[
       If[i === keySize || i < keySize && Mod[j, keySize] === 0, 1, 0], {i, 1, 240}, {j, 1, 320}]]},
   Dynamic[image = ImageDifference @@ (images = CurrentImage[2]);
    new = (Mean[Flatten[ImageData[#]]] > threshold) & /@
      Flatten[ImagePartition[ImageTake[image, keySize], keySize]];
    EmitSound[Sound[SoundNote[notes[[First /@ Position[Transpose[{old, new}], {False, True}]]]]]];
   old = new;
   ImageAdd[images[[2]], overlayImage],
   SynchronousUpdating -> False]]]
POSTED BY: Christopher Carlson
Answer
5 years ago
It’s very neat. Because the camera is for the users to perceive real image – it does not work as a mirror. Meaning if I move to the right my camera image will move to the left. This is a bit inconvenient for playing the instrument though. I'd like to see my screen hand moving in the same direction as my real hand. We could replace this line
images = CurrentImage[2]
with something like this
images = ImageReflect[#, Left -> Right] & /@ CurrentImage[2]
This is fun. Though one probably can’t play Toccata and Fugue in D minor, I’d challenge anyone for Der Flohwalzer. Thanks Chris!
POSTED BY: Vitaliy Kaurov
Answer
5 years ago
Cool.  But one thing I noticed is that because you write

images = CurrentImage[2]

the framerate is divided by two.  A better way to do it might be

image=CurrentImage[];
oldimage=CurrentImage[];
Dynamic[oldimage=image;
image=CurrentImage[]]

Here is a rewritten faster version


 With[{keySize = 40, threshold = 0.05},
  With[{overlay =
     Image[Table[
       If[i === keySize || i < keySize && Mod[j, keySize] === 0, 1,
        0], {i, 1, 240}, {j, 1, 320}]]},
   DynamicModule[{image = ImageReflect[CurrentImage[], Left -> Right],
     oldImage = ImageReflect[CurrentImage[], Left -> Right], imageKeys,
      diff, imageKeysDiff, keysPressed},
    Dynamic[
    oldImage = image;
    image = ImageReflect[CurrentImage[], Left -> Right];
    imageKeys =
     Flatten[ImagePartition[ImageTake[image, keySize], keySize]];
    diff = ImageDifference[oldImage, image];
    imageKeysDiff =
     Flatten[ImagePartition[ImageTake[diff, keySize], keySize]];
    keysPressed =
     Flatten[Position[
       Mean[Flatten[ImageData[#]]] > threshold & /@ imageKeysDiff,
       True]];
    EmitSound[Sound[SoundNote[#, .2, "Xylophone"]]] & /@ keysPressed;
    Show[ImageAdd[image, overlay], ImageSize -> 1000]
    ]]]]
POSTED BY: Christopher Wolfram
Answer
5 years ago
Nice! Maybe incorporate it into a Manipulate app for selection of instrument, octave, etc.
POSTED BY: Jesse Friedman
Answer
5 years ago

Group Abstract Group Abstract