Many fantastic posts in this community describe how to connect external devices to Mathematica and how to read the data. Connecting Mathematica to an Arduino for example allows you to read and then work with data from all kinds of sensors. In most of the cases, when we speak about connected devices, additional hardware is necessary. Smart phones, on the other hand, are our permanent companions and they host a wide array of sensors that we can tap into with Mathematica. For this post, I will be using an iPhone 5 - but a similar approach can be taken with many other smart phones. Björn Schelter and myself have worked on this together.

The first thing we need in order to be able to read the iPhone is a little App which can be purchased on the iTunes App store: it is called Sensor Data. When you open the app you see a screen like this one.

At the top of the screen you see an IP address and a port number (after the colon!). These numbers will be important to connect to the phone and either download data or stream sensor data directly. If you click on the "start capture" the iPhone's data will be stored on the phone and can be downloaded into Mathematica. In this post we are rather interested in the "Streaming" function. If you click on the respective button on the bottom you get to a screen like this:

There you can choose a frequency for the measurements and start the streaming. In fact we also can choose which sensors we want to use with the Config button.

The following Mathematica code will work when all (!) sensors are switched on. Now we are ready to connect to the iPhone. Switch the streaming on and execute the following commands:

ClearAll["Global`*"];
For[i = 1, i < 3, i++, Quiet[InstallJava[]]];
Needs["JLink`"]

and then

LoadJavaClass["java.util.Arrays"];
packet = JavaNew["java.net.DatagramPacket", JavaNew["[B", 1024], 1024];
socket = JavaNew["java.net.DatagramSocket", 10552];
socket@setSoTimeout[10];
listen[] := If[$Failed =!= Quiet[socket@receive[packet], Java::excptn], 
record =JavaNew["java.lang.String", java`util`Arrays`copyOfRange @@ 
packet /@ {getData[], getOffset[], getLength[]}]@toString[] //
Sow];

Next we have to define a ScheduledTask to read the sensors:

RemoveScheduledTask[ScheduledTasks[]];
results = {}; 
RunScheduledTask[AppendTo[results, Quiet[Reap[listen[]][[2, 1]]]]; If[Length[results] > 1200, Drop[results, 150]], 0.01];

We also need to define a streaming function:

stream := Refresh[ToExpression[StringSplit[#[[1]], ","]] & /@ Select[results[[-1000 ;;]], Head[#] == List &], UpdateInterval -> 0.01]

Alright. Now comes the interesting part. Using

(*Compass*)
While[Length[results] < 1000, Pause[2]]; Dynamic[AngularGauge[Refresh[stream[[-1, 30]], UpdateInterval -> 0.01], {360, 0}, 
ScaleDivisions -> None, GaugeLabels -> {Placed["N", Top], Placed["S", Bottom], Placed["E", Right], Placed["W", Left]}, ScaleOrigin -> {{5 Pi/2, Pi/2}, 1}, ScalePadding -> All, ImageSize -> Medium], SynchronousUpdating -> False]

we can measure the bearing of our iPhone. The resulting compass moves as we move the iPhone:

We can also read the (x-,y-,z-) accelerometers

(*Plot Ascelerometers*)
While[Length[results] < 1000, Pause[2]]; Dynamic[Refresh[ListLinePlot[{stream[[All, 2]], stream[[All, 3]], stream[[All, 4]]}, PlotRange -> All], UpdateInterval -> 0.1]]

which gives plots like this one:

The update is a bit bumpy, because the data is only sent every second or so from the iPhone; the measurements, however, are taken with a frequency of up to 100Hz. We can also represent the FFT of the streamed data like so:

(*Plot FFT of accelorometers*)
While[Length[results] < 1000, 
 Pause[2]]; Dynamic[
 Refresh[ListLinePlot[
   Log /@ {Abs[Fourier[Standardize[stream[[All, 2]]]]], 
     Abs[Fourier[Standardize[stream[[All, 3]]]]], 
     Abs[Fourier[Standardize[stream[[All, 4]]]]]}, 
   PlotRange -> {{0, 200}, {-5, 2.5}}, ImageSize -> Large], 
  UpdateInterval -> 0.1]]

Adding a "real time" scale is also quite straight forward:

(Measurements with time scale)

While[Length[results] < 1000, Pause[2]];
starttime = IntegerPart[stream[[2, 1]]];
Dynamic[Refresh[
  ListLinePlot[
   Transpose[{(stream[[Max[-300, -Length[stream]] ;;, 1]] - 
       starttime), stream[[Max[-300, -Length[stream]] ;;, 2]]}], 
   PlotRange -> All, ImageSize -> Large], UpdateInterval -> 0.01]]

Well, then. We can also plot our iPhone's position in space

(*3d Motion*)

While[Length[results] < 1000, Pause[2]]; Dynamic[
 Refresh[ListLinePlot[{stream[[All, 5]], stream[[All, 6]], 
    stream[[All, 7]]}, PlotRange -> All], UpdateInterval -> 0.1]]

While[Length[results] < 1000, Pause[2]]; Dynamic[
 Graphics3D[{Black, 
   Rotate[Rotate[
     Rotate[Cuboid[{-2, -1, -0.2}, {2, 1, 0.2}], 
      stream[[-1, 7]], {0, 0, 1}], -1*stream[[-1, 6]], {0, 1, 0}], 
    stream[[-1, 5]], {1, 0, 0}]}, 
  PlotRange -> {{-3, 3}, {-3, 3}, {-3, 3}}, Boxed -> True], 
 UpdateInterval -> 0.1, SynchronousUpdating -> False]

This looks like so:

Last but not least we can write a little GUI to access all different sensors. (This does run a bit slow though!)

(GUI all sensors)

sensororder = {"Timestamp", "Accel_X", "Accel_Y", "Accel_Z", "Roll", 
   "Pitch", "Yaw", "Quat.X", "Quat.Y", "Quat.Z", "Quat.W", "RM11", 
   "RM12", "RM13", "RM21", "RM22", "RM23", "RM31", "RM32", "RM33", 
   "GravAcc_X", "GravAcc_Y", "GravAcc_Z", "UserAcc_X", "UserAcc_Y", 
   "UserAcc_Z", "RotRate_X", "RotRate_Y", "RotRate_Z", "MagHeading", 
   "TrueHeading", "HeadingAccuracy", "MagX", "MagY", "MagZ", "Lat", 
   "Long", "LocAccuracy", "Course", "Speed", "Altitude", 
   "Proximity"};
While[Length[results] < 1000, Pause[2]]; Manipulate[
 Dynamic[Refresh[
   ListLinePlot[{stream[[All, Position[sensororder, a][[1, 1]]]], 
     stream[[All, Position[sensororder, b][[1, 1]]]], 
     stream[[All, Position[sensororder, c][[1, 1]]]]}, 
    PlotRange -> All, ImageSize -> Full], 
   UpdateInterval -> 0.01]], {{a, "Accel_X"}, 
  sensororder}, {{b, "Accel_Y"}, sensororder}, {{c, "Accel_Z"}, 
  sensororder}, ControlPlacement -> Left, 
 SynchronousUpdating -> False]

This gives a user interface which looks like this:

In the drop down menu we can choose three out of all sensors. These are all available sensors:

"Timestamp", "AccelX", "AccelY", "Accel_Z", "Roll", "Pitch", "Yaw", "Quat.X", "Quat.Y", "Quat.Z", "Quat.W", "RM11", "RM12", "RM13", "RM21", "RM22", "RM23", "RM31", "RM32", "RM33", "GravAcc_X", "GravAccY", "GravAccZ", "UserAccX", "UserAccY", "UserAcc_Z", "RotRateX", "RotRateY", "RotRate_Z", "MagHeading", "TrueHeading", "HeadingAccuracy", "MagX", "MagY", "MagZ", "Lat", "Long", "LocAccuracy", "Course", "Speed", "Altitude", "Proximity"

There are certainly any things that can and should be improved. The main problem seems to be that the data, even if sampled at 100Hz, is sent to the iPhone only every second or so. So it is not really real time. I hope that someone who is better at iPhone programming than I am - I am really rubbish at it- could help and write an iPhone program to stream the data in a more convenient way: one by one rather than in packets.

There are many potential applications for this. Here are some I could come up with:

1. You can carry the iPhone around and measure your movements (acceleration). Attached to your hand you can measure your tremor.
2. The magnetometer is really cool. You can use it to find metal bars in the walls an also electric cables.
3. You can collect GPS data for all sorts of applications; there are ideas to use this for the detection of certain diseases. For example if it takes you longer than usual to find your car when you come from shopping that might hint at early stages of dementia --- or sleep deprivation.
4. When you put the phone on a machine, like a running motor, you can measure the vibrations. When you perform a frequency analysis you can check whether the motor runs alright.
5. Using the accelerometers I was able to measure my breathing (putting the phone on my chest).

I think that there might also be quite some potential for using the Wolfram Cloud here. Deploying a program in the cloud and reading from your phone is certainly quite interesting. The problem is that this particular app only works via WiFi. It would be nice to have one that works via 3G.

So, in summary, it might be quite useful to use the iPhone's sensors. The advantage is that nearly everyone carries a smartphone with them all the time. Making more of your smart phone's sensors with Mathematica seems to be a nice playground for applications. I'd love to hear about your ideas...

Cheers,

Marco

PS: When you are done with the streaming you should execute these commands:

(*Remove Scheduled Tasks and close link*)
RemoveScheduledTask[ScheduledTasks[]]; socket@close[];