Message Boards Message Boards

Using your smart phone as the ultimate sensor array for Mathematica

Many fantastic posts in this community describe how to connect external devices to Mathematica and how to read the data. Connecting Mathematica to an Arduino for example allows you to read and then work with data from all kinds of sensors. In most of the cases, when we speak about connected devices, additional hardware is necessary. Smart phones, on the other hand, are our permanent companions and they host a wide array of sensors that we can tap into with Mathematica. For this post, I will be using an iPhone 5 - but a similar approach can be taken with many other smart phones. Björn Schelter and myself have worked on this together.

The first thing we need in order to be able to read the iPhone is a little App which can be purchased on the iTunes App store: it is called Sensor Data. When you open the app you see a screen like this one.

enter image description here

At the top of the screen you see an IP address and a port number (after the colon!). These numbers will be important to connect to the phone and either download data or stream sensor data directly. If you click on the "start capture" the iPhone's data will be stored on the phone and can be downloaded into Mathematica. In this post we are rather interested in the "Streaming" function. If you click on the respective button on the bottom you get to a screen like this:

enter image description here

There you can choose a frequency for the measurements and start the streaming. In fact we also can choose which sensors we want to use with the Config button.

enter image description here

The following Mathematica code will work when all (!) sensors are switched on. Now we are ready to connect to the iPhone. Switch the streaming on and execute the following commands:

ClearAll["Global`*"];
For[i = 1, i < 3, i++, Quiet[InstallJava[]]];
Needs["JLink`"]

and then

LoadJavaClass["java.util.Arrays"];
packet = JavaNew["java.net.DatagramPacket", JavaNew["[B", 1024], 1024];
socket = JavaNew["java.net.DatagramSocket", 10552];
socket@setSoTimeout[10];
listen[] := If[$Failed =!= Quiet[socket@receive[packet], Java::excptn], 
record =JavaNew["java.lang.String", java`util`Arrays`copyOfRange @@ 
packet /@ {getData[], getOffset[], getLength[]}]@toString[] //
Sow];

Next we have to define a ScheduledTask to read the sensors:

RemoveScheduledTask[ScheduledTasks[]];
results = {}; 
RunScheduledTask[AppendTo[results, Quiet[Reap[listen[]][[2, 1]]]]; If[Length[results] > 1200, Drop[results, 150]], 0.01];

We also need to define a streaming function:

stream := Refresh[ToExpression[StringSplit[#[[1]], ","]] & /@ Select[results[[-1000 ;;]], Head[#] == List &], UpdateInterval -> 0.01]

Alright. Now comes the interesting part. Using

(*Compass*)
While[Length[results] < 1000, Pause[2]]; Dynamic[AngularGauge[Refresh[stream[[-1, 30]], UpdateInterval -> 0.01], {360, 0}, 
ScaleDivisions -> None, GaugeLabels -> {Placed["N", Top], Placed["S", Bottom], Placed["E", Right], Placed["W", Left]}, ScaleOrigin -> {{5 Pi/2, Pi/2}, 1}, ScalePadding -> All, ImageSize -> Medium], SynchronousUpdating -> False]

we can measure the bearing of our iPhone. The resulting compass moves as we move the iPhone:

enter image description here

We can also read the (x-,y-,z-) accelerometers

(*Plot Ascelerometers*)
While[Length[results] < 1000, Pause[2]]; Dynamic[Refresh[ListLinePlot[{stream[[All, 2]], stream[[All, 3]], stream[[All, 4]]}, PlotRange -> All], UpdateInterval -> 0.1]]

which gives plots like this one:

enter image description here

The update is a bit bumpy, because the data is only sent every second or so from the iPhone; the measurements, however, are taken with a frequency of up to 100Hz. We can also represent the FFT of the streamed data like so:

(*Plot FFT of accelorometers*)
While[Length[results] < 1000, 
 Pause[2]]; Dynamic[
 Refresh[ListLinePlot[
   Log /@ {Abs[Fourier[Standardize[stream[[All, 2]]]]], 
     Abs[Fourier[Standardize[stream[[All, 3]]]]], 
     Abs[Fourier[Standardize[stream[[All, 4]]]]]}, 
   PlotRange -> {{0, 200}, {-5, 2.5}}, ImageSize -> Large], 
  UpdateInterval -> 0.1]]

Adding a "real time" scale is also quite straight forward:

(Measurements with time scale)

While[Length[results] < 1000, Pause[2]];
starttime = IntegerPart[stream[[2, 1]]];
Dynamic[Refresh[
  ListLinePlot[
   Transpose[{(stream[[Max[-300, -Length[stream]] ;;, 1]] - 
       starttime), stream[[Max[-300, -Length[stream]] ;;, 2]]}], 
   PlotRange -> All, ImageSize -> Large], UpdateInterval -> 0.01]]

Well, then. We can also plot our iPhone's position in space

(*3d Motion*)

While[Length[results] < 1000, Pause[2]]; Dynamic[
 Refresh[ListLinePlot[{stream[[All, 5]], stream[[All, 6]], 
    stream[[All, 7]]}, PlotRange -> All], UpdateInterval -> 0.1]]

While[Length[results] < 1000, Pause[2]]; Dynamic[
 Graphics3D[{Black, 
   Rotate[Rotate[
     Rotate[Cuboid[{-2, -1, -0.2}, {2, 1, 0.2}], 
      stream[[-1, 7]], {0, 0, 1}], -1*stream[[-1, 6]], {0, 1, 0}], 
    stream[[-1, 5]], {1, 0, 0}]}, 
  PlotRange -> {{-3, 3}, {-3, 3}, {-3, 3}}, Boxed -> True], 
 UpdateInterval -> 0.1, SynchronousUpdating -> False]

This looks like so:

enter image description here

Last but not least we can write a little GUI to access all different sensors. (This does run a bit slow though!)

(GUI all sensors)

sensororder = {"Timestamp", "Accel_X", "Accel_Y", "Accel_Z", "Roll", 
   "Pitch", "Yaw", "Quat.X", "Quat.Y", "Quat.Z", "Quat.W", "RM11", 
   "RM12", "RM13", "RM21", "RM22", "RM23", "RM31", "RM32", "RM33", 
   "GravAcc_X", "GravAcc_Y", "GravAcc_Z", "UserAcc_X", "UserAcc_Y", 
   "UserAcc_Z", "RotRate_X", "RotRate_Y", "RotRate_Z", "MagHeading", 
   "TrueHeading", "HeadingAccuracy", "MagX", "MagY", "MagZ", "Lat", 
   "Long", "LocAccuracy", "Course", "Speed", "Altitude", 
   "Proximity"};
While[Length[results] < 1000, Pause[2]]; Manipulate[
 Dynamic[Refresh[
   ListLinePlot[{stream[[All, Position[sensororder, a][[1, 1]]]], 
     stream[[All, Position[sensororder, b][[1, 1]]]], 
     stream[[All, Position[sensororder, c][[1, 1]]]]}, 
    PlotRange -> All, ImageSize -> Full], 
   UpdateInterval -> 0.01]], {{a, "Accel_X"}, 
  sensororder}, {{b, "Accel_Y"}, sensororder}, {{c, "Accel_Z"}, 
  sensororder}, ControlPlacement -> Left, 
 SynchronousUpdating -> False]

This gives a user interface which looks like this:

enter image description here

In the drop down menu we can choose three out of all sensors. These are all available sensors:

"Timestamp", "AccelX", "AccelY", "Accel_Z", "Roll", "Pitch", "Yaw", "Quat.X", "Quat.Y", "Quat.Z", "Quat.W", "RM11", "RM12", "RM13", "RM21", "RM22", "RM23", "RM31", "RM32", "RM33", "GravAcc_X", "GravAccY", "GravAccZ", "UserAccX", "UserAccY", "UserAcc_Z", "RotRateX", "RotRateY", "RotRate_Z", "MagHeading", "TrueHeading", "HeadingAccuracy", "MagX", "MagY", "MagZ", "Lat", "Long", "LocAccuracy", "Course", "Speed", "Altitude", "Proximity"

There are certainly any things that can and should be improved. The main problem seems to be that the data, even if sampled at 100Hz, is sent to the iPhone only every second or so. So it is not really real time. I hope that someone who is better at iPhone programming than I am - I am really rubbish at it- could help and write an iPhone program to stream the data in a more convenient way: one by one rather than in packets.

There are many potential applications for this. Here are some I could come up with:

  1. You can carry the iPhone around and measure your movements (acceleration). Attached to your hand you can measure your tremor.
  2. The magnetometer is really cool. You can use it to find metal bars in the walls an also electric cables.
  3. You can collect GPS data for all sorts of applications; there are ideas to use this for the detection of certain diseases. For example if it takes you longer than usual to find your car when you come from shopping that might hint at early stages of dementia --- or sleep deprivation.
  4. When you put the phone on a machine, like a running motor, you can measure the vibrations. When you perform a frequency analysis you can check whether the motor runs alright.
  5. Using the accelerometers I was able to measure my breathing (putting the phone on my chest).

I think that there might also be quite some potential for using the Wolfram Cloud here. Deploying a program in the cloud and reading from your phone is certainly quite interesting. The problem is that this particular app only works via WiFi. It would be nice to have one that works via 3G.

So, in summary, it might be quite useful to use the iPhone's sensors. The advantage is that nearly everyone carries a smartphone with them all the time. Making more of your smart phone's sensors with Mathematica seems to be a nice playground for applications. I'd love to hear about your ideas...

Cheers,

Marco

PS: When you are done with the streaming you should execute these commands:

(*Remove Scheduled Tasks and close link*)
RemoveScheduledTask[ScheduledTasks[]]; socket@close[];
POSTED BY: Marco Thiel
15 Replies

I no longer see the sensor data app on the IOS store. But see a few other new ones. Is there one that you recommend?

POSTED BY: Jack I Houng

IS there a way to associate the WM code to the phone running on a separate network? Let say that the Phone is running using its own Data Service but my computer is in a separate network. Where do I place the IP in the code since I do not see this address anywhere in the code.

POSTED BY: Jose Calderon

Marco, excellent ideas and thanks for sharing, I've noticed that after some of the modules executed, they seem to keep running in my application (I am using Mathematica v10 in a OSx) is this expected?

Again thanks,

Salva

POSTED BY: Salvador Romo

How does socket get associated with 192.168.1.95:54388? socket is instanced on Port 10552, but what does that have to do with anything?

Thanks for any hint to get me started.

Fred

POSTED BY: Fred Klingener
Posted 9 years ago

Marco,

Can you explain this code fragment from your JLink processing?

packet = JavaNew["java.net.DatagramPacket", JavaNew["[B", 1024], 1024];
POSTED BY: David G

Hi,

"packet" in this case is an object created from the class"java.net.DatagramPacket". The arguments "JavaNew["[B", 1024]" and "1024" depend on the class you are dealing with. For example, I can have a class "ConnectionUDPClient.class" which I can call by obj = JavaNew["ConnectionUDPClient"].

Furthermore, it is possible to access methods inside the class. For the class above, I can have a method "sendMessage" which I can invoke by "obj@sendMessage". Finally, depending on the implemented method, it could also be possible to pass additional arguments. Hope this helps.

Kind regards,

Ricardo

Posted 9 years ago

Ricardo,

I'm a little foggy on the "[B" argument of the nested JavaNew. I think I've seen that before, but I don't recall that notation. I assume it's specifying a buffer of some sort?

Thanks for the help.

David

POSTED BY: David G

Thanks. I'll try it out. I'm running OSX 10.10.5 on my laptop, and Mathematica 10.3.

But I was also trying to run it through the cloud so I could access it from the phone through the Wolfram Cloud app. Where I was seeing blue was on on dev.wolframcloud.com/.

POSTED BY: Kathryn Cramer

Hi Marco:

I have an iPhone 6 and I'm trying this out. In what environment are you running the Wolfram Language code? Since I don't see a place to plug in the URL for the data, I presume you are running it in on the Wolfram Development Platform using the Wolfram Cloud app on the phone itself.

But functions InstallJava[], LoadJavaClass[], JavaNew[] come up in blue for me, suggesting that they are not currently available on the Development Platform.

So how are you getting this to work? What should I be doing differently?

Kathryn

POSTED BY: Kathryn Cramer

Dear Kathryn,

I am sorry if this didn't work for you. I am attaching the code that works for me and some students of mine. I am very sorry for not annotating it correctly, but I need to prepare a presentation for tomorrow and wanted to reply as swiftly as possible. I ran this piece of code on a couple of independent OSX machines and it appears to be working fine on all of them. I tested it on everything from iPhone 4 to iPhone 6. You are saying that you ran it in the Cloud, which I did not do. I ran it on a local machine, and laptop and iPhone were in the same network. I have not used the Wolfram Cloud app, but an iOS app from the app store that is called "Sensor Data". I think that it is expected that it will not work in the Cloud/Wolfram Development platform.

I just ran that code on MMA10.3 on OSX.

enter image description here

Note that when I run it, there are several functions in blue (e.g. setSoTimeout[10], receive, getData[]) but InstallJava[], LoadJavaClass[], JavaNew[] are black. Here's a screenshot.

enter image description here

As you work at Wolfram I suppose that you also are running MMA10.3. Which operating system are you using? I have asked a colleague of mine at our institute to post his code, which is a self-written code for Androids.

I think that it would be possible to make this "work" in the cloud. If you use, for example, DataDrop you can achieve something similar, with the exception that the update rate of data drop is nominally 2 Hz, so you would not be much faster than that.

If you cannot make it work I would be happy to screen-share and demonstrate how the program works.

Best wishes from Aberdeen,

Marco

Attachments:
POSTED BY: Marco Thiel
Posted 9 years ago
POSTED BY: Alexey Popkov

Hi Alexey,

yes there are loads. We also wrote a little piece of code for Androids. It is somewhere on the Android Store. I will try to "extract" the program from a Post-Doc and post it here.

Cheers,

Marco

POSTED BY: Marco Thiel
Posted 9 years ago

okey, the Mathematica code is written to get the streaming data for any source connected into that ip address and port?

By the way, this will help me out...

Thanks

POSTED BY: Fabio Burgos
Posted 10 years ago

Such a big and informative info.

POSTED BY: Benjamin Stuart
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract