Using your smart phone as the ultimate sensor array for Mathematica

Posted 6 years ago
42240 Views
|
14 Replies
|
33 Total Likes
|
 Many fantastic posts in this community describe how to connect external devices to Mathematica and how to read the data. Connecting Mathematica to an Arduino for example allows you to read and then work with data from all kinds of sensors. In most of the cases, when we speak about connected devices, additional hardware is necessary. Smart phones, on the other hand, are our permanent companions and they host a wide array of sensors that we can tap into with Mathematica. For this post, I will be using an iPhone 5 - but a similar approach can be taken with many other smart phones. Björn Schelter and myself have worked on this together.The first thing we need in order to be able to read the iPhone is a little App which can be purchased on the iTunes App store: it is called Sensor Data. When you open the app you see a screen like this one. At the top of the screen you see an IP address and a port number (after the colon!). These numbers will be important to connect to the phone and either download data or stream sensor data directly. If you click on the "start capture" the iPhone's data will be stored on the phone and can be downloaded into Mathematica. In this post we are rather interested in the "Streaming" function. If you click on the respective button on the bottom you get to a screen like this:There you can choose a frequency for the measurements and start the streaming. In fact we also can choose which sensors we want to use with the Config button. The following Mathematica code will work when all (!) sensors are switched on. Now we are ready to connect to the iPhone. Switch the streaming on and execute the following commands: ClearAll["Global*"]; For[i = 1, i < 3, i++, Quiet[InstallJava[]]]; Needs["JLink"] and then LoadJavaClass["java.util.Arrays"]; packet = JavaNew["java.net.DatagramPacket", JavaNew["[B", 1024], 1024]; socket = JavaNew["java.net.DatagramSocket", 10552]; socket@setSoTimeout[10]; listen[] := If[\$Failed =!= Quiet[socket@receive[packet], Java::excptn], record =JavaNew["java.lang.String", javautilArrayscopyOfRange @@ packet /@ {getData[], getOffset[], getLength[]}]@toString[] // Sow]; Next we have to define a ScheduledTask to read the sensors: RemoveScheduledTask[ScheduledTasks[]]; results = {}; RunScheduledTask[AppendTo[results, Quiet[Reap[listen[]][[2, 1]]]]; If[Length[results] > 1200, Drop[results, 150]], 0.01]; We also need to define a streaming function: stream := Refresh[ToExpression[StringSplit[#[[1]], ","]] & /@ Select[results[[-1000 ;;]], Head[#] == List &], UpdateInterval -> 0.01] Alright. Now comes the interesting part. Using (*Compass*) While[Length[results] < 1000, Pause[2]]; Dynamic[AngularGauge[Refresh[stream[[-1, 30]], UpdateInterval -> 0.01], {360, 0}, ScaleDivisions -> None, GaugeLabels -> {Placed["N", Top], Placed["S", Bottom], Placed["E", Right], Placed["W", Left]}, ScaleOrigin -> {{5 Pi/2, Pi/2}, 1}, ScalePadding -> All, ImageSize -> Medium], SynchronousUpdating -> False] we can measure the bearing of our iPhone. The resulting compass moves as we move the iPhone:We can also read the (x-,y-,z-) accelerometers (*Plot Ascelerometers*) While[Length[results] < 1000, Pause[2]]; Dynamic[Refresh[ListLinePlot[{stream[[All, 2]], stream[[All, 3]], stream[[All, 4]]}, PlotRange -> All], UpdateInterval -> 0.1]] which gives plots like this one:The update is a bit bumpy, because the data is only sent every second or so from the iPhone; the measurements, however, are taken with a frequency of up to 100Hz. We can also represent the FFT of the streamed data like so: (*Plot FFT of accelorometers*) While[Length[results] < 1000, Pause[2]]; Dynamic[ Refresh[ListLinePlot[ Log /@ {Abs[Fourier[Standardize[stream[[All, 2]]]]], Abs[Fourier[Standardize[stream[[All, 3]]]]], Abs[Fourier[Standardize[stream[[All, 4]]]]]}, PlotRange -> {{0, 200}, {-5, 2.5}}, ImageSize -> Large], UpdateInterval -> 0.1]] Adding a "real time" scale is also quite straight forward:(Measurements with time scale) While[Length[results] < 1000, Pause[2]]; starttime = IntegerPart[stream[[2, 1]]]; Dynamic[Refresh[ ListLinePlot[ Transpose[{(stream[[Max[-300, -Length[stream]] ;;, 1]] - starttime), stream[[Max[-300, -Length[stream]] ;;, 2]]}], PlotRange -> All, ImageSize -> Large], UpdateInterval -> 0.01]] Well, then. We can also plot our iPhone's position in space (*3d Motion*) While[Length[results] < 1000, Pause[2]]; Dynamic[ Refresh[ListLinePlot[{stream[[All, 5]], stream[[All, 6]], stream[[All, 7]]}, PlotRange -> All], UpdateInterval -> 0.1]] While[Length[results] < 1000, Pause[2]]; Dynamic[ Graphics3D[{Black, Rotate[Rotate[ Rotate[Cuboid[{-2, -1, -0.2}, {2, 1, 0.2}], stream[[-1, 7]], {0, 0, 1}], -1*stream[[-1, 6]], {0, 1, 0}], stream[[-1, 5]], {1, 0, 0}]}, PlotRange -> {{-3, 3}, {-3, 3}, {-3, 3}}, Boxed -> True], UpdateInterval -> 0.1, SynchronousUpdating -> False] This looks like so:Last but not least we can write a little GUI to access all different sensors. (This does run a bit slow though!)(GUI all sensors) sensororder = {"Timestamp", "Accel_X", "Accel_Y", "Accel_Z", "Roll", "Pitch", "Yaw", "Quat.X", "Quat.Y", "Quat.Z", "Quat.W", "RM11", "RM12", "RM13", "RM21", "RM22", "RM23", "RM31", "RM32", "RM33", "GravAcc_X", "GravAcc_Y", "GravAcc_Z", "UserAcc_X", "UserAcc_Y", "UserAcc_Z", "RotRate_X", "RotRate_Y", "RotRate_Z", "MagHeading", "TrueHeading", "HeadingAccuracy", "MagX", "MagY", "MagZ", "Lat", "Long", "LocAccuracy", "Course", "Speed", "Altitude", "Proximity"}; While[Length[results] < 1000, Pause[2]]; Manipulate[ Dynamic[Refresh[ ListLinePlot[{stream[[All, Position[sensororder, a][[1, 1]]]], stream[[All, Position[sensororder, b][[1, 1]]]], stream[[All, Position[sensororder, c][[1, 1]]]]}, PlotRange -> All, ImageSize -> Full], UpdateInterval -> 0.01]], {{a, "Accel_X"}, sensororder}, {{b, "Accel_Y"}, sensororder}, {{c, "Accel_Z"}, sensororder}, ControlPlacement -> Left, SynchronousUpdating -> False] This gives a user interface which looks like this:In the drop down menu we can choose three out of all sensors. These are all available sensors: "Timestamp", "AccelX", "AccelY", "Accel_Z", "Roll", "Pitch", "Yaw", "Quat.X", "Quat.Y", "Quat.Z", "Quat.W", "RM11", "RM12", "RM13", "RM21", "RM22", "RM23", "RM31", "RM32", "RM33", "GravAcc_X", "GravAccY", "GravAccZ", "UserAccX", "UserAccY", "UserAcc_Z", "RotRateX", "RotRateY", "RotRate_Z", "MagHeading", "TrueHeading", "HeadingAccuracy", "MagX", "MagY", "MagZ", "Lat", "Long", "LocAccuracy", "Course", "Speed", "Altitude", "Proximity" There are certainly any things that can and should be improved. The main problem seems to be that the data, even if sampled at 100Hz, is sent to the iPhone only every second or so. So it is not really real time. I hope that someone who is better at iPhone programming than I am - I am really rubbish at it- could help and write an iPhone program to stream the data in a more convenient way: one by one rather than in packets. There are many potential applications for this. Here are some I could come up with: You can carry the iPhone around and measure your movements (acceleration). Attached to your hand you can measure your tremor. The magnetometer is really cool. You can use it to find metal bars in the walls an also electric cables. You can collect GPS data for all sorts of applications; there are ideas to use this for the detection of certain diseases. For example if it takes you longer than usual to find your car when you come from shopping that might hint at early stages of dementia --- or sleep deprivation. When you put the phone on a machine, like a running motor, you can measure the vibrations. When you perform a frequency analysis you can check whether the motor runs alright. Using the accelerometers I was able to measure my breathing (putting the phone on my chest). I think that there might also be quite some potential for using the Wolfram Cloud here. Deploying a program in the cloud and reading from your phone is certainly quite interesting. The problem is that this particular app only works via WiFi. It would be nice to have one that works via 3G. So, in summary, it might be quite useful to use the iPhone's sensors. The advantage is that nearly everyone carries a smartphone with them all the time. Making more of your smart phone's sensors with Mathematica seems to be a nice playground for applications. I'd love to hear about your ideas...Cheers,MarcoPS: When you are done with the streaming you should execute these commands: (*Remove Scheduled Tasks and close link*) RemoveScheduledTask[ScheduledTasks[]]; socket@close[]; 
14 Replies
Sort By:
Posted 5 years ago
 Such a big and informative info.
Posted 4 years ago
 okey, the Mathematica code is written to get the streaming data for any source connected into that ip address and port?By the way, this will help me out...Thanks
 Is there an analog of the Sensor Data utility (with the data streaming capability) for Android smart phones?
Posted 4 years ago
 Hi Alexey,yes there are loads. We also wrote a little piece of code for Androids. It is somewhere on the Android Store. I will try to "extract" the program from a Post-Doc and post it here.Cheers,Marco
Posted 4 years ago
 Hi Marco:I have an iPhone 6 and I'm trying this out. In what environment are you running the Wolfram Language code? Since I don't see a place to plug in the URL for the data, I presume you are running it in on the Wolfram Development Platform using the Wolfram Cloud app on the phone itself. But functions InstallJava[], LoadJavaClass[], JavaNew[] come up in blue for me, suggesting that they are not currently available on the Development Platform. So how are you getting this to work? What should I be doing differently?Kathryn
Posted 4 years ago
 Dear Kathryn,I am sorry if this didn't work for you. I am attaching the code that works for me and some students of mine. I am very sorry for not annotating it correctly, but I need to prepare a presentation for tomorrow and wanted to reply as swiftly as possible. I ran this piece of code on a couple of independent OSX machines and it appears to be working fine on all of them. I tested it on everything from iPhone 4 to iPhone 6. You are saying that you ran it in the Cloud, which I did not do. I ran it on a local machine, and laptop and iPhone were in the same network. I have not used the Wolfram Cloud app, but an iOS app from the app store that is called "Sensor Data". I think that it is expected that it will not work in the Cloud/Wolfram Development platform. I just ran that code on MMA10.3 on OSX. Note that when I run it, there are several functions in blue (e.g. setSoTimeout[10], receive, getData[]) but InstallJava[], LoadJavaClass[], JavaNew[] are black. Here's a screenshot.As you work at Wolfram I suppose that you also are running MMA10.3. Which operating system are you using? I have asked a colleague of mine at our institute to post his code, which is a self-written code for Androids. I think that it would be possible to make this "work" in the cloud. If you use, for example, DataDrop you can achieve something similar, with the exception that the update rate of data drop is nominally 2 Hz, so you would not be much faster than that. If you cannot make it work I would be happy to screen-share and demonstrate how the program works.Best wishes from Aberdeen,Marco Attachments:
Posted 4 years ago
 Thanks. I'll try it out. I'm running OSX 10.10.5 on my laptop, and Mathematica 10.3.But I was also trying to run it through the cloud so I could access it from the phone through the Wolfram Cloud app. Where I was seeing blue was on on dev.wolframcloud.com/.
Posted 4 years ago
 Hi Marco and Alexey,The app is called "Sensors Prototype" and it can be found in Google Play. We have tried it on the samsung S3 and S4 without issues. We wrote the following code in Java (server side) to test the data streaming: import java.io.*; import java.net.*; class serverUDP { public static void main(String argv[]) throws Exception,IOException { System.out.println(" Server is Running "); System.out.println(" In port :" + argv[0]); DatagramSocket serverSocket = new DatagramSocket(Integer.parseInt(argv[0])); byte[] receiveData = new byte[1024]; while(true) { System.out.println("-----------------------"); DatagramPacket receivePacket = new DatagramPacket(receiveData, receiveData.length); serverSocket.receive(receivePacket); String output = new String(receivePacket.getData()); System.out.println("Received: " + output); System.out.println("-----------------------"); } } } Of course, both mobile and server should be on the same networkRegards Ricardo
Posted 4 years ago
 Marco,Can you explain this code fragment from your JLink processing? packet = JavaNew["java.net.DatagramPacket", JavaNew["[B", 1024], 1024]; 
Posted 4 years ago
 Hi,"packet" in this case is an object created from the class"java.net.DatagramPacket". The arguments "JavaNew["[B", 1024]" and "1024" depend on the class you are dealing with. For example, I can have a class "ConnectionUDPClient.class" which I can call by obj = JavaNew["ConnectionUDPClient"].Furthermore, it is possible to access methods inside the class. For the class above, I can have a method "sendMessage" which I can invoke by "obj@sendMessage". Finally, depending on the implemented method, it could also be possible to pass additional arguments. Hope this helps.Kind regards,Ricardo
Posted 4 years ago
 Ricardo,I'm a little foggy on the "[B" argument of the nested JavaNew`. I think I've seen that before, but I don't recall that notation. I assume it's specifying a buffer of some sort?Thanks for the help.David
Posted 4 years ago
 How does socket get associated with 192.168.1.95:54388? socket is instanced on Port 10552, but what does that have to do with anything?Thanks for any hint to get me started.Fred