Message Boards Message Boards

Classifier for Human Motions with data from an accelerometer

GROUPS:

This project was part of a Wolfram Mentorship Program.

The classification of human motions based on patterns and physical data is of great importance in developing areas such as robotics. Also, a function that recognizes a specific human motion can be an important addition to artificial intelligence and physiological monitoring systems. This project is about acquiring, curating and analyzing experimental data from certain actions such as walking, running and climbing stairs. The data taken with the help of an accelerometer needs to be turned into an acceptable input for the Classify function. Finally, the function can be updated with more data and classes to make it more efficient and whole.

Algorithms and procedures

The data for this project was acquired by programming an Arduino UNO microprocessor with a Raspberry Pi computer, using Wolfram Language. An accelerometer connected to the Arduino sent measurements each time it was called upon, and Mathematica in the Raspberry Pi collected and uploaded the data. The raw data had to be processed for it to be a good input for the classify function. First, it was transformed into an spectrogram (to analyze the frequency domain of the data). Then, the spectrogram's image was put through the IFData function which filters out some of the noise, and finally the images were converted into numerical data with the UpToMeasurements function (main function: ComponentMeasurements). This collection numerical data was put in a classify function under six different classes (standing, walking, running, jumping and waving).

*The IFData function and the UpToMeasurements functions were sent to me by Todd Rowland during the Mentorship. Both functions will be shown at the end of this post.

Example visualization

The following ListLinePlot is an extract from the jumping data

Example data

Next, the data from the plot above is turned into a spectrogram by the function Spectrogram, i.e.:

spectrogramImage = 
 Spectrogram[jumpingData, SampleRate -> 10, FrameTicks -> None, 
  Frame -> False, Ticks -> None, FrameLabel -> None]

Example jumping data spectrogram

Finally, all the spectrogram images are used as input for the UpToMeasurements function, along with some properties for the ComponentMeasurements function:

i.e:

numericalData = 
 N@Flatten[
   UpToMeasurements[
    spectrogramImage, {"EnclosingComponentCount", "Max", 
     "MaxIntensity", "TotalIntensity", "StandardDeviationIntensity", 
     "ConvexCoverage", "Total", "Skew", "FilledCircularity", 
     "MaxCentroidDistance", "ExteriorNeighborCount", "Area", 
     "MinCentroidDistance", "FilledCount", "MeanIntensity", 
     "StandardDeviation", "Energy", "Count", "MeanCentroidDistance"}, 
    1]]

Which outputs a list of real numbers, one for each of the properties:

{0., 1., 1., 1., 1., 19294.9, 0.222164, 0.985741, 31011.8, 15212.5, \
9624.42, -0.0596506, 0.724527, 190.534, 0., 42584.5, 0.364667, \
42584., 0.453101, 0.315209, 0.232859, 0.169549, 0.00909654, 42584., \
98.7136}

These numbers are grouped in a nested list which contains data for all 5 human motions. All the data is lastly classified in a classifier using the Classify function.

After several combinations of both properties and data sets, I was able to produce classifier functions with an accuracy of 91%, and a total size of 269kb.


Attempt on building a classify function using image processing

On the other hand, the image processing capabilities of Mathematica lets us extract data from images, hence it should be possible to create a classifier which recognizes the moving patterns in the frames of a video. First, I had to take the noise out of every image, this proved to be troublesome, since the background can vary greatly between video samples. Then, I binarized the image in order to isolate the moving particles in each frame, and extract their position with ImageData. Lastly, a data set can be formed from all the analyzed frames; this data can essentially be used in the same way as the accelerometer's, but the classifier was unsuccessful in separating the samples accurately. This was mainly because the accelerometer's data is taken at a constant rate and very precisely, whereas the images depend on the camera's frame rate, and many other external factors. This is what made the data different enough to fail being classified with accuracy. Furthermore, if a big dataset is made from videos of people performing certain actions, the data processing can follow similar steps as the ones explained in this report. Thus producing a similar classifier function. This can further increase the functions accuracy, but the process needs an algorithm that can effectively trace the path of "a particle" that moves through each of the frames of the video, and extract precise velocity data from said movement.


Conclusively, the classify function is working very well with the data provided, its accuracy is about 91% for the SupportVectorMachine method. This is a very good result for the human motion classifier. The next step is to add more classes to the function, and test the classifier with data acquired from different sources, such as another accelerometer and various videos of human motion footage.


Code:

  • UpToMeasurements function

    UpToMeasurements[image_,property_,n_]:=MaximalBy[ComponentMeasurements[image,"Count"],Last,UpTo[n]][[All,1]]/.ComponentMeasurements[image,property]
    

*Note: This function simplifies the exploration of properties to input in ComponentMeasurements, also, it outputs a usable list of numerical data retrieved from a given group of images.

  • IFData function:

    imagefunctions=<|1-> (EntropyFilter[#,3]&),
    2-> (EdgeDetect[EntropyFilter[#,3]]&),
    3->Identity,
    4-> (ImageAlign[reference110,#]&),
    5-> (ImageHistogram[#,FrameTicks->None,Frame->False,FrameLabel->None,Ticks->None]&),
    6-> (ImageApply[#^.6&,#]&),
    7-> (Colorize[MorphologicalComponents[#]]&),
    8-> (HighlightImage[#,ImageCorners[#,1,.001,5]]&),
    9-> (HighlightImage[#,Graphics[Disk[{200,200},200]]]&),
    10-> ImageRotate,
    11-> (ImageRotate[#,45Degree]&),
    12->(ImageTransformation[#,Sqrt]&),
    13->(ImageTransformation[#,Function[p,With[{C=150.,R=35.},{p[[1]]+(R*Cos[(p[[1]]-C)*360*2/R]/6),p[[2]]}]]]&),
    14->( Dilation[#,DiskMatrix[4]]&),
    15->( ImageSubtract[Dilation[#,1],#]&),
    16-> (Erosion[#,DiskMatrix[4]]&),
    17-> (Opening[#,DiskMatrix[4]]&),
    18->(Closing[#,DiskMatrix[4]]&),
    19->DistanceTransform,
    20-> InverseDistanceTransform,
    21-> (HitMissTransform[#,{{1,-1},{-1,-1}}]&),
    22->(TopHatTransform[#,5]&),
    23->(BottomHatTransform[#,5]&), 
    24-> (MorphologicalTransform[Binarize[#],Max]&),
    25-> (MorphologicalTransform[Binarize[#],"EndPoints"]&),
    26->MorphologicalGraph,
    27->SkeletonTransform,
    28->Thinning,
    29->Pruning,
    30-> MorphologicalBinarize,
    31-> (ImageAdjust[DerivativeFilter[#,{1,1}]]&),
    32-> (GradientFilter[#,1]&),
    33-> MorphologicalPerimeter,
    34-> Radon
    |>;
    
    reference110=BlockRandom[SeedRandom["110"];Image[CellularAutomaton[110,RandomInteger[1,400],400]]];
    
    IFData[n_Integer]:=Lookup[imagefunctions,n,Identity]
    
    IFData["Count"]:=Length[imagefunctions]
    
    IFData[All]:=imagefunctions
    

Note: This function groups together several image filtering fuctions; it was used to simplify the exploration of functions to be used in the classifier. *This function was written by the the Wolfram team, but was slightly modified for this project.

  • propertyVector function (this function automatically evaluates all the prior necessary code needed to create the classify functions):

    propertyVector[property_]:={walkingvector=N@Flatten[UpToMeasurements[#,property,1]]&/@IFData[6]/@(Spectrogram[#,SampleRate->10,FrameTicks->None,Frame->False,Ticks->None,FrameLabel->None]&/@walk);
    jumpingvector=N@Flatten[UpToMeasurements[#,property,1]]&/@IFData[6]/@(Spectrogram[#,SampleRate->10,FrameTicks->None,Frame->False,Ticks->None,FrameLabel->None]&/@jump);
    standingvector=N@Flatten[UpToMeasurements[#,property,1]]&/@IFData[6]/@(Spectrogram[#,SampleRate->10,FrameTicks->None,Frame->False,Ticks->None,FrameLabel->None]&/@stand);
    runningvector=N@Flatten[UpToMeasurements[#,property,1]]&/@IFData[6]/@(Spectrogram[#,SampleRate->10,FrameTicks->None,Frame->False,Ticks->None,FrameLabel->None]&/@run);
    wavingvector=N@Flatten[UpToMeasurements[#,property,1]]&/@IFData[6]/@(Spectrogram[#,SampleRate->10,FrameTicks->None,Frame->False,Ticks->None,FrameLabel->None]&/@wave);
    stairsvector=N@Flatten[UpToMeasurements[#,property,1]]&/@IFData[6]/@(Spectrogram[#,SampleRate->10,FrameTicks->None,Frame->False,Ticks->None,FrameLabel->None]&/@stairs);
    walkingvectortest=N@Flatten[UpToMeasurements[#,property,1]]&/@IFData[6]/@(Spectrogram[#,SampleRate->10,FrameTicks->None,Frame->False,Ticks->None,FrameLabel->None]&/@testwalk);
    jumpingvectortest=N@Flatten[UpToMeasurements[#,property,1]]&/@IFData[6]/@(Spectrogram[#,SampleRate->10,FrameTicks->None,Frame->False,Ticks->None,FrameLabel->None]&/@testjump);
    standingvectortest=N@Flatten[UpToMeasurements[#,property,1]]&/@IFData[6]/@(Spectrogram[#,SampleRate->10,FrameTicks->None,Frame->False,Ticks->None,FrameLabel->None]&/@teststand);
    runningvectortest=N@Flatten[UpToMeasurements[#,property,1]]&/@IFData[6]/@(Spectrogram[#,SampleRate->10,FrameTicks->None,Frame->False,Ticks->None,FrameLabel->None]&/@testrun);
    wavingvectortest=N@Flatten[UpToMeasurements[#,property,1]]&/@IFData[6]/@(Spectrogram[#,SampleRate->10,FrameTicks->None,Frame->False,Ticks->None,FrameLabel->None]&/@testwave);
    stairsvectortest=N@Flatten[UpToMeasurements[#,property,1]]&/@IFData[6]/@(Spectrogram[#,SampleRate->10,FrameTicks->None,Frame->False,Ticks->None,FrameLabel->None]&/@teststairs);}
    
    Training:=trainingSet=<|"walking"->walkingvector,"running"->runningvector,
    "standing"-> standingvector,
    "jumping"-> jumpingvector,
    "waving"-> wavingvector,
    "stairs"-> stairsvector|>;
    
    Test:=testSet=<|"walking"->walkingvectortest,"running"->runningvectortest,
    "standing"-> standingvectortest,
    "jumping"-> jumpingvectortest,
    "waving"-> wavingvectortest,
    "stairs"-> stairsvectortest|>;
    
  • Example code for the acceleration data acquisition from image processing:

    images=Import["$path"]
    motionData=
    Count[#,1]&/@ 
      (Flatten[     
        ImageData[Binarize[ImageSubtract[ImageSubtract[#[[1]],#[[2]]],ImageSubtract[#[[2]],#[[3]]]]]]&/@
             Partition[images,3,1],1])
    

*Note: before this code can be used, the backgrounds of the frames of the video have to be removed, and the image has to be binarized as much as possible (some examples will be shown in the next section).

  • Example code for the retrieval of raw data from DataDrop:

    rawData=Values[Databin["Serial#", {#}]];
    data=Flatten[rawData["(xacc/yacc/zacc)"]];
    

**Please feel free to contact me or comment if you are interested in the rest of the code ( uploading the C code to the Arduino, the manufacturer's code for the accelerometer, C code switch that lets Mathematica communicate with the Arduino, and the Wolfram Language code used to start each loop in the switch that retrieves data ). Also, I could send the classify function, or any other information that I might have left out; all suggestions welcome.

POSTED BY: Pablo Ruales
Answer
4 months ago

Hi Pablo,

I am interesting in your project, please explain more details. How you attach and test the accelerometer on human, at one position or at different positions like twist, arm, knee or ankle etc. ? Did you use some wireless device for data transfer from Arduino or Raspberry Pi?

Another suggestion for application, with a good classifed data of human motion, Are you plan to do a better calculation for energy consume in every day? That might be helpful for body fit or weight co

POSTED BY: Frederick Wu
Answer
4 months ago

Hello Frederick, thank you for your interest. The data I gathered was all by holding the Raspberry Pi and Arduino with my hand at chest level. The Raspberry Pi runs Mathematica and reads the data from the Arduino, since Mathematica can handle data easily, you can program it to upload to DataDrop or you could save it in a file, and later on extract the files to work on your computer. Since both the raspberry pi and the arduino are small devices they could be held by a chest band (or something like the bands that hold your phone securely attached to your arm while you go running). It can be placed anywhere on the body. I chose to place it at chest level because the data would be more universal, meaning that there would be less noise on the data due the particular way a person moves.

(In order to transfer the data and control the devices from my computer I used PuTTY to connect wirelessly to the Raspberry Pi. Also, I needed a WiFi dongle for the Raspberry)

The classify function just tells you what motion the data is showing, that is its main goal. But besides classifying the data into motion categories, the data could also be processed to estimate the energy consumed for each motion based on the intensity and frequency.

If you are interested in replicating the experiment I would be happy to send you the data I gathered, and all the specifications of the devices I used. I encourage you to get more data though, this would make the classify function more precise.

I am open to any suggestions, and if you are interested in making the image processing part of this project work, I think that would make a very interesting product. It could be a program that analyzes movement (energy consumption, velocity) and categorizes it. I can also send some of the code that I started to extract information from the images. Tell me if you are interested in this, we could keep discussing this matter as an addition to the main project.

Lastly, please do tell if you want me to explain something specific about the project. Unfortunately it is not very easy sharing all the data and visualizations through the community but I will try my best. I can also send you the classify function.

Pablo

POSTED BY: Pablo Ruales
Answer
4 months ago

enter image description here - you have earned "Featured Contributor" badge, congratulations !

This is a great post and it has been selected for the curated Staff Picks group. Your profile is now distinguished by a "Featured Contributor" badge and displayed on the "Featured Contributor" board.

POSTED BY: Moderation Team
Answer
4 months ago

Very nice work! But is it possible to post the complete code? I am not quite sure how training was done? Was this Classify, Predict or NetTrain? Without complete code is hard to comprehend the details.

POSTED BY: Sam Carrettie
Answer
4 months ago

Hello Sam, thank you for your comment. The whole project was done using the built in Classify function, which proved to produce more accurate results than Predict.

After extracting the numerical data from the Spectrogram images using the UpToMeasurements function, the propertyVector function puts it all in an organized array for each of the motion types. These arrays of data are the input for each of the corresponding classes in the Classify function. I hope this answers your question; I didn't post the whole code because there are thousands of lines of data manipulation and extraction which would make this thread lengthy and boring (there was a lot of trial and error involved in this project as well), but I believe I have posted the important parts which produced the results. Anyway, the classify function was built by trying all many combinations of properties for the ComponentMeasurements fuction, so lets say I have a list of properties that I know have produced good results call that list "properties", and I also have a list of all the properties available to the function, call that "allProperties". The property exploration algorithm I used was somewhat as follows:

Do[propertyVector[Append[properties, allProperties[[i]]]]; Training; Test; Print[allProperties[[i]]]; classifier = Classify[trainingSet, PerformanceGoal -> "Quality"]; Print[ClassifierMeasurements[classifier, testSet, "Accuracy"]], {i, 1, Length[allProperties]}]

This way I could see which properties produced a higher accuracy in the classifiers performance. Note that "Training" and "Test" are written above. Once I got a good set of properties I would rebuild the classifier.

propertyVector[properties]; Training; Test; classifier = Classify[trainingSet, PerformanceGoal -> "Quality"];

  • "trainingSet" is inside "Training" *

That is how I built the classifier, please be sure to reply to this comment if you have any other questions. I would be happy to send you my data, the Classify function or one of the test notebooks where I explored the properties and the accuracy they provided in the function.

POSTED BY: Pablo Ruales
Answer
4 months ago

Group Abstract Group Abstract