Message Boards Message Boards

Converting OpenPose for Wolfram Language

Posted 5 years ago
Attachments:
POSTED BY: Kotaro Okazaki
26 Replies

Aaaaand... it works! Thanks so much, Kotaro-San :)

For posterity's sake, here are the things that had to happen to get it working with conda on OSX:

1) Get the right version of python: "conda create -n python3 python=3.7 anaconda"

2) Switch to that python virtualenv: "conda activate python3"

3) Install pytorch for that virtualenv: "conda install pytorch"

4) pqzmq (the python <-> mathematica library) seems to already be installed with the conda distro, but just in case: "conda install pyzmq"

5) Find the path to the conda python install: "which python"

6) Register the conda python with mathematica:

ExternalEvaluate; ExternalEvaluatePrivateresetCache[] RegisterExternalEvaluator["Python", "/opt/anaconda2/envs/python3/bin/python"]

7) Modify Kotaro-San's ExternalEvaluate code to set the directory to the one containing the the .pth file: os.chdir('full path here')

8)I also added a "Normal" to this line of Kotaro-San's code, but I don't know that it was essential: rule = Thread[replacenames -> Normal[parametersReshape]];

That's it. Note that 1-5 are terminal commands, while 6-8 are within Mathematica. Enjoy!

POSTED BY: Jonathan Simon

Kotaro-San,

Thank you so much for the quick reply -- I really appreciate it! I am soooo excited about learning about neural networks and this seems like a terrific opportunity to get into it!

Could you perhaps also let me know what versions of numpy, pkl, and torch you are using? I can certainly switch to a python 3 anaconda package if that will resolve things, but I'd prefer to install all the correct packages this time around :)

Again -- my sincerest appreciation from Chicago, and I hope all is well with you in these difficult times!

Cheers, Jon

POSTED BY: Jonathan Simon

Jonathan-san,

The versions of numpy, pkl, and torch I use are enter image description here

POSTED BY: Kotaro Okazaki

Hi All,

I tried to test out this model, and it failed all over the place :(:( I am wondering if there is a compatibility issue, if links have broken, or something else. To get it to run at all, I first had to make a few changes:

1) "https://www.dropbox.com/s/ae071mfm2qoyc8v/pose_model.pth?\ dl=0" points to posemodel.pth, not posemodelscratch.pth, so I changed the filename in the python ExternalEvaluate[] command sequence to "posemodel.pth" -- is that the wrong model?!

2) I had to register my anaconda 4.8.3 install of python 2.7.16 with mathematica after installing pyzmq; that's my python install that has pytorch installed. The proper external evaluation initialization sequence is now: " ExternalEvaluate ExternalEvaluateImportExportPrivate`$ImporterPythonSession = StartExternalSession["Python" -> "String"] session = StartExternalSession["Python"] "

3) "trainedOpenPose = NetReplacePart[openpose, rule]" fails unless I modify the preceding line to read: rule = Thread[replacenames -> Normal[parametersReshape]]; that is, I have added a "Normal" to strip out the NumericArray[...,"Real32"] commands

At the end everything runs, but the model fails to recognize Ichiro properly, giving only one pair of points that are not even on his body... PLEASE HELP! Thanks!

Attachments:
POSTED BY: Jonathan Simon

Jonathan-san,

Thank you for your comment. I've evaluated your notebook and my notebook in my new environment(v12.1) and they have recognized Ichiro properly. It may be due to the different versions of Python(2.7.16 vs 3.7.4).

I attached my result of your notebook.

Attachments:
POSTED BY: Kotaro Okazaki
Posted 5 years ago

How come I do not know "Person::93r37" where this comes from ???

Entity["Concept", "Person::93r37"]

We are enterprise licensed customers for years and documentation center has nothing on this????

POSTED BY: Wayne Allen
Posted 5 years ago

Tech support responded to this, therefore no worries.

POSTED BY: Wayne Allen
Posted 5 years ago

Step 3 takes a long time on my machine, why?

POSTED BY: Wayne Allen

I'm sorry I have no clear answer to your question. However, it takes a few seconds in my environment (Windows 10 and Mathematica v12.0).

POSTED BY: Kotaro Okazaki
Posted 5 years ago

iMAC 12.0 Mathematica and it hangs for hours.

I am going to submit this to tech support.

POSTED BY: Wayne Allen
Posted 5 years ago

Wolfram techsupport wrote and asked me to ask you Mr.Okazaki to send me your FULL NOTEBOOK to run your code. They like to use that notebook to track the problem.

Let me know if you are willing to do that

Dara

POSTED BY: Wayne Allen

Allen-san,

I have attached my notebook to the post. It takes about one and a half minutes in my environment. Please make sure.

Your problem may be a problem with indentation in ExternalEvaluate.

I hope this will help.

POSTED BY: Kotaro Okazaki
Posted 5 years ago

Thank you Sir!

ExternalEvaluate [ ] has an issue with Python Numpy version so little odd thingy like that.

POSTED BY: Wayne Allen

I saw this post, but don't see any links to OpenPose itself. Is it a Wolfram toolset? Or is there a GitHub repository? Any links would be appreciated.

POSTED BY: David Whitten

David-san,

Thank you for your comment. I refered here.

POSTED BY: Kotaro Okazaki
Posted 5 years ago

Coincidentally I was also looking at converting OpenPose when I came across this post. This is lovely work, thank you very much. It'd be great to see it in the Wolfram Neural Net Repository. I have had 3 models (single shot detectors) submitted and accepted (several more in the pipeline). I don't work for Wolfram, but they were very helpful guiding me through their submission process and they do acknowledge your contribution on their repository site.

Thanks once again,

Julian.

POSTED BY: Julian Francis
Posted 5 years ago

Julian,

Thank you for your kind words. As always, it is a pleasure working with you to get your models in the repository. We always look forward to more submissions from you, Kotaro-san and all our users.

Thanks, Tuseeta

POSTED BY: Test Account

As a somewhat simplistic way of finding multiple poses in an image, I thought it would be interesting to use ImageContents to find people and then use your OpenPose implementation to detect poses for each person.

First, I altered your showPose function to include an origin parameter and removed the last line to make it return the poses only (i.e. a pose function):

pose[img_, origin_: {0, 0}] := 
 Module[{bodylist, size, out, confidences, pts, people, pose},
  bodylist = Range[14];
  size = {368, 368};
  out = trainedOpenPose[img];
  confidences = out[[2]];
  pts = Map[Plus[origin, #] &, 
    maxpts[img, confidences, #] & /@ bodylist];
  pose = Graphics[{Yellow, Thickness[.0125], 
     Line[pts[[#]] & /@ {1, 2}], Green, 
     Line[pts[[#]] & /@ {2, 3, 4, 5}], Cyan, 
     Line[pts[[#]] & /@ {2, 6, 7, 8}], Orange, 
     Line[pts[[#]] & /@ {2, 9, 10, 11}], Magenta, 
     Line[pts[[#]] & /@ {2, 12, 13, 14}], PointSize[Large], Red, 
     Point[pts]}, ImagePadding -> All]]

Then I made a function that retrieves the images and bounding boxes for each person in the original image, computing and translating poses for each image based on the origin of its bounding box:

multiPose[img_] := Module[{people, boxes, poses},
  {people, boxes} = 
   Transpose[
    Normal@ImageContents[img, 
        Entity["Concept", "Person::93r37"]][[All, {"Image", 
         "BoundingBox"}]] // Values];
  poses = 
   Table[pose[people[[i]], boxes[[i, 1]]], {i, Length[people]}]]

Finally, a function to show the poses in context:

showMultiPose[img_] := Show[img, Sequence[multiPose[img]]]

Not suitable for real-time processing or anything, but it was very easy to implement, and it seems to work in seconds, even for a decent number of people:

(From Google search results for

POSTED BY: Brian Wood

Brian-san,

Thanks for your try. Your idea of using ImageContents to estimate multi-person pose is really great. It was unthinkable to me. I think the best part of the community is knowing how to use Wolfram Language inconceivably. Thank you again!

POSTED BY: Kotaro Okazaki

That was very informative, thank you.

This is a great example; thank you for the post! I can imagine all sorts of interesting uses for OpenPose in the Wolfram Language.

I was trying to replicate your process, and I got stuck on the last part of Step 4, which I think may be missing a line of code:

Reshape each one-dimensional list of parameters to the dimension of the corresponding weight or bias.

The subsequent step uses a parametersReshape variable, which is probably meant to be defined here. Could you possibly provide this missing piece?

I'm looking forward to exploring this further!

POSTED BY: Brian Wood

Brian-san,

Thank you very much for taking the time and spotting this. Sorry I have updated my post. There was lines of code missing.

getDimB[layer_] := Dimensions@NetExtract[layer, "Biases"]
getDimW[layer_] := Dimensions@NetExtract[layer, "Weights"]
convs = NetExtract[NetInitialize[openpose], #] & /@ convlayerpos;
dimW = getDimW /@ convs;
dimB = getDimB /@ convs;
dim = Flatten[Transpose[{dimW, dimB}], 1];
parametersReshape = MapThread[ArrayReshape, {parameters, dim}];
POSTED BY: Kotaro Okazaki

Kotaro-san,

Thank you for the update. That was exactly the detail I needed! Looks like your implementation is working perfectly for me.

POSTED BY: Brian Wood

Kotaro-san, what a wonderful work and presentation, thank you for sharing!

POSTED BY: Vitaliy Kaurov

enter image description here - Congratulations! This post is now featured in our Staff Pick column as distinguished by a badge on your profile of a Featured Contributor! Thank you, keep it coming!

POSTED BY: EDITORIAL BOARD
Posted 5 years ago

It is wonderful to see how you converted this model and so soon. Please feel free to reach out to us via the contact us button in the Neural Repo page.

POSTED BY: Test Account
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract