Message Boards Message Boards

GROUPS:

My Robots and Mathematica

Posted 1 year ago
2597 Views
|
7 Replies
|
12 Total Likes
|

Hi there, (Sharing an idea)

I am a hobbyist user of Mathematica. I am using it to control my three robot systems. Firstly there is Jeeves. It is a modified Meccano G15 KS. The Meccano control system has been removed and replaced with another control system (EZ-Robot). This new controller and its PC based control software is capable of starting an external program. In my case that external program is Mathematica. The sensors on the robot read in data and pass these data onto the PC based control software which then fires up Mathematica which in turn carries out some computation and passes a result back to the PC based control software. Data is stored in files. Using this set up means there can be latency. Sometimes quite a bit. For me this is not a problem as the robot is mainly used to test algorithms. Waiting a minute for a result is not an issue. Modified Meccano G15 KS

This robot can read text from a sheet of paper or screen and then repeat what it has seen. It can also read text and evaluate it. For example given the question, “What is the capital of England” the robot will reply “London”. It can recognise objects in an image(potentially thousands). By reading in text similar to the following “AABBCCD”, it will play sounds relating to the musical notes. All this is made possible because of Mathematica. As stated above I use this robot for algorithm testing. In this case Mathematica is used indirectly.

My second robot is a small 16 DOF humanoid robot. This robot is controlled by a 24 channel Pololu Maestro servo controller.
Modified EZ-Robot Rear of modified EZ

Now this robot is controlled directly by Mathematica. All code is contained in a notebook and uses all the features of connected devices contained within Mathematica. Data can be read from and sent to the robot. Just by using the Manipulate function every servo in the robot can be controlled by Mathematica. I have not done so at the moment but to create a robot animation system using Mathematica would only be a few dozen lines of code. That is one of my first tasks. This robot is a modified EZ-Robot now totally under the control of Mathematica.

My third robot is a robot workbench. Built by me to test more complex algorithms. This is controlled by a 24 channel Pololu Maestro servo controller. In turn controlled completely by, you guessed Mathematica.

Robot Workbench 1

Robot Workbench 2

Robot Workbench 3

Robot Workbench 4

Robot Workbench 5

The robot workbench only has a servo control program written at the moment. This control program makes use of the Manipulate function. I have some great plans for this workbench in the future. Under the control of Mathematica it will perform tasks using all the power of the neural networks and machine learning. Image and text processing. To name but a few.

My big goal is to create a Robot Operating System that will contain all the code required for the robots to carry out complex tasks. This would include features such as inverse and forward kinematics. I want to achieve this using only Mathematica.

My reason for putting my robots on the forum is simply to see if anyone is doing anything similar. With only a few lines of Mathematica code I have been able to get the robots to read text and interpret it. I am sure that my Robot Operating System will be thousands of lines of code. I plan to write it in two parts. A front end to carry out general robot tasks, such as movement and manipulation, data processing from sensors and the solving of problems using neural nets and machine learning. A back end that can be made to match up to a specific controller or micro controller. That is my plan. Any code I produce will be made available to this forum.

Well I had better get back to coding.


CODE


The code for the G15 KS simply performs a task and returns a result to the control software of the robot. There is no direct control. The humanoid robot and the workbench code have direct control.

The following is the code that allows the modified G15 KS to read text and repeat what it has seen.

str = TextRecognize[Import["C:\\mathscripts\\images\\img3.jpg"],  Language -> "English"];

str = StringReplace[str, Whitespace -> " "];

str

s = OpenWrite[File["C:\\mathscripts\\jeeves\\output.txt"]]

WriteLine[s, str];

Close[s];

Here the image taken by the robots camera is picked up by mathematica and the TextRecognize function gets the text from the image and stores it in a file ready to be used by the robots control system. Pretty simple coding to get a robot to read text.

The following code allows the G15 KS to read text, evaluate it.

str = TextRecognize[Import["C:\\mathscripts\\images\\img3.jpg"], Language -> "English"];

str = StringReplace[str, Whitespace -> " "];

str

s = OpenWrite[File["C:\\mathscripts\\jeeves\\output2.txt"]]

res = Interpreter["SemanticExpression"][str];

If[NumericQ[res], res = N[res], res]

WriteLine[s, ToString[res]];

Close[s];

The following code allows the G15 KS to identify an object within an image. Actually thousands of objects'

txt = ImageIdentify[Import["C:\\mathscripts\\images\\img3.jpg"]]

s = OpenWrite[File["C:\\mathscripts\\jeeves\\output3.txt"]]

WriteLine[s, ToString[CommonName[txt]]];

Close[s];

The following code allows the G15 KS to read text relating to geographic location. This was used by the robot to read the text generated by a mobile phone app and speak the result.

str = TextRecognize[Import["C:\\mathscripts\\images\\img3.jpg"], Language -> "English"];
str = StringReplace[str, Whitespace -> " "];
str = StringSplit[str];
If[NumberQ[ToExpression[str[[1]]]] && NumberQ[ToExpression[str[[2]]]],
  $GeoLocation = 
   GeoPosition[{ToExpression[str[[1]]], ToExpression[str[[2]]]}];
  country = CountryData[$GeoLocationCountry, "Name"];
  citytown = CityData[$GeoLocationCity, "Name"];
  s = OpenWrite[File["C:\\mathscripts\\jeeves\\output4.txt"]];
  WriteLine[s, country <>  " is the country I am in and the nearest town or city is called " \<>  citytown];
  Close[s];,
  s = OpenWrite[File["C:\\mathscripts\\jeeves\\output4.txt"]];
  WriteLine[s, "Invalid input Please try again " ];
  Close[s];];

The following code allows the robot to play a tune. The text the robot must read is as follows: AABBCCD

str = TextRecognize[Import["C:\\mathscripts\\images\\img3.jpg"], Language -> "English"];

str = StringReplace[str, Whitespace -> ""];

str

s = OpenWrite[File["C:\\mathscripts\\jeeves\\output5.txt"]]

WriteLine[s, str];

Close[s];

It can be seen from the code that most of the work is passing data in files. The code to carry text and image analysis is simply one line of code. It can not get better than that.

The code that follows is the code the is used to control all of the humanoid and workbench servo. Firstly the workbench.

Button["Open connection to Maestro", dev = DeviceOpen["Serial", "COM5"]]

Manipulate[
 DeviceWriteBuffer[
  dev, {132, 1, BitAnd[a*4, 127], 
   BitAnd[BitShiftRight[(a*4), 7] , 127]}], {{a, 1224, "Right Base"}, 
  496, 2016, 1}]

Manipulate[
 DeviceWriteBuffer[
  dev, {132, 0, BitAnd[a*4, 127], 
   BitAnd[BitShiftRight[(a*4), 7] , 127]}], {{a, 1224, "Left Base"}, 
  496, 2016, 1}]

Manipulate[
 DeviceWriteBuffer[
  dev, {132, 3, BitAnd[a*4, 127], 
   BitAnd[BitShiftRight[(a*4), 7] , 127]}], {{a, 1216, "Right Pivot"},
   1008, 2000, 1}]

Manipulate[
 DeviceWriteBuffer[
  dev, {132, 2, BitAnd[a*4, 127], 
   BitAnd[BitShiftRight[(a*4), 7] , 127]}], {{a, 1216, "Left Pivot"}, 
  1008, 2000, 1}]

Manipulate[
 DeviceWriteBuffer[
  dev, {132, 4, BitAnd[a*4, 127], 
   BitAnd[BitShiftRight[(a*4), 7] , 127]}], {{a, 1216, "Right Elbow"},
   1024, 2144, 1}]

Manipulate[
 DeviceWriteBuffer[
  dev, {132, 5, BitAnd[a*4, 127], 
   BitAnd[BitShiftRight[(a*4), 7] , 127]}], {{a, 1270, "Left Elbow"}, 
  1024, 2144, 1}]

Manipulate[
 DeviceWriteBuffer[
  dev, {132, 14, BitAnd[a*4, 127], 
   BitAnd[BitShiftRight[(a*4), 7] , 127]}], {{a, 1240, "Right Wrist"},
   496, 2000, 1}]

Manipulate[
 DeviceWriteBuffer[
  dev, {132, 15, BitAnd[a*4, 127], 
   BitAnd[BitShiftRight[(a*4), 7] , 127]}], {{a, 1182, "Left Wrist"}, 
  496, 2000, 1}]

Manipulate[
 DeviceWriteBuffer[
  dev, {132, 7, BitAnd[a*4, 127], 
   BitAnd[BitShiftRight[(a*4), 7] , 127]}], {{a, 1200, 
   "Right Gripper"}, 1024, 1296, 1}]

Manipulate[
 DeviceWriteBuffer[
  dev, {132, 6, BitAnd[a*4, 127], 
   BitAnd[BitShiftRight[(a*4), 7] , 127]}], {{a, 1200, 
   "Left Gripper"}, 1024, 1296, 1}]

Manipulate[
 DeviceWriteBuffer[
  dev, {132, 8, BitAnd[a*4, 127], 
   BitAnd[BitShiftRight[(a*4), 7] , 127]}], {{a, 1480, "Turn Table"}, 
  992, 2000, 1}]

Manipulate[
 DeviceWriteBuffer[
  dev, {132, 10, BitAnd[a*4, 127], 
   BitAnd[BitShiftRight[(a*4), 7] , 127]}], {{a, 1240, 
   "Move Vertical"}, 496, 2000, 1}]

Manipulate[
 DeviceWriteBuffer[
  dev, {132, 11, BitAnd[a*4, 127], 
   BitAnd[BitShiftRight[(a*4), 7] , 127]}], {{a, 1200, 
   "Move Horizontal"}, 800, 1600, 1}]

CloseDevice[dev];

Now the code to move the humanoid servos using Manipulate.

dev = DeviceOpen["Serial", "COM5"]
Manipulate[
 DeviceWriteBuffer[
  dev, {132, 0, BitAnd[a*4, 127], 
   BitAnd[BitShiftRight[(a*4), 7] , 127]}]; 
 DeviceWriteBuffer[
  dev, {132, 1, BitAnd[b*4, 127], 
   BitAnd[BitShiftRight[(b*4), 7] , 127]}]; 
 DeviceWriteBuffer[
  dev, {132, 2, BitAnd[c*4, 127], 
   BitAnd[BitShiftRight[(c*4), 7] , 127]}]; 
 DeviceWriteBuffer[
  dev, {132, 3, BitAnd[d*4, 127], 
   BitAnd[BitShiftRight[(d*4), 7] , 127]}]; 
 DeviceWriteBuffer[
  dev, {132, 5, BitAnd[e*4, 127], 
   BitAnd[BitShiftRight[(e*4), 7] , 127]}]; 
 DeviceWriteBuffer[
  dev, {132, 4, BitAnd[f*4, 127], 
   BitAnd[BitShiftRight[(f*4), 7] , 127]}]; 
 DeviceWriteBuffer[
  dev, {132, 6, BitAnd[g*4, 127], 
   BitAnd[BitShiftRight[(g*4), 7] , 127]}];
 DeviceWriteBuffer[
  dev, {132, 7, BitAnd[h*4, 127], 
   BitAnd[BitShiftRight[(h*4), 7] , 127]}];
 DeviceWriteBuffer[
  dev, {132, 15, BitAnd[l*4, 127], 
   BitAnd[BitShiftRight[(l*4), 7] , 127]}]; 
 DeviceWriteBuffer[
  dev, {132, 14, BitAnd[k*4, 127], 
   BitAnd[BitShiftRight[(k*4), 7] , 127]}], {{a, 1224, "Left Base"}, 
  496, 2016, 1}, {{b, 1224, "Right Base"}, 496, 2016, 
  1}, {{c, 1216, "Left Pivot"}, 1008, 2000, 
  1}, {{d, 1216, "Right Pivot"}, 1008, 2000, 
  1}, {{e, 1270, "Left Elbow"}, 1024, 2144, 
  1}, {{f, 1270, "Right Elbow"}, 1024, 2144, 
  1}, {{k, 1024, "Right Wrist"}, 496, 2000, 
  1}, {{l, 1024, "Left Wrist"}, 496, 2000, 
  1}, {{g, 1200, "Left Gripper"}, 1024, 1296, 
  1}, {{h, 1200, "Right Gripper"}, 1024, 1296, 1}]

That's all for now at a later date there will be a lot more to come. It can be seen from above that very little code is required to get a great deal of functionality.

7 Replies
Posted 1 year ago

Fascinating stuff Terence. I am just starting to use Mathematica for control of external devices and your work is inspirational. Thanks for sharing!

This is marvelous, thank your for sharing! We consolidated all your comments into the original head post so it reads better. We are looking forward to more of your work!

Posted 12 days ago

In the past year I have been working very hard with my robots. I have been lucky enough to have purchased a NAO robot. This robot is very high end. It comes with a number of SDK's. Python being one of them. With NAO robot comes a visual programming environment called Choregraphe. This entry will show how I have integrated Mathematica into this environment making use of the Wolfram Cloud.

enter image description here

Cloud access and Choregraphe

Being able to integrate Mathematica into the Choregraphe environment would mean that nearly all of the functionality of Mathematica can now be used by the NAO robot. The steps to achieve this are shown below.

1). Create an API Function in Mathematica.

2). Embed this function in the cloud using the Python option.

3). Copy the embedded code to a Python box in Choregraphe.

4). Create any Python code that is required to assist in execution of the function.

5). Finally execute the Python box in Choregraphe to test it

​​Now to examine each step in detail. Steps 1 and 2 can be carried out as shown in the code below. Creating an API function and embedding it is done in one step strictly speaking. However it is good to be aware that there are two steps. The user must create their own function because the cloud address shown in the code image below no longer exists. This is the case simply because if it were still available a user of this site would be using my cloud credits. Cloud credits are required to use cloud functions. Visit Wolfram research for more details on cloud credits.

EmbedCode[ APIFunction[{"x" -> "Integer"}, #x+#x+5 &], "Python"]

enter image description here

Steps 3 and 4 can be carried out as shown in the image below. These steps are much like the first two steps they are carried out in one process. Create a Python box in Choregraphe and copy and paste the code shown above into the Python box as shown below. The red outline at the top is all the code pasted in from Mathematica. The red outline at the bottom is the code that will call the cloud function and get NAO to speak the result

The code that is used by the robot to access the cloud function

I have now created what is called a box library in Choregraphe containing access to a great variety of Mathematica functions. For example my NAO robot can now make use of most of the machine learning functionality provided by Mathematica. The image processing and computer vision functions are also very useful to my robots. Access to all the math functions of which there are thousands. This means I do not have to sit and write code to carry out calculus. I am making use of one of the best computer algebra systems available.

However for the robotics purists out there. There is a small amount of latency that would not be good in a production environment. As I have stated many times before I am about testing and research. I can implement a system to allow the NAO robot to identify hand written digits 0-9 (using MNIST database). I know I could do this in minutes. Not hours or days, minutes. It is easy to see how a small acorn can become a very big oak tree using this setup.

I now apologise for showing code as images displaying the embedded code using the code option did not seem to work. I had to display the Python box used in Choreograph as an image because this will not display using the code option. I have created a small personal website to demonstrate all that I have done with my robots. The web site shows all the code I have produced for my robots so far and demonstrates how I have used it. There is a lot of code there. Far to much to place on a forum. My site can be found here

Anonymous User
Anonymous User
Posted 10 days ago

Hello, Greetings of the day This is great research you shared with us having a lot of information.

POSTED BY: Anonymous User
Answer
Posted 10 days ago

Thanks very much Divya. I would like to have put more code on the forum as I am a great fan of Wolfram Research and Mathematica. I hope I have not offended anyone in the community by placing a link to my site in the discussion above.

Hi Terence,

Very interesting, thanks for sharing! Have you considered hosting your code on Github so others can collaborate and contribute.

Rohit

Posted 9 days ago

Hi Rohit, I did consider it. However I liked the idea of the code in some way being connected to this community.

Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract