[Hackathon] Tic-Tac-Toe with autonomous Delta Robot

[jennymandl1023 (Jenny Mandl)]

Hello! Last weekend I was at a hackathon at Georgetown University where my team and I wrote a python program that allowed our home made delta robot to autonomously play tic tac toe with a human user. The Mathematica program I was working on was written to interface the arduino controlling the delta robot, and the python program which has the tic tac toe algorithm, by relaying information about the user input on the board through image processing.

The user input had to be from the computer however because I wasn’t able to finish the image processing. The img was an image of the board taken by a logitech web cam attached to the main shaft of the robot. It didn’t work reliably but did work generally.

Does anyone have a better way of doing this?
Thanks!

P.S. here is the code: http://paste.ofcode.org/JCyYF2tJtpcmRta8YjMhmW

src = ColorConvert[img, "Grayscale"];
white = Closing[src, DiskMatrix[5]];
srcAdjusted = Image[ImageData[src]/ImageData[white]]
components = 
  ComponentMeasurements[
    ColorNegate@Binarize[srcAdjusted], {"ConvexArea", "Mask"}][[All, 
   2]];
largestComponent = Image[SortBy[components, First][[-1, 2]]]
mask = FillingTransform[Closing[largestComponent, 2]]
lY = ImageMultiply[
   MorphologicalBinarize[
    GaussianFilter[srcAdjusted, 3, {2, 0}], {0.02, 0.05}], mask];
lX = ImageMultiply[
  MorphologicalBinarize[
   GaussianFilter[srcAdjusted, 3, {0, 2}], {0.02, 0.05}], mask]
verticalGridLineMasks = 
  SortBy[ComponentMeasurements[
      lX, {"CaliperLength", "Centroid", "Mask"}, # > 100 &][[All, 2]], #[[2, 1]] &][[All, 3]];
horizontalGridLineMasks = 
  SortBy[ComponentMeasurements[
      lY, {"CaliperLength", "Centroid", "Mask"}, # > 100 &][[All, 2]], #[[2, 2]] &][[All, 3]];
centerOfGravity[l_] := 
 ComponentMeasurements[Image[l], "Centroid"][[1, 2]]
gridCenters = 
  Table[centerOfGravity[
    ImageData[Dilation[Image[h], DiskMatrix[2]]]*
     ImageData[Dilation[Image[v], DiskMatrix[2]]]], {h, 
    horizontalGridLineMasks}, {v, verticalGridLineMasks}];
    Show[src, 
 Graphics[{Red, 
   MapIndexed[{Point[#1], Text[#2, #1, {1, 1}]} &, gridCenters, {2}]}]]


tr = Last@
  FindGeometricTransform[
   Extract[gridCenters, {{4, 1}, {4, 2}, {3, 1}}], {{0, 0}, {0, 1}, {1, 0}}]
ts = Last@
  FindGeometricTransform[
   Extract[gridCenters, {{3, 1}, {3, 2}, {2, 1}}], {{0, 0}, {0, 1}, {1, 0}}]
tt = Last@
  FindGeometricTransform[
   Extract[gridCenters, {{2, 1}, {2, 2}, {1, 1}}], {{0, 0}, {0, 1}, {1, 0}}]
tu = Last@
  FindGeometricTransform[
   Extract[gridCenters, {{4, 2}, {4, 3}, {3, 2}}], {{0, 0}, {0, 1}, {1, 0}}]
tv = Last@
  FindGeometricTransform[
   Extract[gridCenters, {{3, 2}, {3, 3}, {2, 2}}], {{0, 0}, {0, 1}, {1, 0}}]
tv2 = Last@
  FindGeometricTransform[
   Extract[gridCenters, {{2, 2}, {2, 3}, {1, 2}}], {{0, 0}, {0, 1}, {1, 0}}]
tw = Last@
  FindGeometricTransform[
   Extract[gridCenters, {{4, 3}, {4, 4}, {3, 3}}], {{0, 0}, {0, 1}, {1, 0}}]
tx = Last@
  FindGeometricTransform[
   Extract[gridCenters, {{3, 3}, {3, 4}, {2, 3}}], {{0, 0}, {0, 1}, {1, 0}}]
tz = Last@
  FindGeometricTransform[
   Extract[gridCenters, {{2, 3}, {2, 4}, {1, 3}}], {{0, 0}, {0, 1}, {1, 0}}]
pics = {ImageTransformation[src, tr, {200, 200}, DataRange -> Full, 
   PlotRange -> {{0, 1}, {0, 1}}], 
  ImageTransformation[src, ts, {200, 200}, DataRange -> Full, 
   PlotRange -> {{0, 1}, {0, 1}}], 
  ImageTransformation[src, tt, {200, 200}, DataRange -> Full, 
   PlotRange -> {{0, 1}, {0, 1}}], 
  ImageTransformation[src, tu, {200, 200}, DataRange -> Full, 
   PlotRange -> {{0, 1}, {0, 1}}], 
  ImageTransformation[src, tv, {200, 200}, DataRange -> Full, 
   PlotRange -> {{0, 1}, {0, 1}}], 
  ImageTransformation[src, tv2, {200, 200}, DataRange -> Full, 
   PlotRange -> {{0, 1}, {0, 1}}], 
  ImageTransformation[src, tw, {200, 200}, DataRange -> Full, 
   PlotRange -> {{0, 1}, {0, 1}}], 
  ImageTransformation[src, tx, {200, 200}, DataRange -> Full, 
   PlotRange -> {{0, 1}, {0, 1}}], 
  ImageTransformation[src, tz, {200, 200}, DataRange -> Full, 
   PlotRange -> {{0, 1}, {0, 1}}]}
   find[img_] := 
  (croppy = ImageCrop[img, {180, 180}];
   bin = ColorNegate@
     Binarize[ImageAdjust[GaussianFilter[croppy, 5]]];
   components = 
    ComponentMeasurements[
     bin, {"Centroid", "Area", "FilledCircularity", 
      "EquivalentDiskRadius"}, #2 > 100 &]; 
   Show[croppy, 
    Graphics[{Thick {If[#[[3]] > 0.5, Red, Blue], 
          Circle[#[[1]], #[[4]]]} & /@ components[[All, 2]]}]] );

[jennymandl1023 (Jenny Mandl)]

Hello, I edited the post. The python was the tic tac toe algorithm and the goal was to have mathematica interface between the python and the arduino controlling the delta robot which is a 3 axis arm typically used for pick and place operations in manufacturing (we made a fully functioning one with 3D printed parts and stepper motors)

[garrettd (Garrett Ducharme)]

Hello! It sounds like you are trying to identify and find the location of the objects on a tic tac toe board. Here is a notebook which does that. I do not have your exact lighting conditions, so I used an intentionally bad tic tac toe board made in paint. It definitely is not perfect, but maybe it will give you some ideas.

The image I used to test (high-res in the notebook), intentionally bad paint image to try and replicate less than ideal conditions

(*Smoothing out the image, and getting components. This is similar to the original post. Ideally the x's, o's, and grid are separate.*)

srcAdjusted = Erosion[img, DiskMatrix[5]];
components = ComponentMeasurements[ColorNegate@Binarize[srcAdjusted], {"ConvexArea", "Mask"}][[All, 
   2]];

(*Getting the grid component, filling in the center*)

largestComponent = Image[SortBy[components, "First"][[-1, 2]]];
mask = FillingTransform[Closing[largestComponent, 3]]

(*subtracting the component from the filled in center to get just the box, then finding the corners of it*)

box = Opening[ImageSubtract[mask, largestComponent], DiskMatrix[3]];
corners = ImageCorners[box, 20, .01, 25, MaxFeatures -> 4];
Show[box, ListPlot[corners]]

(*remove the grid from components, and obtain the image equivalent of each sparse array*)

nogrid = Delete[SortBy[components, First], -1];
imgcomponents = Map[Image[#[[2]] // Normal] &, nogrid]

(*Sort the corners, get the furthest left/right/top/bottom*)

sortx = SortBy[corners, #[[1]] &];
sorty = SortBy[corners, #[[2]] &];
{left, top, right, bottom} = {Mean[{sortx[[1, 1]], sortx[[2, 1]]}], 
  Mean[{sorty[[3, 2]], sorty[[4, 2]]}], Mean[{sortx[[3, 1]], sortx[[4, 1]]}], 
  Mean[{sorty[[1, 2]], sorty[[2, 2]]}]}

(*Obtain the center of each, figure out what space on the tictactoe board it should be*)

centerOfGravity[l_] := ComponentMeasurements[Image[l], "Centroid"][[1, 2]]
getrowcol[{x_, y_}]  := {If[y >= top, 1, If[y >= bottom, 2, 3]], 
  If[x <= left, 1, If[x <=  right, 2, 3]]  }
locations = Map[{getrowcol[centerOfGravity[#]], 
    If[Length[ImageLines[#, .18, .15]] > 1, "x", "o"]} &, imgcomponents ]

(*Make an empty array, map in the above data, display it in matrix form*)

tictactoe = {{"", "", ""}, {"", "", ""}, {"", "", ""}};
Map[(tictactoe[[#[[1, 1]], #[[1, 2]]]] = #[[2]]) &, locations];
tictactoe // MatrixForm