Try with this syntax:

RegionFunction -> 
 Function[{x, y, z}, 
  1.2 + 0.4 Cos[4 ArcTan[x, y]] < Sqrt[x^2 + y^2] < 
    6 + 0.8 Cos[18 ArcTan[x, y]] && z <= 1]

Have you tried giving the option

RegionFunction -> Function[{x, y, z}, z<=1]

And if you need a 3D shape you can proceed also as follows: Convert to cartesian using FromPolarCoordinates:

In[18]:= gear[\[Theta]_] := {1 + 
   1/8 Sin[18 (\[Theta] - \[Pi]/4)], \[Theta]}
FromPolarCoordinates[gear[\[Theta]]]
ParametricPlot[%, {\[Theta], 0, 2 \[Pi]}]

Out[19]= {Cos[\[Theta]] (1 + 1/8 Sin[18 (-(\[Pi]/4) + \[Theta])]), 
 Sin[\[Theta]] (1 + 1/8 Sin[18 (-(\[Pi]/4) + \[Theta])])}

Insert a 3rd dimension (eg in the y direction) and make a ParametricPlot3D:

In[30]:= Insert[FromPolarCoordinates[gear[\[Theta]]], y, 2]
ParametricPlot3D[%, {\[Theta], 0, 2 \[Pi]}, {y, 0, 1}, 
 PlotPoints -> 25, PlotTheme -> "Marketing",  
 PerformanceGoal -> "Quality"]

Out[30]= {Cos[\[Theta]] (1 + 1/8 Sin[18 (-(\[Pi]/4) + \[Theta])]), y, 
 Sin[\[Theta]] (1 + 1/8 Sin[18 (-(\[Pi]/4) + \[Theta])])}