I still think that the best solution is to use a tool like sox, but if you'd like to find out what FrontEnd`RecordSound does, here's a practical way to do it:

Take the spelunking tools from here. Normally I'd recommend Simon's version, but that one has trouble showing formatted elements, so it's inconvenient in this case. (If you decide to fix this limitation, please send me a pull request.) So use contextFreeDefintion[] from the question itself.

First, use SystemDialogInput["RecordSound"] to trigger loading the code.  Then use contextFreeDefinition[SystemDialogInput].  To make the output a bit more readable, paste it into Workbench, select it, right click, then choose "Format".  Or just paste it into a text editor and replace each ";" with ";\n".  Now try to read the code and figure out how to use FrontEnd`RecordSound.

I don't have the time to do this, but here are a few of the low hanging fruit:

FrontEndExecute[FrontEnd`RecordSound[5]] will give you a list of devices
FrontEndExecute[FrontEnd`RecordSound[6, deviceNumber] will give you a list of supported formats for that device
FrontEndExecute[FrontEnd`RecordSound[1, deviceNumber, formatNumber]]  will probably start recording on the given device with the given format.
FrontEndExecute[FrontEnd`RecordSound[2]] will stop recording and return the duration
FrontEndExecute[FrontEnd`RecordSound[3]] will return the recorded sound data.
FrontEndExecute[FrontEnd`RecordSound[4]] will probably discard the recorded data

I'm not completely sure about these.  If you figure out the rest, let us know.   Also, I think some sort of initialization is needed before this is used for the first time.  I am not sure how to do that but running SystemDialogInput["RecordSound"] once will do it.  The solution might be FrontEndExecute[FrontEnd`RecordSound[7,0]]; FrontEndExecute[FrontEnd`RecordSound[8,0]] but I'm really not sure about this.

Warning: keep in mind that all of this is undocumented for a good reason: it may easily change in future versions and this may not even work the same way on Windows (I'm also on OS X, like you).  It is not meant for end users and I won't be surprised if it's possible to crash the front end or mess up its internal state by using these.

Note: duplicated here.

How would you stop the recording then?  Would you supply a duration when you start it?

EDIT:  I have not tried it, but it seems sox should work and it should be an easy solution (just Run[] it, no MathLink needed).  According to its docs, it has a recording feature and the duration can either be set, or it can just be terminated manually (this latter option is not that useful when calling sox from Mathematica).