I used your suggestion, with a simple modification. So now we have the original and the enhanced.
ParseText[
text_] := (* text is string, result is lower case list of words within text *)
StringSplit[ToLowerCase[text], RegularExpression["\\W+"]]
ParseText2[
text_] := (* text is string, result is lower case list of words within text *)
StringSplit[ToLowerCase[text], Except[WordCharacter | "'"]]
One would think the enhanced solves all problems. Alas, no such luck.
In[227]:= sample3 = "These are the times, that try men's souls.";
In[228]:= ParseText[sample3]
Out[228]= {"these", "are", "the", "times", "that", "try", "men", "s", "souls"}
In[229]:= ParseText2[sample3]
Out[229]= {"these", "are", "the", "times", "", "that", "try", "men's", "souls"}
Merely adding a comma in the sample phrase results in the space after the comma being recognized as a separate word. According to the documentation RegularExpression["\W+"] is equivalent to Except[WordCharacter], so adding the pattern "[RawQuote]" to the Except pattern should not cause problems. It does. If this result is not a bug in Mathematica, I don't understand its logic.