Get the sources for Wolfram's stop words and common words?

Questions
,
1

Hello everyone,

I’m doing some text analysis and making use of Mathematica’s DeleteStopwords and

WordList["CommonWords"]

Where can I find Wolfram’s source for these?

Regards,

Gregory

2

Dear Gregory,

I don’t know the algorithms but I suppose that the list of stop words the WL uses is:

list = Complement[DictionaryLookup["*"], DeleteStopwords[DictionaryLookup["*"]]]

ScreenShot2016-06-07at21.23.38.png
ScreenShot2016-06-07at21.23.38.png695×591 84.4 KB

I can then also build my own function to delete stop words:

Select[TextWords[ExampleData[{"Text", "AliceInWonderland"}]], ! MemberQ[list, #] &]

It is about as efficient as the original:

Select[TextWords[ExampleData[{"Text", "AliceInWonderland"}]], ! MemberQ[list, #] &]; // RepeatedTiming

(0.69 seconds)

DeleteStopwords[TextWords[ExampleData[{"Text", "AliceInWonderland"}]]]; // RepeatedTiming

(0.74 seconds)

Also it does not give quite the same result

Complement[DeleteStopwords[TextWords[ExampleData[{"Text", "AliceInWonderland"}]]], 
Select[TextWords[ExampleData[{"Text", "AliceInWonderland"}]], ! MemberQ[list, #] &]]

ScreenShot2016-06-07at21.30.45.png

and

Complement[Select[TextWords[ExampleData[{"Text", "AliceInWonderland"}]], ! MemberQ[list, #] &],
DeleteStopwords[TextWords[ExampleData[{"Text", "AliceInWonderland"}]]]]

ScreenShot2016-06-07at21.31.57.png
ScreenShot2016-06-07at21.31.57.png704×195 30.2 KB

The last result suggests that this is better:

Select[TextWords[ExampleData[{"Text", "AliceInWonderland"}]], ! MemberQ[ToLowerCase /@ list, ToLowerCase[#]] &]

as you can easily see:

Complement[Select[TextWords[ExampleData[{"Text", "AliceInWonderland"}]], ! MemberQ[ToLowerCase /@ list, ToLowerCase[#]] &], 
DeleteStopwords[TextWords[ExampleData[{"Text", "AliceInWonderland"}]]]]

ScreenShot2016-06-07at22.58.20.png

So they are not quite the same, but the “words” that are different are few. You also notice that the last output contains many words that finish with 's, which is easy to fix, too.

The little function that we built above can be fed with your favourite word list.

I know that this does not answer your question, but I hope it is useful.

Best wishes,

Marco

PS: I quite enjoy the fact that Mathematica appears to treat all in-laws as stop words.

3

Hi Marco,

Much obliged. I’ll definitely have to create a custom stop-words lists. Mathematica’s is too broad, and includes words that have potential positive or negative sentiment (e.g., “against”) in the context of text analysis.

Regards,

Gregory

4

Folks please also take a look at

list = {"cat", "cow"};
StringDelete["cat dog cow horse", list]

which avoids splitting string into words.