Hi Vitaliy,

I write a simple n-gram WordCloud function below. The code quality can still improve, but it works with n-gram idea. Do you have any suggestion to improve the code quality or effiency?

Attached the notebook format for better understand.

nGramWords[text_String, n_Integer: 4, filterLevel_Integer: 2] := 
  Module[{
    words = DeleteStopwords[TextWords[ToLowerCase[text]]], 
    nGramInitial, nGramTable, removeValue, newLine},
   seperate[list_] := 
    With[{l = Length@list}, {Take[list, l - 1], Take[list, -(l - 1)]}];
   nGramInitial = 
    Normal@Table[
      Select[WordCounts[StringRiffle[words], 
        i], # >= filterLevel &], {i, n, 1, -1}];
   nGramInitial = 
    Join[Drop[
      nGramInitial, -1], {({#[[1]]} -> #[[2]]) & /@ 
       Last@nGramInitial}];
   nGramTable = {};
   nGramTable = Append[nGramTable, First@nGramInitial];
   Do[
    removeValue = 
     Flatten@Table[
       Thread[Rule[seperate[First@Last[nGramTable][[i]]], 
         Table[-Last@Last[nGramTable][[i]], {2}]]], {i, 1, 
        Length@Last[nGramTable]}];
    newLine = 
     Sort[Select[
       Flatten@If[
           Length[#] > 
            1, #[[1]][[1]] -> (#[[1]][[2]] + #[[2]][[2]]) , #[[
            1]]] & /@ 
        GatherBy[Join[removeValue, nGramInitial[[j]]], 
         First], #[[2]] >= 1 &], #1[[2]] > #2[[2]] &];
    nGramTable = Append[nGramTable, newLine], {j, 2, n}];
   Sort[{StringRiffle[#[[1]]], #[[2]]} & /@ 
     Select[Flatten[nGramTable], #[[2]] >= filterLevel &], #1[[
       2]] > #2[[2]] &]
   ];

Attachments:

Impressive compactness!

Thanks for posting, @Frederick Wu, I will have to find some time to dig through this. I think if you could add some explanations how your code for n-gram WordCloud works, this actually could make a separate nice post! Especially if the obvious difference between regular WordCloud would be demonstrated in some cases. Please consider this ;-)

Thank you @Jesse ! Yes, I'd like to do that soon. I am considering first giving an upgrade to the function KeywordsGraph that would give am option of counting different grammatical forms as one keyword (such as, for instance, "cat" and "cats"). BTW, great example with ReadabilityScore and American presidents’ inaugural addresses!

Hi Vitaliy,

This is a great idea with many possible extensions. This code could have interesting applications in literature. My incompetence with the Cyrillic alphabet and Slavic language leads me to an idea where many people may be interested in the results.

These are difficult times throughout the world, when we are all hoping for the best possible leadership. It could be a mistake to think that the best voices will be from those who win the contest for public or private office. Maybe you have already guessed what I'm hinting at: yes, Bulgakov, the samizdat hero, from Kiev, Ukraine.

Quickly searching google I find many translations of (in English) "Heart of a Dog", even a few PDF files that could be mined for plaintext. Side by side comparisons of the translations would give us a quantitative idea about the fluctuations between texts. Do you think this is possible using your code? Can your code operate on Cyrillic texts?

"Heart of a Dog" is a masterpiece of science fiction. I'm sure it's a risk to say out loud, but I think more people in the English speaking world should read Bulgakov. That being said, science fiction can sometimes have a gender biased audience. In particular I think the details of this story might be too grotesque and masculine for some women. But I have recently heard that the woman poet Anna Akhmatova is also a worthwhile Ukrainian-born samizdat hero, also active "during the terrible years of the Yezhovshchina". Again many translations exist, so another question: could your method be adapted to compare translations of short form writing such as a poem?

Thanks, Brad.