Hi Sverker:

Topic modeling is not nearly as complicated as you might think. Here is a good introduction: http://www.youtube.com/watch?v=4p9MSJy761Y.

I have used it extensively to analyze research portfolios of the US Department of Agriculture. Some of the work was done with Mathematica, some outside with Mallet. Get in touch if you would like to know more.

Jeff Lapides jrlapides2@verizon.net

Hi Phileas:

You're welcome. This is a very old post. I know a lot more now than I knew then. If you wish to get in touch, I may be able to help more.

For what it is worth, I have now adapted topic modeling to numerical data and applied it to microbiomic studies of Alzheimer's disease. See preprint. The methodology might be useful to you. See the part on LDA.

Jeff Lapides
Adjunct Associate Professor
Drexel University
Working from Annapolis, MD
jeffrey.lapides@gmail.com

Thank you Arno and Jeff!

As regards Topic Modelling I have worked before with different varieties of matrix dimension reduction methods. As I understand MALLET and the discussion around it, it is based on a somewhat more advanced probabilistic model, that generates a more difficult optimization problem. To solve it you need to use some smart heuristic algorithm, and it is this implementations step that it would be nice to have implemented in Mathematica. As I work now I do preprocessing and post-processing in Mathematica, but leave the Topic-Modelling-machinery to MALLET. It is a very non-mathematica-style solution, with this use of textfiles and external software for mathematics of all things that Mathematica could possibly "not" do. :-) I would not be surprised if it is possible to outperform MALLET with smart Mathematica programming, but I cannot do it.

Thank you Arno for the book suggestion! I will go through it carefully, and it seems to me that I will be perfect for a course in Digital Humanities what will be given by the University of Gothenburg next Spring (and then probably on a yearly basis).

I have done some work in this area since my post. My present focus in on a material of Swedish Public Reports. There are about 7000 books, published between 1922 and 2015 (today). I try to use Mathematica as a platform for building tools for "navigating" this material - where navigation should be seen as an extension of "searching". I want to make Topic Modelling part of such navigation, as well as various kinds of "word-clouds". It is difficult to demonstrate what I have, or send you anything, because the tool needs to interact with the material which is several gigabytes large, and needs to be in RAM to get the necessary speed... I have asked WR how I should manage this kind of work on the web, but I suppose there is presently no good solution.

Kind Regards Sverker Lundin