Message Boards Message Boards

[WSS20] Implementing Mutual Information

Posted 4 years ago

Karman vortex street and an estimation of mutual information

POSTED BY: Luigi Brancati
7 Replies

Hello,

Can we use the ResourceFunction["MutualInformation"] function on two lists, one discrete and one continuous ? Do you think it will behave as the algorithm written in the paper "Mutual Information between Discrete and Continuous Data Sets" ? They also use a K-neighbor method, and cite the same paper as you in the documentation page.

Thank you very much !

POSTED BY: Clarisse Wagner
Posted 3 years ago

Brilliant piece of innovation Luigi!

This may seem a really strange question but here goes anyway ...

Is it possible that this concept could be applied to extract some semantic meaning from a corpus of text on a particular topic? For example if there were a series of papers on a concept written in different English language styles over time which described alternate interpretations or meaning of a particular topic I wonder if the textual data sets also could be correlated in the manner you describe and somehow that correlation leads to semantic insights or at least the basis for further Computational experimentation to do so.

What triggered me to ask you this is your statement below:

“Results: We were able to develop a function which calculates mutual information between two sets of data, and relies on a distance function to do so. As any data can be considered to be embedded in a metric space, this implementation works in principle for any kind of data, whether it is images, videos or just time series.”

This is all very “AI’ish” and non-deterministic but I wonder if it makes any sense to you?

I would certainly like to play with this idea of yours on simple data sets like word definitions of complex words with lots of historical alternate descriptions just to see what emerges from relatively simple but possibly large enough number of alternate definitions of the same word.

Cheers ... Syd Geraghty

POSTED BY: Syd Geraghty

enter image description here -- you have earned Featured Contributor Badge enter image description here

Your exceptional post has been selected for our editorial column Staff Picks http://wolfr.am/StaffPicks and Your Profile is now distinguished by a Featured Contributor Badge and is displayed on the Featured Contributor Board. Thank you!

POSTED BY: Moderation Team
Posted 3 years ago

Hello Jim,

You're welcome, and sorry for the late answer. Yes, the code

KraskovI1[list, list+T, 4], with T ranged 0,20

is equivalent to ResourceFunction["MutualInformation"][list,20]. Also note that there's the alternative ResourceFunction["MutualInformation"][list,t,s]. Moreover, note that the 4 in your example is now an option, "KNeighbour", which is used when working with lists of samples.

When working on the project I needed some more flexibility and opted to work directly with the function KraskovI1. Later I wrapped it up in the function MutualInformation and submitted it to the repository, so that's the reason on the discrepancies you see.

As a side note, I'm slowly updating the function to make it better, I updated it to work on any multivariate distribution and added some more error checking. You can always find an updated version here, but hopefully it will be soon on the Wolfram Function Repository.

About your last question, I'm not sure what you mean. As you noticed, the first element is the zero-shift MI, which is not 0, but some of my plots are "clipped" to show the most interesing part, since a peak at position 1 isn't that interesting and was usually too high. Also, ListLinePlot will always plot any list starting from 1, if you need the plot to start from 0 you have to specify an x value along with the y value, e.g.

ListLinePlot[Thread[List[Range[0, 9], Range[10]]], PlotRange -> All]

If you need any further help, please feel free to contact me on linkedin (you can find it on my profile).

And good luck with your work!

POSTED BY: Luigi Brancati
Posted 3 years ago

Thanks so much, Luigi. I think I am able to get it running. But to make certain, may ask you if your Community post syntax:

KraskovI1[list, list+T, 4], with T ranged 0,20

is equivalent to your new ResourceFunction:

ResourceFunction["MutualInformation"][list,20]

A smaller question, I wonder if there's a small index-shift problem? If I do

ListLinePlot[output from your MutualInformation] 

the plot will show the zero-shift MI at position 1 (not zero) in the plot (since the zero shift is the first element in the MI list). The plots in the post were probably shifted by one element from actual value (e.g, for minimum in MI). I suppose this could be fixed if I do my plot shifted so 1st element in MI list is plotted at origin, at 0.

Thanks so much. You have probably moved on from this project, but I am finding it potentially very useful in my Physics research. And I would like to acknowledge your work if/when I ever publish what I'm looking at.

-Jim Clarage, University of St. Thomas Houston, Texas

POSTED BY: Updating Name
Posted 3 years ago

Where is the code for your function "KraskovI1" ? It is missing from your post and notebook. If possible I'd like to use this function, or some function to calculate the Mutual Information of a time series.

Thanks.

POSTED BY: Jim Clarage
Posted 3 years ago

Hi, thanks for the interest! My function is currently under review for submission in the Wolfram Function Repository, but you can find a preliminary (not reviewed) version here. It doesn't explicitely support timeseries yet, but I plan on adding that soon.

(Ignore the comment of the reviewer in the examples section, I just resubmitted it and couldn't delete that)

POSTED BY: Luigi Brancati
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract