Message Boards Message Boards

Kullback–Leibler divergence in Mathematica: how to deal with negative inf?

Posted 9 years ago

I'm trying to implement a function to calculate Kullback-Liebler Divergence but I am running into a problem with complex infinities in my result. Here is how I have implemented it thus far:

pmfA={1/6,1/6,1/6,1/6,1/6,1/6};
pmfB={1/4,1/4,1/8,1/8,1/8,1/8};

Function[{p, q}, p*Log[p] /Log[q]][N@pmfA,N@pmfB]
Total[%]

For some "nice" sample data (containing no zeros), we get the nice result:

1.00526

However, if we use some "not so nice" sample data (containing zeros for some components of the PMF), it won't work, because of a negative infinity for Log[0].

eg. Try it out with...

pmfA={1/6,1/6,1/6,1/6,1/6,1/6};
pmfB={1/4,1/4,1/2,0,0,0};

For the purpose of computing the total, I think it would be acceptable to treat the instances as "0" but I am unsure of how to implement that.

POSTED BY: Caitlin Ramsey
4 Replies
Posted 9 years ago
  • I said "complex infinities" above. I meant to say "negative infinity." :-)
POSTED BY: Caitlin Ramsey
Posted 9 years ago

Maybe building your own log function?

In[1]:= log[x_] := If[x > 0, Log[x], -$MaxMachineNumber]

In[2]:= kld = Function[{p, q}, p*log[p]/log[q]];

In[3]:= kld[1/6, 1/4]

Out[3]= Log[6]/(6 Log[4])

In[4]:= kld[1/6, 0]

Out[4]= 1.661165481546702*10^-309

In[5]:= % // Chop

Out[5]= 0
POSTED BY: David Keith
Posted 9 years ago

Isn't KL Divergence Total[p Log[p/q]], not Total[p Log[p]/Log[q]] ?

POSTED BY: Paul Reiser

I know I'm a bit late to this party, but for anyone who still has the same question in 2019: I wrote a repository functions specifically for this: KullbackLeiblerDivergence on the Wolfram Function Repository. It should work for all sorts of valid WL distributions, but please let me know if you find something lacking.

POSTED BY: Sjoerd Smit
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract