Kullback–Leibler divergence in Mathematica: how to deal with negative inf?

Questions
, ,
1

I’m trying to implement a function to calculate Kullback-Liebler Divergence but I am running into a problem with complex infinities in my result. Here is how I have implemented it thus far:

pmfA={1/6,1/6,1/6,1/6,1/6,1/6};
pmfB={1/4,1/4,1/8,1/8,1/8,1/8};

Function[{p, q}, p*Log[p] /Log[q]][N@pmfA,N@pmfB]
Total[%]

For some “nice” sample data (containing no zeros), we get the nice result:

1.00526

However, if we use some “not so nice” sample data (containing zeros for some components of the PMF), it won’t work, because of a negative infinity for Log[0].

eg. Try it out with…

pmfA={1/6,1/6,1/6,1/6,1/6,1/6};
pmfB={1/4,1/4,1/2,0,0,0};

For the purpose of computing the total, I think it would be acceptable to treat the instances as “0” but I am unsure of how to implement that.

2
  • I said “complex infinities” above. I meant to say “negative infinity.” :slight_smile:
3

Maybe building your own log function?

In[1]:= log[x_] := If[x > 0, Log[x], -$MaxMachineNumber]

In[2]:= kld = Function[{p, q}, p*log[p]/log[q]];

In[3]:= kld[1/6, 1/4]

Out[3]= Log[6]/(6 Log[4])

In[4]:= kld[1/6, 0]

Out[4]= 1.661165481546702*10^-309

In[5]:= % // Chop

Out[5]= 0
4

Isn’t KL Divergence Total[p Log[p/q]], not Total[p Log[p]/Log[q]] ?

5

I know I’m a bit late to this party, but for anyone who still has the same question in 2019: I wrote a repository functions specifically for this: KullbackLeiblerDivergence on the Wolfram Function Repository. It should work for all sorts of valid WL distributions, but please let me know if you find something lacking.