So (for example) is there a simple (textbook) way to derive the entropy from the distribution?
In general, I need to derive the entropy for many discrete probability distributions given a PMF that is determined by combinatorial analysis. (It seems Mathematica calls it the PDF for discrete distributions?)
I suspect that it’s actually not trivial, based on this paper, but I’m very far from being an expert, which is why I’m asking.
Thanks, Mariusz. That question on the Mathematica overflow is closely related, and it helped me find other people asking the same question as me on there!
Henrik, I’m completely new to Mathematica, so I understand most of what you’ve written there, but could you briefly explain, or link to an explanation, of a few of things:
After the end of FunctionExpand, what is that “/.” operator and the expression with Gamma doing? (I know what the Gamma function itself is.)
Thanks, Vitaliy. Yes, you would think that generalization of Shannon entropy would do the job, but I had a quick play with it and wasn’t able to get a result. I probably just don’t know what I’m doing, though.
Mariusz, thank you, that’s a great simplification! It works very well.
It takes several seconds to produce an answer, which is not a problem at the moment, but are there additional constraints/assumptions I could add to help it solve faster? Is it just a matter of experimenting and see what makes a difference?
I successfully solved another pmf, but then I accidentally lost the equation after copying down the result. Now I can’t recreate the result!
This is what I’m trying:
q = ((T - 1)/T)^i;
p = (1/T) q;
pmf = if[i = n, q, p];
-Sum[#, {n, 0, Infinity}] & /@ (pmf Log[pmf] // PowerExpand // Expand)
And I previously got a result of:
-((-1 + T) Log[-1 + T]) + T Log[T]
which I verified numerically against other data I have.
But now when I run the above commands, I get a very ambiguous result that is hardly simplified at all.
What am I going wrong in trying to recreate the result? It seemed to easy the first time.
Thanks, Henrik, yes it took me a while before I realized that Mathematica uses = for assignment. Apologies for the half-baked plea for help, I did figure it out by myself in the end.
n is the size of the discrete random variable minus one, and i is the index you might say of the outcome, so when calculating numerical values of entropy for a drv of finite size, it computes essentially computes p[i] until i == n where it then computes q[i]. But my mathematics is not very strong and I didn’t know how to express that in a way Mathematica would understand, so I actually just got rid of q altogether. The numerical results are spot on, but I’m not sure if the symbolic result is right, given that I dropped the case of when it reaches n (since n is Infinity in the analysis).
I also seemed to be having problems with variable declarations leaking from previous calculations. As in, the k variable from calculating Poisson still existed and so if I used k again in the definition of p then it didn’t work. I can’t explain in detail as I haven’t had time to experiment, but essentially I couldn’t run the Poisson calculation twice – the second time would give me an error.
Finally I made the mistake of changing pmf to p in the Sum without removing the definition fo pmf and so the # symbol still referred to pmf although I didn’t want it to. Rookie mistakes.
This is what I did in the end, but with a lot of Clear commands preceding it:
