Entropy of the multinomial distribution

[Anon]

It doesn't look like the entropy of the multinomial distribution is currently supported, I imagine this is a nontrivial value to calculate, are there plans to support it some time in the future?

[Andreas Noack]

I'm not aware of any plans for supporting this but, as usual, contributions are very welcome.

On Mon, Apr 18, 2016 at 1:07 AM, Anon <esp...@gmail.com> wrote:

It doesn't look like the entropy of the multinomial distribution is currently supported, I imagine this is a nontrivial value to calculate, are there plans to support it some time in the future?

--
You received this message because you are subscribed to the Google Groups "julia-stats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to julia-stats...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Anon]

I asked this question on stats.SE and ended up answering it myself just now, here is the link: http://stats.stackexchange.com/questions/207893/entropy-of-the-multinomial-distribution/208075#208075.

The derivation is fairly straight forward and it scales linearly in both n and k, if this is acceptable then I can code it up and submit a pull request if you'd like, but it's so simple it might be easier for you to just quickly write it up since you're intimately familiar with the package while it would take me a bit of time to get up to speed on the particulars of Distributions.jl.

[Anon]

Also, here is the code I used to make sure the derivation was correct:

using Iterators

multEval(x::Vector{Int}, n::Int, p::Vector{Float64}) = (factorial(n) / prod(gamma(x+1))) * prod(p.^x)

# multinomial entropy

function multEnt(n::Int, p::Vector{Float64})

@assert n >= 0

multientropy = 0

for par in partitions(n)

if length(par) <= length(p)

point = vcat(par, zeros(Int, max(0, length(p)-length(par))))

for perm in Set(permutations(point))

multientropy -= multEval(perm, n, p) * log(multEval(perm, n, p))

end

end

end

return multientropy

end

# efficient multinomial entropy

function func(n::Int, p::Vector{Float64})

@assert n >= 0

multientropy = (-log(factorial(n)) - n*sum(p.*log(p))

+ sum([sum([binomial(n, x) * p[i]^x * (1 - p[i])^(n - x) * log(factorial(x)) for x in 0:n]) for i in 1:length(p)]))

return multientropy

end

On Monday, April 18, 2016 at 2:05:23 PM UTC-7, Andreas Noack wrote:

[Anon]

Ok I just went ahead and wrote the entropy function myself for multinomial.jl and submitted the pull request.

On Monday, April 18, 2016 at 2:05:23 PM UTC-7, Andreas Noack wrote: