negative TE entropy appeared when using kernal entropy

[gaz...@gmail.com]

Hi, thank you for providing us with the useful JIDT.I used kernel entropy in JIDT to calculate TE, but the results were weird.Most of the results were positive, but a few of them were negative(the absolute value of negative TE was quite small). I suppose that the theoretical results should be positive, but negative TE appeared.

My questions are:

1.is this situation normal and acceptable?

2. why did this situation occur?

3. how can I settle this problem?

I put my results and code in the appendix.

Yours,

Alice

[Joseph Lizier]

Hi Alice,

Yes, info theoretic measures on continuous valued data can get a little strange. The entropies can go negative, but mutual information (and conditional mutual information) in theory are still supposed to be non-negative.

Now, what you have to remember is that we are dealing with estimators, which aren't perfect numerically, and there are several things that can make the estimates dip below zero. The most common way we see this is with the bias correction on the KSG estimator, when the uncorrected estimates would have been below the expected bias.

But it can happen on the kernel estimator too. You could have a situation where the TE should be zero, but the estimate of the marginal (conditional) entropies were slightly below the true value, whilst the estimate of the joint (conditional) entropy was slightly above the true value. I can construct something very pathological to demonstrate this, see the attached file (which runs in Matlab, not Python, sorry but that's easier for me and I'm sure you can translate it if you want to run it!). I've constructed this example such that the marginals are constant for almost all points, yet the joint space never has any points in it apart from the search point itself, and so the joint probability is generally lower than the product of the marginals and the result is rather negative (much more negative than the fluctuations you see). You can turn on debugging if you want to see more info for each search point when it runs.

So, to answer your questions, it is a normal numerical issue and you can't really do anything about it (just take those values as consistent with zero).

Now, I notice that you're not setting the kernel width in your example, which suggests you perhaps haven't thought too hard about this parameter? The kernel width is a crucial parameter for this estimator, and makes a very large difference in the values it produces. So large an effect, that you're really better off to go with an estimator that is less parameter dependent. Indeed, the kernel estimator is really only left in JIDT for historical comparative purposes.

May I suggest you switch your analysis over to the KSG estimator, which is considered best of breed these days? The only key parameter there is the number of nearest neighbours, which it is much more stable to, and you can basically leave at the default of 4 anyway (as suggested in the JIDT paper). Another advantage of this estimator is that it is bias corrected, but this does mean that you will see negative values with it too (as described at the link above).

I hope that helps,

--joe

--
JIDT homepage: http://code.google.com/p/information-dynamics-toolkit/
---
You received this message because you are subscribed to the Google Groups "Java Information Dynamics Toolkit (JIDT) discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jidt-discuss+unsubscribe@googlegroups.com.
To post to this group, send email to jidt-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/jidt-discuss.
To view this discussion on the web visit https://groups.google.com/d/msgid/jidt-discuss/8b1d5c01-22c8-4b42-a6ee-00e38ea1cc3a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[sara h]

Dear Joe

thanks for your help

I have used of KSG but all my TE values are negative.what  can I do?

I should mention that I'm calculating transfer entropy of Phase of EEG data. not only for this data but in test on  random data most of TE values were negative.

regards

To unsubscribe from this group and stop receiving emails from it, send an email to jidt-discuss...@googlegroups.com.

[Joseph Lizier]

Hi Sara,

This is related to an answer I provided earlier for you on the thread: "How to solve the pseudo_regression of all-zero series", please refer to that. When the target time series has very strong memory in it, the KSG correction can lead to negative values for no or a very weak interaction. (The bias correction is very good, but not perfect being a bit too large in this instance.) What's most important is the difference to the surrogates generated for that series (even if they are negative) showing us the difference to what the measure would be expected to give under no relationship.

--joe
0408 186 901

Sent from my mobile device

To view this discussion on the web visit https://groups.google.com/d/msgid/jidt-discuss/23b74b66-6453-4275-a877-6f5aa0ab5d60%40googlegroups.com.