Entropy in Hypothesis Testing: Review and Applications

34 downloads 107 Views 381KB Size Report
Shannon's entropy uses h(p(x)) = log 1 p(x). = − log p(x) and is the average surprise on discovering the outcome of a random experiment: H (X) = E[h(X)].
Entropy in Hypothesis Testing: Review and Applications

Overview

This talk consists of three parts: Part I - a review of entropy from the statistical mechanics perspective Part II - an overview of nonparametric approaches for the practical implementation of entropy Part III - some illustrations and examples taken from recent developments

Entropy in Hypothesis Testing: Review and Applications Part I: Entropy - A Review

Statistical Mechanics and Information Functions From the ‘statistical mechanics’ perspective of Shannon (1948), entropy is a measure of uncertainty (‘disorder’, ‘volatility’) associated with a random variable For the example on the last slide, when p(1) = 0 there is no uncertainty, no disorder, and indeed no volatility (the variance of X , ≡ p(1)(1 − p(1)), is 0 when p(1) = 0), so entropy would be zero In this experiment, entropy will increase as p(1) → 1/2 Entropy is not uniquely defined; there exist axiom systems that justify particular entropies Shannon’s ‘information’ or ‘surprise’ function is given by h(x ) = h(p(x )) ≡ log

1 , p(x )

and possesses certain “ideal” properties (for h(p(x )), h(1) = 0, h(0) → ∞, additivity, monotone decreasing); see Hartley (1928), among others

Entropy in Hypothesis Testing: Review and Applications Part I: Entropy - A Review

Example (Shannon’s Information Function)

20 15 10 5 0

h(x)=−log(p(x))

25

30

Shannon's Information/Surprise Function

0.0

0.2

0.4

0.6

p(x) (0 hx