Hellinger distance

Summary

In probability and statistics, the Hellinger distance (closely related to, although different from, the Bhattacharyya distance) is used to quantify the similarity between two probability distributions. It is a type of f-divergence. The Hellinger distance is defined in terms of the Hellinger integral, which was introduced by Ernst Hellinger in 1909.[1][2]

It is sometimes called the Jeffreys distance.[3][4]

Definition

edit

Measure theory

edit

To define the Hellinger distance in terms of measure theory, let   and   denote two probability measures on a measure space   that are absolutely continuous with respect to an auxiliary measure  . Such a measure always exists, e.g  . The square of the Hellinger distance between   and   is defined as the quantity

 

Here,   and  , i.e.   and   are the Radon–Nikodym derivatives of P and Q respectively with respect to  . This definition does not depend on  , i.e. the Hellinger distance between P and Q does not change if   is replaced with a different probability measure with respect to which both P and Q are absolutely continuous. For compactness, the above formula is often written as

 

Probability theory using Lebesgue measure

edit

To define the Hellinger distance in terms of elementary probability theory, we take λ to be the Lebesgue measure, so that dP /  and dQ / dλ are simply probability density functions. If we denote the densities as f and g, respectively, the squared Hellinger distance can be expressed as a standard calculus integral

 

where the second form can be obtained by expanding the square and using the fact that the integral of a probability density over its domain equals 1.

The Hellinger distance H(PQ) satisfies the property (derivable from the Cauchy–Schwarz inequality)

 

Discrete distributions

edit

For two discrete probability distributions   and  , their Hellinger distance is defined as

 

which is directly related to the Euclidean norm of the difference of the square root vectors, i.e.

 

Also,  

Properties

edit

The Hellinger distance forms a bounded metric on the space of probability distributions over a given probability space.

The maximum distance 1 is achieved when P assigns probability zero to every set to which Q assigns a positive probability, and vice versa.

Sometimes the factor   in front of the integral is omitted, in which case the Hellinger distance ranges from zero to the square root of two.

The Hellinger distance is related to the Bhattacharyya coefficient   as it can be defined as

 

Hellinger distances are used in the theory of sequential and asymptotic statistics.[5][6]

The squared Hellinger distance between two normal distributions   and   is:

 

The squared Hellinger distance between two multivariate normal distributions   and   is [7]

 

The squared Hellinger distance between two exponential distributions   and   is:

 

The squared Hellinger distance between two Weibull distributions   and   (where   is a common shape parameter and   are the scale parameters respectively):

 

The squared Hellinger distance between two Poisson distributions with rate parameters   and  , so that   and  , is:

 

The squared Hellinger distance between two beta distributions   and   is:

 

where   is the beta function.

The squared Hellinger distance between two gamma distributions   and   is:

 

where   is the gamma function.

Connection with total variation distance

edit

The Hellinger distance   and the total variation distance (or statistical distance)   are related as follows:[8]

 

The constants in this inequality may change depending on which renormalization you choose (  or  ).

These inequalities follow immediately from the inequalities between the 1-norm and the 2-norm.

See also

edit

Notes

edit
  1. ^ Nikulin, M.S. (2001) [1994], "Hellinger distance", Encyclopedia of Mathematics, EMS Press
  2. ^ Hellinger, Ernst (1909), "Neue Begründung der Theorie quadratischer Formen von unendlichvielen Veränderlichen", Journal für die reine und angewandte Mathematik (in German), 1909 (136): 210–271, doi:10.1515/crll.1909.136.210, JFM 40.0393.01, S2CID 121150138
  3. ^ "Jeffreys distance - Encyclopedia of Mathematics". encyclopediaofmath.org. Retrieved 2022-05-24.
  4. ^ Jeffreys, Harold (1946-09-24). "An invariant form for the prior probability in estimation problems". Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences. 186 (1007): 453–461. Bibcode:1946RSPSA.186..453J. doi:10.1098/rspa.1946.0056. ISSN 0080-4630. PMID 20998741. S2CID 19490929.
  5. ^ Torgerson, Erik (1991). "Comparison of Statistical Experiments". Encyclopedia of Mathematics. Vol. 36. Cambridge University Press.
  6. ^ Liese, Friedrich; Miescke, Klaus-J. (2008). Statistical Decision Theory: Estimation, Testing, and Selection. Springer. ISBN 978-0-387-73193-3.
  7. ^ Pardo, L. (2006). Statistical Inference Based on Divergence Measures. New York: Chapman and Hall/CRC. p. 51. ISBN 1-58488-600-5.
  8. ^ Harsha, Prahladh (September 23, 2011). "Lecture notes on communication complexity" (PDF).

References

edit
  • Yang, Grace Lo; Le Cam, Lucien M. (2000). Asymptotics in Statistics: Some Basic Concepts. Berlin: Springer. ISBN 0-387-95036-2.
  • Vaart, A. W. van der (19 June 2000). Asymptotic Statistics (Cambridge Series in Statistical and Probabilistic Mathematics). Cambridge, UK: Cambridge University Press. ISBN 0-521-78450-6.
  • Pollard, David E. (2002). A user's guide to measure theoretic probability. Cambridge, UK: Cambridge University Press. ISBN 0-521-00289-3.