Let ${\displaystyle X_{1},X_{2},\dots }$  be a sequence of independent identically distributed random variables with values in the state space S with probability distribution P.

Definition

The empirical measure P_n is defined for measurable subsets of S and given by

${\displaystyle P_{n}(A)={1 \over n}\sum _{i=1}^{n}I_{A}(X_{i})={\frac {1}{n}}\sum _{i=1}^{n}\delta _{X_{i}}(A)}$ 

where ${\displaystyle I_{A}}$  is the indicator function and ${\displaystyle \delta _{X}}$  is the Dirac measure.

Properties

• For a fixed measurable set A, nP_n(A) is a binomial random variable with mean nP(A) and variance nP(A)(1 − P(A)).
  • In particular, P_n(A) is an unbiased estimator of P(A).
• For a fixed partition ${\displaystyle A_{i}}$  of S, random variables ${\displaystyle Y_{i}=nP_{n}(A_{i})}$  form a multinomial distribution with event probabilities ${\displaystyle P(A_{i})}$ 
  • The covariance matrix of this multinomial distribution is ${\displaystyle Cov(Y_{i},Y_{j})=nP(A_{i})(\delta _{ij}-P(A_{j}))}$ .

Definition

${\displaystyle {\bigl (}P_{n}(c){\bigr )}_{c\in {\mathcal {C}}}}$  is the empirical measure indexed by ${\displaystyle {\mathcal {C}}}$ , a collection of measurable subsets of S.

To generalize this notion further, observe that the empirical measure ${\displaystyle P_{n}}$  maps measurable functions ${\displaystyle f:S\to \mathbb {R} }$  to their empirical mean,

${\displaystyle f\mapsto P_{n}f=\int _{S}f\,dP_{n}={\frac {1}{n}}\sum _{i=1}^{n}f(X_{i})}$ 

In particular, the empirical measure of A is simply the empirical mean of the indicator function, P_n(A) = P_n I_A.

For a fixed measurable function ${\displaystyle f}$ , ${\displaystyle P_{n}f}$  is a random variable with mean ${\displaystyle \mathbb {E} f}$  and variance ${\displaystyle {\frac {1}{n}}\mathbb {E} (f-\mathbb {E} f)^{2}}$ .

By the strong law of large numbers, P_n(A) converges to P(A) almost surely for fixed A. Similarly ${\displaystyle P_{n}f}$  converges to ${\displaystyle \mathbb {E} f}$  almost surely for a fixed measurable function ${\displaystyle f}$ . The problem of uniform convergence of P_n to P was open until Vapnik and Chervonenkis solved it in 1968.^[1]

If the class ${\displaystyle {\mathcal {C}}}$  (or ${\displaystyle {\mathcal {F}}}$ ) is Glivenko–Cantelli with respect to P then P_n converges to P uniformly over ${\displaystyle c\in {\mathcal {C}}}$  (or ${\displaystyle f\in {\mathcal {F}}}$ ). In other words, with probability 1 we have

${\displaystyle \|P_{n}-P\|_{\mathcal {C}}=\sup _{c\in {\mathcal {C}}}|P_{n}(c)-P(c)|\to 0,}$ 

${\displaystyle \|P_{n}-P\|_{\mathcal {F}}=\sup _{f\in {\mathcal {F}}}|P_{n}f-\mathbb {E} f|\to 0.}$ 

The empirical distribution function provides an example of empirical measures. For real-valued iid random variables ${\displaystyle X_{1},\dots ,X_{n}}$  it is given by

${\displaystyle F_{n}(x)=P_{n}((-\infty ,x])=P_{n}I_{(-\infty ,x]}.}$ 

In this case, empirical measures are indexed by a class ${\displaystyle {\mathcal {C}}=\{(-\infty ,x]:x\in \mathbb {R} \}.}$  It has been shown that ${\displaystyle {\mathcal {C}}}$  is a uniform Glivenko–Cantelli class, in particular,

${\displaystyle \sup _{F}\|F_{n}(x)-F(x)\|_{\infty }\to 0}$ 

with probability 1.

• Empirical risk minimization
• Poisson random measure

• Billingsley, P. (1995). Probability and Measure (Third ed.). New York: John Wiley and Sons. ISBN 0-471-80478-9.
• Donsker, M. D. (1952). "Justification and extension of Doob's heuristic approach to the Kolmogorov–Smirnov theorems". Annals of Mathematical Statistics. 23 (2): 277–281. doi:10.1214/aoms/1177729445.
• Dudley, R. M. (1978). "Central limit theorems for empirical measures". Annals of Probability. 6 (6): 899–929. doi:10.1214/aop/1176995384. JSTOR 2243028.
• Dudley, R. M. (1999). Uniform Central Limit Theorems. Cambridge Studies in Advanced Mathematics. Vol. 63. Cambridge, UK: Cambridge University Press. ISBN 0-521-46102-2.
• Wolfowitz, J. (1954). "Generalization of the theorem of Glivenko–Cantelli". Annals of Mathematical Statistics. 25 (1): 131–138. doi:10.1214/aoms/1177728852. JSTOR 2236518.