Whitening transformation

Summary

A whitening transformation or sphering transformation is a linear transformation that transforms a vector of random variables with a known covariance matrix into a set of new variables whose covariance is the identity matrix, meaning that they are uncorrelated and each have variance 1.[1] The transformation is called "whitening" because it changes the input vector into a white noise vector.

Several other transformations are closely related to whitening:

  1. the decorrelation transform removes only the correlations but leaves variances intact,
  2. the standardization transform sets variances to 1 but leaves correlations intact,
  3. a coloring transformation transforms a vector of white random variables into a random vector with a specified covariance matrix.[2]

Definition

edit

Suppose   is a random (column) vector with non-singular covariance matrix   and mean  . Then the transformation   with a whitening matrix   satisfying the condition   yields the whitened random vector   with unit diagonal covariance.

There are infinitely many possible whitening matrices   that all satisfy the above condition. Commonly used choices are   (Mahalanobis or ZCA whitening),   where   is the Cholesky decomposition of   (Cholesky whitening),[3] or the eigen-system of   (PCA whitening).[4]

Optimal whitening transforms can be singled out by investigating the cross-covariance and cross-correlation of   and  .[3] For example, the unique optimal whitening transformation achieving maximal component-wise correlation between original   and whitened   is produced by the whitening matrix   where   is the correlation matrix and   the diagonal variance matrix.

Whitening a data matrix

edit

Whitening a data matrix follows the same transformation as for random variables. An empirical whitening transform is obtained by estimating the covariance (e.g. by maximum likelihood) and subsequently constructing a corresponding estimated whitening matrix (e.g. by Cholesky decomposition).

High-dimensional whitening

edit

This modality is a generalization of the pre-whitening procedure extended to more general spaces where   is usually assumed to be a random function or other random objects in a Hilbert space  . One of the main issues of extending whitening to infinite dimensions is that the covariance operator has an unbounded inverse in  . Nevertheless, if one assumes that Picard condition holds for   in the range space of the covariance operator, whitening becomes possible.[5] A whitening operator can be then defined from the factorization of the Moore–Penrose inverse of the covariance operator, which has effective mapping on Karhunen–Loève type expansions of  . The advantage of these whitening transformations is that they can be optimized according to the underlying topological properties of the data, thus producing more robust whitening representations. High-dimensional features of the data can be exploited through kernel regressors or basis function systems.[6]

R implementation

edit

An implementation of several whitening procedures in R, including ZCA-whitening and PCA whitening but also CCA whitening, is available in the "whitening" R package [7] published on CRAN. The R package "pfica"[8] allows the computation of high-dimensional whitening representations using basis function systems (B-splines, Fourier basis, etc.).

See also

edit

References

edit
  1. ^ Koivunen, A.C.; Kostinski, A.B. (1999). "The Feasibility of Data Whitening to Improve Performance of Weather Radar". Journal of Applied Meteorology. 38 (6): 741–749. Bibcode:1999JApMe..38..741K. doi:10.1175/1520-0450(1999)038<0741:TFODWT>2.0.CO;2. ISSN 1520-0450.
  2. ^ Hossain, Miliha. "Whitening and Coloring Transforms for Multivariate Gaussian Random Variables". Project Rhea. Retrieved 21 March 2016.
  3. ^ a b Kessy, A.; Lewin, A.; Strimmer, K. (2018). "Optimal whitening and decorrelation". The American Statistician. 72 (4): 309–314. arXiv:1512.00809. doi:10.1080/00031305.2016.1277159. S2CID 55075085.
  4. ^ Friedman, J. (1987). "Exploratory Projection Pursuit" (PDF). Journal of the American Statistical Association. 82 (397): 249–266. doi:10.1080/01621459.1987.10478427. ISSN 0162-1459. JSTOR 2289161. OSTI 1447861.
  5. ^ Vidal, M.; Aguilera, A.M. (2022). "Novel whitening approaches in functional settings". STAT. 12 (1): e516. doi:10.1002/sta4.516. hdl:1854/LU-8770510.
  6. ^ Ramsay, J.O.; Silverman, J.O. (2005). Functional Data Analysis. Springer New York, NY. doi:10.1007/b98888. ISBN 978-0-387-40080-8.
  7. ^ "whitening R package". Retrieved 2018-11-25.
  8. ^ "pfica R package". Retrieved 2023-02-11.
edit
  • http://courses.media.mit.edu/2010fall/mas622j/whiten.pdf
  • The ZCA whitening transformation. Appendix A of Learning Multiple Layers of Features from Tiny Images by A. Krizhevsky.