Tricks for remembering formulas & properties of multivariate normal distribution

Understanding the relationship between univariate and multivariate normal distributions can provide a clearer framework for remembering the definition and properties of the multivariate normal distribution.

First, it’s easier to understand & remember the keys properties of multivariate normal distribution by understanding the Mahalanobis distance. So, to start, recall that the Mahalanobis distance is a measure of the distance between a point and a distribution. It is particularly useful for multivariate data as it takes into account the correlations between variables. Unlike the Euclidean distance, which treats all dimensions equally, the Mahalanobis distance accounts for the variability and correlation in the data, providing a more accurate distance metric in many cases.

For a point $\mathbf{x}$ and a multivariate normal distribution with mean vector $\boldsymbol{\mu}$ and covariance matrix $\boldsymbol{\Sigma}$ , the Mahalanobis distance $D_M(\mathbf{x})$ is defined as:

$D_M(\mathbf{x}) = \sqrt{(\mathbf{x} - \boldsymbol{\mu})^\top \boldsymbol{\Sigma}^{-1} (\mathbf{x} - \boldsymbol{\mu})}$

Now, note that $\mathbf{x} - \boldsymbol{\mu}$ centers the point $\mathbf{x}$ around the mean vector $\boldsymbol{\mu}$ . Next, the term $\boldsymbol{\Sigma}^{-1}$ adjusts for the covariance structure of the data, scaling distances based on the variability and correlations among the variables. So, the quadratic term $(\mathbf{x} - \boldsymbol{\mu})^\top \boldsymbol{\Sigma}^{-1} (\mathbf{x} - \boldsymbol{\mu})$ is a quadratic form, providing a scalar value that represents the squared distance from $\mathbf{x}$ to the mean $\boldsymbol{\mu}$ in units of standard deviations. So:

If $D_M(\mathbf{x})$ is small, the point $\mathbf{x}$ is close to the mean $\boldsymbol{\mu}$ relative to the covariance structure of the distribution.
If $D_M(\mathbf{x})$ is large, the point $\mathbf{x}$ is far from the mean $\boldsymbol{\mu}$ in terms of the distribution’s variability.

Univariate Normal Distribution

The univariate normal distribution is defined by two parameters: the mean $\mu$ and the variance $\sigma^2$ . Its probability density function (pdf) is given by:

$f(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x - \mu)^2}{2\sigma^2}\right)$

Multivariate Normal Distribution

The multivariate normal distribution generalizes the univariate normal distribution to multiple dimensions. It is defined by a mean vector $\boldsymbol{\mu}$ and a covariance matrix $\boldsymbol{\Sigma}$ .

The density function of the multivariate normal distribution can be expressed in terms of the Mahalanobis distance. The Mahalanobis distance between a point $\mathbf{x}$ and the mean vector $\boldsymbol{\mu}$ with respect to the covariance matrix $\boldsymbol{\Sigma}$ is defined as:

$D_M(\mathbf{x}) = \sqrt{(\mathbf{x} - \boldsymbol{\mu})^\top \boldsymbol{\Sigma}^{-1} (\mathbf{x} - \boldsymbol{\mu})}$

Using the Mahalanobis distance, the density function of the $d$ -dimensional multivariate normal distribution is:

$f(\mathbf{x}) = \frac{1}{(2\pi)^{d/2} |\boldsymbol{\Sigma}|^{1/2}} \exp\left(-\frac{1}{2} D_M(\mathbf{x})^2 \right)$

In other words, the probability density function for a $d$ -dimensional multivariate normal distribution is:

$f(\mathbf{x}) = \frac{1}{(2\pi)^{d/2} |\boldsymbol{\Sigma}|^{1/2}} \exp\left(-\frac{1}{2} (\mathbf{x} - \boldsymbol{\mu})^\top \boldsymbol{\Sigma}^{-1} (\mathbf{x} - \boldsymbol{\mu}) \right) ,$

and we can see that the pdf of the multivariate normal is an extension of the univariate normal pdf, where $(x - \mu)^2 / (2\sigma^2)$ generalizes to $(\mathbf{x} - \boldsymbol{\mu})^\top \boldsymbol{\Sigma}^{-1} (\mathbf{x} - \boldsymbol{\mu}) / 2$ .

Key properties of the multivariate normal distribution include:

Symmetry and unimodality: like univariate normal, the distribution of multivariate normal is symmetric around $\boldsymbol{\mu}$
Marginal distributions: Any subset of the multivariate normal variables also has a multivariate normal distribution.
Conditional distributions: Conditional distributions of subsets of variables given others are also multivariate normal.

Discover more from Science Comics

Subscribe to get the latest posts sent to your email.