Example Where Point Estimates Fail

The following example illustrates what can go wrong with point estimates. Three dimensional data vectors x(t) are modelled with linear factor analysis

x(t)=As(t)+n(t), (3.15)

using a single source s(t). The weight matrix A might get a value

$$\mathbf{A} = \left[ \begin{array}{c} 1 \\ 0 \\ 0 \end{array} \right],$$ (3.16)

while the source can just copy the values of the first dimension of x(t)

s(t) = x₁(t). (3.17)

The noise model is Gaussian with zero mean and parameterised variance

$$p(n_k) = \operatorname{N}\left(n_k;0,\sigma_k^2\right) .$$ (3.18)

When the reconstruction error or the noise term is evaluated

$$\mathbf{n}(t) = \mathbf{x}(t)- \mathbf{A}s(t) = \left[ \begin{array}{c} 0 \\ x_2(t) \\ x_3(t) \end{array} \right],$$ (3.19)

one can see that problems will arise with the first variance parameter $\sigma_1^2$. The likelihood of the data

$$p(\mathbf{x}(t)) = p(\mathbf{n}(t)) = \prod_{k=1}^3 \left(2\pi \sigma_k^2\right)^{-1/2}\exp\left(-\frac{x_k(t)^2}{2\sigma_k^2}\right)$$ (3.20)

$$= \left[\frac{1}{\sqrt{2 \pi}}\prod_{k=2}^3 \left(2\pi \sigma_k^2... ...)^{-1/2}\exp\left(-\frac{x_k(t)^2}{2\sigma_k^2}\right)\right]\frac{1}{\sigma_1}$$ (3.21)

goes to infinity when the variance $\sigma_1^2$ goes to zero. Same applies to the posterior density, since it is basically just the likelihood multiplied by a finite factor.

The found model is completely useless and still, it is rated as infinitely good using point estimates. These problems are typical for models with estimates of the noise level or products. They can be sometimes avoided by fixing the noise level or using certain normalisations [2].