Example where point estimates fail

The following example illustrates what can go wrong with point estimates. Three dimensional data vectors ${\mathbf{x}}(t)$ are modelled with the linear factor analysis model ${\mathbf{x}}(t)={\mathbf{a}}s(t)+{\mathbf{n}}(t)$, using a scalar source signal $s(t)$ and a Gaussian noise vector ${\mathbf{n}}(t)$ with zero mean and parameterised variance $p(n_k) = \mathcal N(0,\sigma_k^2)$. Here ${\mathbf{a}}$ is a three-dimensional weight vector.

The weight vector ${\mathbf{a}}$ might get a value ${\mathbf{a}} = \left[ 1 \phantom{o} 0 \phantom{o} 0 \right]^T$, while the source can just copy the values of the first dimension of ${\mathbf{x}}(t)$, that is, $s(t) = x_1(t)$. When the reconstruction error or the noise term is evaluated: ${\mathbf{n}}(t) = {\mathbf{x}}(t)- {\mathbf{a}}s(t) = \left[ 0 \phantom{o} x_2(t) \phantom{o} x_3(t) \right]^T$, one can see that problems will arise with the first variance parameter $\sigma_1^2$. The likelihood goes to infinity as $\sigma_1^2$ goes to zero. The same applies to the posterior density, since it is basically just the likelihood multiplied by a finite factor.

The found model is completely useless and still, it is rated as infinitely good using point estimates. These problems are typical for models with estimates of the noise level or products. They can be sometimes avoided by fixing the noise level or using certain normalisations Attias01RE. When the noise model is nonstationary (see Section 5.1), the problem becomes even worse, since the infinite likelihood appears if the any of the variances goes to zero.