Variance modelling

In many models, variances are assumed to have constant values although this assumption is often unrealistic in practice. Joint modelling of means and variances is difficult in many learning approaches, because it can give rise to infinite probability densities. In Bayesian methods where sampling is employed, the difficulties with infinite probability densities are avoided, but these methods are not efficient enough for very large models. The Bayes Blocks allow to build hierarchical or dynamical models for the variance.

The Bayes Blocks framework was used by Valpola et al. (2004) to jointly model both variances and means in biomedical MEG data. The same approach can be used to translate any model for a mean to a model for a variance, so a large number of models in the literature could be explored as models for variance as well.

The left subfigure of Figure 4.5 shows how linear state-space model (see Section 3.1.6) is built using Bayes Blocks. It can be extended into a model for both means and variances as depicted graphically in the right subfigure of Figure 4.5. The variance sources $\mathbf{u}(t)$ characterise the innovation process of $\mathbf{s}(t)$, in effect telling how much the signal differs from the predicted one but not in which direction it is changing. Both regular sources $\mathbf{s}(t)$ and variance sources $\mathbf{u}(t)$ are modelled dynamically by using one-step recursive prediction model for them. The model equations are:

$$\mathbf{x}(t) = \mathbf{A}\mathbf{s}(t) + \mathbf{a} + \mathbf{n}_x(t)$$ (4.14)

$$\mathbf{s}(t) = \mathbf{B}\mathbf{s}(t-1) + \mathbf{b} + \mathbf{n}_s(t)$$ (4.15)

$$n_{si}(t) = \mathcal{N}\left(n_{si}(t);0,\exp\left[-u_i(t)\right]\right)$$ (4.16)

$$\mathbf{u}(t) = \mathbf{C}\mathbf{u}(t-1) + \mathbf{c} + \mathbf{n}_u(t),$$ (4.17)

where the variance of $n_{si}(t)$, the $i$th component of the noise vector $\mathbf{n}_s(t)$, is determined by the variance source $u_i(t)$.

Figure 4.5: Model structures represented using the blocks in Figure 4.4. Observed variables are shaded. Left: A linear Gaussian state-space model. Right: A dynamic model for the variances of the sources which also have a recurrent dynamic model.