Mixtures of Linear Models

The first approach to extend factor analysis to nonlinear manifolds is to use a mixture model

 

x(t) = A_ks(t) + a_k + n(t), (2.7)

where k is a discrete random variable that selects the kernel or the mixing matrix and the corresponding bias. The bias term a_kis needed here even if the data is centered i.e. it has zero mean. The mixture model can be seen as trying to cover the data points precisely with a distribution containing several Gaussians.

Bishop et al. [7] used this kind of a model with Bayesian inference to image modelling tasks. There was considerable enhancement in the performance when compared to linear methods.

A mixture model is often inadequate with high dimensional data like images. To represent shapes and colours of different object in an image, one would need a componentwise representation of the parameters of the objects. But to represent the multiple objects, one would need several representations active at one time, which is not allowed.

It is possible to restrict the model in () by using only diagonal mixing matrices A_k, leaving out the whole term A_ks(t) or leaving out the noise term n(t). Restrictions simplify the model and thus one can use a greater number of kernels with the same total complexity. Restricted models come close to vector quantisation [1]. Mixture models have also been extended to mixture-of-ICA models [36,45].