1. Introduction

The goal of unsupervised learning is to extract an efficient representation of the statistical structure implicit in the observations. A good model is both accurate and simple in terms of model complexity, i.e., it forms a compact representation of the input data. Sparse coding, independent components, etc., can all be justified from the point of view of constructing compact representations. Learning amounts to searching in a model space by optimization of some cost function that measures both the accuracy of representation and--ideally at least--the model complexity.

In problems with a large number of diverse observations there are often groups of variables which have strong mutual dependences within the group but which can be considered practically independent of the variables in other groups. It can be expected that the larger the problem domain, the more independent groups there are. Estimating a model for each group separately produces a more compact representation than applying the model to the whole set of variables. Compact representations are computationally beneficial and, moreover, offer better generalization.