Independent Component Analysis

The mixing model of independent component analysis (ICA) is similar to that of the FA, but in the basic case without the noise term. The data has been generated from components s(t) through a square mixing matrix A by

 

x(t) = As(t) . (2.5)

The distribution of the components are assumed to be non-Gaussian contrary to FA. Figure  shows an example with super-Gaussian components.

The number of components is typically the same as the number of observation and the observations can be thus reconstructed perfectly. Such an A is searched for that the components s(t)=A^-1x(t) would be 'as independent as possible'. True independence is defined as

$$p(s_1,s_2,\dots,s_n) = p_1(s_1)p_2(s_2)\dots p_n(s_n),$$ (2.6)

which means that the density is factorised to a product of the marginal densities of the components. In practice, the independence can be maximised e.g. by maximising non-Gaussianity of the components or minimising mutual information [31].

ICA has many forms [31]. It can be approached from different starting points. In some extensions the number of independent components can exceed the number of dimensions of the observations making the basis overcomplete [46,31]. The noise term can be taken into the model. ICA can be viewed as a generative model when the one dimensional distributions for the components are modelled with for example mixtures-of-Gaussians [2,10,49].

Experiments have shown that some components found by ICA tend to be active, i.e. nonzero simultaneously in most natural data. Taking this into account in the model has led to topographic ICA and independent subspace analysis [30,29,31]. In topographic ICA, the components are organised in a grid. The model states that the energies s_i(t)² and s_j(t)² of components iand j are positively correlated, if the components are close to each other in the grid. In independent subspaces, the components are grouped to feature subspaces such that inside the group the probability distribution is spherically symmetric and the energies are correlated. The groups are considered mutually independent.