Basis of the algorithm

In this chapter we derive a new algorithm for finding sparse codes. Emphasis is laid on computational efficiency. The derivation is based on the reconstruction error minimisation scheme, which was discussed in section 2.1. When we are using this scheme, we have to define

1. the reconstruction mapping
2. the reconstruction error function and
3. suitable constraints for the outputs.

Equations 2.1 and 2.2 then provide the framework for derivation of the algorithm. We shall choose a linear reconstruction mapping and quadratic reconstruction error function, because they are mathematically tractable and are able to yield various algorithms depending on the constraints for the outputs. These algorithms have been shown to include versions which yield local, sparse, or dense codes.

We want to choose the constraints so that the resulting code is sparse. We shall formulate a competition mechanism, which is used to impose restrictions to the outputs. The competition mechanism fixes the ratios between the outputs of the neurons, but the overall magnitude is left arbitrary. The minimisation in equation 2.1 is then done over the set of outputs, which has the given ratios between outputs. In other words, the minimisation fixes the overall magnitude of the outputs.