We could go on formulating reconstruction mappings and solving the output mappings, but this would become increasingly difficult. Moreover, it is not at all clear that we would get a network which would have the kind of competition behaviour we sketched in the beginning of this section. Figure 3.3 shows an example where the simple linear reconstruction does not yield a sparse code. The weight vector is much closer to the input than the two other weight vectors, but according to a linear reconstruction it would not be used in the representation. This is due to the fact that linear reconstruction does not promote sparsity in any way. If we use linear reconstruction and find the optimal outputs using equation 2.1, we obtain a dense code, because the reconstruction error is minimised when all the neurons are allowed to participate in the representation of the input. We need constraints for the outputs which assure the sparsity of the representation.

**Figure 3.3:**
An example of a situation where the reconstruction error
minimisation does not yield a sparse code. All vectors are
supposed to be nearly parallel and 3-dimensional. The figure
shows a view from top (cf. a map of a small area on globe). The
reconstruction is optimal when neurons 1 and 3 represent the
input. However, the code would be more sparse if the neuron 2
represented the input.

It should be reminded that we are using reconstruction error minimisation only because it allows easy derivation for algorithms and assures that the representation generated by the network contains information about the input. Our ultimate goal is not to find an exact reconstruction mapping, but to find a mapping which can generate sparse codes. If we can find outputs that approximately minimise the reconstruction error in equation 2.1, we can use the result in equation 2.2 to derive the learning rule and even an approximate minimisation will ensure that the outputs contain information about the inputs.

We shall now leave the reconstruction mapping for a moment and
consider a way to formulate the constraints for the outputs. We have
already found out that the winning strengths in
equation 3.5 are able to measure the competition between
two neurons. If we have more than two neurons we can still measure
the competition between each pair of neurons and try to combine these
competitions to a final winning strength. By combining equations
3.4 and 3.5 we can define a winning ratio
between each pair of neurons *i* and *j*:

where and . If there were only two neurons in the network we could write and we would get the same solution as in equation 3.5. When we have more than two neurons we need a way to combine the winning ratios in order to obtain the winning strengths .

One can think that the winning ratios are fuzzy truth
values for propositions ``Neuron *i* is a winner when it
competes with neuron *j*''. This suggests the use of some kind of
fuzzy and-operation to combine the winning ratios: neuron *i* is a
winner if it wins the competition with neuron 1 and 2 and and
*n*. A simple choice for and-operation is the product of truth
values. We could simply calculate the product of over index
*j* to obtain the winning strengths for neuron *i*.

However, there would be a small problem. Suppose that neuron 1 has
lost the competition with neuron 2, that is, . It follows
that . This is reasonable, since the neuron cannot be a
winner if it has totally lost in competition with another neuron. The
problem is that for other neurons than 2 may differ from one.
This means that other neurons are competing with a neuron that has
already lost. Since the lost neuron does not convey any information
it would be reasonable that it would not affect the outputs of other
neurons. This can be fixed by weighing the winning ratios in
the product by the winning strengths. We can now examine the
solutions to the *preliminary winning strengths* in the
following equation:

We have named the values preliminary winning strengths, since they do not necessarily satisfy the property , but we can use to compute values which do satisfy it, as will be shown later on. The weights are put to the exponent, because we are dealing with a product of numbers. We have also introduced a parameter which can be used to control the amount of competition.

The preliminary winning strengths appear in both sides of
equation 3.7 and it is impossible to find a solution in a
closed form. However, this time we can prune most of the neurons from
the computation of . If for some neuron *i* the winning ratio
with another neuron *j* is zero, that is, , then we can
omit the neuron *i* from the computations, since will anyway be
zero and it does not affect the of other neurons. This allows
us to find a *set of winners*: the set of neurons *i*, for which
the winning ratio for all *j*. Only these neurons will
have positive preliminary winning strengths. For all the other
neurons the preliminary winning strength *m* = 0.

Thu May 9 14:06:29 DST 1996