We shall now return back to the reconstruction mapping. We derived
the winning ratios by starting with two neurons and the
reconstruction mapping in equation 3.3. The optimal
outputs were then defined by equation 2.1. The relation
between the winning strength and the output *y* was defined in
equation 3.5 to be . The reconstruction
mapping was thus

There are no guarantees, however, that we could use the defined in equation 3.7 in place of . There is no reason why the reconstruction defined in equation 3.8 would be even close to the optimal. It turns out, however, that we are already quite close and can get even closer by making a few corrections to . The corrections are such that they approximately preserve the relations between the winning strengths, but correct their overall magnitudes so that we get an agreement with the reconstruction equation 3.8.

Again it should be emphasised that we want to get an agreement with the reconstruction mapping only in order to be able to derive the learning rule and to make sure that the outputs contain information about the inputs. For these purposes even an approximation will suffice.

We would now like to find a mapping from to . It should
be such that equation 3.8 yields a fairly good
reconstruction. We shall make two corrections to the preliminary
winning strengths . We obtain the first one by considering *n*
neurons, which have nearly parallel weight vectors and which are
equally far from the input , that is, and
for all *i*, *j*. Then it follows from the definition of
the winning ratios in equation 3.6 that . By symmetry, the solution to equation 3.7 has to be
. From these initial conditions we get

On the other hand, we know that for nearly parallel weight vectors the
sum of should be near to one in order to minimise the
reconstruction error in equation 2.1 if we are using the
reconstruction mapping in equation 3.8. Thus the optimal
solution is . Solving 1/*n* from
equation 3.9 gives the first correction to .

After the first correction the winning strengths are . The correction function defined in equation 3.10 is monotonically increasing and it will therefore leave the order of the outputs unchanged although the relations may change.

We get the second correction when we notice that if the were optimal in terms of equation 3.8, then in the following equation

should equal . If it means that should
be multiplied by to make the equation 3.8 hold.
However, for those neurons *j* that most contribute to the
correlation is biggest, which means that for them . We shall therefore make the correction by
to the neuron *i* only.

The equation 3.12 will now give the final corrected winning strengths . The last corrections to the magnitudes of the winning strengths are not globally uniform, because the ratios may vary in different directions. The corrections are locally uniform, however. For neurons with similar weight vectors the ratios are also similar, and therefore the last correction preserves the local relations between winning strengths.

If equations 3.11 and 3.12 were iterated, the solution would approach the optimal solution of equation 3.8. We don't want to do this, however, since the simple linear reconstruction mapping does not promote sparsity in any way. We have obtained a sparse coding by using equation 3.7 and we don't want to lose it. Even a single application of equation 3.12 suffices to bring the outputs to a satisfactory agreement with equation 3.8.

Thu May 9 14:06:29 DST 1996