Next: Finding the set of Up: Derivation of the new Previous: Combining the competitions

# Finding the outputs

We shall now return back to the reconstruction mapping. We derived the winning ratios by starting with two neurons and the reconstruction mapping in equation 3.3. The optimal outputs were then defined by equation 2.1. The relation between the winning strength and the output y was defined in equation 3.5 to be . The reconstruction mapping was thus

There are no guarantees, however, that we could use the defined in equation 3.7 in place of . There is no reason why the reconstruction defined in equation 3.8 would be even close to the optimal. It turns out, however, that we are already quite close and can get even closer by making a few corrections to . The corrections are such that they approximately preserve the relations between the winning strengths, but correct their overall magnitudes so that we get an agreement with the reconstruction equation 3.8.

Again it should be emphasised that we want to get an agreement with the reconstruction mapping only in order to be able to derive the learning rule and to make sure that the outputs contain information about the inputs. For these purposes even an approximation will suffice.

We would now like to find a mapping from to . It should be such that equation 3.8 yields a fairly good reconstruction. We shall make two corrections to the preliminary winning strengths . We obtain the first one by considering n neurons, which have nearly parallel weight vectors and which are equally far from the input , that is, and for all i, j. Then it follows from the definition of the winning ratios in equation 3.6 that . By symmetry, the solution to equation 3.7 has to be . From these initial conditions we get

On the other hand, we know that for nearly parallel weight vectors the sum of should be near to one in order to minimise the reconstruction error in equation 2.1 if we are using the reconstruction mapping in equation 3.8. Thus the optimal solution is . Solving 1/n from equation 3.9 gives the first correction to .

After the first correction the winning strengths are . The correction function defined in equation 3.10 is monotonically increasing and it will therefore leave the order of the outputs unchanged although the relations may change.

We get the second correction when we notice that if the were optimal in terms of equation 3.8, then in the following equation

should equal . If it means that should be multiplied by to make the equation 3.8 hold. However, for those neurons j that most contribute to the correlation is biggest, which means that for them . We shall therefore make the correction by to the neuron i only.

The equation 3.12 will now give the final corrected winning strengths . The last corrections to the magnitudes of the winning strengths are not globally uniform, because the ratios may vary in different directions. The corrections are locally uniform, however. For neurons with similar weight vectors the ratios are also similar, and therefore the last correction preserves the local relations between winning strengths.

If equations 3.11 and 3.12 were iterated, the solution would approach the optimal solution of equation 3.8. We don't want to do this, however, since the simple linear reconstruction mapping does not promote sparsity in any way. We have obtained a sparse coding by using equation 3.7 and we don't want to lose it. Even a single application of equation 3.12 suffices to bring the outputs to a satisfactory agreement with equation 3.8.

Next: Finding the set of Up: Derivation of the new Previous: Combining the competitions

Harri Lappalainen
Thu May 9 14:06:29 DST 1996