next up previous contents
Next: About this document Up: Derivations Previous: Derivation of the equation

Derivation of the equation 3.4

Next we prove that equation 3.4:


where tex2html_wrap_inline1619 and tex2html_wrap_inline1621 , is the solution to the system of the following equations:









We assume here that the weight vectors tex2html_wrap_inline1603 are column vectors with the same dimensionality as tex2html_wrap_inline1467 .

We know that the equations do not have a unique solution if the weight vectors are identical. In that case the correlation between the weight vectors would equal one. We assume that this degenerate case can be omitted, and we always have tex2html_wrap_inline2375 .

The derivation is very similar to the previous one. At first we combine equations A.10, A.12 and A.13, which yields


We can further manipulate the norm appearing on the right hand side of the equation. We denote the norm by tex2html_wrap_inline2377 .


Using the notations tex2html_wrap_inline1619 , tex2html_wrap_inline2381 and tex2html_wrap_inline1621 , and substituting tex2html_wrap_inline2385 we can write



Let us denote the square matrix tex2html_wrap_inline2387 by S. We can now write


Note that because S is positive definite (we assumed tex2html_wrap_inline2375 ), tex2html_wrap_inline2377 is a parabolic function.

We first find the solution without the constraints A.14 and A.15. If tex2html_wrap_inline1455 is the solution to the equation, then the gradient of tex2html_wrap_inline2377 at point tex2html_wrap_inline1455 has to be a zero vector tex2html_wrap_inline2403 . The gradient of N is


and therefore the solution tex2html_wrap_inline1455 has to satisfy


The computation of tex2html_wrap_inline2409 yields


and thus the solution without the constraints is


Next we take into consideration the constraints A.14 and A.15. If either of the tex2html_wrap_inline1629 is less than or equal to zero, then tex2html_wrap_inline1605 = 0 and the solution to the other output reduces to the case of one neuron, which has already been solved. We can thus assume that both tex2html_wrap_inline1669 . The constraints bound a rectangular area of allowed outputs. If the unconstrained solution given by equation A.16 lies in that area, it is also the solution to the constrained minimisation. Those cases, where the unconstrained solution does not lie in the allowed area, are left to examine. Due to the symmetry, the solutions to tex2html_wrap_inline2417 and tex2html_wrap_inline1627 are identical except for the interchanged indices. We can therefore consider only the solution to tex2html_wrap_inline2417 . Let us see what happens if tex2html_wrap_inline2423 . It is evident that the solutions to both of the outputs tex2html_wrap_inline1605 in equation A.16 exceed the boundaries:


Because tex2html_wrap_inline2377 is parabolic, the best constrained solution lies in the corner of the allowed area: tex2html_wrap_inline2429 and tex2html_wrap_inline2431 .

Now only those cases are left where tex2html_wrap_inline2433 , and c > 0. Let us consider what happens if equation A.16 gives a solution tex2html_wrap_inline2437 . We now that tex2html_wrap_inline2439 , and therefore tex2html_wrap_inline2441 . We see that if the unconstrained solution to one of the outputs is under the bound, then the other is over, and vice versa. Again the constrained solution is found in the corner of the allowed area. This time one of the outputs is zero and the other is tex2html_wrap_inline1629 . The solution to the case tex2html_wrap_inline2445 can thus be formulated


Collecting together all the different cases yields the equation 3.4. This concludes the proof.


next up previous contents
Next: About this document Up: Derivations Previous: Derivation of the equation

Harri Lappalainen
Thu May 9 14:06:29 DST 1996