Let . The expectation to be approximated is then

Let us first concider the latter term. The logarithm of the sum is a strictly convex function ofThe expectation is similar to equation 1 and equals to

Let us denote this by . At this point, the approximation equals to The terms inside the expectations are functions ofNext, let us concider a second order Taylor's series expansion about and . Notice that the first order terms and all second order crossterms disappear in the expectations and only the constant and the pure second order terms remain. This is because the variables are independent in the ensemble.

(2) |

(3) |

(4) |

Some care has to be taken with the approximation resulting from the
Taylor's series expansion because it utilises only local information
about the shape of the posterior pdf. For instance, if the mean
happens to be in a valley between two Gaussians, the
term in equation 4 will be negative. It then looks like the
Kullback-Leibler information can be decreased by increasing
. This only holds for small , however. At
some point after the distribution of *s*_{i}(*t*) has become broader than
the separation between the two Gaussians, the Kullback-Leibler
information starts to increase as increases.

In order to avoid the problem, only positive terms of equations 3 and 4 will be included. The final approximation is thus equation 2 plus the positive terms of equations 3 and 4.