Updating $q(s)$ for the Gaussian node followed by a nonlinearity

A Gaussian variable has its own terms in the cost function and it affects the cost function of its children. In case there is a nonlinearity attached to it, only the latter is changed. The cost function of the children can be written in the form

$${\cal C}_{\operatorname{ch}(s),p} = M \left< f(s) \right> + V [(\... ...ht>- \left< f(s) \right>_{\text{current}})^2 + \mathrm{Var}\left\{f(s)\right\}]$$ (64)

where $\left< f(s) \right>_{\text{current}}$ stands for the expectation using the current posterior estimate $q(s)$, and $M$ and $V$ are constants.

The posterior $q(s)=\mathcal N(s;\overline{s},\widetilde{s})$ is updated to minimise the cost function. For $\widetilde{s}$ we get a fixed point iteration for the update candidate:

$$v \widetilde{s}_{new} = \left[ \left< \exp v \right> + \frac{4 V ... ... \frac{M}{2V})\left< f(s) \right>}{\left(2\widetilde{s} + 1\right)^{2}} \right.$$

$$\left. - \frac{4 V \left(1- 4\overline{s}^2 + 4\widetilde{s}\right) \left< [f(s)]^2 \right>}{\left(4\widetilde{s} + 1\right)^{2}} \right]^{-1}$$ (65)

And for $\overline{s}$ we have an approximated Newton's iteration update candidate

$$\overline{s}_{new} = \overline{s} - \widetilde{s}_{new} \left[ \l... ...- \frac{\left< [f(s)]^2 \right>}{4\widetilde{s} + 1}\right) \right] %\nonumber$$ (66)

These candidates guarantee a direction, in which the cost function decreases locally. As long as the cost function is about to increase in value, the step size is halved. This guarantees the convergence to a stable point.