COMPUTING THE RIEMANNIAN METRIC TENSOR

When applying natural gradients to approximate inference, the geometry is defined by the approximation $q(\boldsymbol{\theta}\vert \boldsymbol{\xi})$ and not the full model as usually. If the approximation $q(\boldsymbol{\theta}\vert \boldsymbol{\xi})$ is chosen such that disjoint groups of variables are independent, that is,

$$q(\boldsymbol{\theta}\vert \boldsymbol{\xi}) = \prod_i q_i(\boldsymbol{\theta}_i\vert \boldsymbol{\xi}_i),$$ (5)

the computation of the natural gradient is simplified as the Fisher information matrix becomes block-diagonal. The required matrix inversion can be performed very efficiently because

$$\mathrm{diag}(A_1,\dots,A_n)^{-1}=\mathrm{diag}(A_1^{-1},\dots,A_n^{-1}).$$ (6)

The dimensionality of the problem space is often so high that inverting the full matrix would not be feasible.