3. ${\rm IVGA_{VQ}}$: algorithm for IVGA with VQs as data models

Any IVGA algorithm consists of two parts, (1) grouping variables, and (2) constructing a separate model for each variable group. An independent variable grouping is obtained by comparing models with different groupings using a suitable cost function. In principle any model can be used, if the necessary cost function is derived for the model family.

As a cost function one can use negative log-likelihood of the data given the model, namely $-\ln p({\boldsymbol x}\vert H)$. The total model cost $L_{\rm tot}$ needed for comparing variable groupings is the sum of costs of individual variable groups $L_{\rm tot}=\sum_g L_g =\sum_g -\ln p({\boldsymbol x}_g\vert H_g)$, where g is the index of a group of variables, and x_g and H_g are the data and the model related to that variable group, respectively.

In all cases where the IVGA approach is used, the same problem arises: it is computationally infeasible to try all possible different groupings of D variables into G distinct groups, where G varies from 1 to D. Thus, especially when D is large, some heuristic optimization strategy has to be utilized.

In our experiments, all the variables were initially assigned to groups of their own and then moved one by one from group to another if the movement reduced the value of the cost $L_{\rm tot}$. In addition, every now and then (1) IVGA was recursively run for one group or the union of two groups (the depth of recursion was limited to one) or (2) a merge of two groups was considered.