Marginalisation principle

While Bayes' rule specifies how the learning system should update its beliefs as new data arrives, the marginalisation principle provides for the derivation of probabilities of new propositions given existing probabilities. This is useful for prediction and inference.

Suppose the situation is the same as in the example with Bayes' rule, but now the learning system tries to compute the probability of making observation B before it has actually made the observation, that is, the learning system tries to predict the new observation.

Suppose $A_1, A_2, \ldots$ are exhaustive and mutually exclusive propositions, in other words, exactly one of A_i is true while the rest are false. As before, assume that A_i are possible explanations for B and the prior assumptions and experience C are such that both P(B | A_i C) and P(A_i | C) are determined. The marginalisation principle then states the following:

$$P(B \vert C) = \sum_i P(A_i \vert C) P(B \vert A_i C) \, .$$ (2)

The probability of B thus depends on the prior probabilities P(A_i | C) of the different explanations and the probability P(B | A_i C)which each explanation gives to B.

Notice also that P(B | C) appears in Bayes' rule, but the marginalisation principle shows that it can be computed from P(A_i | C) and P(B | A_i C) alone. Therefore P(A_i | C) and P(B | A_i C) suffice for computing the posterior probability P(A_i | BC):

$$P(A_i \vert BC) = \frac{P(A_i \vert C) P(B \vert A_i C)}{\sum_j P(A_j \vert C) P(B \vert A_j C)} \, .$$ (3)