Bayesian Probability Theory

The definition of probabities using frequencies will run into problems in some cases. What is the probability of six when a player throws a dice? One out of six. But what if the dice was biased? That is a good hypothesis and it should be tested. After trying a hundred throws the player is somewhat confident that the dice is biased. What is the probability of another six? How does it change exactly?

Another example is that a mathematician has a hypothesis, which he thinks is probably true. After writing the first half of the proof and sketching the other half he is somewhat more certain that the proof exists and the hypothesis is true. It is clear that in this case the probability is a subjective degree of belief. The frequency of a certain mathematical hypothesis to be true or not is quite absurd as a concept in this example.

In contrast to the traditional definition of probability using relative frequencies, the Bayesian probability theory interprets probability as a degree of belief. This offers a way to put all the hypotheses to prior beliefs $\mathcal{H}$. The data or the observations Xare in the player example the dice throws and the parameters $\boldsymbol{\theta}$ are the studied properties of the dice.

Cox has shown [11] that from very general requirements of consistency and compatibility with common sense, the basic laws concerning beliefs are equivalent to

$$p(A,B\mid C)=p(A \mid C)p(B \mid A,C)$$ (3.1)

$$p(A\mid B) + p(\bar{A} \mid B) = 1,$$ (3.2)

from which the two basic rules of Bayesian inference, Bayes rule and the marginalisation principle can be derived.