9.6.2: Probability Formula

Last updated
Save as PDF

Page ID: 51695

Paul Penfield, Jr.
Massachusetts Institute of Technology via MIT OpenCourseWare

\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \) \( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)\(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\) \(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\)\(\newcommand{\AA}{\unicode[.8,0]{x212B}}\)

The probability distribution \(p(A_i)\) we want has been derived by others. It is a function of the dual variable \(\beta\):

\(p(A_i) = 2^{-\alpha} 2^{-\beta g(A_i)} \tag{9.12}\)

which implies

\(\log_2 \Big(\dfrac{1}{p(A_i)} \Big) = \alpha + \beta g(A_i) \tag{9.13}\)

where \(\alpha\) is a convenient abbreviation\(^2\) for this function of \(\beta\):

\(\alpha = \log_2 \Big( \displaystyle \sum_{i} 2^{-\beta g(A_i)} \Big) \tag{9.14}\)

Note that this formula for \(\alpha\) guarantees that the \(p(A_i)\) from Equation 9.12 add up to 1 as required by Equation 9.8.

If \(\beta\) is known, the function \(\alpha\) and the probabilities \(p(A_i)\) can be found and, if desired, the entropy \(S\) and the constraint variable \(G\). In fact, if \(S\) is needed, it can be calculated directly, without evaluating the \(p(A_i)\)—this is helpful if there are dozens or more probabilities to deal with. This short-cut is found by multiplying Equation 9.13 by \(p(A_i)\), and summing over \(i\). The left-hand side is \(S\) and the right-hand side simplifies because \(\alpha\) and \(\beta\) are independent of \(i\). The result is

\(S = \alpha + \beta G \tag{9.15}\)

where \(S\), \(\alpha\), and \(G\) are all functions of \(\beta\).

\(^2\)The function \(\alpha(\beta)\) is related to the partition function \(Z(\beta)\) of statistical physics: \(Z = 2^{\alpha}\) or \(\alpha = \log_2 Z\).