9.6.5: Examples

For the Berger’s Burgers example, suppose that you are told the average meal price is $2.50, and you want to estimate the probabilities \(p(B)\), \(p(C)\), \(p(F)\), and \(p(T)\). Here is what you know:

\(\begin{align*} 1 &= p(B) + p(C) + p(F) + p(T) \tag{9.24} \\ 0 &= $1.00p(B) + $2.00p(C) + $3.00p(F) + $8.00p(T) − $2.50 \tag{9.25} \\ S &= p(B) \log_2 \Big(\dfrac{1}{p(B)} \Big) + p(C) \log_2 \Big(\dfrac{1}{p(C)} \Big) + p(F) \log_2 \Big(\dfrac{1}{p(F)} \Big) + p(T) \log_2 \Big(\dfrac{1}{p(T)} \Big) \tag{9.26} \end{align*}\)

The entropy is the largest, subject to the constraints, if

where

\(\alpha = \log_2(2^{−\beta $1.00} + 2^{− \beta $2.00} + 2^{− \beta $3.00} + 2^{−\beta $8.00}) \tag{9.31}\)

and \(\beta\) is the value for which \(f(\beta)\) = 0 where

\(f(\beta) = $0.50 × 2^{−$0.50\beta} + $5.50 × 2^{−$5.50\beta} − $1.50 × 2^{$1.50\beta} − $0.50 × 2^{$0.50\beta} \tag{9.32}\)

A little trial and error (or use of a zero-finding program) gives \(\beta\) = 0.2586 bits/dollar, \(\alpha\) = 1.2371 bits, \(p(B)\) = 0.3546, \(p(C)\) = 0.2964, \(p(F)\) = 0.2478, \(p(T)\) = 0.1011, and \(S\) = 1.8835 bits. The entropy is smaller than the 2 bits which would be required to encode a single order of one of the four possible meals using a fixed-length code. This is because knowledge of the average price reduces our uncertainty somewhat. If more information is known about the orders then a probability distribution that incorporates that information would have even lower entropy.

For the magnetic dipole example, we carry the derivation out with the magnetic field \(H\) set at some unspecified value. The results all depend on \(H\) as well as \(E\).

\begin{align*}
1 &=p(U)+p(D) \tag{9.33} \\
\widetilde{E} &=e(U) p(U)+e(D) p(D) \\
&=m_{d} H[p(U)-p(D)] \tag{9.34} \\
S &=p(U) \log _{2}(\frac{1}{p(A)})+p(D) \log _{2}(\frac{1}{p(D)}) \tag{9.35}
\end{align*}

The entropy is the largest, for the energy \(\widetilde{E}\) and magnetic field \(H\), if

\begin{align*}
p(U) &= 2^{-\alpha} 2^{-\beta m_{d} H} \tag{9.36} \\
p(D) &= 2^{-\alpha} 2^{\beta m_{d} H} \tag{9.37}
\end{align*}

where

\(\alpha=\log _{2}\Big (2^{-\beta m_{d} H}+2^{\beta m_{d} H}\Big ) \tag{9.38}\)

and \(\beta\) is the value for which \(f(\beta)\) = 0 where

\(f(\beta)=(m_{d} H-\widetilde{E}) 2^{-\beta(m_{d} H-\widetilde{E})}-(m_{d} H+\widetilde{E}) 2^{\beta(m_{d} H+\widetilde{E})} \tag{9.39}\)

Note that this example with only one dipole, and therefore only two states, does not actually require the Principle of Maximum Entropy because there are two equations in two unknowns, \(p(U)\) and \(p(D)\) (you can solve Equation 9.39 for \(\beta\) using algebra). If there were two dipoles, there would be four states and algebra would not have been sufficient. If there were many more than four possible states, this procedure to calculate \(\beta\) would have been impractical or at least very difficult. We therefore ask, in Chapter 11 of these notes, what we can tell about the various quantities even if we cannot actually calculate numerical values for them using the summation over states.