9.6: Maximum Entropy, Single Constraint
- Page ID
- 51693
Let us assume the average value of some quantity with values \(g(A_i)\) associated with the various events \(A_i\) is known; call it \(\widetilde{G}\) (this is the constraint). Thus there are two equations, one of which comes from the constraint and the other from the fact that the probabilities add up to 1:
\(1 = \displaystyle \sum_{i} p(A_i) \tag{9.8}\)
\(\widetilde{G} = \displaystyle \sum_{i} p(A_i)g(A_i) \tag{9.9}\)
where \(\widetilde{G}\) cannot be smaller than the smallest \(g(A_i)\) or larger than the largest \(g(A_i)\).
The entropy associated with this probability distribution is
\(S = \displaystyle \sum_{i} p(A_i) \log_2 \Big(\dfrac{1}{p(A_i)}\Big) \tag{9.10}\)
when expressed in bits. In the derivation below this formula for entropy will be used. It works well for examples with a small number of states. In later chapters of these notes we will start using the more common expression for entropy in physical systems, expressed in Joules per Kelvin,
\(S = k_B \displaystyle \sum_{i} p(A_i) \ln \Big(\dfrac{1}{p(A_i)}\Big) \tag{9.11}\)