7.5: Capacity

Last updated
Save as PDF

Page ID: 50199

Paul Penfield, Jr.
Massachusetts Institute of Technology via MIT OpenCourseWare

\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \) \( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)\(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\) \(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\)\(\newcommand{\AA}{\unicode[.8,0]{x212B}}\)

In Chapter 6 of these notes, the channel capacity was defined. This concept can be generalized to other processes.

Call \(W\) the maximum rate at which the input state of the process can be detected at the output. Then the rate at which information flows through the process can be as large as \(WM\). However, this product depends on the input probability distribution \(p(A_i)\) and hence is not a property of the process itself, but on how it is used. A better definition of process capacity is found by looking at how \(M\) can vary with different input probability distributions. Select the largest mutual information for any input probability distribution, and call that \(M_{max}\). Then the process capacity \(C\) is defined as

\(C = WM_{max} \tag{7.32}\)

It is easy to see that \(M_{max}\) cannot be arbitrarily large, since \(M ≤ I\) and \(I ≤ \log_2 n\) where \(n\) is the number of distinct input states.

In the example of symmetric binary channels, it is not difficult to show that the probability distribution that maximizes \(M\) is the one with equal probability for each of the two input states.