15.3: How fminsearch Works

Last updated
Save as PDF

Page ID: 84554

\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \) \( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)\(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\) \(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\)\(\newcommand{\AA}{\unicode[.8,0]{x212B}}\)

According to the MATLAB documentation, fminsearch uses the Nelder-Mead simplex algorithm. You can read about it at https://greenteapress.com/matlab/nelder, but you might find it overwhelming.

To give you a sense of how it works, I will present a simpler algorithm, the golden-section search. Suppose we’re trying to find the minimum of a function of a single variable, \(f(x)\).

As a starting place, assume that we have evaluated the function at three places, \(x_1\), \(x_2\), and \(x_3\), and found that \(x_2\) yields the lowest value. Figure 15.4 shows this initial state.

15.4.jpg — Figure 15.4: Initial state of a golden-section search

We will assume that \(f(x)\) is continuous and unimodal in this range, which means that there is exactly one minimum between \(x_1\) and \(x_3\).

The next step is to choose a fourth point, \(x_4\), and evaluate \(f(x_4)\). There are two possible outcomes, depending on whether \(f(x_4)\) is greater than \(f(x_2)\) or not. Figure 15.5 shows the two possible states.

15.5.jpg — Figure 15.5: Possible states of a golden-section search after evaluating \(f(x_4)\)

If \(f(x_4)\) is less than \(f(x_2)\) (shown on the left), the minimum must be between \(x_2\) and \(x_3\), so we would discard \(x_1\) and proceed with the new triple \((x_2, x_4, x_3)\).

If \(f(x_4)\) is greater than \(f(x_2)\) (shown on the right), the local minimum must be between \(x_1\) and \(x_4\), so we would discard \(x_3\) and proceed with the new triple \((x_1, x_2, x_4)\).

Either way, the range gets smaller and our estimate of the optimal value of \(x\) gets better.

This method works for almost any value of \(x_4\), but some choices are better than others. You might be tempted to bisect the interval between \(x_2\) and \(x_3\), but that turns out not to be the best choice. You can read about a better option at https://greenteapress.com/matlab/golden.