Skip to main content
Engineering LibreTexts

8.3: Non-linear Optimization

  • Page ID
    22410
  • \( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \) \( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)\(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\) \(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\)\(\newcommand{\AA}{\unicode[.8,0]{x212B}}\)

    Various conditions and situations are not adequately described using linear systems. In this case, nonlinear optimization may be applied. Unlike linear optimization, the optimal operating condition does not exist at the boundaries.

    Quadratic Optimization

    \[f(x)=c-x^{T} b+\frac{1}{2} x^{T} A x\nonumber \]

    To optimize, it is necessary to find when the gradient of f is equal to zero.

    \[\nabla f(x)=0\nonumber \]

    \[\nabla f(x)=b-A x\nonumber \]

    \[x_{*}=A^{-1} b\nonumber \]

    It may be possible to solve the optimal quad x_*by a linear equation, approximated by a Taylor series.

    \[f\left(x_{*}\right)=f(x)+\left(x_{*}-x\right)^{\prime} \nabla f(x)+\frac{1}{2}\left(x_{*}-x\right)^{\prime} \nabla \nabla f(x)\left(x_{*}-x\right)+\ldots\nonumber \]

    Iterative Methods

    When direct methods cannot solve the equation (i.e. A is not symmetric positive definite), iterative methods are possible [1].

    By starting with an initial guess of quad x_i, an algorithm may lead to a quad x_{i+1}that better satisfies the equation. Through iteration, theoretically, quad x_\infty=x.

    Applications

    • Finance: Portfolio optimization
    • Businesses: Optimize inventory
    • Engineering: Rigid body dynamics
    • Biochemistry: Kinetic modeling [2]

    Example: Typical Nonlinear 3d Curves

    onlinear.jpg

    (Image from [1])

    As observed, the optimal condition does not necessarily exist at the boundary of the curve.

    Example: Quadratic Optimization

    \[f(x)=\vec{c}^{T} \vec{x}+\frac{1}{2} \vec{x}^{T} Q \vec{x}\nonumber \]

    where

    \[\vec{c}^{T}=\left(c_{1}, c_{2}, \ldots, c_{n}\right)\nonumber \]

    \[\vec{x}^{T}=\left(x_{1}, x_{2}, \ldots, x_{n}\right)\nonumber \]

    For a quadratic system, \(n=2\), thus, \(Q\) (the quadratic term constant) is defined as a symmetric matrix as follows.

    \[Q=\left[\begin{array}{ll}
    Q_{1} & Q_{3} \\
    Q_{3} & Q_{2}
    \end{array}\right]\nonumber \]

    Thus, multiplying out the \(f\),

    \[f(x)=\left(c_{1} x_{1}+c_{2} x_{2}\right)+\frac{1}{2}\left(Q_{1} x_{1}^{2}+2 Q_{3} x_{1} x_{2}+Q_{2} x_{2}^{2}\right)\nonumber \]

    References

    1. Lippert, Ross A. "Introduction to non-linear optimization." D.E. Shaw Research, February 25, 2008. http://www.mit.edu/~9.520/spring08/Classes/optlecture.pdf
    2. Mendes, Pedro and Kell, Douglas B. "Non-linear optimization of biochemical pathways: application to metabolic engineering and parameter estimation." Journal of Bioinformatics, Volume 14, 869-883. 1998.
    3. "Introduction to Non-linear optimization." Georgia Institute of Technology Systems Realization Laboratory. www.srl.gatech.edu/education/ME6103/NLP-intro.ppt

    This page titled 8.3: Non-linear Optimization is shared under a CC BY 3.0 license and was authored, remixed, and/or curated by Matthew Goh, Andrew King, Edwin Yik, & Edwin Yik via source content that was edited to the style and standards of the LibreTexts platform; a detailed edit history is available upon request.