Chapter 10: Fitting, Regressions, and Confidence Bands

Method of Least Squares

In the method of least squares we will be working with two variables x, which will be considered the independent or input variable and y, the dependent or response variable. We will assume that the total uncertainty in y will be much greater than x which in most experimental conditions is a fairly reasonable assumption and we will start by analyzing the regression curve of y on x as a linear regression curve

\begin{equation}
y = a +bx
\end{equation}

where a and b are fitting constants. Then the error e_i in predicting a y value will be given by

\begin{equation}
e_i = y_i - y(x_i)
\end{equation}

We will thereby select a and b such that \(\sum_{i=1}^n e_i\) is as close to 0 as possible to minimize the error.

So the principle of least squares will give us

\begin{equation}
\sum_{i=1}^n e_i = \sum_{i=1}^n [y_{i} - y(x_{i})]^{2}
\end{equation}

Before we can proceed with this process it will be extremely convenient here to define some critical values specifically

\begin{equation}
S_{xx} = \sum_{i=1}^n (x_i-\overline{x})^2
\end{equation}

and similarly

\begin{equation}
S_{yy} = \sum_{i=1}^n (y_i-\overline{y})^2
\end{equation}

and

\begin{equation}
S_{xy} = \sum_{i=1}^n (x_i-\overline{x})(y_i-\overline{y})
\end{equation}

Then we can calculate the error sum of squares which we seek to minimize as

\begin{equation}
SSE = S_{yy} - \frac{S_{xy}^2}{S_{xx}}
\end{equation}

This calculation will give us fitting parameters but we can also obtain estimates of the confidence intervals of those fitting parameters as well.

Let’s look at some plots of raw data and then we can perform some linear fits

Here we can see some example of raw data and we can now proceed to attempt to fit the data based on functions that we believe will best fit the values. We set the function and then proceed to allow the program to run through the above calculations to determine the fitting parameters in the linear function

We can see the fits here and we can also plot the 95% confidence interval for these fits as well as seen here

Non-Linear Fitting:

We can extend these tools to also perform non-linear fits that can range from exponential expressions, logarithmic expressions, polynomials, and beyond.

Here we can see some nonlinear fits of the previous data set

Non-Linear Fitting:

We can extend these tools to also perform non-linear fits that can range from exponential expressions, logarithmic expressions, polynomials, and beyond.

Here we can see some nonlinear fits of the previous data set

Correlation

So far we have assumed that the total uncertainty in x was extremely small and close to 0, while this holds for my real life scenarios there are many many many scenarios where this is not the case in which not true.

In which case the purpose of plotting x vs y in the most general sense is to observe some type of correlation between the variables. One can quantitatively calculate this via the sample correlation coefficient r which is defined as

\begin{equation}
r= \frac{S_{xy}}{\sqrt{S_{xx} \cdot S_{yy}}}
\end{equation}

here if r = 1 we have perfect positive correlation if r = −1 we have perfect negative correlation, and if r = 0 we have no correlation, i.e. there is no linear association. Additionally the limits of r is between −1 and 1.

We can also calculate our r² value which varies from 0-1 and is defined as

\begin{equation}
r^2 = \frac{S_{xy}^2}{S_{xx}S_{yy}}
\end{equation}

We can see these values tabulated here for our previous fits to show correlations, both r and r² values.

Looking at our previous fits we can run regression analysis to investigate the correlations as well.