14.6: Summary

Summary: Solving Systems of Linear Equations

This chapter delves into the fundamental concepts and practical methods for solving systems of linear equations, a cornerstone of mathematics, science, and engineering. A linear system can be generally represented as Ax=b, where A is the coefficient matrix, x is the vector of variables, and b is the constant vector. The nature of the solution depends critically on the relationship between the number of equations (m) and the number of variables (n).

1. Well-Determined Systems (m=n)

When the number of equations equals the number of variables, the system is considered well-determined. If the coefficient matrix A is non-singular (i.e., its determinant is non-zero, and it has full rank), there exists a unique solution. This solution can be found directly by matrix inversion (\(x = A^{−1} b\)) or, more robustly and efficiently in practice, using numerical solvers like numpy.linalg.solve in Python, which employ techniques like LU decomposition.

2. Underdetermined Systems (m<n)

In underdetermined systems, there are fewer equations than variables. Such systems typically have infinitely many solutions, provided they are consistent (i.e., a solution exists). The solution set forms an affine subspace. To describe all possible solutions, we combine two components:

• A Particular Solution (xp​): This is often the minimum-norm solution, meaning the solution vector with the smallest Euclidean length. It can be found using the pseudoinverse of A (\(A^+\)), where \(x_p​ = A^+ b\), or by using numpy.linalg.lstsq, which yields this minimum-norm solution for underdetermined cases.
• The Null Space ( \(x_h\)​ ): This is the set of all vectors that, when multiplied by A, result in the zero vector (\(Ax_h = 0\)). The null space provides the "homogeneous" part of the solution. A basis for the null space can be found using functions like scipy.linalg.null_space.

The general solution for an underdetermined system is the sum of a particular solution and any vector from the null space: \(x = x_p​ + x_h\)​.

3. Overdetermined Systems (m>n)

Overdetermined systems have more equations than variables. Due to potential inconsistencies or errors in real-world data, these systems generally do not have an exact solution that satisfies all equations simultaneously. Instead, the objective is to find the "best approximate" solution. This is achieved through the least squares method, which minimizes the sum of the squares of the differences between the observed values and the values predicted by the model (\(\min_x ​∥Ax − b∥^2\)). The numpy.linalg.lstsq function is the standard tool in Python for computing this least-squares solution, which is robustly found via Singular Value Decomposition (SVD).

4. General Solution Concept

The concept of a "general solution" is most relevant for underdetermined systems, where it encompasses all possible solutions by combining a specific solution with the vectors that lie in the null space of the coefficient matrix. For well-determined systems, the general solution is simply the unique solution itself. For overdetermined systems, since an exact solution is rare, the "solution" refers to the best approximate (least squares) fit.

In summary, understanding the relationship between the number of equations and variables is crucial for determining the nature of solutions to linear systems. Python's NumPy and SciPy libraries provides powerful and efficient tools like np.linalg.solve, np.linalg.pinv, and np.linalg.lstsq to handle all types of linear systems, whether they are well-determined, underdetermined, or overdetermined.