12.1: Functions for Model Building and Regression

Regression and Model Building

Python offers several powerful libraries for regression and model building, each with its own set of functions and methodologies. Here's a breakdown of the most commonly used ones:

Regression

NumPy

While not a dedicated regression library, NumPy is fundamental for numerical operations in Python and is often used for foundational regression calculations or when implementing algorithms from scratch.

Relevant Functions:

• numpy.polynomial.Polynomial.fit(x, y, deg): (Replaces numpy.polyfit) Fits a polynomial of degree deg to points (x, y). Returns the coefficients of the polynomial.
• numpy.polyval(p, x): Evaluates a polynomial p at specific x values.
• numpy.linalg.lstsq(a, b): Solves the equation ax = b by computing a vector x that minimizes the Euclidean 2-norm ||b - ax||^2. This is the core of ordinary least squares.
• numpy.corrcoef(x, y): Returns the Pearson product-moment correlation coefficients.
• numpy.mean(), numpy.std(): For calculating means and standard deviations, useful in manual calculations of regression coefficients.

SciPy

SciPy builds on NumPy and provides more advanced scientific computing capabilities, including optimization, signal processing, and statistics.

Relevant Functions:

• scipy.stats.linregress(x, y): Calculates a linear least-squares regression for two sets of measurements. Returns slope, intercept, R-value, p-value, and standard error. It's great for simple linear regression.
• scipy.optimize.curve_fit(f, xdata, ydata): Uses non-linear least squares to fit a function f to data. Useful for more complex, non-linear regression models where you define the function form.

Model Building

Scikit-learn (sklearn)

Scikit-learn is the go-to library for model building and machine learning in Python, providing a vast array of algorithms for supervised and unsupervised learning, including many regression models.

Commonly Used Regression Models/Functions:

• sklearn.linear_model.LinearRegression:
  • LinearRegression(): Initializes a linear regression model.
  • .fit(X, y): Fits the linear model to the training data.
  • .predict(X_new): Predicts target values for new data.
  • .coef_: Returns the estimated coefficients for the features.
  • .intercept_: Returns the independent term in the linear model.
  • .score(X, y): Returns the coefficient of determination (R2) of the prediction.
• sklearn.linear_model.Ridge:
  • Ridge(alpha=1.0): Initializes a Ridge regression model (L2 regularization). alpha controls the strength of regularization.
  • .fit(X, y): Fits the model.
  • .predict(X_new): Predicts.
  • .coef_, .intercept_, .score(X, y): Similar to LinearRegression.
• sklearn.linear_model.Lasso:
  • Lasso(alpha=1.0): Initializes a Lasso regression model (L1 regularization). alpha controls the strength of regularization.
  • .fit(X, y): Fits the model.
  • .predict(X_new): Predicts.
  • .coef_, .intercept_, .score(X, y): Similar to LinearRegression.
• sklearn.linear_model.ElasticNet:
  • ElasticNet(alpha=1.0, l1_ratio=0.5): Initializes an Elastic Net regression model (combination of L1 and L2 regularization). l1_ratio controls the mix between L1 and L2.
  • .fit(X, y): Fits the model.
  • .predict(X_new): Predicts.
• sklearn.linear_model.BayesianRidge:
  • BayesianRidge(): Initializes a Bayesian Ridge regression model.
• sklearn.linear_model.SGDRegressor:
  • SGDRegressor(): Implements linear regression with stochastic gradient descent.
• sklearn.ensemble.RandomForestRegressor:
  • RandomForestRegressor(n_estimators=100): Initializes a Random Forest Regressor.
• sklearn.ensemble.GradientBoostingRegressor:
  • GradientBoostingRegressor(): Initializes a Gradient Boosting Regressor.
• sklearn.tree.DecisionTreeRegressor:
  • DecisionTreeRegressor(): Initializes a Decision Tree Regressor.
• sklearn.svm.SVR:
  • SVR(): Initializes a Support Vector Regressor.

Helper Functions for Model Building and Evaluation:

• sklearn.model_selection.train_test_split(X, y, test_size=0.2, random_state=42): Splits data into training and testing sets.
• sklearn.preprocessing.PolynomialFeatures(degree=2): Generates polynomial features for polynomial regression.
• sklearn.pipeline.make_pipeline(*steps): Creates a pipeline for chaining multiple processing steps and models.
• sklearn.metrics.mean_squared_error(y_true, y_pred): Calculates Mean Squared Error.
• sklearn.metrics.r2_score(y_true, y_pred): Calculates the R2 score.
• sklearn.metrics.mean_absolute_error(y_true, y_pred): Calculates Mean Absolute Error.
• sklearn.model_selection.GridSearchCV: For hyperparameter tuning.
• sklearn.model_selection.KFold, sklearn.model_selection.StratifiedKFold: For cross-validation.