Earlier we covered Ordinary Least Squares regression. In this posting we will build upon this foundation and introduce an important extension to linear regression, regularization, that makes it applicable for ill-posed problems (e.g. number of predictors >> number of samples) and helps to prevent overfitting.

This is part of a series of blog posts showing how to do common statistical learning techniques with Python. We provide only a small amount of background on the concepts and techniques we cover, so if you’d like a more thorough explanation check outIntroduction to Statistical Learning or sign up for the free online course run by the book’s authors here.

Regularized Linear Regression

In a previous posting we introduced linear regression and polynomial regression. Polynomial regression fits a n-th order polynomial to our data using least squares. There’s a question that we didn’t answer: which order of the polynomial should we choose? Clearly, the higher the order of the polynomial, the higher the complexity of the model. This is true both computationally and conceptually because in both cases we now have a higher number of adaptable parameters. The higher the complexity of a model the more variance it can capture. Given that computation is cheap, should we always pick the most complex model? As we will show below, the answer to this question is no: we have to strike a balance between variance and (inductive) bias: our model needs to have sufficient complexity to model the relationship between the predictors and the response, but it must not fit the idiosyncrasies of our training data, idiosyncrasies which will limit its ability to generalize to new, unseen cases.

This is best illustrated using a simple curve fitting example, which is adopted from C. Bishop’s Pattern Recognition and Machine Learning (2007). Let’s create a synthetic dataset by adding some random gaussian noise to a sinusoidal function.

Now let’s see how different polynomials can approximate this curve.

In the plot above we see that the polynomial of degree zero is just a constant approximation, the polynomial of degree one fits a straight line, the polynomial of degree three nicely approximates the ground truth, and finally, the polynomial of degree nine has nearly zero training error but does a poor job approximating the ground truth because it already fits the variance induced by the random gaussian noise that we added to our data.

If we plot the training and testing error as a function of the degree of the polynomial we can see what’s happening: the higher the degree of the polynomial (our proxy for model complexity), the lower the training error. The testing error decreases too, but it eventually reaches its minimum at a degree of three and then starts increasing at a degree of seven.

This phenomenon is called overfitting: the model is already so complex that it fits the idiosyncrasies of our training data, idiosyncrasies which limit the model’s ability to generalize (as measured by the testing error).

In the above example, the optimal choice for the degree of the polynomial approximation would be between three and six. However, there is an alternative to manually selecting the degree of the polynomial: we can add a constraint to our linear regression model that constrains the magnitude of the coefficients in the regression model. This constraint is called the regularization term and the technique is often called shrinkage in the statistical community because it shrinks the coefficients towards zero. In the context of polynomial regression, constraining the magnitude of the regression coefficients effectively is a smoothness assumption: by constraining the L2 norm of the regression coefficients we express our preference for smooth functions rather than wiggly functions.

A popular regularized linear regression model is Ridge Regression. This adds the L2 norm of the coefficients to the ordinary least squares objective:

J(β)=1n∑ni=0(yi–βTx′i)2+α∥β∥2

where β is the vector of coefficients including the intercept term and x′i is the vector of the predictors of the i-th data point including a constant predictor for the intercept. The L2 norm term is weighted by a regularization parameter alpha: if alpha=0 then you recover the Ordinary Least Squares regression model. The larger the alpha the higher the smoothness constraint.

Below you can see the approximation of a sklearn.linear_model.RidgeRegression estimator fitting a polynomial of degree nine for various values of alpha (left) and the corresponding coefficient loadings (right). The smaller the value of alpha the higher the magnitude of the coefficients, so the functions we can model can be more and more wiggly.

Regularization techniques

In the above example we used Ridge Regression, a regularized linear regression technique that puts an L2 norm penalty on the regression coefficients. Another popular regularization technique is the LASSO, a technique which puts an L1 norm penalty instead. The difference between the two is that the LASSO leads to sparse solutions, driving most coefficients to zero, whereas Ridge Regression leads to dense solutions, in which most coefficients are non-zero. The intuition behind the sparseness property of the L1 norm penalty can be seen in the plot below. The plot shows the value of the penalty in the coefficient space, here a space with two coefficients w0 and w1. The L2 penalty appears as a cone in this space whereas the L1 penalty is a diamond. The objective function of a regularized linear model is just the ordinary least squared solution plus the (weighted) penalty term (the point that minimizes the objective function is where those two error surfaces meet), so in the case of the L1 penalty this is usually at the spike of the diamond, a sparse solution because some coefficients are zero. For the L2 penalty, on the other hand, the optimal point generally has non-zero coefficients. Another popular regularization technique is the Elastic Net, the convex combination of the L2 norm and the L1 norm. It too leads to a sparse solution.

L2 and L1 regularization differ in how they cope with correlated predictors: L2 will divide the coefficient loading equally among them whereas L1 will place all the loading on one of them while shrinking the others towards zero. Elastic Net combines the advantages of both: it tends to either select a group of correlated predictors in which case it puts equal loading on all of them, or it completely shrinks the group.

Scikit-learn provides separate classes for LASSO and Elastic Net: sklearn.linear_model.Lasso andsklearn.linear_model.ElasticNet. In contrast to RidgeRegression, the solution for both LASSO and Elastic Net has to be computed numerically. The classes above use an optimization technique called coordinate descent. Alterntively, you can also use the class sklearn.linear_model.SGDRegressor which uses stochastic gradient descent instead and often is more efficient for large-scale, high-dimensional and sparse data.

Regularization Path Plots

Another handy diagnostic tool for regularized linear regression is the use of so-called regularization path plots. These show the coefficient loading (y-axis) against the regularization parameter alpha (x-axis). Each (non-zero) coefficient is represented by a line in this space. The example below is taken from the scikit-learn documentation. You can see that the smaller the alpha (i.e. the higher the –log(alpha), the higher the magnitude of the coefficients and the more predictors selected). You can also see that the Elastic Net tends to select more predictors, distributing the loading evenly among them, whereas L1 tends to select fewer predictors.

 

 

Regularization path plots can be efficiently created using coordinate descent optimization methods but they are harder to create with (stochastic) gradient descent optimzation methods. Scikit-learn provides a number of convenience functions to create those plots for coordinate descent based regularized linear regression models: sklearn.linear_model.lasso_path andsklearn.linear_model.enet_path.