Regularized Linear Regression via Robust Optimization Lens

In preparation with Vineet Goyal and Garud Iyengar.

In this paper, we build upon the work done by Bertsimas and Copenhaver (2014) where they seek to obtain a robust optimization based explanation for why regularized linear regression methods perform well in the face of noise, even when these methods do not produce reliably sparse solutions. We explore this connection to more general loss functions used widely in Statistics, Machine Learning and Econometrics literature. In particular, we consider Huber loss, ϵ-insensitive loss and hinge loss functions which are some of the most used methods in literature. We derive tight regularized regression bounds for the corresponding robust problems with convex, positive homogeneous loss functions and Fenchel convex loss functions on Frobenius norm bounded uncertainty sets. And based on the regularized regression bounds, we propose a principled way to choose the regularization parameter λ to balance bias-variance trade-off for the regularized linear regression problem. Moreover, we construct a feature-wise uncoupled uncertainty set in the case of SLOPE and formulate a robust regression problem equivalent to the SLOPE regularization model