Linear Models Pdf
Mckenzie Witting
In statistics, the term linear model refers to any model which assumes linearity in the system. The most common occurrence is in connection with regression models and the term is often taken as synonymous with linear regression model. However, the term is also used in time series analysis with a different meaning. In each case, the designation "linear" is used to identify a subclass of models for which substantial reduction in the complexity of the related statistical theory is possible.
Given that estimation is undertaken on the basis of a least squares analysis, estimates of the unknown parameters β j \displaystyle \beta _j are determined by minimising a sum of squares function
where again the quantities ε i \displaystyle \varepsilon _i are random variables representing innovations which are new random effects that appear at a certain time but also affect values of X \displaystyle X at later times. In this instance the use of the term "linear model" refers to the structure of the above relationship in representing X t \displaystyle X_t as a linear function of past values of the same time series and of current and past values of the innovations.[1] This particular aspect of the structure means that it is relatively simple to derive relations for the mean and covariance properties of the time series. Note that here the "linear" part of the term "linear model" is not referring to the coefficients ϕ i \displaystyle \phi _i and θ i \displaystyle \theta _i , as it would be in the case of a regression model, which looks structurally similar.
There are some other instances where "nonlinear model" is used to contrast with a linearly structured model, although the term "linear model" is not usually applied. One example of this is nonlinear dimensionality reduction.
Linear models describe a continuous response variable as a function of one or more predictor variables. They can help you understand and predict the behavior of complex systems or analyze experimental, financial, and biological data.
Linear regression is a statistical method used to create a linear model. The model describes the relationship between a dependent variable \(y\) (also called the response) as a function of one or more independent variables \(X_i\) (called the predictors). The general equation for a linear model is:
Simple linear regression is commonly done in MATLAB. For multiple and multivariate linear regression, see Statistics and Machine Learning Toolbox. It enables stepwise, robust, and multivariate regression to:
To create a linear model that fits curves and surfaces to your data, see Curve Fitting Toolbox. To create linear models of dynamic systems from measured input-output data, see System Identification Toolbox. To create a linear model for control system design from a nonlinear Simulink model, see Simulink Control Design.
See also:Statistics and Machine Learning Toolbox, Curve Fitting Toolbox, machine learning, linearization, data fitting, data analysis, mathematical modeling, time series regression, linear model videos, Machine Learning Models, MANOVA
The following are a set of methods intended for regression in whichthe target value is expected to be a linear combination of the features.In mathematical notation, if \(\haty\) is the predictedvalue.
LinearRegression fits a linear model with coefficients\(w = (w_1, ..., w_p)\) to minimize the residual sumof squares between the observed targets in the dataset, and thetargets predicted by the linear approximation. Mathematically itsolves a problem of the form:
The coefficient estimates for Ordinary Least Squares rely on theindependence of the features. When features are correlated and thecolumns of the design matrix \(X\) have an approximately lineardependence, the design matrix becomes close to singularand as a result, the least-squares estimate becomes highly sensitiveto random errors in the observed target, producing a largevariance. This situation of multicollinearity can arise, forexample, when data are collected without an experimental design.
It is possible to constrain all the coefficients to be non-negative, which maybe useful when they represent some physical or naturally non-negativequantities (e.g., frequency counts or prices of goods).LinearRegression accepts a boolean positiveparameter: when set to True Non-Negative Least Squares are then applied.
The complexity parameter \(\alpha \geq 0\) controls the amountof shrinkage: the larger the value of \(\alpha\), the greater the amountof shrinkage and thus the coefficients become more robust to collinearity.
Note that the class Ridge allows for the user to specify that thesolver be automatically chosen by setting solver="auto". When this optionis specified, Ridge will choose between the "lbfgs", "cholesky",and "sparse_cg" solvers. Ridge will begin checking the conditionsshown in the following table from top to bottom. If the condition is true,the corresponding solver is chosen.
It might seem questionable to use a (penalized) Least Squares loss to fit aclassification model instead of the more traditional logistic or hingelosses. However, in practice, all those models can lead to similarcross-validation scores in terms of accuracy or precision/recall, while thepenalized least squares loss used by the RidgeClassifier allows fora very different choice of the numerical solvers with distinct computationalperformance profiles.
RidgeCV and RidgeClassifierCV implement ridgeregression/classification with built-in cross-validation of the alpha parameter.They work in the same way as GridSearchCV exceptthat it defaults to efficient Leave-One-Out cross-validation.When using the default cross-validation, alpha cannot be 0 due to theformulation used to calculate Leave-One-Out error. See [RL2007] for details.
The Lasso is a linear model that estimates sparse coefficients.It is useful in some contexts due to its tendency to prefer solutionswith fewer non-zero coefficients, effectively reducing the number offeatures upon which the given solution is dependent. For this reason,Lasso and its variants are fundamental to the field of compressed sensing.Under certain conditions, it can recover the exact set of non-zerocoefficients (seeCompressive sensing: tomography reconstruction with L1 prior (Lasso)).
For high-dimensional datasets with many collinear features,LassoCV is most often preferable. However, LassoLarsCV hasthe advantage of exploring more relevant values of alpha parameter, andif the number of samples is very small compared to the number offeatures, it is often faster than LassoCV.
Alternatively, the estimator LassoLarsIC proposes to use theAkaike information criterion (AIC) and the Bayes Information criterion (BIC).It is a computationally cheaper alternative to find the optimal value of alphaas the regularization path is computed only once instead of k+1 timeswhen using k-fold cross-validation.
However, such criteria need a proper estimation of the degrees of freedom ofthe solution, are derived for large samples (asymptotic results) and assume thecorrect model is candidates under investigation. They also tend to break whenthe problem is badly conditioned (e.g. more features than samples).
The MultiTaskLasso is a linear model that estimates sparsecoefficients for multiple regression problems jointly: y is a 2D array,of shape (n_samples, n_tasks). The constraint is that the selectedfeatures are the same for all the regression problems, also called tasks.
The following figure compares the location of the non-zero entries in thecoefficient matrix W obtained with a simple Lasso or a MultiTaskLasso.The Lasso estimates yield scattered non-zeros while the non-zeros ofthe MultiTaskLasso are full columns.
The MultiTaskElasticNet is an elastic-net model that estimates sparsecoefficients for multiple regression problems jointly: Y is a 2D arrayof shape (n_samples, n_tasks). The constraint is that the selectedfeatures are the same for all the regression problems, also called tasks.
Least-angle regression (LARS) is a regression algorithm forhigh-dimensional data, developed by Bradley Efron, Trevor Hastie, IainJohnstone and Robert Tibshirani. LARS is similar to forward stepwiseregression. At each step, it finds the feature most correlated with thetarget. When there are multiple features having equal correlation, insteadof continuing along the same feature, it proceeds in a direction equiangularbetween the features.
If two features are almost equally correlated with the target,then their coefficients should increase at approximately the samerate. The algorithm thus behaves as intuition would expect, andalso is more stable.
Because LARS is based upon an iterative refitting of theresiduals, it would appear to be especially sensitive to theeffects of noise. This problem is discussed in detail by Weisbergin the discussion section of the Efron et al. (2004) Annals ofStatistics article.
LassoLars is a lasso model implemented using the LARSalgorithm, and unlike the implementation based on coordinate descent,this yields the exact solution, which is piecewise linear as afunction of the norm of its coefficients.
The Lars algorithm provides the full path of the coefficients alongthe regularization parameter almost for free, thus a common operationis to retrieve the path with one of the functions lars_pathor lars_path_gram.
OrthogonalMatchingPursuit and orthogonal_mp implement the OMPalgorithm for approximating the fit of a linear model with constraints imposedon the number of non-zero coefficients (ie. the \(\ell_0\) pseudo-norm).
OMP is based on a greedy algorithm that includes at each step the atom mosthighly correlated with the current residual. It is similar to the simplermatching pursuit (MP) method, but better in that at each iteration, theresidual is recomputed using an orthogonal projection on the space of thepreviously chosen dictionary elements.
This can be done by introducing uninformative priorsover the hyper parameters of the model.The \(\ell_2\) regularization used in Ridge regression and classification isequivalent to finding a maximum a posteriori estimation under a Gaussian priorover the coefficients \(w\) with precision \(\lambda^-1\).Instead of setting lambda manually, it is possible to treat it as a randomvariable to be estimated from the data.
93ddb68554