Linear Model - Linear Regression Models

Linear Regression Models

For the regression case, the statistical model is as follows. Given a (random) sample the relation between the observations Y_i and the independent variables X_ij is formulated as

where may be nonlinear functions. In the above, the quantities ε_i are random variables representing errors in the relationship. The "linear" part of the designation relates to the appearance of the regression coefficients, β_j in a linear way in the above relationship. Alternatively, one may say that the predicted values corresponding to the above model, namely

are linear functions of the β_j.

Given that estimation is undertaken on the basis of a least squares analysis, estimates of the unknown parameters β_j are determined by minimising a sum of squares function

From this, it can readily be seen that the "linear" aspect of the model means the following:

• the function to be minimised is a quadratic function of the β_j for which minimisation is a relatively simple problem;
• the derivatives of the function are linear functions of the β_j making it easy to find the minimising values;
• the minimising values β_j are linear functions of the observations Y_i;
• the minimising values β_j are linear functions of the random errors ε_i which makes it relatively easy to determine the statistical properties of the estimated values of β_j.