Omnibus Test - Omnibus Tests in Logistic Regression - Model Fitting: Maximum Likelihood Method

Model Fitting: Maximum Likelihood Method

The omnibus test, among the other parts of the logistic regression procedure, is a likelihood-ratio test based on the maximum likelihood method. Unlike the Linear Regression procedure in which estimation of the regression coefficients can be derived from least square procedure or by minimizing the sum of squared residuals as in maximum likelihood method, in logistic regression there is no such an analytical solution or a set of equations from which one can derive a solution to estimate the regression coefficients. So logistic regression uses the maximum likelihood procedure to estimate the coefficients that maximize the likelihood of the regression coefficients given the predictors and criterion. The maximum likelihood solution is an iterative process that begins with a tentative solution, revises it slightly to see if it can be improved, and repeats this process until improvement is minute, at which point the model is said to have converged.. Applying the procedure in conditioned on convergence ( see also in the following "remarks and other considerations ").

In general, regarding simple hypotheses on parameter θ ( for example): H₀: θ=θ₀ vs. H₁: θ=θ₁ ,the likelihood ratio test statistic can be referred as:

,where L(y_i|θ) is the likelihood function, which refers to the specific θ.

The numerator corresponds to the maximum likelihood of an observed outcome under the null hypothesis. The denominator corresponds to the maximum likelihood of an observed outcome varying parameters over the whole parameter space. The numerator of this ratio is less than the denominator. The likelihood ratio hence is between 0 and 1.

Lower values of the likelihood ratio mean that the observed result was much less likely to occur under the null hypothesis as compared to the alternative. Higher values of the statistic mean that the observed outcome was more than or equally likely or nearly as likely to occur under the null hypothesis as compared to the alternative, and the null hypothesis cannot be rejected.

The likelihood ratio test provides the following decision rule:

If do not reject H₀,

otherwise

If reject H₀

and also reject H₀ with probability q if ,

whereas the critical values c, q are usually chosen to obtain a specified significance level α, through the relation: .

Thus, the likelihood-ratio test rejects the null hypothesis if the value of this statistic is too small. How small is too small depends on the significance level of the test, i.e., on what probability of Type I error is considered tolerable The Neyman-Pearson lemma states that this likelihood ratio test is the most powerful among all level-α tests for this problem.