Linear Classifier - Generative Models Vs. Discriminative Models

Generative Models Vs. Discriminative Models

There are two broad classes of methods for determining the parameters of a linear classifier . Methods of the first class model conditional density functions . Examples of such algorithms include:

• Linear Discriminant Analysis (or Fisher's linear discriminant) (LDA)—assumes Gaussian conditional density models
• Naive Bayes classifier—assumes independent binomial conditional density models.

The second set of methods includes discriminative models, which attempt to maximize the quality of the output on a training set. Additional terms in the training cost function can easily perform regularization of the final model. Examples of discriminative training of linear classifiers include

• Logistic regression—maximum likelihood estimation of assuming that the observed training set was generated by a binomial model that depends on the output of the classifier.
• Perceptron—an algorithm that attempts to fix all errors encountered in the training set
• Support vector machine—an algorithm that maximizes the margin between the decision hyperplane and the examples in the training set.

Note: Despite its name, LDA does not belong to the class of discriminative models in this taxonomy. However, its name makes sense when we compare LDA to the other main linear dimensionality reduction algorithm: Principal Components Analysis (PCA). LDA is a supervised learning algorithm that utilizes the labels of the data, while PCA is an unsupervised learning algorithm that ignores the labels. To summarize, the name is a historical artifact (see, p. 117).

Discriminative training often yields higher accuracy than modeling the conditional density functions. However, handling missing data is often easier with conditional density models.

All of the linear classifier algorithms listed above can be converted into non-linear algorithms operating on a different input space, using the kernel trick.