K nearest neighbour

Question

Is the k-nearest neighbour algorithm a discriminative or a generative classifier? My first thought on this was that it was generative, because it actually uses Bayes' theorem to compute the posterior. Searching further, it seems like it is a discriminative model. But I couldn't find the explanation.

So is KNN discriminative first of all? And if it is, is that because it doesn't model the priors or the likelihood?

Before I suggest the answer directly -- what did you find that makes you think it's discriminative? are you sure k-NN always involves Bayes's theorem or priors? — Sean Owen, Dec 14 '14 at 11:26
The problem is that I'm not sure about the definition of these two types classifiers, my book isn't clear. That being said, I think that the generative classifier must be able to generate data poins as well, and to do that it learns the joint probability $p(x,C_{k})$. The discriminative classifier does not learn the joint probability, and so does KNN. While I see that it does use Baye's theorem, it only computes the posterior. Tell me if this is correct or not, but also please tell me the definitive difference between the two classifiers because I'm confused. — 101, Dec 14 '14 at 11:32
Also from what I understand in the book, it always uses Baye's theorem to compute the posteriors. The class priors $p(C_{k})$ are given as just $\frac{N_{k}}{N}$ where the numerator is the number of points in class k, and N is the total number of points in our set. — 101, Dec 14 '14 at 11:51

score 4 · Accepted Answer · edited Apr 13 '17 at 12:44

See a similar answer here. To clarify, k nearest neighbor is a discriminative classifier.

The difference between a generative and a discriminative classifier is that the former models the joint probability where as the latter models the conditional probability (the posterior) starting from the prior.

In the case of nearest neighbors, the conditional probability of a class given a data point is modeled. To do this, one starts with the prior probability on the classes.

K nearest neighbour

1 Answers1