Understanding Locally Weighted Linear Regression

Question

I'm having problem understanding how we choose the weight function. In Andrew Ng's notes, a method for calculating a local weight, a standard choice of weights is given by: What I don't understand is, what exactly is the x here? Apparently

Note that the weights depend on the particular point x at which we’re trying to evaluate x.

But I don't get it.

Take the example of housing prices predicted by number of rooms and size in square feet. So each x^(i) is a [roomnum, size] array. So what's in x? I guess that also should be a [roomnum, size] array, but what's in it? Is it even a vector? Or is it the target variable? If so, why isn't it marked with y? I don't get it, please help!

EDIT Ok so what I want is to make create a regression line like this:

How would I choose the x-es? What would they be in an algorithm? Do I have to make guesses for each x? How can I produce a line like this?

Answer 1

Locally weighted linear regression is a non-parametric method for fitting data points. What does that mean?

• Instead of fitting a single regression line, you fit many linear regression models. The final resulting smooth curve is the product of all those regression models.
• Obviously, we can't fit the same linear model again and again. Instead, for each linear model we want to fit, we find a point x and use that for fitting a local regression model.
• We find points that closest to x to fit each of our local regression model. That's why you'll see the algorithm is also known as nearest neighbours algorithm in the literature.

Now, if your data points have the x-values from 1 to 100: [1,2,3 ... 98, 99, 100]. The algorithm would fit a linear model for 1,2,3...,98,99,100. That means, you'll have 100 regression models.

Again, when we fit each of the model, we can't just use all the data points in the sample. For each of the model, we find the closest points and use that for fitting. For example, if the algorithm wants to fit for x=50, it will put higher weight on [48,49,50,51,52] and less weight on [45,46,47,53,54,55]. When it tries to fit for x=95, the points [92,93,95,96,97] will have higher weight than any other data points.

Do you see the pattern? The points closer the where you want to fit have higher weight and the points further have lower weight (zero if too far). That's what the weight function is for.

To answer your question directly:

x are the data points for each local regression model. They are usually (but not always) the data points in your sample.

Answer 2

As the article has mentioned:

In the original linear regression algorithm, to make a prediction at a query point $x$ (i.e., to evaluate $h(x)$)...

$x$ you want is the exact data point you want to predict, in your case, $x$ is a [roomnum, size] tuple.

Assuming you want to find a $f(r, s)$, where $f(roomnum_i, size_i) = price_i, \forall i$. You have a dataset, and an algorithm to fit $f(r,s)$.

The most interesting part of locally weighted linear regression is that, the model changes when $x$ changes (keep in mind $x$ is the data point you want to query).

Assume $x=(R,S)=(3,30)$, the algorithm becomes:

Find $\theta$ to minimize

$\sum_{i} exp(-\frac{|x^{(i)}-x|^2}{2\tau^2})(y^{(i)}-\theta^T x^{(i)})$

where $x^{(i)}$ and $y^{(i)}$ is your dataset, $x=(3, 30)$ is the point you want to query.

The article shows that clearly:

The (unweighted) linear regression algorithm that we saw earlier is known as a parametric learning algorithm, because it has a fixed, finite number of parameters (the θi ’s), which are fit to the data.

However,

“non-parametric” (roughly) refers to the fact that the amount of stuff we need to keep in order to represent the hypothesis $h$ grows linearly with the size of the training set.

Answer 3

$x$ could be the position of the center of the peak in the Bell-shaped function for defining the weights. Note that the weights depend on the particular point $x$ at which we’re trying to evaluate $x$.

Moreover,

if $|x^{(i)} − x|$ is small, then $w(i)$ is large (close to 1)

and

if $|x^{(i)} − x|$ is large, then $w(i)$ is small (close to 0).