11

Suppose I have a smooth function like $f(x, y) = x^2+y^2$. I have a training set $D \subsetneq \{((x, y), f(x,y)) | (x,y) \in \mathbb{R}^2\}$ and, of course, I don't know $f$ although I can evaluate $f$ wherever I want.

Are regression trees capable of finding a smooth model of the function (hence a tiny change in the input should only give a tiny change in the output)?

From what I've read in Lecture 10: Regression Trees it seems to me that regression trees basically put the function values into bins:

For classic regression trees, the model in each cell is just a constant estimate of Y .

As they write "classic" I guess there is a variant where the cells do something more interesting?

Martin Thoma
  • 18,880
  • 35
  • 95
  • 169

3 Answers3

2

Regression trees, particularly gradient boosting (essentially many trees), tend to do very well on continuous predictions, often outperforming models that are truly continuous like linear regression when. This is especially true when there are variable interactions and when you have a sufficiently large dataset (over 10,000 records) so that overfitting is less likely. If your primary objective is simply predictive power, then whether the model is 100% continuous or pseudo continuous should be irrelevant. If making your regression trees more continuous enhances out of sample predictive power, then you can simply increase tree depth or add more trees.

Ryan Zotti
  • 4,149
  • 3
  • 19
  • 32
  • 1
    I agree. My boosted trees almost always outperform very painstakingly crafted and optimized GLMs. Of course, you lose interpretability when you gain predictive power. – prooffreader Jan 18 '16 at 12:44
0

If you slightly extend the question to include general gradient boosting techniques (in contrast with the special case of boosted regression trees), then the answer is yes. Gradient boosting has been successfully used as an alternative for variable selection. A good example is mboost package. The key is that the class of base learners used for boosting consists of continuous models to start with. This tutorial describes typical classes of base learners as follows:

The commonly used base-learner models can be classified into a three distinct categories: linear models, smooth models and decision trees. There is also a number of other models, such as markov random fields (Dietterich et al., 2004) or wavelets (Viola and Jones, 2001), but their application arises for relatively specific practical tasks.

Note that it particularly mentions wavelets. Trees and wavelets has been successfully combined before into tree-based wavelets.

user3605620
  • 121
  • 6
0

In classic regression trees you have a one value in the leaf, but in the leaf you can have a linear regression model, check this ticket.

You can also use ensemble of trees (Random Forest or Gradient Boosting Machines) to have continuous output value.

Community
  • 1
pplonski
  • 361
  • 3
  • 7
  • 1
    The first paragraph is useful, but the second one is not true. Tree ensembles, while "smoother" than a single tree, are still piecewise constant with rectangular regions. – Ben Reiniger Oct 21 '20 at 14:04