Forward or Backward Stepwise Regression

Question

I understand the process and logic of why to perform stepwise regression. To me they should always arrive at the same function, just one adds coefficients and tests for significance while the other removes coefficients as it tests for significance. Are there any significant trade-offs between using one versus the other?

Answer 1

Imagine you have 20 coefficients to test for and also have target accuracy (or whatever metric you're interested in) that you aim to beat. It acts as a threshold.

One tradeoff could be that performing "backwards regression" means you would in theory start with you maximum accuracy and be decreasing each time you remove a parameter. You might cut your threshold after removing only 3 parameters! So don't necessarily need to continue.

Conversely, starting with a single parameter and working your way up, you would have to try 17 models 5o reach your threshold.

This scenario could of course be reversed, so it is important to understand your data and make an informed decision about what suits your situation best.