2

From my understanding, PCA assumes that redundancy in features can be explained by linear relationships. It also finds orthogonal bases, so when the variance of your data is maximized along non-orthogonal directions PCA isn't going to give you what you hope for. In my, albeit limited, experience I've never worked on a dataset where I can safely assume the above two conditions hold. At the same time, when working with audio or video it has amazing results.

What is it about audio and video that allow those assumptions to hold? When working outside those domains, how can you know PCA isn't just giving you random stuff?

Thanks!

user5046480
  • 23
  • 3

2 Answers2

2

PCA does not assume your data can be written as a linear combination. It simply finds an orthogonal basis for your data, oriented by decreasing variation. Because of orthogonality you have also a linear combination associated.

If your data have low dimensionality you need less dimensions to describe it, or at least to approximate it. I suppose this is what you mean by “it works”.

Often data from audio and video have lower dimensions compared to raw data. This is the reason why often works.

rapaio
  • 4,763
  • 20
  • 35
  • Thanks for your answer! Please correct me where I'm wrong. If we are finding orthogonal bases we're de-correlating the features, i.e. making them linearly independent. The point of kernel pca is to extend PCA to non-linear relationships. That's why I believe PCA only removes redundancy that can be explained by linear relationships – user5046480 Sep 06 '21 at 21:56
  • The new basis may have less axes/features ... so in a way the new feature may have less correlations with the other one from the new feature set. – Malo Sep 06 '21 at 22:08
0

PCA will find a new orthogonal basis and new features for your data, maximizing variance along the new features/axes.

Moreover by selecting a limited number of the most usefull/meaningfull new features/axis you will be able to do dimension reduction.

Note that two linearly related features (x1 and x2) will probably be replaced by a unique new feature (xn) in the new basis. So the new feature (xn) may be not/less correlated to the other new features.

If you train/test your model after applying PCA and keeping only most relevant features (try aroud 50 or adjust the number) you should gain efficiency as it will be faster to train and also generalize better.

So It should 'work' in most case.

Malo
  • 376
  • 1
  • 9