9

So I draw a pairplot/heatmap from the feature correlations of a dataset and see a set of features that bears Zero-correlations both with:

  • every other feature and
  • also with the target/label

.Reference code snippet in python is below:

corr = df.corr()
sns.heatmap(corr) # Visually see how each feature  is correlate with other (incl. the target)
  1. Can I drop these features to improve the accuracy of my classification problem?
  2. Can I drop these features to improve the accuracy of my classification problem, if it is explicitly given that these features are derived features?
karthiks
  • 342
  • 1
  • 2
  • 10

3 Answers3

8

Can I drop these features to improve the accuracy of my classification problem?

If you are using a simple linear classifier, such as logistic regression then yes. That is because your plots are giving you a direct visualisation of how the model could make use of the data.

As soon as you start to use a non-linear classifier, that can combine features inside the learning model, then it is not so straightforward. Your plots cannot exclude a complex relationship that such a model might be able to exploit. Generally the only way to proceed is to train and test the model (using some form of cross-validation) with and without the feature.

A plot might visually show a strong non-linear relationship with zero linear correlation - e.g. a complete bell curve of feature versus target would have close to zero linear correlation, but suggest that something interesting is going on that would be useful in a predictive model. If you see plots like this, you can either try to turn them into linear relationships with some feature engineering, or you can treat it as evidence that you should use a non-linear model.

In general, this advice applies whether or not the features are derived features. For a linear model, a derived feature which is completely uncorrelated with the target is still not useful. A derived feature may or may not be easier for a non-linear model to learn from, you cannot easily tell from a plot designed to help you find linear relationships.

Neil Slater
  • 28,918
  • 4
  • 80
  • 100
  • That clears the air. Thanks. I've also added a follow-up question. Do you mind answering it as well? Thanks in advance. – karthiks Nov 02 '18 at 10:03
  • @karthiks: You should not really edit/extend your question after it has been answered. In this case it is only a small extension and the answer is basically "yes, it is the same", so I have edited that in. – Neil Slater Nov 02 '18 at 10:43
  • I second that and added another one, only because it is more an extension question that I missed out. – karthiks Nov 02 '18 at 10:48
4

If I understand you well, you are asking if you can remove features having zero-correlation either :

  1. With other features
  2. With the label you want to predict


Those are two different cases :

1. We usually recommend to remove features having correlation between them (stabilize the model). If they are ZERO-correlated, you cannot conclude here. This is by training your model that you will see is the feature is worth or not.

Don't drop those ones.


2. If a feature is strongly correlated with your label, this means a linear function (or model) should be able to predict well the latter. Even if it is not correlated, it doesn't tell you that a non-linear model wouldn't perform well by using this feature.

Don't drop this one either !


I hope I answered your question.

Atani
  • 66
  • 4
  • Modified question for pressing clarity. I meant that a set of features bearing ZERO-correlation with all other features including the target/label. Hope that clarifies.. – karthiks Nov 02 '18 at 09:58
  • Thank you for your clarification. Edited my answer accordingly. – Atani Nov 02 '18 at 10:09
4

These uncorrelated features might be important for target in connection with other non-target features. So, it might be not a good idea to remove them, especially if your model is a complex one.

It might be a good idea to remove one of the highly correlated between themselves non-target features, because they might be redundant.

Still, it might be better to use feature reduction technics like PCA, because PCA maximize variance, without removing the whole feature, but including it into principal component.

In case of ordinals or binary features, correlation won't tell you a lot. So I guess, the best way to test if a feature is important in case it's not correlated with target is to directly compare performance of a model with and without the feature. But still different features might have different importance for different algorithms.

DmytroSytro
  • 517
  • 2
  • 14