0

Given that I have a machine learning model.

I evaluated the model over several labeled datasets and acquired the accuracy (or any other metrics) for these datasets.

Now I receive a new dataset without labels. I run the model and got the prediction.

Is there any way to estimate the accuracy of my model on the new dataset?

I was thinking of measuring the similarity of the datasets (in the feature space), then based on the similarity to predict the accuracy of the new dataset. For instance, if the new dataset is not far from the old dataset, the accuracies might be similar.

mommomonthewind
  • 161
  • 1
  • 5

6 Answers6

3

Is there any way to estimate the accuracy of my model on the new dataset?

No.

If you want to evaluate the classification accuracy of your model, you need to know the ground truth to compare, simple as that. Estimating the similarity of the datasets is totally irrelevant, unless there is prior knowledge that feature similarity leads to label similarity.

pcko1
  • 3,940
  • 2
  • 15
  • 29
1

I have not heard about such an estimation. Think about it like this: You need a method, that given a similar dataset will predict labels for another dataset. Then, you will use the labels as the groundtruth. But how such a method is different from a machine learning model?

In other words, any estimated accuracy on such labels will tell you not how close is your model to the real annotations, but how is it close to another model used to generate the pseudo-groundtruth. This will make sense if your 'label generator' performs with a near 100% accuracy, but I am afraid such tasks are very rare. At least, outside labs.

Dmytro Prylipko
  • 836
  • 5
  • 10
1

I'm not a machine learning expert by any means, but I feel like that's what the error measurement is on your training data. Or at least, in theory.

If it turns out that this isn't the case, then the data you used for training wasn't distributed in a way that represented the thing you're trying to model or you overfit your training data.

You can get a good "estimate" for the performance of your model by using a validation set, which basically helps you to determine if either of these two things have occurred.

wheresmycookie
  • 213
  • 1
  • 8
0

Please read about semi-supervised learning and label propagation. It's what you was looking for. Bunch of code:

from sklearn.semi_supervised import LabelPropagation
label_prop_model = LabelPropagation()

random_unlabeled_points=#
features=#

labels = old_data_set_labels
labels[random_unlabeled_points] = -1
label_prop_model.fit(features, labels)

Sources:

fuwiak
  • 1,373
  • 8
  • 13
  • 26
  • That's why my question is "to estimate", not "to measure". I am asking if there is a technique that estimates (or predicts if you like) the performance of the model if we perform the model on a particular dataset without the label. – mommomonthewind Jan 19 '19 at 12:27
0

Technically there is no direct way to do that. But let's us say you are doing some binary classification and the response rate in your training data set is 30% . Now when you use your model to score the new data set (without labels). Ideally average score should be around 30% (considering new data set is similar to training data set) .

In case of regression problems, the response rate can be replaced by average of dependent variable.

Rupesh Prasad
  • 1
0

There is no direct way to estimate the accuracy. I would highly suggest Question on bias-variance tradeoff and means of optimization for how variance/bias is evaluated/defined in ML modelling, which can at least provide some context of how "accurate" (I am hesitant to use that word) should be defined here.

Canbo
  • 103
  • 3