How do you overcome results oriented thinking from a client?

Question

I am working with a client on a special project that I am going to obfuscate in this question. Basically, I'm trying to overcome some short-term, results-oriented thinking from my client.

Let's say you have a model to forecast the performance of a racehorse. Your model tells your client to sell racehorse X because the probability of its performance is low (<10%). The horse is sold and said racehorse goes on to win 3 races. Your client says, "See, we should have never let go of that racehorse! The model is wrong!"

As data scientist we can understand that anomalies happen and that this horse may have also lost some races - we were just on the wrong side of probability this time. But how do you overcome those objections with the client? How do you turn short-term thinking into a long-term outlook of predictive modeling?

Answer 1

The scenario you described with one horse is basically anecdotal evidence. To objectively evaluate what you're doing, you need to agree with the client to make this an experiment.

I understand you don't trust my horse predictions and that's totally understandable. It is human nature to worry about the ones that got away. Let's handle this objectively and run an experiment. I'll give you buy/sell ratings on 50 horses. We then track their performance for six months. If on average my ratings outperform a coin flip, we'll know we're onto something at a macro level even if a few got away. If my models end up being worse than a coinflip, we'll know the model needs to be improved or you need to fire me.

Answer 2

I think that you should manage the customer expectations.

First, try to define and use a common terminology. You spoke about a low probability of performance while your customer spoke about winning races. Can you tie the terms together? For example, if the probability of performance is the probability of winning a race, once that you have a common terminology you can explain to the customer that the probability of a low performer to win 3 races in not zero and given X horses, you are expecting to see Y such horses, even if the model is perfect.

Short term vs. long term issues, should be discussed and incorporated into the concept that you would like to predict. I'm not familiar with horse races but I assume that there are horses that are sprinter and can win a single race while there are horses with more stamina that can win a full match. If there are such differences of interest, they should guide the whole research process.

Next, you should explain to the customer that the model being wrong, doesn't means it is not useful. Indeed, what I have in mind in Box's

"essentially, all models are wrong, but some are useful"

You can help the customer see the benefit by comparing your model to some benchmark model (e.g., assuming that all horses are fast/slow, a simple approximation, a common domain knowledge estimation, etc).

Now that we understand that perfect prediction is not a requirement to benefit, give your customer additional power. Tell your customer that your model will have errors, but he can choose to which side he prefers that the errors will fall. He can do that by assigning a cost to false positive and false negatives. Given the costs, you can tune the model to better fit his needs (e.g., by choosing a confidence level that optimize the cost).

Once that you have a model, compute the confusion matrix and discuss it with the customer. Explain to him the precision and recall and their meaning (e.g., the model will recommend to sell x% horses that are indeed good ones).

Last but not least, incase that it is an ongoing project, you should use the errors as a source of improvement. The details of improvement methods depend on the specific project. However, at least go over these cases and try to understand why did your model fail on them.

Good luck!