0

I understand the need for Explainability in AI. However, I am uncertain of what is meant by 'making AI explainable'.

What needs to be explainable? Is it the output of a model? Does it refer to the model itself? Does it refer to the user interface of the tool that the AI is a part of? Is it all of the above? If so, what is not included in Explainable AI?

What do we strive for when making AI 'explainable'? Are there commonly referred to definitions of explainability of AI, which go beyond 'to understand how a decision was made'?

Robin van Hoorn
  • 2,366
  • 1
  • 10
  • 33
  • depends on the case, but usually is "why is the model acting like that"... so for example, you would like to know that your resnet is recognizing a cat because the picture has some whiskers and ears, more than some other features – Alberto Oct 24 '23 at 16:45

1 Answers1

1

As a very rough answer, we need to be able to confirm a model that we came up with. So that we need to convince some one that our model is working logically well. What I meant by "logically well" is that the decision-making process should be clear either for a single data or the general model. I would suggest taking a look at this page: https://towardsdatascience.com/decrypting-your-machine-learning-model-using-lime-5adc035109b5

M SA
  • 11
  • 1
  • Thanks for your answer. I was aware of this 'ambiguous' goal. My question is looking more for definitions (i.e. from literature) that define it, for example, in a multi-faceted way. – Robin van Hoorn Oct 30 '23 at 12:43
  • I would clarify one aspect in this answer: The difference between interpretable models (what you describe) and explainable models. A model is interpretable if its inner workings (intermediate results) are human understandable. A model is explainable if you can at least apply a post-hoc method that extends the models output/input with an explanation. While all interpretable models are also explainable, not all explainable models are interpretable, so explainability is a weaker property than interpretability. – Chillston Oct 30 '23 at 16:12
  • 1
    Im not sure if I would agree with that @Chillston. I always had the idea that interpretable is a subset of explainable; where 'interpretable' refers to whether you can make sense of the output of a model, and 'explainable' refers to whether you can also make sense of how the model gets to that output. – Robin van Hoorn Nov 29 '23 at 15:19
  • Yeah, I am sure there is a lot of room for debate, oftentimes interpretability and explainability are regarded as the same thing which it is certainly not. I agree with you that interpretability is a subset of explainability and thus a stronger statement about a model. Where I have a different opinion is that to 'make sense' of an output, you have to make sense of how the model gets to that output first. Without that, how would you make sense of anything, given only an output without simply 'trusting' the model? Therefore, your description of 'interpretability' wouldn't work for me like that. – Chillston Dec 10 '23 at 15:41