0

Why it is not easy to just train model for factual accuracy and punish for any invented facts or mistakes, award for admitting ignorance in difficult cases?

Anixx
  • 321
  • 1
  • 11

2 Answers2

3

Why it is not easy to just train model for factual accuracy and punish for any invented facts or mistakes, award for admitting ignorance in difficult cases?

Detecting "truth" in a general sense from natural language is not a solved problem by any means. It is not possible to automate this kind of training, except in very limited domains.

For example, it would be possible to automate production of billions of maths calculations and their correct answers, but not a great use for an LLM, because any pocket calculator is far more efficient at such tasks. Tuning heavily on such constructed data would very likely cause the model to get better at maths calculations at the expense of most of the other text generations.

Other than such limited generative/combinatorial true-statement engines, the best models for processing text data and assessing truth more generally are the existing LLMs . . .

The reinforcement fine tuning used by e.g. ChatGPT can reward for factually correct output, but requires human trainers to assess responses, so is much harder in terms of resources required. In addition, the core technology of LLMs is not really designed to store/recall such factual and reasoning information efficiently, so although this approach is used and does improve the model significantly, it is not a complete answer to the issue, but instead a useful piece of practical fine tuning.

Scaling up the reinforcement learning stage might improve the LLM model a little practically over where things stand today, but I would suspect that other architectures and mechanisms for world modelling, memory recall and correct reasoning are where more major improvements lie.

Neil Slater
  • 32,068
  • 3
  • 43
  • 64
  • But does not factual training in one sphere (like math) reduce made up answers in other spheres? Another sphere is programming, where a testing software can simply run the generated program and check the result. I think, the problem of inventing fake code that only looks real has been solved already. Most LLMs produce if not etirely correct, but at least, sensible code. Similarly, they rarely invent words for natural language. – Anixx Mar 11 '24 at 05:23
  • 2
    @Anixx: No training in one factual "sphere" does not really impact results in another one, except stylistically. There is no known inherent signal for "The Truth", that can be learned in a cross-domain way. Definitely code generation has got better, but your assertion is false - I can tell you, as a developer who uses ChatGPT for code tasks regularly, that the language model still frequently makes mistakes that invalidate the functionality of the code. The mistakes are rarely with syntax or structure of well-known functions, but they still occur. – Neil Slater Mar 11 '24 at 07:08
1

You're asking 2 questions here. You should avoid asking more than one question per post.

They are a big problem because misinformation/disinformation can be harmful.

For instance, suppose someone uses these models to get medical, military, political, financial, etc., advice, and they get a factually wrong answer. This can have bad consequences. Of course, no intelligent person should blindly use these models, if you care about facts.

Moreover, these models are not really trained to give you facts, but to give you the most likely answer given the training data. As stated in the other answer, you can fine-tune them with RL to punish them for factually wrong answers, but, as far as I know, there's nothing that prevents models like gpt-4 or gpt-3-turbo from sometimes producing factually wrong statements (if there was, it would not really be working). If they give you facts, it's just a byproduct, it's just chance. The next time they could give you wrong info, and this can depend on the input you pass to them, among other parameters, like the temperature.

nbro
  • 40,472
  • 12
  • 105
  • 192
  • 1
    I think it is just one question. "Why are hallucinations a huge problem?" could be restated "Why can we not simply train away hallucinations?" and then the title would match the body. So "a huge problem" not in the sense of damage caused, but in the sense of not being easy to resolve. – Neil Slater Mar 10 '24 at 22:24
  • Ok, I might have misunderstood it then, if that's the case. I can remove this answer if it doesn't really address the question – nbro Mar 10 '24 at 23:41
  • Indeed, I asked why it is a huge problem in the sense of resolving it, not regarding damage caused, as @Neil Slater points out. – Anixx Mar 11 '24 at 05:17
  • @Anixx Can you please edit your post to clarify that, i.e. change the title to be more specific? – nbro Mar 11 '24 at 14:23