1

GPT-1 mentions both Semi-supervised learning and Unsupervised pre-training but it seems like the same to me. Moreoever, "Semi-supervised Sequence Learning" of Dai and Le also more like self-supervised learning. So what the key differences between them?

Inhyeok Yoo
  • 33
  • 4

1 Answers1

1

Semi-supervised learning is having label for a fraction of data, but in self-supervised there is no label available. Imagine a huge question/answer dataset. No one labels that data but you can learn question answering right? Because you are able to retrieve relation between question and answer from data.

Or in modeling documents you need sentences which are similar and sentences which are dissimilar in order to learn document embedding but these detailed labels are usually not available. In this case you count sentences from same document as similar and sentences from two different documents as dissimilar and train your model (example idea: you can run a topic modeling on data and make similar/dissimilar labels more accurate). It is called self training.

Kasra Manshaei
  • 6,570
  • 1
  • 21
  • 45
  • Then can I consider the pre-training phase of BERT as 'self-supervised learning', and the fine-tuning phase of BERT as 'semi-supervised learning? – Inhyeok Yoo May 28 '21 at 08:32
  • I would call the fine-tuning part still "supervised" but you are right. In fine tuning you have a set of labeld data (however small) and you learn only from them so it is classic "supervised". Semi supervised means there are data in your training set for which you do not have labels. In general your high-level understanding is correct – Kasra Manshaei May 28 '21 at 09:01
  • Got it. The overall process of BERT, ELMo, etc. can be considered as 'Semi-supervised learning' because they including both LM(self-supervised) and supervised learning. The training process of LM is self-supervised learning. Thank you. – Inhyeok Yoo May 29 '21 at 10:32
  • Welcome my friemd. If it answered your question you may consider accepting answer or upvoting it. – Kasra Manshaei May 29 '21 at 10:51
  • Yeap definitely. Thank you – Inhyeok Yoo May 29 '21 at 12:00