Pre-trained language models can be used for tasks in Natural Language Understanding such as Named Entity Recognition, Extraction-based Question Answering, Sentiment analysis, Document Classification, and Natural Language Inference.
As well as tasks in Natural Language Generation such as Machine translation, Automatic summarization, Reading Comprehension, and Data-to-Text Generation.
This article introduces the most representative ELMO, BERT, and GPT-2 models by listing the characteristics of currently mainstream pre-trained language models; using the most concise text descriptions to allow everyone to easily compare the differences.
ELMO（Embeddings from Language Model）
- RNN-based Language Models
- Trained through a large number of sentences, without the need for annotation
- Predict the next Token
- Obtain Contextualized word embeddings from the hidden layer of the RNN
- Concatenating forward and backward embeddings yields the context embedding
- Finally, sum the embeddings from each layer, with subsequent tasks learning the weighting parameters
- 94M parameters
Source: https://arxiv.org/abs/1802.05365
BERT（Bidirectional Encoder Representations from Transformers）
- Belongs to the Encoder part of the Transformer
- Only need to train the Transformer Encoder (one-to-one input and output)
- Trained through a large number of sentences, without the need for annotation
- Given a sequence of words, each word outputs an embedding
- Chinese is more suitable using characters as units, because using one-hot encoding results in too many words; there are approximately 4,800 commonly used Chinese characters, while Chinese words are several times higher than this number
- Masked LM: 15% of the words in the input word sequence are randomly replaced with a special Token [Mask], and predictions are made
- Predict the next sentence: Introduce [SEP] to represent the boundary between two sentences, and [CLS] to represent the position for outputting classification results
- Both of the above methods take the extracted [Mask] or [CLS] Vector and feed it into a Linear Multi-class Classifier to predict the word
- The above two methods must be used simultaneously
- 340M parameters
Source: https://arxiv.org/abs/1810.04805
GPT-2（Generative Pre-Training）
- Belongs to the Decoder part of the Transformer
- Predict the next Token
- Trained on 40GB of text
- Can achieve Zero-shot Learning, requiring no training data, achieving Reading Comprehension (F-score=55, close to Dr.QA), Summarization (similar to random), and Translation (similar to random)
- 1542M parameters
