​​REALM: Integrating Retrieval into Language Representation Models
by google research

A new paper from google with a novel approach for language model pre-training, which augments a language representation model with a knowledge retriever.

The idea is the following: we take a sentence or a piece of text and augment it with additional knowledge (pass original text and additional texts to the model).

An example:
The masked text is:

We paid twenty __ at the Buckingham Palace gift shop.


Knowledge retriever could add the following information to it:
Buckingham Palace is the London residence of the British monarchy.
The official currency of the United Kingdom is the Pound.


blog post: https://ai.googleblog.com/2020/08/realm-integrating-retrieval-into.html
paper: https://arxiv.org/abs/2002.08909
github: https://github.com/google-research/language/tree/master/language/realm

#nlp #languagemodel #knowledgeretriever #icml2020