For large documents like research papers from different sources, news, emails and other such text based data, text summarisation tools can be very handy. They can save the reader the hours of effort required in going through the papers themselves. Until recently, effective text summarisation tools were in the realm of science fiction only. However, thanks to the ever surprising flexibility of Deep Learning models, the wait may not be that long for tools to be able to summarise multi-page documents into a few sentences, at least within a specific subject area. Here in this blog we will introduce you to the basic technological foundation on which these text summarisation models are being built.
Text Summarisation is very closely related to information understanding which is the key to information science and retrieval. The technology of text summarisation can improve information extraction systems (which today can have multiple use cases) and hence also allows the reader to quickly review a large number of documents without missing out on important bits. Indeed, automatic summarisation has recently been recognised as one of the most important natural language processing (NLP) tasks, yet one of the most difficult ones too.
Approaches
In theory, we have two major approaches to text summarisation:
- Abstractive Text Summarisation
- Extractive Text Summarisation
Extractive summarisation systems selects a number of segments from the source document to make up a summary. The advantage of this approach is that the resulting text is guaranteed to be grammatically correct. Abstractive methods on the other hand are more advanced and involve generating a summary that captures salient ideas of the input text document. The adjective ‘abstractive’ is used to denote a summary that is not a mere selection of a few existing passages or sentences from the source, but a compressed paraphrasing of the main contents of the document, potentially using vocabulary unseen in the source document.
While Extractive Text Summarisation is arguably well suited for identifying the most relevant information, such techniques may lack the fluency and coherency of human-generated summaries. In theory Abstractive summarisation is lot more promising and should be the go-to method in the near future when the related technology becomes more mature. But as of today, it hasn’t delivered convincing performances. In general, extractive systems have achieved higher scores and are more reliable than the abstractive summarisation ones, until now.
General Underlying Technology
The basic neural network architectures that make it possible to perform summarisation are the Seq2Seq architectures, the LSTM recurrent neural networks (RNN), the BERT and the Transformer models as well as the attention mechanism.
Training such a complex hierarchical network from scratch is impractical because it requires an unrealistic amount of document-summary pairs. The popular strategy to train such complex network with limited data is using Transfer Learning. Transfer learning is a technique where a deep learning model trained on a large dataset is used to perform similar tasks on another dataset. Google’s BERT is commonly used for that purpose which is then fine-tuned to a specific domain.