Natural Language Processing (NLP) is a branch of AI that helps computers or machines to understand, interpret and manipulate human languages to analyse the context and derive its meaning. NLP helps data scientists to organise and structure knowledge to perform tasks like translation, summarization, spam detection , sentiment analysis , named entity recognition, relationship extraction, speech recognition, topic segmentation, etc. Putting it more technically, NLP combines Artificial Intelligence (AI) and computational linguistics so that computers and humans can talk or understand each other seamlessly.
Steps for a NLP Pipeline
Data for the NLP can be generated either by speech or direct text in the form of emails, feeds and news. The commonality between these two is that they are both unstructured data. They are fed into the NLP Pipeline and undergo the steps as shown in the figure.
The very first step is Lexical Analysis
Lexical Analysis involves identifying and analysing the structure of words. Lexicon by definition means the collection of words and phrases in a language. Lexical analysis is dividing the whole chunk of input text into paragraphs, sentences, and words.
The most common lexicon practices are Stemming and Lemmatization::
- Stemming: Stemming is a rudimentary rule-based process of stripping the suffixes (“ing”, “ly”, “es”, “s” etc) from a word.
- Lemmatization: Lemmatization, on the other hand, is an organised & step by step procedure of obtaining the root form of the word. It makes use of vocabulary (dictionary importance of words) and morphological analysis (word structure and grammar relations).
The second step is Syntactic Analysis which is used to check grammar, arrangements of words, and the interrelationship between the words.
The third in sequence is Semantic Analysis in which a structure is created by the syntactic analyzer which assigns meanings. This component transfers linear sequences of words into structures. It shows how the words are associated with each other.
Semantics focuses only on the literal meaning of words, phrases, and sentences. This only abstracts the dictionary meaning or the real meaning from the given context. The structures assigned by the syntactic analyzer always have assigned meaning.
Next step in the whole process is called Discourse Integration which focuses on the properties of the text as a whole that convey meaning by making connections between component sentences. It means a sense of the context. The meaning of any single sentence depends upon that sentence. It also considers the meaning of the following sentence. For example, the word “that” in the sentence “He wanted that” depends upon the prior discourse context.
The last step is Pragmatic Analysis and it deals with the overall communicative and social content and its effect on interpretation. It means abstracting or deriving the meaningful use of language in situations. In this analysis, the main focus is always on “what was said” and on “what is meant.”
Applications of NLP
Classification of news coming in:
In today’s world, every market participant is bombarded with a lot of information through news. It is not humanly possible to go through all the information. Therefore the task of classifying news as important or not so important can be crucial in only the presenting news likely to have an impact . In today’s NLP world we can do exactly this with a very high level of accuracy.
The other utility of NLP classification can the positive or negative classification. We can mine and analyse news headlines and content and classify it as either positive or negative according to any number of rules depending on the context. This coupled with sentiment analysis can very effectively help market participants focus on the news that is likely to be impactful and market moving.
