Traditionally in enterprise search domain we have relied on keyword based searches. If a document matched or contained a certain keyword(s), then the document would be listed in the search results. This is known as Heuristic Search and it further evolved into smarter search models. The “smartness” was achieved by integrating some other techniques like Fuzzy search, Proximity search and so on. Even after these innovations, the search was still primarily a heuristic one (i.e. the basis of the search was input keywords only). In the last decade as the number of documents grew, search models evolved further, taking context and meaning into account. With this and the evolution of several language models such as BERT, USE, GLOVE, GPT etc., the text search began to take baby steps towards what we today know as Semantic Search. It’s still in infancy and still very much work in progress.
Let’s look at a basic architecture to implement the semantic search.
Implementation
We have divided the entire implementation into 2 parts.
- Training & Storing
- Query Module
Training & Storing
We get the documents containing ids, URLs and text. We parse the documents and provide a JSON structure to the document. The text part of document is vectorised (the machine only understands the vectors and not strings). This vectorization or embedding ( NLP term) is done through a language model. The language model can be a base model easily downloadable or fine-tuned model – especially curated to take care of domain specific terms. The output of the language model is called embeddings (low-dimensional vector of vocabulary terms). These vectors are stored in a special document store which has the capability to store dense vectors. A lot of dense vectors stores are available in the open source world such as Solr or Elasticsearch. This process is followed for many documents creating a robust all-encompassing data store.
Query Module
Now that we have the store we can use it to search the documents. The user query is also vectorized by an embedder (e.g. TensorFlow service). These embeddings are now passed to the store for similarity matching. These similarity can be calculated by different similarity algorithms like cosine similarity. Once we have similar documents, we can get the ids and URLs and present it to users.
Future for Semantic Search
The above describes a basic implementation of semantic enterprise search. Now the question is where the search paradigm heading towards. The newest addition to this is the paradigm of Machine Reasoning. Machine reasoning will bring in a new dimension called “intent” or “intent based inferencing” . We will keep that topic for a future blog.