Precision and Recall in search results.
Precision can be defined as “Fraction of results that are relevant” and recall is defined as “Fraction of relevant results that are returned “
Suppose the search engine returns 30 results, only 20 of which are relevant, while failing to return 40 additional relevant results, its precision is 20/30 = 2/3, which tells us how valid the results are, while its recall is 20/60 = 1/3, which tells us how complete the results are.
Precision and recall may be very relevant in other machine learning algorithm but in case of text search these matrices need to be modified to make it relevant to the text search algorithms.
Let’s go through the ideas behind developing these evaluation matrices.
Position matters
In the research it is found that about 90% users don’t look beyond the first page and half searchers don’t even look beyond the first three results. Therefore in the text search paradigm the most important thing to look for is the position of the search results. The argument is simple. If a user is not seeing the relevant results then the results don’t matter to him. So the conclusion can be drawn that the precision matters for the top results.
There are two categories of evaluation parameters that can be derived.
One is with human judgement other one is without human judgement. Let’s have a look at them.
With human judgement.
Mean Average Precision(mAP) @ k
The mean Average Precision or mAP score is calculated by taking the mean AP over all classes. Let’s look the way to calculate this matrix.Let’s first get Precision @k. We take k as 10 in following example.
If 5 of the top 10 results are relevant, then precision @ 10 is 50%.
Average Precision@k.
Average of all Precision@1,Precision@2…Precision@k
For example, if the first 5 of 10 results are relevant, then average precision at 10 is
(1 + 1 + 1 + 1 + 1 + 5/6 + 5/7 + 5/8 + 5/9 + 5/10) / 10 = 0.82.
Now if we repeat the experiment for number of queries(take 100) then
mAP@k will be
Average Precision@k /|Q|
Where |Q| is number of queries.
The higher the value of mAP@k then better your results from the search engine.
NDCG(Normalised Discounted Cumulative Gain)
This matrix takes into account the position significance into account.
Say for example the ideal Discounted Cumulative Gain (iDCG) is
[3,3,2,2,1]
And the search result gives [2,3,3,1,2]
The higher the NDCG the better is result.
Without Human Judgement
Click Through Rate (CTR) or conversion rate.
This is simply the percentage of results that got clicked.
Mean reciprocal Rank (MRR )
The mean reciprocal rank is the average of the reciprocal ranks of results for a sample of queries Q.
