This was by far the most significant iteration of the ongoing exercise where I challenge an audience to produce a keyword search that works better than technology-assisted review (also known as predictive coding or supervised machine learning). There were far more participants than previous rounds, and a structural change in the challenge allowed participants to get immediate feedback on the performance of their queries so they could iteratively improve them. A total of 1,924 queries were submitted by 42 participants (an average of 45.8 queries per person) and higher recall levels were achieved than in any prior version of the challenge, but the audience still couldn’t beat TAR.
In previous versions of the experiment, the audience submitted search queries on paper or through a web form using their phones, and I evaluated a few of them live on stage to see whether the audience was able to achieve higher recall than TAR. Because the number of live evaluations was so small, the audience had very little opportunity to use the results to improve their queries. In the latest iteration, participants each had their own computer in the lab at the 2019 Ipro Tech Show, and the web form evaluated the query and gave the user feedback on the recall achieved immediately. Furthermore, it displayed the relevance and important keywords for each of the top 100 documents matching the query, so participants could quickly discover useful new search terms to tweak their queries. This gave participants a significant advantage over a normal e-discovery scenario, since they could try an unlimited number of queries without incurring any cost to make relevance determinations on the retrieved documents in order to decide which keywords would improve the queries. The number of participants was significantly larger than any of the previous iterations, and they had a full 20 minutes to try as many queries as they wanted. It was the best chance an audience has ever had of beating TAR. They failed.
To do a fair comparison between TAR and the keyword search results, recall values were compared for equal amounts of document review effort. In other words, for a specified amount of human labor, which approach gave the best production? For the search queries, the top 3,000 documents matching the query were evaluated to determine the number that were relevant so recall could be computed (the full population was reviewed in advance, so the relevance of all documents was known). That was compared to the recall for a TAR 3.0 process where 200 cluster centers were reviewed for training and then the top-scoring 2,800 documents were reviewed. If the system was allowed to continue learning while the top-scoring documents were reviewed, the result was called “TAR 3.0 CAL.” If learning was terminated after review of the 200 cluster centers, the result was called “TAR 3.0 SAL.” The process was repeated with review of 6,000 documents instead of 3,000 so you can see how much recall improves if you double the review effort. Participants could choose to submit queries for any of three topics: biology, medical industry, or law.
The results below labeled “Avg Participant” are computed by finding the highest recall achieved by each participant and averaging those values together. These are surely somewhat inflated values since one would probably not go through so many iterations of honing the queries in practice (especially since evaluating the efficacy of a query would normally involve considerable labor instead of being free and instantaneous), but I wanted to give the participants as much advantage as I could and including all of the queries instead of just the best ones would have biased the results to be too low due to people making mistakes or experimenting with bad queries just to explore the documents. The results labeled “Best Participant” show the highest recall achieved by any participant (computed separately for Top 3,000 and Top 6,000, so they may be different queries).
| Biology | Recall | |
| Top 3,000 | Top 6,000 | |
| Avg Participant | 54.5 | 69.5 |
| Best Participant | 66.0 | 83.2 |
| TAR 3.0 SAL | 72.5 | 91.0 |
| TAR 3.0 CAL | 75.5 | 93.0 |
| Medical | Recall | |
| Top 3,000 | Top 6,000 | |
| Avg Participant | 38.5 | 51.8 |
| Best Participant | 46.8 | 64.0 |
| TAR 3.0 SAL | 67.3 | 83.7 |
| TAR 3.0 CAL | 80.7 | 88.5 |
| Law | Recall | |
| Top 3,000 | Top 6,000 | |
| Avg Participant | 43.1 | 59.3 |
| Best Participant | 60.5 | 77.8 |
| TAR 3.0 SAL | 63.5 | 82.3 |
| TAR 3.0 CAL | 77.8 | 87.8 |
As you can see from the tables above, the best result for any participant never beat TAR (SAL or CAL) when there was an equal amount of document review performed. Furthermore, the average participant result for Top 6,000 never beat the TAR results for Top 3,000, though the best participant result sometimes did, so TAR typically gives a better result even with half as much review effort expended. The graphs below show the best results for each participant compared to TAR in blue. The numbers in the legend are the ID numbers of the participants (the color for a particular participant is not consistent across topics). Click the graph to see a larger version.
The large number of people attempting the biology topic was probably due to it being the default, and I illustrated how to use the software with that topic.
One might wonder whether the participants could have done better if they had more than 20 minutes to work on their queries. The graphs below show the highest recall achieved by any participant as a function of time. You can see that results improved rapidly during the first 10 minutes, but it became hard to make much additional progress beyond that point. Also, over half of the audience continued to submit queries after the 20 minute contest, while I was giving the remainder of the presentation. 40% of the queries were submitted during the first 10 minutes, 40% were submitted during the second 10 minutes, and 20% were submitted while I was talking. Since there were roughly the same number of queries submitted in the second 10 minutes as the first 10 minutes, but much less progress was made, I think it is safe to say that time was not a big factor in the results.
In summary, even with a large pool of participants, ample time, and the ability to hone search queries based on instant feedback, nobody was able to generate a better production than TAR when the same amount of review effort was expended. It seems fair to say that keyword search often requires twice as much document review to achieve a production that is as good as what you would get TAR.