Automatic Enemy Item Detection Using Natural Language Processing

This study explores the effectiveness of using Natural Language Processing (NLP) techniques in automatically detecting enemy item pairs within item banks. The overarching goal was to compare and evaluate the performance of various automatic enemy item detection procedures using an operational item pool. To answer the research questions, this study examined the classification results across three conditions: (a) NLP techniques, including Vector Space Model, Latent Semantic Analysis, and Latent Dirichlet Allocation; (b) classification algorithms, including the logistic regression classifier and the Artificial Neural Network classifier, and (c) probability cutoffs ranging from .60 to .90. The classification results were further evaluated by subject matter experts (SMEs), and the models were re-trained using the input from the SMEs. The findings from this study showed the robustness of the NLP techniques in automatic identification of enemy item pairs. The automatic detection process successfully identified additional enemy relationships previously untagged in the item bank. The classification results from the numerous conditions suggested that the LSA and the VSM models consistently outperformed the LDA models and yielded optimal results at the cutoff of .90. Integrating feedback from SMEs further improved the performance. This iterative process greatly reduced the time and manual labor needed for enemy relationship monitoring and offered flexibility for SME review.