University of Illinois Chicago
Browse

Scientific Natural Language Inference

thesis
posted on 2024-08-01, 00:00 authored by Mobashir Sadat
Natural Language Inference (NLI) aims at recognizing the semantic relationship between a pair of sentences---whether one sentence entails the other sentence, contradicts it, or they are semantically independent. NLI was introduced in the literature to facilitate the evaluation of Natural Language Understanding (NLU) that significantly impacts the performance of many natural language processing tasks such as text summarization, question answering, information retrieval, and commonsense reasoning. Despite that substantial progress has been made on NLI (with developments in terms of datasets and models), several challenges remain. Most existing works on NLI focus on a general domain (e.g., image captions or face-to-face conversations) and ignore specialized domains e.g., the scientific domain. The scientific text captured in research papers brings additional challenges not only in terms of the language, vocabulary and more complex sentence structure but also the inferences that exist in it. For example, a sentence can present the reasoning behind the conclusion made in the previous sentence, while other sentences indicate a contrast or entailment with the preceding sentence. Thus, the traditional NLI task is not suitable to track the progress of NLU of models on scientific text. Furthermore, existing methods for modeling NLI rely on the availability of large amounts of human annotated data, which may be infeasible to obtain for a domain of interest. In this thesis, we specifically address the above challenges. First, we propose a new task, scientific NLI that aims at predicting the semantic relation between a pair of sentences extracted from research articles. To capture the inference relations which are prevalent in scientific text but are unavailable in the traditional NLI task, we introduce two new classes for scientific NLI---Contrasting and Reasoning. We thoroughly study this task across a diverse range of scientific domains and find that: a) scientific NLI is indeed more suitable to track the progress of NLU on scientific text; b) scientific NLI is more challenging than traditional NLI; and c) unlike traditional NLI, scientific NLI can aid in improving the performance of downstream tasks in the scientific domain. Next, we propose two novel semi-supervised learning (SSL) approaches for NLI. SSL is a popular technique for reducing the reliance on expensive human annotation by leveraging unlabeled data for training. However, despite its substantial success on single sentence classification tasks where the challenge in making use of unlabeled data is to assign "good enough'' pseudo-labels, for NLI tasks, the nature of unlabeled data is more complex: one of the sentences in the pair along with the class label are missing from the data and require human annotations, which makes SSL for NLI more challenging. Thus, we propose an SSL framework for NLI where the missing sentence is generated using a Large Language Model (LLM), and then a self-training method is employed to assign high quality pseudo-labels which are utilized in further model training. Although the self-training method shows promising performance, we observe that the reliance on a single model results in error accumulation in the pseudo-labels. Thus, we propose a novel co-training approach for NLI in which we train two different classifiers simultaneously. Unlike existing co-training approaches which exchange pseudo-labels between the classifiers, we exchange the classifiers' beliefs in the quality of the pseudo-labels, and encourage a divergence between them to ensure that they learn complementary information from the data. Both self-training and co-training approaches utilize a small amount of human labeled data and leverage a large amount of automatically labeled data to further improve the performance on NLI, thereby reducing human annotation effort.

History

Advisor

Cornelia Caragea

Department

Computer Science

Degree Grantor

University of Illinois Chicago

Degree Level

  • Doctoral

Degree name

Doctor of Philosophy

Committee Member

Elena Zheleva Natalie Parde Fabio Miranda Doina Caragea

Thesis type

application/pdf

Language

  • en

Usage metrics

    Categories

    No categories selected

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC