SpanBERT

Description

SpanBERT is an advanced AI model developed by Facebook Research, designed to enhance pre-training techniques by focusing on the representation and prediction of text spans. This repository provides the necessary code and models for researchers and developers to utilize and evaluate SpanBERT's capabilities.

The project offers pre-trained models in both base and large cased versions. These models share the same configuration as BERT but incorporate distinct masking schemes and training objectives, as detailed in the accompanying paper. The SpanBERT (base & cased) model features a 12-layer architecture with 768 hidden units and 12 heads, totaling 110 million parameters. The larger SpanBERT (large & cased) model boasts a 24-layer architecture, 1024 hidden units, 16 heads, and 340 million parameters. These models are designed for seamless integration, adopting the format of HuggingFace BERT models, allowing for easy substitution.

This repository includes comprehensive code for fine-tuning SpanBERT on a variety of Natural Language Processing (NLP) tasks. Specific scripts are provided for fine-tuning on SQuAD 1.1 and SQuAD 2.0 question answering datasets, TACRED for relation extraction, and MRQA benchmarks which encompass datasets like NewsQA, TriviaQA, SearchQA, HotpotQA, and NaturalQuestions. Additionally, code for fine-tuning on GLUE tasks, such as RTE, is available. For coreference resolution tasks, a separate TensorFlow implementation is linked.

Performance benchmarks are presented, showcasing SpanBERT's superior results compared to BERT on SQuAD 1.1, SQuAD 2.0, Coref, and TACRED datasets. For instance, SpanBERT (large) achieves an F1 score of 94.6 on SQuAD 1.1 and 88.7 on SQuAD 2.0, outperforming BERT (large). The project also facilitates the download of fine-tuned models for downstream tasks, simplifying the adoption process for users.

SpanBERT is released under the CC-BY-NC 4.0 license, which applies to both the code and the pre-trained models. This makes it accessible for non-commercial research and development. The repository encourages contributions and provides contact information for inquiries and support.

SpanBERT Highlights

• Pre-trained SpanBERT models (base and large, cased)

• Code for using and evaluating SpanBERT

• Fine-tuning scripts for SQuAD 1.1 and SQuAD 2.0

• Fine-tuning scripts for TACRED relation extraction

• Fine-tuning scripts for MRQA datasets (NewsQA, TriviaQA, etc.)

• Fine-tuning scripts for GLUE tasks (RTE)

• Compatibility with HuggingFace BERT model formats

• Coreference resolution fine-tuning code available

• Performance benchmarks against BERT

• Downloadable fine-tuned models for downstream tasks

• SpanBERT: Improving Pre-training by Representing and Predicting Spans paper implementation

Getting Started with SpanBERT

1. 1

   Access code: Clone the SpanBERT GitHub repository.

2. 2

   Set up environment: Install required dependencies, including Apex (specific commit recommended).

3. 3

   Load pre-trained models: Utilize provided scripts to load base or large cased SpanBERT models.

4. 4

   Fine-tune models: Execute provided Python scripts for tasks like SQuAD, TACRED, MRQA, or GLUE.

5. 5

   Integrate via API: Adapt model usage for custom applications, leveraging HuggingFace compatibility.

6. 6

   Evaluate performance: Run evaluation scripts to assess model performance on specific NLP tasks.

7. 7

   Download fine-tuned models: Use provided scripts to obtain pre-trained models for downstream tasks.

SpanBERT's Use Cases

• Question Answering
• Relation Extraction
• Coreference Resolution
• Natural Language Understanding
• Text Span Prediction
• Research and Development
• Benchmark Evaluation

FAQ from SpanBERT

• What is SpanBERT and how does it improve pre-training?

  SpanBERT is an AI model that enhances pre-training by focusing on representing and predicting spans of text, leading to improved performance on various NLP tasks compared to traditional BERT models.

• What types of pre-trained SpanBERT models are available?

  The repository offers pre-trained SpanBERT models in both base and large cased versions, with configurations similar to BERT but with distinct masking and training objectives.

• How can I use SpanBERT for question answering?

  You can use the provided Python scripts to fine-tune SpanBERT on SQuAD 1.1 and SQuAD 2.0 datasets for question answering tasks.

• Is SpanBERT compatible with HuggingFace models?

  Yes, SpanBERT models are designed to have the same format as HuggingFace BERT models, allowing for easy integration and replacement.

• What are the parameter counts for SpanBERT models?

  SpanBERT (base & cased) has 110 million parameters, while SpanBERT (large & cased) has 340 million parameters.

• Can SpanBERT be used for relation extraction?

  Yes, the repository includes code for fine-tuning SpanBERT on the TACRED dataset for relation extraction tasks.

• What is the license for SpanBERT?

  SpanBERT and its pre-trained models are released under the CC-BY-NC 4.0 license, which permits non-commercial use.

• Where can I find code for coreference resolution with SpanBERT?

  A separate TensorFlow implementation for coreference resolution is available at https://github.com/mandarjoshi90/coref.

• How do I evaluate SpanBERT's performance on downstream tasks?

  You can evaluate the models by setting '--do_train' to false and '--do_eval' to true in the respective Python fine-tuning scripts.

• What datasets are covered by the MRQA fine-tuning script?

  The MRQA fine-tuning script supports datasets such as NewsQA, TriviaQA, SearchQA, HotpotQA, and NaturalQuestions.

• Can I download fine-tuned SpanBERT models?

  Yes, you can download fine-tuned models for tasks like SQuAD 1.1/2.0 and TACRED using the provided script.

• What is the primary benefit of using SpanBERT over BERT?

  SpanBERT offers improved performance on various NLP tasks due to its novel approach to representing and predicting spans during pre-training.