Description
SpanBERT is an advanced AI model developed by Facebook Research, designed to enhance pre-training techniques by focusing on the representation and prediction of text spans. This repository provides the necessary code and models for researchers and developers to utilize and evaluate SpanBERT's capabilities.
Read More
The project offers pre-trained models in both base and large cased versions. These models share the same configuration as BERT but incorporate distinct masking schemes and training objectives, as detailed in the accompanying paper. The SpanBERT (base & cased) model features a 12-layer architecture with 768 hidden units and 12 heads, totaling 110 million parameters. The larger SpanBERT (large & cased) model boasts a 24-layer architecture, 1024 hidden units, 16 heads, and 340 million parameters. These models are designed for seamless integration, adopting the format of HuggingFace BERT models, allowing for easy substitution.
This repository includes comprehensive code for fine-tuning SpanBERT on a variety of Natural Language Processing (NLP) tasks. Specific scripts are provided for fine-tuning on SQuAD 1.1 and SQuAD 2.0 question answering datasets, TACRED for relation extraction, and MRQA benchmarks which encompass datasets like NewsQA, TriviaQA, SearchQA, HotpotQA, and NaturalQuestions. Additionally, code for fine-tuning on GLUE tasks, such as RTE, is available. For coreference resolution tasks, a separate TensorFlow implementation is linked.
Performance benchmarks are presented, showcasing SpanBERT's superior results compared to BERT on SQuAD 1.1, SQuAD 2.0, Coref, and TACRED datasets. For instance, SpanBERT (large) achieves an F1 score of 94.6 on SQuAD 1.1 and 88.7 on SQuAD 2.0, outperforming BERT (large). The project also facilitates the download of fine-tuned models for downstream tasks, simplifying the adoption process for users.
SpanBERT is released under the CC-BY-NC 4.0 license, which applies to both the code and the pre-trained models. This makes it accessible for non-commercial research and development. The repository encourages contributions and provides contact information for inquiries and support.
- CategoryAI Models & LLMs
- Pricing ModelOpen source
- DeploymentCloud, Self-hosted
- PlatformsWeb
SpanBERT's Core Features
Pre-trained SpanBERT models (base and large, cased)
Code for using and evaluating SpanBERT
Fine-tuning scripts for SQuAD 1.1 and SQuAD 2.0
Fine-tuning scripts for TACRED relation extraction
Fine-tuning scripts for MRQA datasets (NewsQA, TriviaQA, etc.)
Fine-tuning scripts for GLUE tasks (RTE)
Compatibility with HuggingFace BERT model formats
Coreference resolution fine-tuning code available
Performance benchmarks against BERT
Downloadable fine-tuned models for downstream tasks
SpanBERT: Improving Pre-training by Representing and Predicting Spans paper implementation
Getting Started with SpanBERT
Access code: Clone the SpanBERT GitHub repository.
Set up environment: Install required dependencies, including Apex (specific commit recommended).
Load pre-trained models: Utilize provided scripts to load base or large cased SpanBERT models.
Fine-tune models: Execute provided Python scripts for tasks like SQuAD, TACRED, MRQA, or GLUE.
Integrate via API: Adapt model usage for custom applications, leveraging HuggingFace compatibility.
Evaluate performance: Run evaluation scripts to assess model performance on specific NLP tasks.
Download fine-tuned models: Use provided scripts to obtain pre-trained models for downstream tasks.
SpanBERT's Use Cases
- Question Answering
- Relation Extraction
- Coreference Resolution
- Natural Language Understanding
- Text Span Prediction
- Research and Development
- Benchmark Evaluation
