bigscience/bloom

Description

BLOOM, developed by BigScience, is an autoregressive large language model (LLM) designed to advance and democratize artificial intelligence through open-source and open science. Released on July 11, 2022, this model is capable of generating coherent text across 46 natural languages and 13 programming languages, making it a versatile tool for various applications in natural language processing.

The architecture of BLOOM is based on a transformer model, modified from Megatron-LM GPT2. It features 176 billion parameters, with 70 layers and 112 attention heads, allowing it to process and generate text effectively. The model was trained on a substantial dataset of 1.6TB of pre-processed text, which includes 350 billion unique tokens. This extensive training enables BLOOM to perform text generation tasks, including information extraction, question answering, and summarization, even for tasks it hasn't been explicitly trained for.

BLOOM is intended for public research on large language models, making it accessible for researchers, educators, and developers. It can be easily integrated into applications using Hugging Face's ecosystem, requiring the installation of transformers and accelerate. The model's intended users include the general public, researchers, students, and engineers, among others. However, it is essential to note that the model should not be used in high-stakes settings or for critical decision-making, as it may produce outputs that appear factual but can be incorrect.

The training of BLOOM was conducted on the Jean Zay Public Supercomputer, utilizing 384 A100 80GB GPUs, and was funded by the French government. The environmental impact of the training process is mitigated through the use of nuclear energy, with heat generated being reused for campus housing. Overall, BLOOM represents a significant step forward in the field of AI, providing a powerful tool for language generation and research.

bigscience/bloom Highlights

• Model Type: Transformer-based Language Model

• Parameters: 176B

• Languages Supported: 46 natural languages, 13 programming languages

• License: RAIL License v1.0

• Release Date: July 11, 2022

• Training Data Size: 1.6TB

• Training Tokens: 350B unique tokens

• Compute Infrastructure: Jean Zay Public Supercomputer

Getting Started with bigscience/bloom

1. 1

   Access page: Visit the Hugging Face website for BLOOM.

2. 2

   Load model: Download the BLOOM model using Hugging Face's transformers library.

3. 3

   Configure environment: Ensure transformers and accelerate are installed in your environment.

4. 4

   Integrate: Use the model for text generation tasks as needed.

5. 5

   Fine-tune: Optionally fine-tune the model for specific applications.

bigscience/bloom's Use Cases

• Text Generation
• Information Extraction
• Question Answering
• Summarization
• Language Research

FAQ from bigscience/bloom

• What is BLOOM?

  BLOOM is a multilingual autoregressive large language model developed by BigScience.

• How many languages does BLOOM support?

  BLOOM supports 46 natural languages and 13 programming languages.

• What is the training data size for BLOOM?

  BLOOM was trained on 1.6TB of pre-processed text.

• What is the release date of BLOOM?

  BLOOM was released on July 11, 2022.

• What type of license does BLOOM have?

  BLOOM is licensed under the RAIL License v1.0.

• Who developed BLOOM?

  BLOOM was developed by BigScience.

• What is the purpose of BLOOM?

  BLOOM is intended for public research on large language models.

• Can BLOOM be used for critical decision-making?

  No, BLOOM should not be used in high-stakes settings or for critical decisions.