Transformers

Description

State-of-the-art Machine Learning for PyTorch, TensorFlow and Jax.

Home page

https://github.com/huggingface/transformers

Documentation

https://huggingface.co/docs/transformers/

License

Apache License 2.0

Usage

Transformers uses the PyTorch backend and should (ideally) be run on the GPU nodes (using the "cuda" module). It's configured for offline use and points to a local Huggingface cache.

Use

module avail Transformers

to see which versions of Transformers are available. Use

module load Transformers/version

to get access to Transformers.

Huggingface cache

The module sets environment variables (HF_DATASETS_OFFLINE, HF_HUB_OFFLINE, TRANSFORMERS_OFFLINE) for offline usage in TSD.
The module sets "HB_HOME=/cluster/software/huggingface" to point to a local repository for models and data. Models and data are distributed under various licenses, so please read the license for your models prior to usage. To add models and data to the central repository submit a request to tsd-drift.

You can list the available models using the following command:

huggingface-cli scan-cache

Overwrite the "HB_HOME" environment variable to point to your own imported models/data:

export HB_HOME=/path/to/your/model

Example

Please note that the example below is model specific. Scripts will require modification to fit the needs of your analysis and data. The PyTorch default device is the cpu, so the cuda device needs to be set in the code. Please consult the model documentation on Huggingface for details and examples.

The following example from Huggingface will help you to get started. Create the following Slurm script "transformers.sm":

#!/bin/bash
#SBATCH --job-name=transformers
#SBATCH --account=pXX_tsd
#SBATCH --partition=accel
#SBATCH --gres=gpu:1
#SBATCH --time=0:05:00
#SBATCH --mem-per-cpu=4G

set -o errexit
module purge --quiet
module load Transformers/4.37.2-foss-2021a-CUDA-11.3.1

python transformers_lmm.py

Then create the "transformers_lmm.py" file:

from transformers import AutoModelForCausalLM
from transformers import AutoTokenizer

# Specify the model
model_id = "mistralai/Mistral-7B-v0.1"

# Set "local_files_only" to skip online lookup attempts
model = AutoModelForCausalLM.from_pretrained(
    model_id, device_map="auto", load_in_4bit=True, local_files_only=True
)
tokenizer = AutoTokenizer.from_pretrained(model_id, padding_side="left", local_files_only=True)

print('... processing single input data')

model_inputs = tokenizer(["A list of colors: red, blue"], return_tensors="pt").to("cuda")
generated_ids = model.generate(**model_inputs, max_new_tokens=50)
print(tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0])

print('... processing batch input data')

# Most LLMs don't have a pad token by default
tokenizer.pad_token = tokenizer.eos_token
model_inputs = tokenizer(
    ["A list of colors: red, blue", "Portugal is"], return_tensors="pt", padding=True
).to("cuda")
generated_ids = model.generate(**model_inputs, max_new_tokens=50)
print(tokenizer.batch_decode(generated_ids, skip_special_tokens=True))

print('... done')

Finally submit the job:

sbatch transformers.sm

Which returns something along the lines of:

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /cluster/software/EASYBUILD/Transformers/4.37.2-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda113.so
CUDA SETUP: CUDA runtime path found: /cluster/software/EASYBUILD/CUDA/11.3.1/lib/libcudart.so.11.0
CUDA SETUP: Highest compute capability among GPUs detected: 8.0
CUDA SETUP: Detected CUDA version 113
CUDA SETUP: Loading binary /cluster/software/EASYBUILD/Transformers/4.37.2-foss-2021a-CUDA-11.3.1/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda113.so...
Loading checkpoint shards: 100%|¨€¨€¨€¨€¨€¨€¨€¨€¨€¨€| 2/2 [00:41<00:00, 20.85s/it]
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.

... processing single input data
A list of colors: red, blue, green, yellow, orange, purple, pink, brown, black, white, gray, silver, gold, tan, beige, cream, ivory, tan, and more.

... processing batch input data
['A list of colors: red, blue, green, yellow, orange, purple, pink, brown, black, white, gray, silver, gold, tan, beige, cream, ivory, tan, and more.\n\nA list of colors: red, blue, green', 'Portugal is a country in southwestern Europe, on the Iberian Peninsula. It is the westernmost country of mainland Europe, being bordered by the Atlantic Ocean to the west and south and by Spain to the north and east.\n\nPort']

... done