huggingface/alignment-handbook is a repository that collects training recipes for aligning language models with human and AI preferences. Alignment refers to training that shifts a model's answers in the direction that people or other AI prefer more. There is no public latest release; it is written in Python and published under the Apache-2.0 license.
The scripts directory holds procedures for model training and evaluation: continued pretraining, SFT, DPO, and ORPO. Continued pretraining is the stage that teaches a finished model additional knowledge, while SFT, DPO, and ORPO are training procedures that align the model's responses. Training scripts, configs, and data processing methods are bundled together, so the fine-tuning process can be reproduced. Full-model training supports DeepSpeed ZeRO-3 and LoRA/QLoRA. ZeRO-3 is a way of splitting the memory needed for training across multiple devices, and LoRA/QLoRA is a way of training only small additional modules while keeping the body of a large model frozen. The recipes directory contains YAML configs that reproduce models such as Zephyr 7B, and also explains how to train a chat model with your own dataset. The repository was created on August 25, 2023 and has 5697 stars.
However, no benchmark figures demonstrating the performance of the alignment training are given. The freshness, technology, and quality scores are left blank, and only the momentum score remains, at a low 1.0. So this repository is worth treating as reference material for trying out an alignment training procedure, but it is hard to judge alignment quality from this repository alone.