nvidia/megatron-lm is NVIDIA's public repository for training transformer models across multiple GPUs. Its GitHub description says it continues to research and train transformer models at scale, and the latest release of this repository is core_v0.19.2.
The repository is split into two parts. Megatron Core provides GPU-optimized components, transformer components and model architectures so users can build custom training pipelines, while Megatron-LM is a reference example of pre-configured training scripts that use that core. Parallelism covers tensor (TP), pipeline (PP), data (DP), expert (EP) and CP, and the mixed-precision training mixes FP16, BF16, FP8 and FP4. The use of this repository is that you can study, in the form of ready-made parts, how to spread training across several GPUs. The primary language is Python, the GitHub star count is 18216, and it was created on March 21, 2019.
However, the collected material contains no changelog for core_v0.19.2, and no performance figures or benchmarks are given, so what this release changed cannot be confirmed. The license is only recorded as NOASSERTION, so the actual terms of use cannot be known, and on the 10-point scale the technology, quality and novelty scores are blank while only momentum is 4.7. All that can be judged at this point is that parallelization components for large-scale training have been made public.