AMD releases fully open-source MoE large language model Instella-MoE, with a 16B-parameter scale challenging mainstream open-source models

BlockBeats message, July 25: AMD announced the release of a fully open-source mixture-of-experts (MoE) model, Instella-MoE. The model has 16 billion total parameters, with 2.8 billion parameters activated per token, claiming leading performance among open-source language models of the same scale.

AMD said Instella-MoE was trained from scratch entirely on its own AMD Instinct MI300X and MI325X GPUs, using the ROCm software stack, and it adopts architectural innovations such as Gated Multi-head Latent Attention (Gated MLA) and FarSkip-Collective to improve training and inference efficiency.

Performance tests show that the Instella-MoE-16B-A3B base model achieved an average score of 76.7, ranking among the leading models among fully open-source models. It surpasses models such as SmolLM3-3B and OLMo-3-7B, and while activating only 2.8 billion parameters, it competes with larger-scale models.

In addition, the model supports 64K token long-context processing and completes an end-to-end training pipeline, including pretraining, mid-stage training, long-context extension, supervised fine-tuning (SFT), direct preference optimization (DPO), and reinforcement learning (RL).

AMD also released all Instella-MoE model weights, training configurations, data mixing ratios, intermediate checkpoints, and inference code at this time, to promote open AI model research and reproducibility.

AMD said Instella-MoE demonstrates the capability to train large-scale MoE models based on AMD hardware and an open software ecosystem. In the future, AMD will continue to advance the development of larger-scale, stronger inference performance, and higher-efficiency open-source language models.