Google releases music model Magenta RealTime 2, local latency on Mac is below 200 milliseconds

According to Beating Monitoring, Google DeepMind has released an open-source real-time music generation model, Magenta RealTime 2 (abbreviated as MRT2). The new model is optimized for Apple-chip (M series) MacBooks and supports local ultra-low-latency operation. Users can use MIDI signals, text prompts, or audio snippets to perform and control the model as an instrument in real time with latency below 200 milliseconds.

Musicians can guide music generation through a variety of interaction methods. In the MIDI-guided mode, the accompaniment system automatically generates accompaniment combinations such as orchestral arrangements based on the chords the performer plays. In the text-to-synthesizer mode, users can enter style descriptions such as “disco funk” to generate playable MIDI instruments. In the audio cloning mode, users only need to drag in a short audio sample, and the system can simulate the corresponding timbre. The sound generation process also supports dynamic modulation via low-frequency oscillators, MIDI controllers, or even camera input, providing rich interaction dimensions for real-time sound design.

To meet the needs of different hardware configurations, the development team has released two versions with different parameter specifications. The 230 million-parameter mrt2_small version enables smooth real-time inference on all Apple-chip computers, including MacBook Air, while the 2.4 billion-parameter mrt2_base version offers higher sound quality but requires high-end M series Pro or Max chips to ensure real-time audio-stream synthesis.

At present, the inference library magenta-rt and the C++-based inference engine magentart::core have both been open-sourced on GitHub. The official team also provides standalone applications and digital audio workstation (DAW) plugins that can be run directly on macOS.