OpenAI gives voice AI a sense of sight

Author: Li Hailun, Tencent Technology

At around 3:00 AM Beijing time on July 9, OpenAI officially launched a new generation of voice models, the GPT-Live series, which includes two versions: GPT-Live-1 and GPT-Live-1 mini. These models provide support for ChatGPT's voice features.

The core upgrade of GPT-Live is its "full-duplex architecture," which emphasizes the voice AI's understanding of conversational dynamics. The model can simultaneously process listening and speaking. During a conversation, it can indicate it is listening through short phrases, support rapid back-and-forth exchanges, and remain quiet when the user is thinking.

It is closer to the state of real human conversation, with a bit more sense of proportion in dialogue exchanges.

OpenAI officially stated: "During a conversation, GPT-Live can use responses like 'mm-hmm' or 'yeah' to indicate it is listening, and can engage in fast back-and-forth exchanges; when you need a moment to think, it can also stay quiet. The end result is a more relaxed and natural voice interaction experience."

When encountering tasks that require web search, deep reasoning, or complex operations, GPT-Live delegates the work to a backend frontier model, and brings the results back into the conversation when ready, keeping the conversation uninterrupted.

At launch, the backend model is GPT-5.5. OpenAI said it will continuously update the underlying support as new models are released.

Chat and work simultaneously

In the past, GPT relied on a real-time voice capability at its core, using a low-latency audio streaming system: the user speaks, the model quickly understands and responds; the user interrupts, the model promptly stops.

The most critical upgrade in GPT-Live this time is the separation of voice interaction and deep reasoning into two independent components.

When a user makes a request that requires search, reasoning, or invoking stronger agent capabilities, GPT-Live hands the task to the backend model, while the front-end voice model continues to maintain the conversation with the user, striving to avoid interrupting the communication rhythm.

This "delegation model" is essentially an architectural division of labor: the front end uses a voice-native model optimized for real-time interaction to handle responses, pauses, and maintaining natural conversation; the back end uses an independent reasoning model to handle search, calculations, tool invocation, and complex tasks.

The two operate independently, and the reasoning engine can be swapped out at any time as technology advances, while the voice model's experience can be upgraded without retraining.

For enterprise applications, the value of the full-duplex architecture is that the voice agent can complete backend tasks such as querying databases, web searches, and multi-step reasoning without interrupting the conversation rhythm. Previously, such operations often caused several seconds of silence, making users feel that "the system is processing." Now, these can be hidden within a more natural communication rhythm.

Evaluation data reflects the effectiveness of the separation design.

According to official information, in a head-to-head comparison between GPT-Live-1 and Advanced Voice Mode, the former significantly outperformed in overall preference, turn-taking, interruption handling, conversation fluency, and naturalness per interaction in conversations lasting 5 to 10 minutes, with an overall preference of 75.7%.

In conversation fluency tests, GPT-Live-1 scored 4.96, while Advanced Voice Mode scored 3.80. In pleasantness, GPT-Live-1 scored 5.19, and Advanced Voice Mode scored 3.82.

In the expert-level scientific reasoning test GPQA, Advanced Voice Mode scored 45.3%, GPT-Live-1 (high) reached 84.2%, and GPT-Live-1 mini scored 74.9%.

In the agent web search test BrowseComp, Advanced Voice Mode scored only 0.7%, while GPT-Live-1 (high) reached 75.2%, and GPT-Live-1 mini scored 31.6%.

The significant gap shows that the strategy of delegating reasoning tasks to the backend frontier model is indeed effective.

Additionally, OpenAI said it has re-recorded the nine voices in ChatGPT for GPT-Live.

Users can now choose three reasoning levels for answers: instant mode for quick responses, and medium and high-intensity modes for scenarios requiring more thought.

During conversations, ChatGPT can display visual cards for topics like weather, stocks, and sports, allowing users to browse while chatting. Voice features continue to support search, memory, images, and file uploads.

From a product evolution perspective, ChatGPT voice has developed from a basic feature into an independent product experience. Users use it to practice languages, tell bedtime stories, chat during commutes, or get hands-free daily help.

The release of GPT-Live makes this experience closer to the vision OpenAI has described: collaborating with AI as smoothly and responsively as working with a real person, while reasoning and complex tasks happen seamlessly in the background.

Safety

In voice conversations, safety boundaries are also very important. Especially in real-time conversations, the model must not only judge what the user says, but also handle the complex risks brought by tone, emotion, interruptions, and continuous questioning.

To this end, OpenAI has conducted targeted training in key risk areas based on existing safety work, and has designed a separate set of protective measures specifically for the voice scenario.

Regarding safety testing, OpenAI expanded audio-native evaluations, adding real user voice samples and audio prompts generated comprehensively for areas such as self-harm, psychosis and mania, emotional dependence on AI, violence, and sexual content.

Internal experts also conducted red team testing on the model for voice-specific risks.

In a comprehensive evaluation, GPT-Live-1 showed significant improvements over Advanced Voice Mode in several areas: preventing illegal acts rose from 0.63 to 0.97, preventing self-harm from 0.72 to 0.98, and preventing hate speech from 0.87 to 1.00.

In more ambiguous production prompt evaluations, GPT-Live-1 was comparable or better than Advanced Voice Mode in most categories, but emotional dependence dropped slightly from 0.88 to 0.82. OpenAI noted that this change is not statistically significant.

In terms of real-time protection, when the system detects potentially unsafe output, it can intervene while the model is speaking, guiding toward a safer response, displaying additional safety information or resources, or ending the voice conversation in high-risk situations.

For example, regarding topics involving "self-harm," ChatGPT's support process has been adjusted for the voice environment, including providing expert-reviewed crisis hotline support.

For youth protection, OpenAI designed additional measures, training age-appropriate behavior into the model. Parents can use parental controls to decide whether teenagers can use ChatGPT voice, and in high-risk situations, the associated parent may receive a notification.

Notably, OpenAI has introduced long-term measurement and post-deployment monitoring for emotional dependence, based on previous research on emotional usage and emotional well-being. This is an acknowledgment that the conversational naturalness GPT-Live aims for may itself bring new risks.

Additionally, GPT-Live uses predefined voices in ChatGPT, with safeguards against imitating real human voices.

The background of this statement is the controversy in May 2024 during the release of GPT-4o, when the "Sky" voice resembled actress Scarlett Johansson's voice. Johansson said she had declined an invitation to voice the system, and after the product launch, she felt "shocked, angered, and incredulous." OpenAI subsequently removed the voice and apologized.

Three generations of evolution

Currently, full-duplex voice interaction is rapidly becoming a standard feature of consumer AI products.

Google's Gemini Live already supports full-duplex conversations as well as camera and screen sharing, the latter two of which were not supported at GPT-Live's launch. In March, Google released Gemini 3.1 Flash Live, providing low-latency voice interaction for developers. Also, NVIDIA's PersonaPlex, released in January, brings customizable voice and character control into full-duplex models.

Let's review the three generations of evolution in ChatGPT voice technology over roughly two years.

The initial ChatGPT voice feature launched in 2023, using a cascading system that connected three models: a voice-to-text model transcribed speech to text, a large language model generated text responses, and a text-to-speech model converted the response back to audio.

In this architecture, OpenAI used the Whisper speech recognition model for voice-to-text, GPT-4 as the underlying language model, and a companion TTS model for final speech synthesis.

This method enabled conversations with a frontier AI model for the first time, but at a significant cost: each handoff introduced latency and could lose information, making responses slow and rigid.

Then, in May 2024, OpenAI released GPT-4o, the company's first native multimodal model capable of directly processing audio input and output, no longer requiring external voice-to-text and text-to-speech modules. The Advanced Voice Mode built on GPT-4o was released to paying users in a "limited" manner in July 2024, and more broadly in September.

However, it still operated in discrete turns. The model had to wait for the user to stop speaking before responding. Turn detection relied on silence detection, and short pauses or background noise could be misinterpreted as the end of the turn, causing the model to interrupt at inappropriate times.

Now, GPT-Live solves this problem with its full-duplex architecture, able to generate output while continuously processing input. The model can make multiple interaction decisions per second: speak, continue listening, pause, interrupt, or call tools.

This allows the model to naturally insert conversational acknowledgments, recognize natural pauses without jumping in prematurely, and handle rapid interruptions without derailing the conversation.

When you need a moment to think, ChatGPT waits instead of jumping in. Under background noise such as passing traffic or nearby conversations, ChatGPT can better focus on the user's voice.

From two years ago, speaking into a microphone and waiting nearly two seconds for a rigid reply, to a year ago with smoother but still turn-based experience, to today's full-duplex interaction close to real conversation, the trajectory of technological evolution is clear.

GPT-Live may not be the end point, but it brings the end point closer.