Original source: AIGC Open Community
Image source: Generated by Unbounded AI
On October 27, Boston Dynamics, the world's top robotics developer, showed off a new study on its website that developed a talking guide-guided robot dog by combining ChatGPT, Spot, and other AI models.
It is reported that the robot dog can talk to humans according to text and voice prompts, and provides a visual question and answer function, which can analyze the picture taken by the camera and automatically generate image descriptions.
Boston Dynamics says big-language models such as ChatGPT have demonstrated powerful control and output capabilities, which have inspired them to control the behavior and decision-making functions of physical robots. For example, entering some specific attraction data, pictures, etc., allows the robot to provide logical and organized tour guide functions.
In addition, this also enhances the anthropomorphic ability of physical robots, such as telling you a cold joke and doing a funny action that makes you happy. If you use the traditional hair method, it will be a very difficult thing.
Principles of Robot Dog Technology for Tour Guide
Boston Dynamics uses its self-developed quadruped robot Spot as a physical robot framework to realize basic functions such as walking, monitoring, navigation, and scanning.
In order to allow the robot dog to speak and interact with humans, the developers equipped it with a Respeaker V2 speaker, a ring array microphone with LEDs, which can be connected to Spot's EAP 2 via USB for data transmission.
1) Spot EAP 2) Speaker 3) Bluetooth speaker 4) Point arm and fixture camera
A computer was used as the control brain of the robot dog and interacted with it through the Spot SDK. In order to make the robot dog have anthropomorphic actions, such as nodding and stretching its neck, Spot's point arm and clamp camera are used.
Get the robot dog talking with ChatGPT
After the physical hardware environment is built, in order to make the robot dog have dialogue function, the R & D personnel use GPT-3.5 and GPT-4 combined with the Spot SDK, and carry out simple instruction fine-tuning, so that the robot dog has the function of primary guide judgment and dialogue.
Next, to allow Spot to interact with humans and the environment, VQA and speech-to-text software are integrated. At the same time, the robot's gripper camera and front-facing body camera are fed into the BLIP-2 and displayed in a visual question-and-answer mode (e.g. "What's so interesting about this picture?"). and other simple questions) or image caption mode to run it.
The process runs approximately once per second, and the results are entered directly into the prompt.
In order to allow the robot dog to realize the function of "listening", the developers input the microphone data into OpenAI's speech model Whisper into English text.
In order for the robot dog to have a conversation with a human, it needs to convert human voice commands into text for prompting ChatGPT, so a conversion tool is also needed. After trying out a variety of tools, the developers decided to use the cloud service ElevenLabs.
Surprising phenomena in experiments
In the course of multiple test conversations, the developers found a surprising phenomenon, the robot dog seems to have simple self-decision-making ability.
For example, when you ask the robot dog who Marc Raibert (executive director of Boston Dynamics) is? It replied: "I don't know who he is, let's go to the help desk and ask." ”**
Once at the service desk, the robot dog continued to ask the service staff who Marc Raibert was?
When developers ask: Who are your parents? The robot dog walked to the Spot V1 and Big Dog display areas, and considered these robots to be its fathers. In fact, they do have a correlation**.
The robot dog also shows an interesting side, and can actively ask passers-by if there are any mysterious creatures around.
Boston Dynamics said that it will continue to optimize the functions of the product in the future. The combination of ChatGPT and physical robots opens the door for the physical landing of large language models, which can be widely used in tour guides, entertainment, logistics, companionship and other fields.
About Boston Dynamics
Boston Dynamics was founded in 1992 and is headquartered in Boston, Massachusetts. What began as a branch of the Massachusetts Institute of Technology (MIT) grew into an independent company.
Boston Dynamics is known for developing highly advanced, flexible and practical robots, and its products are used in industrial, research and consumer fields, mainly representative robots include: BigDog, Atlas, Spot, Handle, etc.
