ChatGPT: Revolutionising Robotics with Natural-Language Commands

ChatGPT is revolutionising robotics.

It unlocks a new paradigm that replaces coding with natural-language commands.

This is how Microsoft is opening the door to an unexpected future:

— Alex Banks (@thealexbanks) March 2, 2023

Microsoft has used ChatGPT for:

• Controlling a drone
• Moving a robot arm
• Acting as a home assistant

This is only the beginning of what's possible.

Let's dive in. pic.twitter.com/epbLQVSPCA

— Alex Banks (@thealexbanks) March 2, 2023

1/ Challenges in robotics today

Current robotics requires an engineer "in the loop".

They need to translate a task's requirements into code.

• Slow — you need to write low-level code
• Expensive — requires skilled robotics knowledge
• Inefficient — multiple interactions pic.twitter.com/gFWw8GzhxK

— Alex Banks (@thealexbanks) March 2, 2023

2/ How ChatGPT can help

ChatGPT unlocks a new robotics paradigm.

Instead of a technical engineer, a non-technical user can sit "on the loop".

They provide high-level feedback to the large language model (LLM) while monitoring the robot's performance. pic.twitter.com/ITqSfGfSC8

— Alex Banks (@thealexbanks) March 2, 2023

3/ Controlling a drone

This is an example of zero-shot task planning:

The model performs a task without any specific training on that task.

ChatGPT wrote complex code structures for the drone.

This allowed it to do a zig-zag pattern to visually inspect the shelves. pic.twitter.com/pRGZJ9bRg5

— Alex Banks (@thealexbanks) March 2, 2023

4/ Moving a robot arm

ChatGPT not only recalled the logo from its internal knowledge base.

It was able to ‘draw’ the logo as SVG code.

Then used its learned skills to determine which existing robot actions can create its physical form. pic.twitter.com/Y0vzq9HKe4

— Alex Banks (@thealexbanks) March 2, 2023

5/ Acting as a home assistant

This is an example of perception-action loops:

Robots that sense the world before they act.

A textual description of the camera image at each step was fed into the chat.

The model was able to control the robot until it reached an object. pic.twitter.com/BGdPdGmLqs

— Alex Banks (@thealexbanks) March 2, 2023

Good prompt engineering is crucial for the success of LLMs such as ChatGPT for robotics tasks.

Luckily Microsoft recently introduced "PromptCraft".

This is an open-source platform where anyone can share examples of prompting strategies for different robotics categories.

— Alex Banks (@thealexbanks) March 2, 2023

They also released a robotics simulator: ChatGPT-AirSim.

This is built on Microsoft AirSim with ChatGPT integration.

Anyone can get started with these ideas.

I believe integrating natural language interfaces—such as ChatGPT—will allow robots to be used by everyone.

— Alex Banks (@thealexbanks) March 2, 2023

Follow me @thealexbanks for more on AI.

If you liked this thread, you'll love the newsletter.

Subscribe here:https://t.co/Kre1UavZe3

— Alex Banks (@thealexbanks) March 2, 2023

Help everyone learn and retweet this thread: https://t.co/YAJEKLxU4y

— Alex Banks (@thealexbanks) March 2, 2023

You can check out PromptCraft here:https://t.co/Pm97A451zY

And ChatGPT-AirSim here:https://t.co/pVGYPm2jbs

— Alex Banks (@thealexbanks) March 2, 2023