ChatGPT: Revolutionising Robotics with Natural-Language Commands
ChatGPT is revolutionising robotics by replacing coding with natural-language commands. This is how Microsoft is opening the door to an unexpected future. Let's dive in to find out what's possible.
![Author profile image](https://pbs.twimg.com/profile_images/1635316385260744705/h9sFTnKS_normal.jpg)
Alex Banks
Building a better future with AI.
![Header media for article](https://pbs.twimg.com/tweet_video_thumb/FqN9w0QWwAAvLFu.jpg)
-
ChatGPT is revolutionising robotics.
— Alex Banks (@thealexbanks) March 2, 2023
It unlocks a new paradigm that replaces coding with natural-language commands.
This is how Microsoft is opening the door to an unexpected future: -
Microsoft has used ChatGPT for:
— Alex Banks (@thealexbanks) March 2, 2023
• Controlling a drone
• Moving a robot arm
• Acting as a home assistant
This is only the beginning of what's possible.
Let's dive in. pic.twitter.com/epbLQVSPCA -
1/ Challenges in robotics today
— Alex Banks (@thealexbanks) March 2, 2023
Current robotics requires an engineer "in the loop".
They need to translate a task's requirements into code.
• Slow — you need to write low-level code
• Expensive — requires skilled robotics knowledge
• Inefficient — multiple interactions pic.twitter.com/gFWw8GzhxK -
2/ How ChatGPT can help
— Alex Banks (@thealexbanks) March 2, 2023
ChatGPT unlocks a new robotics paradigm.
Instead of a technical engineer, a non-technical user can sit "on the loop".
They provide high-level feedback to the large language model (LLM) while monitoring the robot's performance. pic.twitter.com/ITqSfGfSC8 -
3/ Controlling a drone
— Alex Banks (@thealexbanks) March 2, 2023
This is an example of zero-shot task planning:
The model performs a task without any specific training on that task.
ChatGPT wrote complex code structures for the drone.
This allowed it to do a zig-zag pattern to visually inspect the shelves. pic.twitter.com/pRGZJ9bRg5 -
4/ Moving a robot arm
— Alex Banks (@thealexbanks) March 2, 2023
ChatGPT not only recalled the logo from its internal knowledge base.
It was able to ‘draw’ the logo as SVG code.
Then used its learned skills to determine which existing robot actions can create its physical form. pic.twitter.com/Y0vzq9HKe4 -
5/ Acting as a home assistant
— Alex Banks (@thealexbanks) March 2, 2023
This is an example of perception-action loops:
Robots that sense the world before they act.
A textual description of the camera image at each step was fed into the chat.
The model was able to control the robot until it reached an object. pic.twitter.com/BGdPdGmLqs -
Good prompt engineering is crucial for the success of LLMs such as ChatGPT for robotics tasks.
— Alex Banks (@thealexbanks) March 2, 2023
Luckily Microsoft recently introduced "PromptCraft".
This is an open-source platform where anyone can share examples of prompting strategies for different robotics categories. -
They also released a robotics simulator: ChatGPT-AirSim.
— Alex Banks (@thealexbanks) March 2, 2023
This is built on Microsoft AirSim with ChatGPT integration.
Anyone can get started with these ideas.
I believe integrating natural language interfaces—such as ChatGPT—will allow robots to be used by everyone. -
Follow me @thealexbanks for more on AI.
— Alex Banks (@thealexbanks) March 2, 2023
If you liked this thread, you'll love the newsletter.
Subscribe here:https://t.co/Kre1UavZe3 -
Help everyone learn and retweet this thread: https://t.co/YAJEKLxU4y
— Alex Banks (@thealexbanks) March 2, 2023 -
You can check out PromptCraft here:https://t.co/Pm97A451zY
— Alex Banks (@thealexbanks) March 2, 2023
And ChatGPT-AirSim here:https://t.co/pVGYPm2jbs -
What robot applications are you most looking forward to?
— Alex Banks (@thealexbanks) March 2, 2023
Let me know in the comments.
I'll start: bringing me toilet paper when I get caught off guard.