ChatGPT Interviews Top Video Game Developers

I recently interviewed some of the world's top video game developers—with the help of ChatGPT

The experiment: Have the AI write 1 question for each developer

The result: 9 questions asked, 9 answers given. BUT.... 4 of the questions had errors

Let's break this down...

— Stephen Totilo (@stephentotilo) April 14, 2023

I published this test across 4 editions of the @axios Gaming newsletter (Subscription recommended!)

Initial conundrum: When testing, ChatGPT's questions were kinda dull

The solution: Mandate a word that had to be in the question. "Tomato", for example https://t.co/GICynnmsAs pic.twitter.com/7ig1rtbPFp

— Stephen Totilo (@stephentotilo) April 14, 2023

My interviews were all in-person, so I decided it'd be fun to let the interviewee pick the word and see the AI generate their question

First 3 interviews were with indie devs who picked words that generated solid, straightforward questions https://t.co/hzNSU50DoI pic.twitter.com/2NAFnigww8

— Stephen Totilo (@stephentotilo) April 14, 2023

Things got a little wilder with the second set of interviews. Devs for Immortality gave the AI a tricky word. It basically fell for it and asked an erroneous question

(Chat GPT fared better using "banana" via two God of War developers)https://t.co/g0G3vHlIax pic.twitter.com/KTvquyOYm4

— Stephen Totilo (@stephentotilo) April 14, 2023

But by the final 3 interviews, ChatGPT was falling apart

It volunteered an inaccurate summary of Pentiment, called John Romero the creator (not co-creator) of Doom and thought this year's first-time GDC Awards host had hosted the show beforehttps://t.co/4SlcSoHDLG pic.twitter.com/LJUyzcq5cb

— Stephen Totilo (@stephentotilo) April 14, 2023

Key takeaways:
- The AI can be confidently wrong
- Interviewees are too nice to call that out and, regardless, are always up for offering an interesting answer
- I think my job is safe!

— Stephen Totilo (@stephentotilo) April 14, 2023

Thank you to the devs who put up with my nonsense: @ItsMadsTalking, @cosmoddd, @aryodarm, @MkentAudio, @GoKupsGo, @mrsambarlow, @nataliewatson, Connor Carson, @swann_mr, @jesawyer, @hothambone, @romero, and @leslee_annsh

— Stephen Totilo (@stephentotilo) April 14, 2023