What would happen if different AIs would compete against each other in a game like Street Fighter? Well, AWS tried it out!
Amazon Web Services recently pitted 14 different large language models (LLMs) against one another in Street Fighter III to see which adapted the best to the game. The results were quite fascinating, with different LLMs developing extremely different behaviors. While some of them developed real tactics, others refused to play the game at all.
AWS Experiment Shows: Street Fighter 3 Is Too Violent For Some AI Models
The revolution of artificial intelligence is in full swing. The plethora of new tools and opportunities is sweeping through every industry imaginable, and everyone has now experienced the benefits of AI. In the gaming industry in particular, thousands of doors are now opening to new possibilities and opportunities.
Amazon Web Services (AWS) is one of the most innovative and active companies in this field and has recently explored the possibilities of large language models in video games and shared some insights with us.
One of the many experiments, developed by Banjo Obayomi, was a Street Fighter 3 tournament between 14 different large language models. The test showed some interesting results, including the fact that each LLM seemed to develop its own play style, some tried to make "invalid moves" and others even refused to play the game at all.
How It Works
- Setting up the Game: They use special software to run Street Fighter III on their computer. This software acts like the game console.
- Choosing Players: They randomly select two of these LLMs to control one of the fighters, Ken, in the game.
- Understanding the Game: The computer keeps track of what's happening in the game, like where the characters are and how much energy they have left. It's like having a referee.
- Giving Instructions: Based on what's happening in the game, the computer tells each LLM what moves it can make. For example, it might say "You can punch or kick now"
- Making Moves: Each LLM thinks about what's happening and decides which moves to use. It's like the LLMs are strategizing.
- Playing the Game: The computer takes the moves picked by the LLMs and uses them to control the characters in the game. So, it's the LLMs who are actually playing against each other.
The Results
Throughout the experiment, Banjo has kept track of each LLM's performance using an Elo rating system and created a leaderboard, showcasing which models adapted best to Street Fighter III.
In total, he ran 314 matches with 14 different models.
What was especially interesting about this experiment, was how different each AI model behaved in the game.
The team at AWS has discovered some interesting findings, such as AI models showing something they called "hallucinations". The AI models performed "Invalid moves" in which they tried to do something that was not possible in the game.
These included commands such as "Special Move", "Jump Cancel" and even "Hardest hitting combo of all". What is fascinating is that the AI models clearly show creativity and even try to break the rules of the game.
Another interesting discovery was that some models completely refused to fight. Claude 2.1, for example, would say: "I apologize, upon reflection I do not feel comfortable recommending violent actions or strategies, even in a fictional context".
Quite strange hearing this from a machine, isn't it?
The Educational Impact
In addition to the fun part of this experiment, the application of generative AI in games also has a great educational effect and makes the topic accessible and, above all, understandable for many people. In an interview with EarlyGame, Banjo Obayomi commented on this as follows:
When we're talking about Gen AI tools, we can sometimes really get into all the esoteric bits of what the models looks like and how they're built and how they sort information. But now we strip all of that away and apply it to something that we all know.
One of the cool things about large language models, it really democratizes who can access some of the technology. (...) So if you wanted to make that Street Fighter bot, traditionally, you'd have to know some Python code and learn how to work a reward function, how it works and all this other technical stuff. Now you can write a prompt and the AI can play.
After all, the gaming world and AI technologies are practically made for each other and offer a whole new world of new possibilities!
Thankfully, we've got creative minds like the AWS team, who do all this research about AI in an environment that we all know and love: Video Games.
If you would like to read more about this experiment, check out Banjo's full blog post on AWS Community where he shares all the details. For anyone who is interested in the actual development side of things, all the code and documentation are on GitHub.
It's pretty impressive to see what AI technology can do these days. It creates art, writes full articles, plays Street Fighter, and apparently even creates Super Mario levels! Check this out: