Traps, Tricks & Mistakes: AI Blunder Festival
In Wikipedia I found an accurate definition about what we call Artificial Intelligence: “Artificial Intelligence (AI) is the capability of computational systems to perform tasks typically associated with human intelligence, such as learning, reasoning, problem-solving, perception and decision-making. It is a field of research in computer science that develops and studies methods and software that enable machines to perceive their environment and use learning and intelligence to take actions that maximize their chances of achieving defined goals.”
As every software tool, it’s convenient to evaluate AI performance and scientists and engineers have developed benchmark tests to measure it. However, as a cutting-edge field of research, AI develops very fast and current benchmarks are struggling to keep pace with modern models.
Current benchmarks are helpful in measuring model performance on specific tasks. However, as training models use internet data, it’s hard to know if those models are actually solving problems or just remembering answers previously seen. Because of that, scientists are going further in the search for new ways to evaluate.
In this respect Google LLC have launched “Kaggle Game Arena”, a public AI benchmarking platform where AI models compete head-to-head in strategic games.
Why games? Games provide a clear, unambiguous signal of success: win, loss or draw. Their structured nature and measurable outcomes make them a perfect testbed for evaluating models. Games force models to demonstrate skills such as strategic reasoning, long-term planning and dynamic adaptation against an intelligent opponent, which is a robust signal of their general problem-solving intelligence. The value of games as a benchmark is further enhanced by their scalability—difficulty increases with the opponent’s intelligence.
Engineers believe that Kaggel will provide a glimpse of the AIs’ way of thinking, helping to understand the way they approach complex problems.
You probably have read/heard about (even used!!) Stockfish. It is a specialized engine for playing chess at a very high level. Kaggle will no test Stockfish and other specialized models that would beat every general-purpose AI models easily. Kaggle will only test large language models (LLM) not built to specialize in any specific games.
As introductory event, on August 5-7, Kaggle hosted a three-day AI Exhibition Chess Tournament featuring eight of the world’s leading AI models. Competitors included LLMs from Google, OpenAI, Anthropic, along with other labs.
The tournament was a single-elimination knockout with individual matches of 4 games. AI models used a universal controller to “see” the positions and make moves. Because the LLMs participating were not specialized game models, they had to comply a key rule concerning legality of moves. They had four total attempts to make a legal move. If they failed to make a legal move in the fourth attempt, they lost the game.
On day one, all games finished with decisive results (no draws!) But the quality of moves was quite poor. As an example, below is one of the games played by “DeepSeek R1” vs “o4mini” (OpenAI). Some comments written by both models have been added in quotation marks next to their moves.
Looking only at the first few moves you would say that is a game between experienced players. But at some point, the quality of play go down really fast.
You can follow the course of that game along with comments displayed bay both models in this video.
This was the trend throughout the tournament. A few good opening moves followed by hallucination and a series of blunders.
LLMs have displayed three key weaknesses when it comes to playing chess: visualizing the entire board, understanding how pieces interact with each other, and making legal moves.
Maybe you play chess but are unable to attend a local club. If that is your case, then join our virtual Chess.com club. On our site, we regularly play online tournaments and team matches. More than 1370 members enjoy that chance and new players join every week. For the coming months we have new tournaments scheduled waiting for players! Join now! Additionally, we have a puzzle section.
