Moonshot AI's Kimmy K3.1 Open Model Surprises Experts

Last week, a mysterious new AI model, initially codenamed Kibitz, emerged on Chapot Arena, exhibiting frontier-level performance. This model, widely suspected to be from Chinese company Moonshot AI, was officially unveiled as Kimmy K3.1 on Friday. Boasting three trillion parameters and slated for full public download on November 4th, Kimmy K3.1 is challenging the current landscape of large language models, particularly given ongoing controversies surrounding model training.

Kimmy K3.1’s unexpected performance has ignited debate, especially considering recent efforts by Western AI giants to curb alleged “distillation attacks.” Anthropic and OpenAI have reportedly invested heavily in blocking Chinese companies from stealing model outputs and reasoning tokens, which are then used to train open-source models. According to Fireship, Anthropic recently claimed to have blocked 90-99% of such attacks from China. If these claims are accurate, then the emergence of an open Chinese model like Kimmy K3.1, performing at a frontier level, presents a significant paradox, according to Fireship.

The Benchmark Anomaly

Kimmy K3.1 has demonstrated impressive results on key benchmarks. Fireship notes that the model is within one point of Opus 5.5 on Terminal Bench 4.0, a feat no other open model has achieved. Independent analysis by Artificial Analysis reportedly confirmed a 67% score for Kimmy, the highest they’ve observed. Fireship also highlights Kimmy’s success on S. Marathon, a benchmark designed to test a model’s ability to handle complex, multi-hour software requirements that would typically take humans weeks to complete. These scores are particularly noteworthy given the context of alleged distillation blocks, leading Fireship to question how such performance is possible if the purported theft of big AI outputs has been largely neutralized.

However, Fireship also cautions that benchmark results can vary wildly depending on the “harness” used to evaluate models, implying that the specific testing environment plays a crucial role in the outcomes. This variability means that while Kimmy’s raw scores are compelling, a deeper understanding of the testing methodology is essential for a complete picture.

Strategic Censorship for Compliance

A notable aspect of Kimmy K3.1’s development is Moonshot AI’s deliberate decision to remove cybersecurity-related data from its training dataset. The goal was to reduce the model’s hacking capabilities, a move Moonshot internally verified by observing a drop from 90.5% to 81% in solving “illegal exploit tests” compared to an unfiltered checkpoint. This strategic move, Fireship explains, is a calculated business decision to preempt potential bans. Just days prior, the US government banned China’s Deepseek model from use by federal civilian agencies, demonstrating a clear regulatory trend. Fireship suggests Moonshot has no choice but to “lobotomize” Kimmy’s hacking abilities to avoid a similar fate and prevent giving US authorities a reason to ban all Chinese models, which would further benefit Western AI companies.

Despite these limitations, Kimmy still topped all publicly available models on the new Cyber Index by Artificial Analysis. However, this was largely because established models like Claude and GPT-6 reportedly declined most prompts on safety grounds, effectively taking a lower score by refusing to engage.

The Thoroughbench Horse Test

Beyond traditional benchmarks, Fireship introduced a unique, albeit unconventional, evaluation method called “Thoroughbench,” designed to assess a model’s ability to generate a realistic, interactive 3D horse using 3JS. This bespoke test involves a simple prompt: “create a realistic interactive 3D horse with 3JS.” Models are then scored from 0 to 10 based on proprietary criteria.

On Thoroughbench, Kimmy K3.1 achieved a score of 4.20, impressive but still trailing other frontier models like GPT-6 Astra (5.21) and Opus (6.39). While Kimmy’s horse was praised for its aesthetic beauty, realistic coloring, and appropriate anatomy, it fell short in animation quality. Fireship observed that Kimmy’s animation was unsophisticated, simply warping the original mesh, in contrast to Claude, which built a complex mesh with 95,000 vertices tied to 35 independently animated bones, resulting in more natural movement. Even Gemini Argon used bones, albeit with an extra, unexplained one. This suggests that while Kimmy excels in visual generation, its understanding of complex animation mechanics may still be developing.

Key Takeaways

• Kimmy K3.1, a new open-weight model from Moonshot AI, is performing at frontier levels on standard AI benchmarks.
• Its performance is surprising given recent claims by Western AI companies that they have largely blocked distillation attacks.
• Moonshot AI deliberately removed cybersecurity data from Kimmy’s training to avoid US government bans.
• On Fireship’s unique “Thoroughbench” for 3D horse generation, Kimmy scored well for aesthetics but lagged in animation sophistication.

Why It Matters

The emergence of Kimmy K3.1 at a frontier level, amidst claims of successful anti-distillation efforts, raises profound questions about the efficacy of current protective measures and the true origins of open-source model capabilities. Fireship’s insights into this paradox underscore a potential shift in the global AI landscape, where Chinese models might be achieving parity or even surpassing Western counterparts, irrespective of previous dependencies or alleged data theft.

Furthermore, Moonshot AI’s pre-emptive censorship of cybersecurity capabilities highlights the growing geopolitical pressures shaping AI development. Companies are making strategic decisions not just based on technical prowess but also on compliance with international regulations and national security concerns. This trend could lead to more specialized or “lobotomized” AI models, each tailored to specific regulatory environments. Finally, unique evaluation methods like “Thoroughbench” emphasize the need for diverse and creative ways to assess AI capabilities beyond conventional benchmarks, particularly as models become more adept at complex, multi-modal tasks. The ongoing development of open models like Kimmy K3.1 will likely continue to challenge established notions of AI training, performance, and regulation.

Watch the original video

Source video by Fireship on YouTube: watch it here. This article is Astro Signal's own summary and analysis.