Over the weekend, Meta dropped two new Llama 4 fashions: a smaller mannequin named Scout, and Maverick, a mid-size mannequin that the corporate claims can beat GPT-4o and Gemini 2.0 Flash “throughout a broad vary of broadly reported benchmarks.”
Maverick rapidly secured the number-two spot on LMArena, the AI benchmark website the place people examine outputs from completely different methods and vote on the perfect one. In Meta’s press launch, the corporate highlighted Maverick’s ELO rating of 1417, which positioned it above OpenAI’s 4o and slightly below Gemini 2.5 Professional. (The next ELO rating means the mannequin wins extra usually within the enviornment when going head-to-head with opponents.)
The achievement appeared to place Meta’s open-weight Llama 4 as a critical challenger to the state-of-the-art, closed fashions from OpenAI, Anthropic, and Google. Then, AI researchers digging by way of Meta’s documentation found one thing uncommon.
In effective print, Meta acknowledges that the model of Maverick examined on LMArena isn’t the identical as what’s out there to the general public. Based on Meta’s personal supplies, it deployed an “experimental chat model” of Maverick to LMArena that was particularly “optimized for conversationality,” TechCrunch first reported.
“Meta’s interpretation of our coverage didn’t match what we count on from mannequin suppliers,” LMArena posted on X two days after the mannequin’s launch. “Meta ought to have made it clearer that ‘Llama-4-Maverick-03-26-Experimental’ was a personalized mannequin to optimize for human choice. Because of that, we’re updating our leaderboard insurance policies to bolster our dedication to truthful, reproducible evaluations so this confusion doesn’t happen sooner or later.“
A spokesperson for Meta, Ashley Gabriel, mentioned in an emailed assertion that “we experiment with all kinds of customized variants.”
“‘Llama-4-Maverick-03-26-Experimental’ is a chat optimized model we experimented with that additionally performs effectively on LMArena,” Gabriel mentioned. “Now we have now launched our open supply model and can see how builders customise Llama 4 for their very own use instances. We’re excited to see what they’ll construct and sit up for their ongoing suggestions.”
Whereas what Meta did with Maverick isn’t explicitly towards LMArena’s guidelines, the location has shared considerations about gaming the system and brought steps to “forestall overfitting and benchmark leakage.” When firms can submit specially-tuned variations of their fashions for testing whereas releasing completely different variations to the general public, benchmark rankings like LMArena turn out to be much less significant as indicators of real-world efficiency.
”It’s probably the most broadly revered common benchmark as a result of the entire different ones suck,” impartial AI researcher Simon Willison tells The Verge. “When Llama 4 got here out, the truth that it got here second within the enviornment, simply after Gemini 2.5 Professional — that actually impressed me, and I’m kicking myself for not studying the small print.”
Shortly after Meta launched Maverick and Scout, the AI neighborhood began speaking a few rumor that Meta had additionally educated its Llama 4 fashions to carry out higher on benchmarks whereas hiding their actual limitations. VP of generative AI at Meta, Ahmad Al-Dahle, addressed the accusations in a put up on X: “We’ve additionally heard claims that we educated on take a look at units — that’s merely not true and we might by no means do this. Our greatest understanding is that the variable high quality persons are seeing is because of needing to stabilize implementations.”
“It’s a really complicated launch typically.”
Some additionally observed that Llama 4 was launched at an odd time. Saturday doesn’t are usually when large AI information drops. After somebody on Threads requested why Llama 4 was launched over the weekend, Meta CEO Mark Zuckerberg replied: “That’s when it was prepared.”
“It’s a really complicated launch typically,” says Willison, who intently follows and paperwork AI fashions. “The mannequin rating that we bought there’s fully nugatory to me. I can’t even use the mannequin that they bought a excessive rating on.”
Meta’s path to releasing Llama 4 wasn’t precisely easy. In accordance to a latest report from The Info, the corporate repeatedly pushed again the launch because of the mannequin failing to fulfill inside expectations. These expectations are particularly excessive after DeepSeek, an open-source AI startup from China, launched an open-weight mannequin that generated a ton of buzz.
Finally, utilizing an optimized mannequin in LMArena places builders in a troublesome place. When deciding on fashions like Llama 4 for his or her functions, they naturally look to benchmarks for steering. However as is the case for Maverick, these benchmarks can replicate capabilities that aren’t truly out there within the fashions that the general public can entry.
As AI growth accelerates, this episode exhibits how benchmarks have gotten battlegrounds. It additionally exhibits how Meta is raring to be seen as an AI chief, even when meaning gaming the system.
Replace, April seventh: The story was up to date so as to add Meta’s assertion.