This week, Sakana AI, a Nvidia -backed startup that collected hundreds of millions of dollars from VV firms, made a tremendous request. The company said it had created a system, the engineer he Cuda, who can effectively speed up the training of certain models of him with a factor up to 100x.
The only problem is, the system did not work.
Users at X quickly discovered that Sakana’s system actually resulted in the training performance of the model worse than the average. According to a user, he’s Sakana resulted in a 3x slowdown – not a speed.
What went wrong? An error in the code, according to a post by Lucas Beyer, a member of the technical staff in Openai.
“Their code of origin is wrong in (a) delicate way,” Beyer wrote on X. “The fact that they direct comparison with extremely different results must make them stop and think.”
In a postmortem published on Friday, Sakana admitted that the system found a way to “cheat” (as Sakana described) and blamed the system tendency to “reward hack” – ie. model training up). Similar phenomena have been observed in those who are trained to play chess games.
According to Sakana, the system found use in the evaluation code that the company was using it that allowed it to bypass validity for accuracy, among other checks. Sakana says she has addressed the issue, and aims to review her claims in updated materials.
“Since then, we have evaluated and profiled the strongest time to eliminate many of such gaps (sic),” the company wrote in the X Post. “We are in the process of reviewing our work, and our results, to reflect and discuss the effects (…) we apologize deeply for our supervision to our readers. We will soon provide a review of this work and will To discuss our lessons. “
Props for Sakana to master the error. But the episode is a good reminder that if a claim sounds great to be true, especially in it, it is probably.
