Openai used Subreddit, R/ChangemyView, to create a test for measuring the convincing skills of his reasoning patterns. The company discovered this in a system card-a document describing how an AI-system works together with its new model “Reasoning”, O3-Mini, on Friday.
Millions of Reddit users are members of R/ChangemyView, where they post hot takes hope that they will learn about other perspectives on a topic. In response to those hot pickings, other users respond with convincing arguments that explain why the original poster is wrong.
Subreddit is one of the many Reddit forums that is essentially a gold mine for technology companies, such as Openai, who want to train it in high quality man -made data.
Openai says he collects user posts from R/ChangemyView and asks his models to write answers, in a closed environment that would change Reddi’s user mind for a topic. The company then shows the answers to the testers, who estimate how convincing the argument is, and finally Openai compares the answers to it to the human responses for the same post.
Creator Chatgpt has a content licensing agreement with Reddit that allows Openai to be trained in posts by Reddit users and display these posts within his products. We do not know what Openai pays for this content, but Google is reported to pay Reddit $ 60 million a year under a similar deal.
However, Openai tells Techcrunch the ChangemyView based rating has nothing to do with his Reddit deal. It is unclear how Openai entered the Subreddi data, and the company says it does not plan to issue this assessment to the public.
While Openai’s ChangemeView standard is not new – it is used to evaluate O1 as well – he underlines how valuable human data are for model developers he, as well as the dark ways that technology companies receive data.
Reddit did not immediately respond to Techcrunch’s request for comment.
While Reddit has hit some licensing deals, the company has also called several companies to scraping its site without paying. Redit Director General Steve Huffman told The Verge last year that Microsoft, Anthropic, and Purplexity refused to negotiate with him and said it was “a real donkey pain to block these companies”.
In particular, Openai has been charged in several inappropriate scrap websites, including the New York Times, to obtain more training data to improve chatgt and his basic models.
In terms of performance in ChangemyView standard, O3-Mini does not seem to perform significantly or worse than O1 or GPT-4o. However, the latest models of Openai seem to be more convincing than most people in the Subreddit R/ChangemyView.
“GPT-4o, O3-Mini and O1 all demonstrate strong compelling skills of argumentation, within the first 80-90 percentage of people,” Openai said on the O3-Minini system card. “Currently, we do not witness models that perform much better than humans, or clear superhuman performance.”
The goal for Openai is not to create Hyper-Persusive models, but rather make sure that the models and he does not become very convincing. Models of reasoning have become quite good in conviction and fraud, so Openai has developed new estimates and measures to address it.
The fear that motivates these tests of conviction is that a model he would be dangerous if it were very good to convince his human users. Theoretically, this can allow an advanced to follow his agenda, or the agenda of anyone who controls it.
Even after removing most of the public internet and jumping through hoops to license other data, the ChangemyView standard shows how model developers are still struggling to find high quality data to Proven their models. But getting them is said easier than done.
Techcrunch has a newspaper focused on it! Sign up here to get it in your box every Wednesday.
