13:36Machine-translated from Azerbaijani by AIRead in Azerbaijani
Startup Turns Gamers’ Actions Into AI Training Data
British startup “Worldmodeldata” turns gamers’ actions in video games into data for training artificial intelligence. The goal is to develop “world models” that learn about the physical environment and the consequences of actions. The startup, which has Yann LeCun as an adviser, collects footage, the buttons players press, and other control signals from game studios. It then delivers them to artificial intelligence laboratories as a unified and organized dataset.
Large language models are trained primarily on text. As a result, they can struggle with tasks that require precise actions in the physical world. For example, there is little information online about how much force a robotic arm should use to grasp an object or how it should rotate it. Laboratories collect such examples by attaching sensors to people and robots, but obtaining large amounts of data this way is difficult. Recreating rare and dangerous situations from the real world is also not easy. According to “Worldmodeldata,” 3D games can partially address this shortcoming. Game footage is paired with the action the player takes at that moment. This allows the system to learn the relationship between an action and its outcome. The company has licensed approximately 1 million hours of data from the studios behind various popular games, but it has not disclosed the studios’ names. It also plans to pay individual gamers in the future for the data they share. The startup’s CEO, Rhea Loucas, believes that game data could eventually become a primary training source for world models. These models could then be adapted using real-world data for a specific robot, vehicle, or manufacturing task. Games contain many different scenarios, including rare situations. It is important for vehicles, drones, and factory equipment to learn how to behave correctly in such situations because mistakes can have serious consequences. However, not everyone believes that game data is sufficient for teaching systems about the physical world. When developing world models, “Nvidia” favors a specialized software system that simulates the actual laws of physics. Company expert Ming-Yu Liu says that games often do not calculate fine details such as the pressure with which an object is held between the fingers. Therefore, this data may be more useful for creating realistic-looking videos and 3D environments than for teaching robots precise movements. According to experts, although games depict some laws of physics, they are simplified simulations of the real world. The effectiveness of the startup’s method has also not yet been confirmed through extensive testing. At the same time, the lack of large and diverse datasets remains one of the main challenges in training world models. Whether game data can address this shortage will depend on future results.
Large language models are trained primarily on text. As a result, they can struggle with tasks that require precise actions in the physical world. For example, there is little information online about how much force a robotic arm should use to grasp an object or how it should rotate it. Laboratories collect such examples by attaching sensors to people and robots, but obtaining large amounts of data this way is difficult. Recreating rare and dangerous situations from the real world is also not easy. According to “Worldmodeldata,” 3D games can partially address this shortcoming. Game footage is paired with the action the player takes at that moment. This allows the system to learn the relationship between an action and its outcome. The company has licensed approximately 1 million hours of data from the studios behind various popular games, but it has not disclosed the studios’ names. It also plans to pay individual gamers in the future for the data they share. The startup’s CEO, Rhea Loucas, believes that game data could eventually become a primary training source for world models. These models could then be adapted using real-world data for a specific robot, vehicle, or manufacturing task. Games contain many different scenarios, including rare situations. It is important for vehicles, drones, and factory equipment to learn how to behave correctly in such situations because mistakes can have serious consequences. However, not everyone believes that game data is sufficient for teaching systems about the physical world. When developing world models, “Nvidia” favors a specialized software system that simulates the actual laws of physics. Company expert Ming-Yu Liu says that games often do not calculate fine details such as the pressure with which an object is held between the fingers. Therefore, this data may be more useful for creating realistic-looking videos and 3D environments than for teaching robots precise movements. According to experts, although games depict some laws of physics, they are simplified simulations of the real world. The effectiveness of the startup’s method has also not yet been confirmed through extensive testing. At the same time, the lack of large and diverse datasets remains one of the main challenges in training world models. Whether game data can address this shortage will depend on future results.