Culture
Roblox knows child pornography could be its downfall
“The success of our business model is contingent upon our ability to provide a safe online environment for children to experience and if we are not able to continue to provide a safe environment, our business will suffer dramatically.”
On Thursday, game platform Roblox filed to go public. The platform is particularly popular among minors, but its user-generated games come from an array of demographics. Adults on the platform generally remain insular to friends and family, but Roblox must also play whack-a-mole with inappropriate interactions between adults and children. In its filing with the SEC, the company included several entry points of sexual predation on children as a risk to its business.
Roblox is complicated — on one hand, it encourages and inspires creativity from users, from game design to avatar skin creation, in order to create the kind of whimsical experiences where you can receive hugs from giant floating heads.
On the other hand, the platform is constantly playing catchup trying to take down highly sexual rooms, and even worse, deal with multiple incidents of more direct child predation. A 48-year old man recently coerced an eight-year-old girl into sending explicit videos, and a couple of years ago, a hack allowed the avatar of a seven-year-old girl to be raped in a game. The nature of the platform makes policing these sorts of things difficult, but finding a way could prove the crux that decides whether or not the company can thrive when it goes public, and it knows it.
Moderation is hard — The SEC filing also draws attention to its mix of technology and human moderators which work to prevent child sexual abuse. As Motherboard notes, though this moderation system is far from transparent, the platform is a part of Microsoft’s Project Artemis. This project aims to review chatroom conversations and rate them on how much of a grooming risk they are with certain levels being passed off to human moderators.
Roblox is both a company that has grown astronomically during the pandemic and one that is still unprofitable. Sales grew 68 percent to $588.7 million in the first nine months of the year, but the company posted a net loss of $203.2 million.
Manual interventions — Various types of platform purchases require a higher level of scrutiny given the age range of the majority of its users, and that introduces friction and, with it, additional costs. Finding a way to keep those costs down while still growing the user base and keeping existing players loyal is no small task.
By tapping into such a young demographic, Roblox has grown quickly. Those young users, however, will make or break the company depending on how well they’re protected.