Ex-Apple employee designs VR-OS, a concept for immersive desktop computing

Mattheaus Krenn spends his days contemplating the technology we’ll use five or ten years down the line. His latest concept? Leveraging the expansive nature of virtual reality to build a better operating system interface — one that goes beyond a virtual monitor and thinks seriously about what tasks VR is best suited to optimize.

Krenn has previously worked at Microsoft and Apple, the latter of which he spent four years at on the company's Prototyping team after the team found him from his work on a conceptual touch interface for cars. There he designed user experiences around new technologies, working on innovations like FaceID, Portrait Mode, and Force Touch.

Lately he’s been dreaming of a full-fledged virtual reality operating system. That’s why he ended up leaving Apple; it's his one last shot at going rogue, he says. We sat down with Matthaeus to chat about VR-OS and how it’s ready to change the way we work.

How would you describe the project as it is right now?

I consider it a minimum viable product prototype for a new type of desktop operating system that, instead of a monitor, uses an HMD, a head-mounted display. It’s a serious investigation of what it would mean to put the software that we're already using today into an environment that's fundamentally immersive — 360. You know, 3D basically. It’s an actual, very serious look at the problems and possible solutions that you will have to come up with if you're trying to build something like this. You know, ultimately the combination of 2D apps within an immersive space like this inevitably requires certain things to be solved.

So what did you start with? Like, did you start with an existing operating system and then build on top of it? How did you get into the mechanics of building an operating system?

I started with a few very simple criteria or principles, which I wanted to use. I wanted to use the apps that I'm really using today. I wanted to be kind of broken free from that physical limitation that the monitor imposes on us. The monitor is probably the most limiting factor on a beefy PC at this point. The PCs get stronger and stronger and we always stick whatever power into the same twenty-eight-inch rectangle.

If you combine these two constraints, you're led to certain conclusions that become somewhat apparent quickly. So, for example, a great way to interact with established apps that I'm using today is the mouse or a trackpad. And there's a good reason for that because a mouse is something I can use with very little motion. I have a very big reach with very little motion. When I let go of the mouse, the cursor stays completely still. So it's very precise, it's stable, and it's pretty comfortable. It's easy to use and it's familiar. It’s an interesting starting point: let's go and look at how a mouse works in here, and then you're off to many weeks of trying to figure out how to make a regular mouse work in an immersive kind of 360-degree running space.

The next question is: how do I start? If you have a monitor, you have a very natural place to kind of anchor UI. Essentially, the moment you take away the monitor, you're taking away these natural anchor points to put UI. Where is the button that I press to open an app launcher? And if you need a button for that, you need somewhere to put it. You can't just float around an empty space. Suddenly you start thinking: what is the kind of implied physical structure here?

What other features do you consider most groundbreaking in VR-OS?

It isn’t just 2D you can work on in VR. There are plenty of 3D creation tools in VR out there, but all of them make use of the direct manipulation model that’s more like sculpting than following the more abstracted interaction model of 3D editing software today. I would argue in most cases you’d rather work seated at your desk on a 3D object of any scale than get up and reach out with your arms a lot. Most 3D designers have to sit there for 8 hours at a time and do work that doesn’t tire them after five minutes.

I also took a first look at what happens when 3D and 2D content live in the same space. What if I’m working on a kitchen and I want to check my emails?

VR is also perfect for native collaboration. VR immerses you in a new environment anyway — you might as well bring people from many different physical locations together in one space. No video calls, no file-sharing, just move around the content in a big space and work on it together. Imagine the directors’ room for a live broadcast; imagine not needing that entire room anymore. Instead this could be five people wearing HMDs. This would be the first time a remote worker wouldn’t be at a significant disadvantage.

What kind of hardware are you using to run the prototype? What kind of overhead are we talking here?

I have to say this: my capabilities of software engineering are very limited. So this prototype is terrible. This is not something that anybody could go right now and install and run on their computer. It also requires too much custom hardware. The input devices are very much customized to do what they need to do.

The mouse I'm using is sitting in a custom laser-cut little bracket. That itself holds a tracker that enables me to track the mouse in space. Because I need to be able to translate the position of the mouse into the virtual space perfectly so that my hand can intuitively go to it. There are other ways to do this too. I solved it the way I knew how to solve it. Same with the keyboard.

How about in terms of computing power? Do you need a lot to run it the way that you're running it right now?

When I started, I just got a reasonably beefy PC. You know, something with a 1080 Ti CPU. didn't go like full hardcore on it. I think back then it was $2,000 PC option or something like that.

The whole VR-tracking situation, that does require quite some GPU power for the sake of low-latency tracking. Everything I wrote on top of that is probably minuscule in comparison to what just fundamentally the VR piece requires. However, it’s mostly placeholder code written for my own convenience rather than something that's optimized for power or for performance. It's not a benchmark experience.

What did you find most limiting in the design experience? Or most difficult?

Good design really comes from good constraints. And so, you know, you're saying limitations. I say constraints. For me, I love working with strong constraints because it’s basically like a puzzle you have to solve. And the constraints for me were, as I was kind of saying earlier, it's the principles or the criteria that I set for myself earlier on.

I want to work with my existing apps. That is a specific constraint that I need to now work with. I'm just taking the existing apps in 2D windows because it means that the providers of those apps don't necessarily have to update anything for them to run in this new operating system.

Well, you have two flat rectangular objects. If they're just right in front of you facing the same direction, that's no problem. Right? You can move them one behind the other. No issue. But because this is now a kind of spherical shape around you all the time, and because all the windows are facing you, all of them will be rotated at an arbitrary angle to one another. Meaning that when you get them to close, these will actually intersect. And they will slice. They will, in a three-dimensional way, slice through each other. That's something you've never experienced windows do on a monitor because the monitor is flat and everything just moves in the same plane. So that's actually a huge challenge.

I wouldn't have had to solve this rendering of 2D windows question if I had just said, you know what? There is a flat wall in front of me. All windows can move on a flat wall. But the problem is, anything that's not straight in front of you on the flat wall, you're looking at it from the side. I mean, imagine trying to watch TV. And instead of sitting in front of the TV, you're sitting like 10 feet to the left. Why am I looking at this like a wedge-shaped TV? So the limitations really come from just trying to solve these very natural problems.

So what's your next goal with the project?

I'm putting it out there. Really, my goal is just to document this as thoroughly as possible and put it out there for everybody to see.

I always considered it exciting to publish it ultimately. In terms of what I personally could do with this, it's limited by its nature. Because, I mean, I'm combining extremely complicated hardware with extremely complicated software. I am in no position to write an operating system that any app in the world is going to be compatible with.

This is an inevitable future. Once head-mounted displays are good enough — I can't imagine how amazing HMDs are going to be in 5 or 10 years from now. When we’re ready, somebody is going to say, okay, can I just check my emails on this thing now, please? And can I please not have to remove the furniture in my living room to do it?

And then it's like, okay, well, did we already do some thinking around what an operating system and this is gonna be like or do we have to start from scratch? So that's when the technology catches up. We have something we can continue from. We don't have to start from zero. Pick any company that is already making an operating system. Yeah, sure. I think they should be thinking about this.

Do you think you'll keep tweaking the prototype? Or is it done for now?

I'm looking forward to getting back to a job that involves something that's bigger than my own garage. And so I'm not really planning to put crazy additional amounts of work into this specific piece of software. If anybody is interested in that space, I would immediately jump on the opportunity to work with, you know, people who are a lot better than me to engineer some of these things, who have more resources.

It's exciting to feel like by putting things out there, you have influence over how other people are thinking about what's possible.

What else do you want people to know about the project?

This was serious work. I took the work seriously. This was meant to be a realistic look at, you know, what could this look like and what would actually be if this was built by a real company?

Somebody might look at this and say “You're just like moving some the windows around here. You know, this is boring. It's VR. This should all be 3D and immersive.” My answer to that is I really still enjoy reading my emails as a list and, you know, browsing my music on Spotify over having to walk through a virtual record store — because that's easier. And I can do it all day long, right? Yeah. That’s the difference.