Culture
Researchers used AI to trick our perception of time
By decomposing layers of the same video, researchers are playing around with time itself.
A team of Google researchers and the University of Oxford scholars have collaborated to do a highly trippy trick: distort time and our perception of it. In a research project titled Layered Neural Rendering For Retiming People in Video, both teams are toying around with the concept of "retiming" people's movements in different clips. Make it fast or slow it down, anything's possible.
Imagine two videos of the same event, like school children dancing and playing together. With this neural network, the researchers are able to speed up one pair of kids and slow another pair down. You might think you're high as heck looking at these clips but it's just a little bit of machine learning and layering.
Here's an example of a family playing in water and how the neural network creates layers of the same clip.
Playing tricks with time — Think of the entire venture as a joint process of neural rendering and basic graphics. With the help of deep learning, human-specific movements can be understood as stand-alone elements and then used to render a different movement for every frame. Essentially, it can be understood as a decomposition of different levels and layers in a video.
The network attempts to detangle these motions and then generate its own version of shadows, speed, momentum, angles, even the texture of moving fabric, color, and more. Take this example from Synced.
The original video depicts a woman strolling forward while a man jumps high above. The woman represents Layer 1 to the neural network while the man is decoded as Layer 2. The background is separated similarly. These layers are then retimed and depicted in different speeds.
This can be done to all kinds of physical activities, including dancing, running, jumping, fighting, hand gestures, and more. You can read the full paper here.
This could be useful — Retimed video clips could prove to be helpful in the case of choreographed dancing, off-beat movement, and other contexts where you need to bring one individual up to speed with others in the same clip. But don't expect it to be perfect. "Achieving a temporally coherent result is challenging," the researchers write, "small errors such as subtle misalignment between frames immediately show up as visual artifacts when the frames are viewed as a video."
