360 Style Transfer Animation

Today I present to you my first 360 Style Transfer video:

This video attempts to be a 360 version of the Style Transfer animation I presented in ITP's Spring Show in 2018. The stylization isn't quite identical, partly because of technical shortcomings with what I created in 2018 that I did not wish to repeat.

All source images come from Google using the Google Street View Image API. The style is a mixture of Picasso's Seated Nude (1909) and a photograph by Cat Connor of a sunset at Yosemite National Park.

The YouTube video is 4K but truthfully it is a 3K video that I scaled up to 4K. I'll also confess that I now find the style used to be mediocre. I did it this way because making this specific video with this style is a personally significant achievement relating to last year's Spring Show. I very much wanted to see the goal of that project finally come to fruition. In any case, the next 360 video will be better.

While watching this video, take note of the temporal coherence, or in other words, the lack of flickering. Often times style transfer videos flicker from one frame to the next because there is no relationship between how the style is applied to neighboring frames. The technique I used to address this in 2018 is to use the DeepFlow optical flow model to add an optimization constraint to the stylization process. To the extent that the optical flow model was successful, this approach enforces temporal coherence from one frame to the next. The paper Artistic style transfer for videos by Ruder, Dosovitskiy, and Brox explains this in more detail.

In the case of this new 360 video, the temporal coherence is achieved without the use of an optical flow model. I came up with a simple and clever trick to make this happen that I'll explain at a later date. In the mean time, give this video a try, preferably with a VR headset.