360 Style Transfers

360 Style Transfer Animation

Jim Schmitz — Tue, 06 Aug 2019 16:21:29 GMT

Today I present to you my first 360 Style Transfer video:

This video attempts to be a 360 version of the Style Transfer animation I presented in ITP's Spring Show in 2018. The stylization isn't quite identical, partly because of technical shortcomings with what I created in 2018 that I did not wish to repeat.

All source images come from Google using the Google Street View Image API. The style is a mixture of Picasso's Seated Nude (1909) and a photograph by Cat Connor of a sunset at Yosemite National Park.

The YouTube video is 4K but truthfully it is a 3K video that I scaled up to 4K. I'll also confess that I now find the style used to be mediocre. I did it this way because making this specific video with this style is a personally significant achievement relating to last year's Spring Show. I very much wanted to see the goal of that project finally come to fruition. In any case, the next 360 video will be better.

While watching this video, take note of the temporal coherence, or in other words, the lack of flickering. Often times style transfer videos flicker from one frame to the next because there is no relationship between how the style is applied to neighboring frames. The technique I used to address this in 2018 is to use the DeepFlow optical flow model to add an optimization constraint to the stylization process. To the extent that the optical flow model was successful, this approach enforces temporal coherence from one frame to the next. The paper Artistic style transfer for videos by Ruder, Dosovitskiy, and Brox explains this in more detail.

In the case of this new 360 video, the temporal coherence is achieved without the use of an optical flow model. I came up with a simple and clever trick to make this happen that I'll explain at a later date. In the mean time, give this video a try, preferably with a VR headset.

4K Resolution

Jim Schmitz — Thu, 20 Jun 2019 17:43:08 GMT

After a brief hiatus, I have returned to working on this project. The ITP thesis process was intense and I needed some time to get my life back in order.

My first order of business was to create high resolution 360 style transfers. Previously, all of the 360 images on this site provided 2.5K resolution. Now all but one of them are in 4K.

These were created using cloud computing because higher-end machines are necessary. As resolution increases, both memory and execution time scale with \(O(n^2)\) complexity.

To push the limits of my code I also created 8K and 16K resolution style transfers. I'm glad I did that because at 8K I found a bug I wouldn't have found any other way. I can't present either in A-Frame, however, because the files are too big and provide more resolution that would be necessary for any current 360 viewer. I'll save that for later on this summer when I redesign this website using other 360 tools.

This week I also experimented with style transfer videos. The experiment was a success but there were some unexpected problems that I want to fix before sharing. So, you'll have to wait to see this.

Next steps for me are to redesign this website and continue experimenting with 360 video.

Thesis Presentation

Jim Schmitz — Tue, 21 May 2019 14:48:17 GMT

Last week's thesis presentation was recorded and has now been uploaded to Vimeo. You can watch it here:

First Post

Jim Schmitz — Wed, 15 May 2019 02:21:23 GMT

Hello there!

While a grad student in NYU's ITP program, I developed a new method of applying style transfers to 360 photos and video. I created this blog to document my progress writing a research paper and creating an open source library. I hope to achieve both goals by the end of 2019. By writing about it here, I will keep myself accountable to these goals.