It is only a few months ago that neural network based style transfer was the latest thing. Now we have a short and serious movie which contains sections that were generated by style transfer. The director is actress Kristen Stewart who also co-authors the paper.
It is highly probable that most of the heavy lifting was done by Adobe software engineer Bhautik Joshi, but the credits to Stewart and producer David Shapiro were probably justified on creative grounds and simply being able to understand and use a cutting edge technique enough to realize their artistic intent. The film, Come Swim was written and directed by Stewart and makes use of a painting by her to establish the style.
You can see a tiny clip from the film in the video, although there are no style-transferred portions:
Style transfer is achieved by training a neural network to recognize the style of a particular painter and then extracting the features that characterize the style. These can then be applied to a new image to transform it into the same style.
In this case the processing pipeline used lots of GPUs provided by Amazon EC2. Even then the resolution was only 1024 pixels wide and this needed to be upscaled to 2048 pixles to be incorporated into the rest of the movie. The processing time was about 40 minutes per frame per instance used.
The problem to be solved is how do you provide the creative control to the director? Using the technical parameters of the network doesn't make it easy to achieve a desired result. The key factor in this work is that a measure of "unrealness" u was derived, allowing the strength of the style transfer to be controlled.
The content of the paper doesn't rate a "breakthrough" headline, and to be honest most of the attention it has received is more to do with the inclusion of actress Kristen Stewart - wow an actress authors an AI paper - but it is an indication of how fast advanced AI is making inroads into what we usually think of as the creative arts.
For a long while installation art and other more esoteric "happenings" have needed their creators to know about languages such as Processing and hardware such as the Arduino or the Raspberry Pi. Now you can add neural networks to the list. Ever since Google's inception network started to dream up surrealistic images, AI has been a creative force in its own right.
We all know the Arthur C. Clarke quote about any sufficiently advanced technology looking like magic, but perhaps our own technology is already enough to look like magic if presented in the right way. [ ... ]