Training a neural network to transfer style from one image to another

Neural Style uses VGG-19 (more info can be found here) and transfers style from a given style image to a given content image.

The source can be found here. This tutorial assumes you already have Spell installed, if not, follow the instructions here.

In the below images, we took the style from this Ralph Steadman illustration:

Ralph Steadman Saiga Antelope, 2017 source: Critical Critters

And transfered it onto this image of our teammate's beloved cat:

Uploading Images

Let's start with uploading the images we'd like to use, both as style to extract and as content to transfer style onto. To stay organized, we can create a directory called neural-style-imgs containing two sub-directories: content and styles. Put in the neural-style-imgs/style directory any and all "style" images that you want to transfer, and put in the neural-style-imgs/content directory any and all "content" images that you want to have styles transfered onto.

$ ls neural-style-imgs
content   styles
$ spell upload neural-style-imgs

Downloading Datasets

In order to run neural-style-tf, you will need the required VGG-19 model weights. This pre-trained model is about 500MB and can be downloaded quickly on a cheap CPU machine using Spell:

$ spell run --machine-type CPU 'wget -O imagenet-vgg-verydeep-19.mat "http://www.vlfeat.org/matconvnet/models/imagenet-vgg-verydeep-19.mat"'

A truncated response to this run:

✨ Casting spell #1…
Run created -- waiting for a CPU machine.
...
✨ Run is running
http://www.vlfeat.org/matconvnet/models/beta16/imagenet-vgg-verydeep-19.mat
...
Length: 576042600 (549M) [text/plain]
Saving to: 'imagenet-vgg-verydeep-19.mat'
...
(15.8 MB/s) - 'imagenet-vgg-verydeep-19.mat' saved [576042600/576042600]
...
✨ Run 1 complete

The run identifier for this run (in this example it's 1) is important, so don't forget it! We will be using this identifier to mount the downloaded dataset results into the file system of a new run using the mount flag --mount runs/1/imagenet-vgg-verydeep-19.mat.

Training with neural_style.py

In order to train a machine learning model, you’ll need to choose a --framework and a --machine-type that fit for your run. We'll use --framework tensorflow to specify the AI framework TensorFlow that this fast style transfer implementation uses, and --machine-type K80 to specify a K80 GPU to use (a V100 would work too and be faster, but more expensive). Read more about machine types here

Now we are ready to run our trained model on our images! This call is a combination of all of the required Spell flags as well as the flags for neural_style.py detailed in the neural-style-tf docs. Choose an interesting style image from your uploaded neural-style-imgs/styles directory, and use the --style_imgs flag to specify the name of your style image.

Also don't forget to change runs/1/imagenet-vgg-verydeep-19.mat to your run id from Downloading Datasets above. If you don't remember your run id, you can find it in your list of previous runs using spell ps or by checking your runs list on the runs detail page online.

$ spell run --github-url https://github.com/cysmith/neural-style-tf \
            --mount runs/1/imagenet-vgg-verydeep-19.mat \
            --mount uploads/neural-style-imgs/styles:styles \
            --mount uploads/neural-style-imgs/content:image_input \
            --machine-type K80 \
  "python neural_style.py \
  --style_imgs antelope.jpg \
  --content_img cat.jpg"
✨ Casting spell #2…

The transfering will take a bit of time (around 5 minutes on a K80 for 512x512pixels, the default size) and will result in the newly generated image living inside image_output/result/result.png. To stop viewing live logs, type ^C.

Downloading and Viewing Your Images

You can download your output image with the spell cp command. You'll need the path to your output folder from your previous run. In our case, the run id of our previous run is 2. You can also specify where you want the images to be downloaded to in the second part of your command. In the code below, the images from our style transfer run will save into our local folder images/output

To list and download the generated image use the spell ls and spell cp commands. (Note the use of run id 2 here, which would mean the above run returned ✨ Casting spell #2…).

$ spell ls runs/2
0        -              image_output/
$ spell ls runs/2/image_output/result
1734608  Jun 14 19:44   content.png
1734608  Jun 14 19:44   init.png
342      Jun 14 19:44   meta_data.txt
2442801  Jun 14 19:44   result.png
2422704  Jun 14 19:44   style_0.png
$ spell cp runs/2/image_output/result/result.png images/output
Downloading
Copying file to images/output/result.png
✔ Copied 1 files.
17K

Further Experimentation

As with any machine learning algorithm, you can tweak the training parameters to get different outputs. For style transfer, the interesting factors to change are:

  • --content_weight: Weight of content in loss function. Default: 5e0.
  • --style_weight: Weight of style in loss function. Default: 1e4.
  • --tv_weight: Weight of total variation term in loss function. Default: 1e-3.
  • --temporal_weight: Weight for the temporal loss function. Default: 2e2.

Try changing the default numbers when you train your model and see what happens to the output. You can even use Spell's Hyperparameter Search feature to procedurally experiment with different values to see what kinds of interesting art we can make. The details of how to do this are in the Transferring Style with Hyperparameter Search guide here on learn.spell.run.

And that's it! Now you know how to transfer style with neural-style-tf and Spell!

Have more questions? Submit a request

Ready to Get Started?

Create an account in minutes or connect with our team to learn how Spell can accelerate your business.