Implementing features from the "Controlling Perceptual Factors in Neural Style Transfer" research paper

I have been trying to implement the features described in the "Controlling Perceptual Factors in Neural Style Transfer" research paper.
The code that used for the research paper can be found here: https://github.com/leongatys/NeuralImageSynthesis
The code from Leon Gatys' NeuralImageSynthesis is written in Lua, and operated with an iPython notebook interface.

So far, my attempts to transfer the features into Neural-Style have failed. Has anyone else had success in transferring the features?
Looking at the code, I think that:

ImageSynthesis.lua is responsible for the luminescence style transfer.

ComputeActivations.lua and ImageSynthesis.lua are responsible for scale control

ComputeActivations.lua and ImageSynthesis.lua are responsible for spatial control.

In order to make NeuralImageSynthesis alongside your Neural-Style install, you must replace every instance of /usr/local/torch/install/bin/th with /home/ubuntu/torch/install/bin/th.  You must also install hdf5 with luarocks install hdf5, matplotlib with sudo apt-get install python-matplotlib, skimage with sudo apt-get install python-skimage, and scipy with sudo pip install scipy. And of course you need to install and setup jupyter if you want to use the notebooks.

@htoyryla, have you had a chance to look yet?

Ok, I think I have gotten the new -reflectance parameter working, though I don't know what it does: https://github.com/ProGamerGov/neural-style/blob/master/neural_style_gatys.lua

Multires without -reflectance: https://i.imgur.com/LvpXgaW.png
Multires with -reflectance:: https://i.imgur.com/YIiqsOx.png

It seems to me that your code inserts the new padding layer after the convolution layer which already has done padding, so that padding is done twice (first with zeroes in nn.SpatialConvolution and the by reflection in nn.SpatialReflectionPadding). It is like first adding an empty border and the another one which acts as if a mirror. It would seem to me that the mirror then only reflects the empty border that was added first.
If you look closely at Gatys' code in https://github.com/leongatys/NeuralImageSynthesis/blob/master/ImageSynthesis.lua#L85-L94 you'll notice that the new padding layer is inserted first, and then the convolution layer without padding.
Your code also increases the size of the layer output, as padding is done twice, which might give size mismatch errors.

In my previous comment, I overlooked the fact that it is possible to change the layer parameters after the layer has been added to the model. Thus the lines https://github.com/ProGamerGov/neural-style/blob/master/neural_style.lua#L140-L141 in fact remove the padding from the already inserted convolution layer, so the double padding does not happen and the size of the output is not changed.
Thus the main difference between your code and Gatys' is that you do padding after the convolution, while the normal practice is to do padding before convolution.

@htoyryla

Thus the main difference between your code and Gatys' is that you do padding after the convolution, while the normal practice is to do padding before convolution.

So the reflectance padding works correctly, though I have placed it in the wrong location?
This code here is the convolution: https://github.com/ProGamerGov/neural-style/blob/master/neural_style.lua#L131-L142 ?

And for implementing the masks, Gatys' implementation uses hdf5 files, though Neural-Style does not:
cmd:option('-mask_file', 'path/to/HDF5file', 'Spatial mask to constrain the gradient descent to specific region')
    -- Load mask if specified
    local mask = nil
    if params.mask_file ~= 'path/to/HDF5file' then
        local f = hdf5.open(params.mask_file, 'r')
        mask = f:all()['mask']
        f:close()
        mask = set_datatype(mask, params.gpu)
    end

I have been trying to figure out how to modify the above code for Neural-Style masks, but non of my attempts to replace the hdf5 requirement have worked thus far. Any ideas?

The code you now linked looks better, now the padding is inserted (line #127) before the convolution (line #141). Most of what you have highlighted is NOT the convolution but related to selecting between max and avg pooling. But if you follow the if logic, if the layer is convolution it will be inserted to the model in line 141 of your present code.
I cannot guarantee that it now works but now the padding and convolution come in the correct order.

"I have been trying to figure out how to modify the above code for Neural-Style masks, but non of my attempts to replace the hdf5 requirement have worked thus far. Any ideas?"
The code you cited does not implement any mask functionality, it only loads a mask from an existing hdf5 file.

I ran a quick test with the -reflectance option. The change is not particularly obvious at first glance, but it does appear to cause a change. More testing, and different parameter combinations could be needed to farther understand it's affect on artistic outputs.
On the left is the control test with -reflectance false, and on the right is -reflectance true:

Direct link to the comparison: https://i.imgur.com/YGCOCiu.png
False: https://i.imgur.com/0oQNsxl.png
True: https://i.imgur.com/a7fQTLb.png
Command used:
th neural_style.lua -seed 876 -reflectance -num_iterations 1500 -init image -image_size 640 -print_iter 50 -save_iter 50 -content_image examples/inputs/hoovertowernight.jpg -style_image examples/inputs/starry_night.jpg -backend cudnn -cudnn_autotune

Keeping track of the new features, and potential implementation into Neural-Style

Working:
cmd:option('-reflectance', false, 'if true, use reflectance padding')
From Repository: ProGamerGov/neural-style.

Not working:
cmd:option('-mask_file', 'path/to/HDF5file', 'Spatial mask to constrain the gradient descent to specific region')
cmd:option('-aesth_input', 'path/to/HDF5file', 'file with aesthetics prediction weights for different layers')
cmd:option('-loss_file', 'path/to/HDF5file', 'Name of file in which the tracked loss is saved')
cmd:option('-dilation', '1', 'comma separated list what dilations to return')
cmd:option('-nnf', 'path/to/HDF5file', 'the file with the nearest neibour field')
Color Control/Luminescence Style Transfer.
Scale Control Ex: Independent Style_scale values for different styles.
Spatial Control.
Transfer of the color of a single style image, when multiple style images are used.
From Repository: leongatys/NeuralImageSynthesis.

Are Gatys' Grad related functions different that Neural-Styles? I'm looking for where the style masks come into play. Or should I be looking at different functions for implementing these features like masks?

From what I can see, luminescence style transfer requires the LUV color space, which unlike YUV, it has no easy to use function in the image library.
Style masks seem to require a modifying deeper levels of the Neural-Style code.

For the independent style_scale control with multiple style images, it seems like we only need a way to disable content loss:
From the research paper:

We initialise the optimisation procedure with the coarse-scale image and omit the content loss entirely, so that the fine-scale texture from the coarse-style image will be fully replaced.

And then a simple sh script similar to multires.sh should do the trick. That runs your style images through Neural-Style first should do the trick, but such a script needs a way to disable content loss.
I am thinking that a parameter like:
cmd:option('-content_loss', true, 'if set to false, content loss will be disabled')
@htoyryla Which part of the content loss code should this be implemented on to achieve the desired effect?

I must be missing something when trying to replicate the "Naive scale combination" from here: https://github.com/leongatys/NeuralImageSynthesis/blob/master/ExampleNotebooks/ScaleControl.ipynb
Following the steps on the research paper:

Should result in something like this output that I made running Gatys' iPython code: https://i.imgur.com/boz8PhW.jpg
And the styled style image from his code: https://i.imgur.com/6xEumk0.jpg

But instead I get this:
The styled style image: https://i.imgur.com/30HUeOH.png
And here is the final output: https://i.imgur.com/SWhzMn0.png
I tried this code to create the styled style image: https://gist.github.com/ProGamerGov/53979447d09fe6098d4b00fc8e924109
And then ran:
th neural_style_c.lua -original_colors 1 -output_image out.png -num_iterations 1000 -content_image fig4_content.jpg -style_image out7.png -image_size 640 -save_iter 50 -print_iter 50 -seed 876 -init image -backend cudnn -cudnn_autotune

What am I doing wrong here?

Ok, so analyzing the styled style image from Gatys' code:
The outputs have the parameters used, and the values used, in the name:
[scimg_fig4_content.jpg_spimg_fig4_style2.jpg_simg_fig4_style3.jpg_hrpt_layer_relu4_1_hrsz_1024_model_norm_pad_ptw_1.0E+05](https://i.imgur.com/6xEumk0.jpg)
From another experiment using his code:
`cimg_cm_fig4_content.jpg_scimg_fig4_content.jpg_spimg_Amazing-Nature_3840x2160.jpg_simg_raime.jpg_pt_layer_relu2_1_sz_512_model_norm_pad_sw_2.0E+08_cw_1.0E+05_naive_scalemix.jpg
The enlarged version (I think 1 step multires?):
cimg_cm_fig4_content.jpg_scimg_fig4_content.jpg_spimg_raime.jpg_simg_Amazing-Nature_3840x2160.jpg_pt_layer_relu2_1_sz_512_hrsz_1024_model_norm_pad_sw_2.0E+08_cw_1.0E+05_naive_scalemix.jpg.filepart
And:
cimg_cm_fig4_content.jpg_scimg_fig4_content.jpg_spimg_raime.jpg_simg_Amazing-Nature_3840x2160.jpg_pt_layer_relu2_1_sz_512_model_norm_pad_sw_2.0E+08_cw_1.0E+05_naive_scalemix.jpg

On the subject of style and content weights:
*Neural-Style normalising the Gram Matrices differently, as it additionally divides by the number of features, when compared with Gatys' code. This means that the style loss weights for the different layers in Neural-Style and Gatys' code are a little different:
In a layer l with n_l = 64 features, a style loss weight of 1 in Neural-Style, is a style loss weight of 1/64^2 in Gatys' code.

"Neural-Style is normalising the Gram Matrices differently, as it additionally divides by the number of features, when compared with Gatys' code. This means that the style loss weights for the different layers in Neural-Style and Gatys' code are a little different:
In a layer l with n_l = 64 features, a style loss weight of 1 in Neural-Style, is a style loss weight of 1/64^2 in Gatys' code."

I am not familiar with Gatys's code, but what you wrote is confusing. First you say that Neural_style divides the Gram matrix by the number of features, but in your example you don't do this division.
If Gatys' normalizes by 1/C^2 where C is the number of features, it makes sense to me as the size of the Gram matrix is CxC.
In neural_style, the gram matrix is normalized for style loss in the line https://github.com/jcjohnson/neural-style/blob/master/neural_style.lua#L534
Here, input:nElements() is not C but CxHxW, where C,H,W are the dimensions of the layer to which the Gram matrix is added, so that in practice neural-style ends up with a smaller value for the normalized style loss than 1/C^2.
Dividing instead by self.G:nElements() would implement division by C^2 so if that's what you want, try it.
I don't know if this use of input:nElement() instead of self.G:nElements() here is intentional or an accident. @jcjohnson ?
There has been an earlier discussion about this division but there was nothing on this in particular: https://github.com/jcjohnson/neural-style/issues/90

I ran a quick test with the -reflectance option. The change is not particularly obvious at first glance, but it does appear to cause a change.

As padding only means adding a few pixels around the image I wouldn't expect large changes. Mostly this should be visible close to the edges, and indeed there appears to be a difference along the left hand side.

Changing line https://github.com/jcjohnson/neural-style/blob/master/neural_style.lua#L534 to divide by self.G:nElement(), I ran neural-style with defaults and got this.

whereas with the original the resulting image was

Now, they are obviously different but as the style weight now effective increased, we should not read too much into this difference. Anyway, this is worth more testing and the idea of normalizing this way makes intuitively sense to me.

Concerning YUV... I was under the impression that Y is the luminance.
When you want to disable content_loss, why not simply set content_weight to 0?

It looks like the 1/C^2 style normalization favors the lowest layers which have smaller C (64 for conv1 as opposed to 512 for conv5). The original neural-style behavior 1/(CxHxW) penalizes less the higher layers because H and W decrease when going to higher layers.

When you want to disable content_loss, why not simply set content_weight to 0?

I will try that as well later today. I think my settings from before were to different from Gatys' settings.
The other issue is that I think transferring the color from a third image, might be needed, as I would imagine that Gatys' would have used something similar to -original_colors 1 if it were the better solution.

I think I figure out the style combination:

The styled style image: https://i.imgur.com/G1eZerW.png
This was used to produce the final image:
th neural_style.lua -original_colors 1 -style_weight 10000 -output_image out3.png -num_iterations 1000 -content_image fig4_content.jpg -style_image out1_200.png -image_size 512 -save_iter 0 -print_iter 50 -seed 876 -init image -backend cudnn -cudnn_autotune
And this was used to produce the styled style image:
th neural_style_c.lua -content_weight 0 -style_weight 10000 -output_image out1.png -num_iterations 1000 -content_image fig4_style3.jpg -style_image fig4_style1.jpg -image_size 2800 -content_layers relu2_1 -style_layers relu2_1 -save_iter 50 -print_iter 50 -seed 876 -init image -backend cudnn -cudnn_autotune

A note on 1/C^2 gram matrix normalization: this line also needs to be changed https://github.com/jcjohnson/neural-style/blob/master/neural_style.lua#L553 so that the backward pass too will use the normalized matrix.
This will require quite different weights, like content_weight 1e3 and style_weight 1, it can take some 300 iterations before the image starts really to develop, but to me the results look good. I am talking about plain neural_style with modifed Gram matrix normalization. Haven't really looked deeper into the Gatys project.

ProGamerGov, just a little suggestion: since GPU handling is already implemented in "function setup_gpu(params)" (line 324), maybe it's possible to use that function instead of new "set_datatype(data, gpu)"?
It could make the code more maintainable – in case of any changes someone will have to modify only one function instead of two.
For example: pad_layer = nn.SpatialReflectionPadding(padW, padW, padH, padH):type(dtype)
(see how nn.SpatialAveragePooling(kW, kH, dW, dH):type(dtype) is added in line 136).
Currently I can not test it on GPU, but I can confirm that it does work on CPU.

@VaKonS
I'll take a look. I originally pasted in Gatys GPU handling code at the time because I couldn't get the reflection function to work with this line of code:
pad_layer = set_datatype(pad_layer, params.gpu)
As I couldn't figure out how to use function setup_gpu with the code.

@ProGamerGov, yes.
And to delete function set_datatype(data, gpu) at line 611, as it will not be needed anymore.

@VaKonS , I made a version that contains other padding types: https://gist.github.com/ProGamerGov/0e7523e221935442a6a899bdfee033a8
When using -padding, you can try 4 different types of padding: default, reflect, zero, or pad. In my testing, the pad option seems to leave untouched edges on other either side of the image.

Types 'reflect' and 'replication' make sense, although with the typical padding width = 1 as in VGG19 the result is identical.
Type 'zero' is superfluous as the convolution layer already pads with zeroes.
Type 'pad' only pads in one dimension so it hardly makes sense.
You should read nn documentation when using the nn layers. The nn.Spatial.... layers are meant to work with two-dimensional data like images. nn.Padding provides a lower level access for padding of tensors.

The code could also be structured like this (to avoid duplicating code and making the same checks several times). Here I used 'reflect' and 'replicate' as they are shorter, you may prefer 'replication' and 'reflection' as in the layer names. But having one as a verb and the other as a noun is not a good idea.
local is_convolution = (layer_type == 'cudnn.SpatialConvolution' or layer_type == 'nn.SpatialConvolution')
if is_convolution and params.padding ~= 'default' then
              local padW, padH = layer.padW, layer.padH
			  if params.padding == 'reflect' then
                  local pad_layer = nn.SpatialReflectionPadding(padW, padW, padH, padH):type(dtype)
              elseif params.padding == 'replicate' then
                  local pad_layer = nn.SpatialReplicationPadding(padW, padW, padH, padH):type(dtype)
			  else
			      error('Unknown padding type')
			  end
              net:add(pad_layer)
              layer.padW = 0
              layer.padH = 0
end

@htoyryla, reflective padding probably takes pixels starting from 1 pixel distance: [ x-2, x-1, x ] [ x-1, x-2 ].

Yes, I just realized that when I did a small test. That explains why it made a difference also with padding of one row/column.
Lähetetty iPadista

Valentin<EMAIL_ADDRESS>kirjoitti 15.2.2017 kello 16.35:
@htoyryla, reflective padding probably takes pixels starting from 1 pixel distance: [ x-2, x-1, x ] [ x-1, x-2 ].
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

I have been trying to get this python script to work for the linear color feature found in Gatys' code here: https://github.com/leongatys/NeuralImageSynthesis/blob/master/ExampleNotebooks/ScaleControl.ipynb
https://gist.github.com/ProGamerGov/5fc5ef9035edc9a026e41925f733a45c
The idea is that making this feature into a simple python script will be easier and less messy than implementing into neural_style.lua. But I can't figure out the python parameters so that the image is fed into the function properly.

About your first script https://gist.github.com/ProGamerGov/5fc5ef9035edc9a026e41925f733a45c
To make it process the images and save the result you need something like this. You did not pass the images to your function and you did not use the resulting image returned by the function. Remember that the function parameters target_img and source_img are totally separate from the variables with the same names, usually it is a good practice to avoid using the same names for both.
The numpy imports were needed, on the other hand I had to use skimage.io instead of PIL for reading and saving the image, probably they use a different format for the image inside python. Anyway, Gatys used imread() and not Image.open().
This works in principle but the resulting image is probably not what one would expect. It could be that some kind of pre/deprocessing is needed which was not obvious to me (not being familiar with the process you are trying to duplicate).
import scipy
import h5py
import skimage
import os
from skimage import io,transform,img_as_float
from skimage.io import imread,imsave
from collections import OrderedDict
#from PIL import Image, ImageFilter
import numpy as np
from numpy import eye
import decimal
#import click

target_img = imread('to.png')
source_img = imread('from.png')

def match_color(target_img, source_img, mode='pca', eps=1e-5):
    ....
    return matched_img

output_img = match_color(target_img, source_img)
imsave('result.png', output_img)

OK, by still changing the two imread lines to
target_img = imread('to.png').astype(float)/256
source_img = imread('from.png').astype(float)/256

from these two images

I get this (don't know if this is what is expected but it looks ok)

Just noticed that there was already an import for img_as_float so these work as well
target_img = img_as_float(imread('to.png'))
source_img = img_as_float(imread('from.png'))

The script seems to work like the outputs Gatys' code produced in the iPython interface now:
The source image:

The target images:

The images I used can be found in Gatys respository here, and in my Imgur album here: https://imgur.com/a/PrKtg.

Python interpreter is useful for testing small things (just like th in lua):
Python 2.7.6 (default, Oct 26 2016, 20:30:19)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import skimage
>>> from skimage import io,transform,img_as_float
>>> from skimage.io import imread,imsave
>>> img_as_float
<function img_as_float at 0x7f3bc9cad230>
>>> img = imread('to.png')
>>> img
array([[[255, 255, 255],
        [255, 255, 255],
        [255, 255, 255],
        ...,
        [133, 119, 112],
        [101,  84,  85],
        [ 54,  45,  44]],
>>> img_as_float(img)
array([[[ 1.        ,  1.        ,  1.        ],
        [ 1.        ,  1.        ,  1.        ],
        [ 1.        ,  1.        ,  1.        ],
        ...,
        [ 0.52156863,  0.46666667,  0.43921569],
        [ 0.39607843,  0.32941176,  0.33333333],
        [ 0.21176471,  0.17647059,  0.17254902]],

I got the script to accept user specified parameters: https://gist.github.com/ProGamerGov/d0917848a728bceb4131272734f61e8b
Only the target and source image are required, but you can also control the eps value and the transfer mode. Though the --eps parameter currently only accepts values in scientific notation.

It seems you don't understand how functions work. When you define a function like match_color you specify the parameters that are input to the function when it is called.
When you call the function you give the actual values for those parameters.
What you are doing now is defining a function so that the default values of transfer_mode and eps are defined from user input. It works when you only run the function once but it is confusing. That is not the way to pass values into a function.
You should change the def line as it was and add the actual values of transfer_mode and eps to the line where the function is called (like I already suggested).
output_img = match_color(target_img, source_img, mode=transfer_mode, eps=int(float(eps_value)))

BTW, I don't understand the int() for eps... first we give something like 1e-5, then float it and finally int which gives 0. So you limit eps to integer values only?

It seems you don't understand how functions work.

It works when you only run the function once but it is confusing. That is not the way to pass values into a function.

I went for making the code work, without putting a lot of focus on how. Which is a terrible way to go about coding.

You should change the def line as it was and add the actual values of transfer_mode and eps to the line where the function is called (like I already suggested).

Yea, I see that now. Not sure what I was thinking at time when I made such an embarrassing and obvious mistake.

BTW, I don't understand the int() for eps... first we give something like 1e-5, then float it and finally int which gives 0. So you limit eps to integer values only? Why the int?

It was the first that worked, which I now think was because I fixed a bracket placement error. I have removed the integer limitation.

I think I am getting close to the research paper's results:
Layers relu2_1,relu3_1:

Control test:

Can you give the commands how to run the whole process, would like to test.

@htoyryla

Images used:
fig4_content.jpg: https://github.com/leongatys/NeuralImageSynthesis/blob/master/Images/ControlPaper/fig4_content.jpg
Fine style: https://github.com/leongatys/NeuralImageSynthesis/blob/master/Images/ControlPaper/fig4_style1.jpg
Course style: https://github.com/leongatys/NeuralImageSynthesis/blob/master/Images/ControlPaper/fig4_style2.jpg

Step 1:
python linear-color-transfer.py --target_image coarse_style.png --source_image fig4_content.jpg --output_image coarse_pca.png
python linear-color-transfer.py --target_image fine_style.png --source_image fig4_content.jpg --output_image fine_pca.png
Step 2 (Texture transfer):
 th neural_style.lua -tv_weight 0 -content_weight 0 -style_weight 10000 -output_image out5.png -num_iterations 550 -content_image coarse_pca.png -style_image fine_pca.png -image_size 1536 -content_layers relu2_1,relu4_2 -style_layers relu2_1,relu4_2 -save_iter 50 -print_iter 50 -seed 876 -init image -backend cudnn -cudnn_autotune
Step 2.5 (I don't think Gatys' code does this, but I thought it would make the colors look better):
python linear-color-transfer.py --target_image out5.png --source_image fig4_content.jpg --output_image out5_pca.png
Then I tried to mimic Gaty's two step process where the first image is generated at 512px:
th neural_style.lua -style_weight 10000 -output_image out_final.png -num_iterations 1000 -content_image fig4_content.jpg -style_image out5_pca.png -image_size 512 -save_iter 0 -print_iter 50 -seed 876 -init image -backend cudnn -cudnn_autotune

th neural_style.lua -style_weight 10000 -output_image out_final_hr.png -num_iterations 550 -content_image fig4_content.jpg -init_image out_final.png -style_image out5_pca.png -image_size 1536 -save_iter 0 -print_iter 50 -seed 876 -init image -backend cudnn -cudnn_autotune

Those commands in that order should give you the exact same output as I got.

After making more tests with different models, I was wrong: the noise is not added by padding.
It's a quality of some models: vgg19 from crowsonkb's repository makes clean images with or without padding, and images made with Illustration2Vec, for example, have noisy borders even with default padding.

Examining the outputs produced by ScaleControl.ipynb:

Gatys' Scale Control code produce 3 different outputs, each follows a two step Multires process. I am not sure if these are 3 different ways of doing Scale Control, or if 1 or 2 of them are meant to showcase ways that don't work?
For each of the 3 options in the iPython script, I ran the code and generated the images. Each produced a low resolution 648x405 image and then a 1296x810 "hr" resolution image.
Gatys' iPython code names the images with the parameters used to create them, and as such I have included the image file names.
"Stylemix images" are what Gatys calls the resulting combination style image made of both the coarse and fine style images.

Combine 2 images with fine and coarse scale:

512 and hr res: https://imgur.com/a/D7AcK

File names:

512:
cimg_cm_fig4_content.jpg_scimg_fig4_content.jpg_spimg_fig4_style2.jpg_simg_fig4_style3.jpg_pt_layer_relu2_1_sz_512_model_org_pad_sw_1.0E+03_cw_1.0E+00.jpg

hr:
cimg_cm_fig4_content.jpg_scimg_fig4_content.jpg_spimg_fig4_style2.jpg_simg_fig4_style3.jpg_pt_layer_relu2_1_hrpt_layer_relu4_1_sz_512_hrsz_1024_model_org_pad_sw_1.0E+03_cw_1.0E+00.jpg

norm net:

512 and hr res: https://imgur.com/a/oTB1k

File names:

cimg_cm_fig4_content.jpg_scimg_fig4_content.jpg_spimg_fig4_style2.jpg_simg_fig4_style3.jpg_pt_layer_relu2_1_sz_512_model_norm_pad_sw_2.0E+08_cw_1.0E+05.jpg

cimg_cm_fig4_content.jpg_scimg_fig4_content.jpg_spimg_fig4_style2.jpg_simg_fig4_style3.jpg_pt_layer_relu2_1_hrpt_layer_relu4_1_sz_512_hrsz_1024_model_norm_pad_sw_2.0E+08_cw_1.0E+05.jpg

Naive scale combination:

512 and hr res: https://imgur.com/a/LbqJQ

File names:

cimg_cm_fig4_content.jpg_scimg_fig4_content.jpg_spimg_fig4_style2.jpg_simg_fig4_style3.jpg_pt_layer_relu2_1_sz_512_model_norm_pad_sw_2.0E+08_cw_1.0E+05_naive_scalemix.jpg

cimg_cm_fig4_content.jpg_scimg_fig4_content.jpg_spimg_fig4_style2.jpg_simg_fig4_style3.jpg_pt_layer_relu2_1_sz_512_hrsz_1024_model_norm_pad_sw_2.0E+08_cw_1.0E+05_naive_scalemix.jpg

The code in addition to creating the two images for each example, also created 4 stylemix images:
Stylemix image: https://i.imgur.com/m7nRgKP.jpg
Name:
scimg_fig4_content.jpg_spimg_fig4_style2.jpg_simg_fig4_style3.jpg_hrpt_layer_relu4_1_hrsz_1024_model_norm_pad_ptw_1.0E+05.jpg

Stylemix image: https://i.imgur.com/XPt6N52.jpg
Name:
scimg_fig4_content.jpg_spimg_fig4_style2.jpg_simg_fig4_style3.jpg_pt_layer_relu2_1_sz_512_model_norm_pad_ptw_1.0E+05

Stylemix image: https://i.imgur.com/Vf3mg2n.jpg
Name:
spimg_fig4_style2.jpg_simg_fig4_style3.jpg_hrpt_layer_relu4_1_hrsz_1024_model_org_pad_ptw_1.0E+03.jpg

Stylemix image: https://i.imgur.com/c1ZNDoZ.jpg
Name:
spimg_fig4_style2.jpg_simg_fig4_style3.jpg_pt_layer_relu2_1_sz_512_model_org_pad_ptw_1.0E+03.jpg

I am not sure why there are 3 Examples of Scale Control, and 4 stylemix images. But I assume one of the examples must use 2 stylemix images?

Ok, so trying both models from Gatys repository which are the normalized VGG19, and the VGG-19 Conv model, I can't seem to get the parameters right.

Using Gatys' weights for Scale Control in Neural-Style seems to work pretty well:

@ProGamerGov, by the way, the code can probably be reimplemented in Torch and maybe even will not be too different:

Python
Torch

numpy.mean
torch.mean

numpy.linalg.cholesky
torch.potrf

numpy.linalg.eigh
torch.symeig

numpy.eye
torch.eye

numpy.dot
torch.dot

numpy.transpose
transpose / permute

.T
t()

numpy.reshape
torch.reshape

numpy.linalg.inv
torch.inverse

numpy.diag
torch.diag

numpy.sqrt
torch.sqrt

@VaKonS Thanks, I'll take a look at that.

@htoyryla I have started trying to extract the python code responsible for luminescence style transfer: https://gist.github.com/ProGamerGov/08c5d25bb867e4313821a45b2e3b2978
As I understand it, the research paper basically describes converting your content/style images to LUV or YIQ, before running them through the style transfer network. In his python code, Gatys appears to use LUV, so I'll start with that.
Testing those 3 functions:
rgb2luv creates this:

luv2rgb creates this:

lum_transform results in this error whenever I try to use it:
ubuntu@ip-Address:~/neural-style$ python lum2.py --input_image fig4_content.jpg
Traceback (most recent call last):
  File "lum2.py", line 47, in <module>
    output_img = lum_transform(input_img)
  File "lum2.py", line 32, in lum_transform
    img = tile(lum[None,:],(3,1)).reshape((3,image.shape[0],image.shape[1]))
NameError: global name 'tile' is not defined
ubuntu@ip-Address:~/neural-style$

These functions come from Gatys' Color Control code here: https://github.com/leongatys/NeuralImageSynthesis/blob/master/ExampleNotebooks/ColourControl.ipynb
I am not sure if lum_transform is needed to perform the LUV change to and from for style transfer.

Edit: "tile" is part of numpy, and just like "eye" from linear-color-transfer.py, it needs a "np." appended to it's front.

Good you found it. I just woke up and was about to comment :)

How do the Gatys models fail when you try them?
I can load the conv model using the prototxt, both from the links you gave.
th> require "loadcaffe"
{
  load : function: 0x416b6098
  C : userdata: 0x41383dc8
}
th> model_file = "VGG_ILSVRC_19_layers_conv.caffemodel"
                                                                      [0.0001s]
th> proto_file = "VGG_ILSVRC_19_layers_deploy.prototxt"
                                                                      [0.0001s]
th> cnn = loadcaffe.load(proto_file, model_file, "nn")
Successfully loaded VGG_ILSVRC_19_layers_conv.caffemodel
conv1_1: 64 3 3 3
conv1_2: 64 64 3 3
conv2_1: 128 64 3 3
conv2_2: 128 128 3 3
conv3_1: 256 128 3 3
conv3_2: 256 256 3 3
conv3_3: 256 256 3 3
conv3_4: 256 256 3 3
conv4_1: 512 256 3 3
conv4_2: 512 512 3 3
conv4_3: 512 512 3 3
conv4_4: 512 512 3 3
conv5_1: 512 512 3 3
conv5_2: 512 512 3 3
conv5_3: 512 512 3 3
conv5_4: 512 512 3 3
                                                                      [0.2703s]

@htoyryla How do the Gatys models fail when you try them?
The style loss function does not work with them for me with any variation of this command for both models:
th neural_style.lua -content_weight 0 -style_weight 10000 -image_size 1024 -output_image out_norm_hr.png -num_iterations 500 -content_image result_2.png -style_image result.png -content_layers relu2_1,relu4_1 -style_layers relu2_1,relu4_1 -model_file models/vgg_normalised.caffemodel -proto_file models/VGG_ILSVRC_19_layers_deploy.prototxt.lua -save_iter 50 -print_iter 50 -seed 876 -init image -backend cudnn -cudnn_autotune

For me this
th neural_style.lua -model_file ../test/pg/VGG_ILSVRC_19_layers_conv.caffemodel -proto_file ../test/pg/VGG_ILSVRC_19_layers_deploy.prototxt

works on a clean copy of neural-style and produces this image at 400 iterations.

I see you tried the normalized... I have almost never used the normalized models, they probably require different weight values.

Normalized works too, but the losses are very small (which I think is typical for a normalized model)
Iteration 50 / 1000
  Content 1 loss: 0.000000
  Style 1 loss: 4.361065
  Style 2 loss: 0.464805
  Style 3 loss: 0.069460
  Style 4 loss: 0.017796
  Style 5 loss: 0.015553
  Total loss: 4.928679
Iteration 100 / 1000
  Content 1 loss: 0.000000
  Style 1 loss: 3.661895
  Style 2 loss: 0.397874
  Style 3 loss: 0.057522
  Style 4 loss: 0.017153
  Style 5 loss: 0.015032
  Total loss: 4.149477

Looking into where the functions are located in Gatys' code here: https://github.com/leongatys/NeuralImageSynthesis/blob/master/ExampleNotebooks/ColourControl.ipynb
lum_transform seems to come before the style transfer process:
 if cp_mode == 'lum':
        org_content = imgs['content'].copy()
        for cond in conditions:
            imgs[cond] = lum_transform(imgs[cond])
        imgs['style'] -= imgs['style'].mean(0).mean(0)
        imgs['style'] += imgs['content'].mean(0).mean(0)
        for cond in conditions:
            imgs[cond][imgs[cond]<0] = 0
            imgs[cond][imgs[cond]>1] = 1

And then rgb2luv and luv2rgb are used:
#execute script
    !{script_name}
    output = deprocess(get_torch_output(output_file_name))
    if cp_mode == 'lum':
        org_content = rgb2luv(org_content)
        org_content[:,:,0] = output.mean(2)
        output = luv2rgb(org_content)
        output[output<0] = 0
        output[output>1]=1
    imshow(output);gcf().set_size_inches(8,14);show()
    imsave(result_dir + result_image_name, output)

But I am not sure exactly what the code is doing. Though I haven't been able to get the code that uses rgb2luv and luv2rgb to work properly yet.

@htoyryla If they are working for you, then it could have been something weird with the instance I was running, or a corrupted/incorrect prototxt file with the same name.

I downloaded the models and the prototxt from the links you gave, otherwise a fresh copy of neural-style. Maybe you have some modifications in your neural-style? The conv model worked just as usual, the normalized one didn't produce a good image with the default content and style weights, but I've seen that before too.

Difficult to say what a code snippet is doing without being familiar with the whole of it.
But from the paper the color control looks basically simple.

"The modification is simple. The luminance channels LS and LC are first extracted from the style and content images. Then the Neural Style Transfer algorithm is applied to these images to produce an output luminance image Lˆ. Using the YIQ colour space, the colour information of the content image is represented by the I and Q channels; these are combined with Lˆ to produce the final colour output image (Fig. 3(d))."

In other words, one makes luminance-only versions of content and style images, runs them through neural-style and finally applies the color information from the content image to it. I am not too familiar with the these color schemes, but it would be simple to try how this works using YUV and neural-style.
Gatys then goes deeper into using histogram matching for the cases where the histograms of the two images are quite different.

Speaking of "vgg_normalised.caffemodel": together with much higher style weight, it probably requires "normalize_gradients". In above examples with this model, the content weight was 100, style weight 300000, and gradients were "semi-normalized" with scale 0.7.

Related to the discussion above, I made a quick try to run neural-style on luminance (Y) only when -original_colors == 1, then add color (UV) from the content image https://gist.github.com/htoyryla/38a4d6b2280ed5b4e47fc8d67b304f9f
Using my modified code,  Gatys VGG19 conv model and neural-style defaults, style transfer with luminance only, followed by transferring color from the content image:

For comparison, original_colors == 0 (style transfer with color):

For comparison, unmodified neural_style  with original_colors == 1 (style transfer with color, followed by transferring color from the content image)

In above examples with this model, the content weight was 100, style weight 300000, and gradients were "semi-normalized" with scale 0.7.

I can't see which examples you are referring to, but never mind. I was only suggesting that probably there is nothing wrong with the models.

I further tried to add the first histogram adjustment by Gatys (formula 10 in the paper), adjusting the luminance-only style image (before preprocessing) so that its mean and variance match the luminance-only content image:
 for _, img_path in ipairs(style_image_list) do
    local img = image.load(img_path, 3)
    img = image.scale(img, style_size, 'bilinear')
    if params.original_colors == 1 then
	  -- use luminance only
	  img = image.rgb2yuv(img)[{{1, 1}}]
	 -- match historgram
	  cmean = torch.mean(content_image)
	  smean = torch.mean(img)
	  cvar = torch.var(content_image)
	  svar = torch.var(img)
	  img = img:add(-smean):mul(cvar/svar):add(cmean)
    end
    local img_caffe = preprocess(img):float()
    table.insert(style_images_caffe, img_caffe)

I guess something went wrong, but somehow I like how this looks:

In this version I separated style transfer (color or luminance), histogram matching (none, whole tensor, channel-wise) and original_colors, allow trying different approaches (for instance doing style transfer with color, histogram matching per channel and finally restoring original colors. Histogram matching is by mean and var only.
Note that I have not really looked at Gatys' code other than some snippets, this is simply based on reading a page of Gatys' paper. Do not assume that my params work like Gatys'.
-- quick hack to make neural-style do
-- 1 either luminance-only or color based style transfer (--transfer lum|color)
-- 2 match style histogram to content (--histogram no|all|rgb)
-- no: no histogram matching
-- all: match whole tensor (use this with --transfer lum)
-- rgb: match each channel separately
-- 3 restore original colors can be combined with the above
-- NOT ALL COMBINATIONS ARE GUARANTEED TO WORK
https://gist.github.com/htoyryla/af9de7a712d74d12f5d3acc7725e6229

So using this terrible and messy code: https://gist.github.com/ProGamerGov/ba9a9d54bae53e84ebf0116262df6758
I think I have achieved luminescence style transfer without editing neural_style.lua:

Images used:
https://github.com/leongatys/NeuralImageSynthesis/blob/master/Images/ControlPaper/fig3_style1.jpg
https://github.com/leongatys/NeuralImageSynthesis/blob/master/Images/ControlPaper/fig3_content.jpg

Step 1:
First you transfer the color from your content image, to your style image like so:
python linear-color-transfer.py --target_image fig3_style1.jpg --source_image fig3_content.jpg --output_image style_colored_pca.png

Then you run that through the lum_transfer.py script like this:
python lum_transfer.py --content_image fig3_content.jpg --style_image style_colored_pca.png --cp_mode lum --output_style_image output_lum_style_pca.png --output_content_image output_lum_style_pca.png

Now you run your content image through lum_transfer.py like this:
python lum_transfer.py --content_image fig3_content.jpg  --style_image fig3_content.jpg --cp_mode lum --output_content_image out_lum_transfer.png

Step 2:
Now you run both newly created images through Neural-Style like this:
th neural_style.lua -original_colors 0 -image_size 1000 -content_weight 1 -style_weight 1e3 -output_image out_lum6_test.png -content_image out_lum_transfer.png -style_image output_lum_style_pca.png -num_iterations 1500 -save_iter 50 -print_iter 50 -seed 876 -init image -backend cudnn -cudnn_autotune

Once you've finished with Neural-Style, then you use the lum_transfer.py script again like this:
python lum_transfer.py --output_lum2 out_lum6_test_400_r.png --content_image out_lum6_test_400.png --style_image output_style_colored.png --cp_mode lum2 --output_style_image output_style_2.png --output_content_image output_content_2.png --output_image out_combined.png --org_content fig3_content.jpg

Now you can leave your image as is, or you can try to change the colors slightly like this:
python linear-color-transfer.py --target_image out_combined.png --source_image fig3_content.jpg --output_image out_combined_lct_pca.png

python linear-color-transfer.py --mode sym --target_image out_combined.png --source_image fig3_content.jpg --output_image out_combined_lct_sym.png

python linear-color-transfer.py --mode chol --target_image out_combined.png --source_image fig3_content.jpg --output_image out_combined_lct_chol.png

And here are my outputs generated by this process:
The final unmodified output:

--mode pca:

--mode sym:

--mode chol:

Album of the full resolution final images: https://imgur.com/a/AW6qU
Screenshot from the research paper:

Control Output using -original_colors 1 and the unmodified style/content images:

Looking at the research paper's examples, and my examples, it seems really difficult to actually see the difference between Luminescence Style Transfer, and normal style transfer.
@htoyryla Your modifications to neural_style.lua created this:

And using linear-color-transfer.py has little to no effect on the output with all 3 modes: https://imgur.com/a/wmNyO

The -original_colors 1 parameter in Neural-Style uses YUV, where the "Y" part deals with luminance. So this parameter's effect on our results should be noted.

-original-colors 1 does not affect the style transfer at all, it only adjusts the output image so that luminance comes from the output image and color from the content image.
One of Gatys' proposals is to make style transfer using luminance only and only afterwards add color just like with original-color 1.

The modification is simple. The luminance channels LS and LC are first extracted from the style and content images. Then the Neural Style Transfer algorithm is applied to these images to produce an output luminance image Lˆ. Using the YIQ colour space, the colour information of the content image is represented by the I and Q channels; these are combined with Lˆ to produce the final colour output image (Fig. 3(d)).

In my latter version this is done with -transfer lum -original-colors 1.
Gatys then goes on

If there is a substantial mismatch between the luminance histogram of the style and the content image, it can be help- ful to match the histogram of the style luminance channel LS to that of the content image LC before transferring the style. For that we simply match mean and variance of the content luminance.

In my latter version, this is done (although in YUV space) by -transfer lum -histogram all -original-colors 1
I have not implemented the color matching method described in section 5.2 of the paper. However, when he later writes

In comparison, when using full style transfer and colour matching, the output image really consists of strokes which are blotches of paint, not just variations of light and dark.

I wanted to try the something like this although with simple histogram matching, setting -transfer color -histogram rgb -original_colors 1
Note that when doing histogram matching, the change in the levels of the style image may require different style weight settings.
I hope this clarifies the logic behind my code.

@htoyryla, I was reffering to those examples. Just a few tests with different styles and models, my mistake was that I didn't test other models at first, and made wrong assumptions.
@ProGamerGov, looks awesome!
Just to be sure: is line 101 supposed to be "content_img = lum_transform(style_img)"?
In original code there is "imgs[cond] = lum_transform(imgs[cond])", which probably means that content_img should be converted from content_img, not from style_img.
Anyway, results look great!
Second, I think I've found a way to get rid of noise near the borders.
It' very simple - to add 1 pixel gray border around the image at the very beginning, right after "local net = nn.Sequential()" and before "nn.TVLoss" layer.
Something like:
  if params.padding ~= 'default' then
    print('Padding image with zeroes')
    net:add(nn.SpatialZeroPadding(1, 1, 1, 1):type(dtype))
  end

Results (above are regular versions, below are with gray border, variants with "default-reflected-replicated" paddings):

And it doesn't seem to add noise to previously clean images, as far as I can tell:

Disadvantage: gray border appears around the edge with some models, so it should be optional:

It looks like there are many variant procedures possible based on Gatys' paper. Like these lines in @ProGamerGov's code https://gist.github.com/ProGamerGov/ba9a9d54bae53e84ebf0116262df6758#file-lum_transfer-py-L98-L103 correspond to luminance only style transfer (assuming the correction by @VaKonS above) but unlike my implementation based on a paragraph in the paper quoted above, here the mean of style image is additionally adjusted to match the content image.
My -histogram option then does that but also adjusts the variance according to formula 10 in the paper.

@VaKonS isn't padding with zeroes actually padding with black pixels. It seems straightforward though to modify SpatialZeroPadding to pad with a given value, like 0.5 for gray.

@htoyryla, nn.SpatialZeroPadding pads with zeroes, but images are converted to mean-centered first, therefore zeroes in processing layers are actually mean values in images, which should be gray pixels in most cases. At least I think so.

@htoyryla, images mean values in "Neural-style" are not "should be gray", but exactly RGB = 123.68, 116.779, 103.939, if I'm not mistaken.

Hmmm.... I have been under impression that in Torch images pixel values are between 0 and 1. You can test this by creating a torch.Tensor(3,512,512):fill(0) and torch.Tensor(3,512,512):fill(1) and saving them (I'll check that too).
Images need to be adjusted for models bu I don't think that can change the meaning of 0 and 1 in a Torch image.

Experimenting with my code, I have been thinking of what Gatys et al wrote: "If there is a substantial mismatch between the luminance histogram of the style and the content image, it can be help- ful to match the histogram of the style luminance channel LS to that of the content image LC before transferring the style."
I think here is an example. This is a style image which was itself created by neural-style.

Using it as a style image without histogram matching can result in something like this

while with histogram matching one can get something like this (not really like the style but still looks better)

But it is not so obvious when histogram matching works and when not. From the same two images, with different settings, I get this with histogram matching (far too dark)

and now I get quite close to the original style without histogram matching

In these examples I did not use luminance transfer nor original_colors, just histogram on and off. My feeling based on this is that histogram matching does not necessarily help getting the exact look of the original style, but it can help getting a good balance between style and content in some problematic cases. And it can also be used in creating pictures with a strong interesting style which is noticeably different from the original style.

@htoyryla, I thought that histogram matching tricks were to make stylized image use original content colors while keeping some elements from style whose colors were not present in content image.
p. s. It's my mistake with image padding – I shouldn't probably have used "image padding", because it's "layer padding", as you have correctly noticed.

Looking at @ProGamerGov's lum_transfer.py, which if I understand correctly is used to produce an image file for subsequent input into neural_style. In this process it seems to me that the image is stored as an image file (png?) so the question is: is it safe to put, say, LUV image data into a png file and recover the correct data when read into neural-style later. I don't know about png but it seems to me that it might be for greyscale or RGB only.

I thought that histogram matching tricks were to make stylized image use original content colors while keeping some elements from style whose colors were not present in content image.

Reading carefully, yes, that's what the paper says. My first example, I think, works exactly that way. In the second example, histogram matching gave too extreme results but leaving it off gave a very good (in my eyes) result preserving even the sense of depth in the style, even if the original coloring is not preserved.
Personally, I lean towards original artistic application of these techniques, and don't look for a single perfect tool for everything, but rather a versatile toolbox. For an example of an interesting result using histogram matching see my cubistic portrait from yesterday. Funny... that too was made from the same photo as these examples today.

So I have now made I refined version of my lum_transfer.py script: https://gist.github.com/ProGamerGov/2e7a0fe7a5ef6e117dc0be81df243331
Now the process only takes 4 commands (including neural_style.lua) to complete:
python linear-color-transfer.py --target_image fig3_style1.jpg --source_image fig3_content.jpg --output_image style_colored_pca.png

python lum_transfer.py --content_image fig3_content.jpg --style_image style_colored_pca.png --cp_mode lum --output_style_image output_lum_style_pca.png --output_content_image output_lum_content_pca.png --org_content fig3_content.jpg

th neural_style.lua commands here...

python lum_transfer.py --output_lum2 out_lum6_test_400_r.png --cp_mode lum2 --output_image out_combined.png --org_content fig3_content.jpg

For lum_transfer.py, the outputs and required inputs are now dependent on the --cp_mode you choose. Though for --cp_mode lum2 and --output_lum2 there is an issue. One must resize either their Neural-Style output to match their original content image's size, or vice version. Though I would image that in order to preserve the quality of your Neural-Style output, you would want to resize your content image.
Does anyone know how I can resize the content image in the script, to match that of the Neural-Style output image?

@ProGamerGov, here, for example, images are transformed like this:

from skimage import io, transform
im = io.imread(input_name)
im = transform.resize(im, (im.shape[0]*scale, im.shape[1]*scale), order=3)
io.imsave(output_name, im)

@ProGamerGov, this also works for me (note that "transform" must be imported from skimage, it seems, or scipy will not find its "misc" part):
import scipy
from skimage import io, transform
im = io.imread(input_name)
im = scipy.misc.imresize(im, (im.shape[0]*2, im.shape[1]*2))
io.imsave(output_name, im)

@VaKonS This is the specific part of the script in which the resizing needs to take place:
elif cp_mode == 'lum2':
            output = args.output_lum2
            org_content = args.org_content
	    org_content = imread(org_content).astype(float)/256
	    output = imread(output).astype(float)/256
            org_content = rgb2luv(org_content)
            org_content[:,:,0] = output.mean(2)
            output = luv2rgb(org_content)
            output[output<0] = 0
            output[output>1]=1
	    imsave(output_a_name, output)

Do I need to read both images before they are converted to the required float value?

This is the control image with no modifications:

This the control image run through the last lum_transfer.py step:

This is the output I made following the steps I have outlined above:

This one is the same one as the above image, except for the fact that I swapped the content and style images for the first step, so linear-color-transfer.py transferred the color from the style image, to the content image:

I am not sure if it's the content and style images that I am using, but the luminance changes seem very subtle to me.

Maybe it is just the style image's luminance?
Following the luminance transfer steps on this new style image, results in this:

Control Test with the last lum_transfer step and -original_colors 1:

Looking at the images side by side (control on the right):

The difference is much more apparent with this style image. For example, the lighting on the grass is different between the two images

@ProGamerGov, scipy.misc.imresize seems to automatically convert output to array of [0...255] integers, therefore if you need floats, you'll have to resize image first, and then convert if to ".astype(float)/256".
On the contrary, it looks like skimage.transform.resize accepts and returns array of floats [0...1], so with it you'll need to convert to float/256 at first, and after that resize.

@ProGamerGov, shouldn't image arrays be divided by 255, by the way?

Theses are the results of my luminance tests for iteration count, and -image_size:

Full comparison image here: https://i.imgur.com/xLLukch.png

@VaKonS Thanks for the help, using skimage.transform.resize did the trick:
org_content = skimage.transform.resize(org_content, output.shape)
https://gist.github.com/ProGamerGov/2e7a0fe7a5ef6e117dc0be81df243331
The script now doesn't require the user to do any manual resizing.

I repeat my question about @ProGamerGov's process, as it seems to me that in order to do luminance only transfer, a python script will produce the luminance-only copies of the images which are then supposed to undergo style transfer in plain neural style. There are some problems in this process as I see it and I am not sure if they have been addressed properly.
How do we get these luminance only images into neural style?
How do we get neural-style to understand that the input is luminance-only and not RGB, and further, how does neural-style process luminance-only data when the model expects three channels (RGB).
If we just save output for luminance conversion into a png file, then an unmodified neural-style will read it assuming RGB and the result will be something quite different from luminance-only transfer.
The simple solution would be, I think, to save the luminance-only images as RGB so that the L channel is copied, possible scaled, into R,G and B channels. This is what I do in my modified neural-style.

@htoyryla

python scripts with neural-style, in order to do luminance only transfer

Looking at the naming of the feature in Gatys' code, he calls it "Color Control", which leads me to believe it's not just about the luminance transfer. His code does not produce the same outputs as that of the research paper, which makes me wonder if the changes are intentional as the code seems to produce the same outputs for the other features in the research paper.
For instance, when comparing the results of your neural_style.lua modifications, to his results, it's like the only difference is that he replaced the color white, with the color yellow:
Results with your code:

From the research paper:

As to my earlier note wondering whether png can be used to contain yuv data instead of rgb (assuming that the receiving end knows that the contents are yuv instead of rgb), I made a test converting to yuv, saving into png, reading back, converting into rgb and saving again. There was an easily noticeable color shift.

So would these results indicate that png can be used to contain yuv data instead of rgb?

So would these results indicate that png can be used to contain yuv data instead of rgb?

My result shows that using png to contain YUV changes the color content. The color nuances in the original image have been replaced by shades of pink when the image has passed from rgb -> yuv -> png -> read as yuv -> rgb.
But using png to carry YUV content is not the main issue, there is also more serious issue that neural-style and VGG are designed to work with RGB, not YUV (to which my solution is converting Y to RGB by setting R,G,B = Y).
You seem to suggest that Gatys' code implements some other scheme than described in the paper. It can well be, but I am not convinced that your approach duplicates his thinking, if you cannot even explain how your process handles the problems I mentioned.
After all, I was just raising some valid questions about the process, and to me nicely looking results are not a proof that the process is correctly implemented. Making correctly working software requires an understanding of both the process and all details of the implementation. If the implementation is divided between separate programs, one needs to understand how to make them working correctly together and passing information correctly between them. For instance, in this particular case one needs to be aware in which format the image is being processed at any point in the process.
It seems to me that I am wasting my time now, so I'll most likely leave this thread.

@htoyryla, @ProGamerGov, if I'm not mistaken, then the code actually does not save YUV values in PNGs.
line 126: "output = luv2rgb(org_content)" converts YUV colorspace back to RGB before saving.

I think that at least two functions from "Controlling Perceptual Factors in Neural Style Transfer" can already be used in Neural-style, improving the quality of results and not overcomplicating it:

luminance-matched style transfer with original colors (like if artist drew a picture in his unique manner, but used original-colored paints) – it can be made as extended "-original_colors" option;
padding option, allowing to improve the quality of edges in rendered image.

Though both need to be polished, tested and reimplemented in Torch to use in Neural-style.

@VaKonS Yes, the code converts the custom gray scale images back to the RGB after working with them in the LUV colorspace.  I am currently finalizing my analysis of how the scripts work in relation to Gatys' examples, and Neural-Style's code.

You seem to suggest that Gatys' code implements some other scheme than described in the paper. It can well be, but I am not convinced that your approach duplicates his thinking, if you cannot even explain how your process handles the problems I mentioned.

@htoyryla Sorry, I should have dug into the inner workings of the how, and why of the script far sooner. My bad.

**The following assumes that I have correctly implemented Gatys' code in both scripts. I used ImageMagick to both analyze the inputs before they were run through the scripts, and the outputs after they were run through the scripts: **
The linear-color-transfer.py script makes the RGB channel means, standard deviations, and overall mean/standard deviation, extremely close to that of values of your chosen source image. Min, max, and gamma remain unaffected. With this in mind, it does not seem like this script is crucial to the function of the luminance transfer process. Though this script does seem make the style image's colors work better for style transfer, as seen above in the experiments with Picasso's Seated Nude artwork.
As for the lum-transfer.py script, things are more complicated than it first appears:
There are many different methods and meanings of the word "grayscale", but I think that Gatys is using a custom method the focuses on luminance. Grayscale images only contain shades of grey, and no color. Each pixel has a "luminance" value. Other names for this "luminance" value, are "intensity", and "brightness".
This means that a grayscale in RGB form, represents the luminance of the image. So one can run images representing luminance through Neural-Style. So the scripts always output in the RGB color space, even though they are working with both a grey scale color space and the LUV color space.
I can break the lum-transfer.py and the linear-transfer-color.py code by giving either script a grey scale image with every grey scale algorithm I have tried. This is because Gatys is not using the rec709luma algorithm, or any other algorithm known to ImageMagick. ImageMagick is listed the script's gray scale output in RGB form has it's intensity value listed as, "Undefined" while the intensity value for my control test is listed as "rec709luma".
The lum-transfer.py script's lum mode first converts your input image that's in the RGB color space, to a grey scale format using the grey scale format as way to preserve/focus on the luminance of the input image.
The lum-transfer.py script's lum2 mode takes the RGB output from Neural-Style composed of the two grey scale images you ran through Neural-Style. It also takes your original RGB content image. Your original content image is then converted from the RGB color space to the LUV color space. This RGB to LUV conversion is what the rgb2luv function does. After this, the script the original content image in LUV form, and applies it to the RGB gray scale output image. This is what the luv2rgb function does. The luv2rgb function basically uses the luminance from the LUV color space, to apply the colors onto the grey scale luminance output image.
In summary, the Python scripts I have made, and Gatys' code, are using a gray scale color space, in the RGB color space to bypass the limitations of the pre-trained caffemodel. In order to add the color back to the gray scale luminance output, the LUV color space is used along with the original content image.
Now, the question is are the differences between the Python script aided outputs, and Gatys' outputs because of the neural_style.lua parameters I used, or am I missing a step in the luminance transfer process?

I am still learning how histograms work, so I don't know exactly where, if at all, does histogram matching play a role in what I described above. I also may have missed some things, so please feel free to let me know where I messed up.

Comparing what I have learned/figured out, to what the research paper says, I think the linear-color-transfer.py script is for histogram matching:

If there is a substantial mismatch between the luminance histogram of the style and the content image, it can be helpful to match the histogram of the style luminance channel LS to that of the content image LC before transferring the style. For that we simply match mean and variance of the content luminance.

I think that the lum mode on the lum-transfer.py is responsible for this luminance histogram matching described by the research paper.
For the linear-color-transfer.py Python script, this the relevant part of the research paper:

The one choice to be made is the colour transfer procedure. There are many colour transformation algorithms to choose from; see [5] for a survey. Here we use linear methods, which are simple and effective for colour style transfer. Given the style image, each RGB pixel is transformed as:

where A is a 3 × 3 matrix and b is a 3-vector. This transformation is chosen so that the mean and covariance of the RGB values in the new style image match those of [11] (Appendix B). In general, we find that the colour matching method works reasonably well with Neural Style Transfer (Fig. 3(e)),

In contrast to the grey scale method, @htoyryla 's method does not use the gray scale color space to transfer luminance information, I think?

@VaKonS
I was mainly looking at lum_transform when cp_mode = lum. But must say I have not had time to look deep enough. Just raised a concern about an additional complexity when using this kind of mixed approach.

Pull images directly from Gatys' code, the content images before the style transfer process are exactly the same as as the ones made by --cp_mode lum:
![]https://i.imgur.com/1OqEsRol.png)
These are the outputs from from the lum-transfer.py script for comparison (ignore the size difference mistake):

After style transfer, you get pretty much the same output as with Neural-Style:

For adding the color to the output image, lum-transfer.py, and Gatys'  code both do the exact same (Script on the left, Gatys' code on the right):

Full image size album here: https://imgur.com/a/HSdcj
Seeing these outputs, I do believe I have replicated Gatys' luminance transfer code in Neural-Style by using the lum-transfer.py script. The differences between his outputs and mine, arise from both differences in the style transfer parameters, and because he uses a two step process.
@htoyryla I think I have with the best of my knowledge, explained how the process works, and now I have proven that there are no missing steps from the process. So I think it is safe to say that I have gotten luminance transfer working in Neural-Style.

My problem is that with the limited time I have, I cannot follow your process as it evolving. Like when you now say "after style transfer, you get pretty much the same output as with Neural-Style", I am confused because I believed your process used neural-style for style-transfer proper.
Therefore, no point in my continuing to participate. The early part of the thread gave me important impulses but now I think my participation is counterproductive for all of us.

@htoyryla

Like when you now say "after style transfer, you get pretty much the same output as with Neural-Style", I am confused because I believed your process used neural-style for style-transfer proper.

Sorry, my bad. I was referring to running Gatys' code from the iPython notebook example and how it compared with doing the same in Neural-Style + the Python scripts.

Therefore, no point in my continuing to participate. The early part of the thread gave me important impulses but now I think my participation is counterproductive for all of us.

The luminance stuff is pretty much done, but the next feature (and final feature I think?) is called Spatial Control, and it looks as though I need to modify the deeper levels of neural_style.lua, which you know far better than I do, due to your own experimentation with the code. I previously thought I could use some simple Python code for masks, but upon examining the supplementary material for the research paper, it appears I cannot do things in Python.

Using masks in torch based style processing is of interest to me and it would be quite simple to a add mask before the gram matrix. But full spatial control, as I understand it, would require the use of multiple gram matrices each with its own mask. I faintly recall that Gatys put the multiple gram matrices in a tensor adding just one more dimension, which sounds simple enough in principle. That also feels interesting for my own purposes but not something to do in one hour, so it'll have to wait.
Using masks for spatial control is also not so easy if you have to make the masks manually. I know because I have used neural-doodle for semantic style transfer. To be really useful, one would need an automatic solution for creating the masks (belonging to the general category of image segmentation). I had a look at one promising solution, but the code was in matlab; the model was caffe but used custom layers not recognized by loadcaffe, using it would have required using a custom build of caffe to run the model for making the masks, so I didn't go further with it.

@ProGamerGov, @htoyryla, I think I've made "match_color" in Torch.
In case if someone needs it.
https://github.com/VaKonS/neural-style/blob/7682e2e24b650206afeddc576a6ca0778d11260c/match_colors.lua
Results seem to be byte-idenctical to ProGamerGov's script.
Does anybody knows how to remove commits from pull request?

This might be of interest https://arxiv.org/pdf/1701.08893v1.pdf

@VaKonS, I was trying to figure out where in the process this match_color would belong. To convert the style image to match the content image before style-transfer? So it is used in full-color style transfer, not in luminance-only?

Assuming that match_color is meant to adjust style image before style transfer (in color), I added it to my code. Use parameter -histogram matchcolor (transfer should be left to color and originalcolors I believe shall be off).
This loads match_color from a file match_colors.lua in the same directory, but unlike @VaKonS's code, the file should contain only the function, no main and no parameter handling.
This should also work with multiple style images but I have not tested it.
Here's a sample output using neural-style default and Gatys' VGG_ILSVRC_19_layers_conv.caffemodel

I made a version of Neural-Style with two currently working options for your content loss and style loss functions (I think that's the correct term?).
I modified LossLayers.lua and neural_style.lua so that one can now use the default Neural-Style Content/Style loss functions, or Gatys Content/Style Loss Functions. Currently 2 of the 3 currently implemented options work with the new parameter called -gram:
https://gist.github.com/ProGamerGov/fb09f2e89609c7333aea2c781190d7b4

-gram lin Will compute content and style loss in-place with linear transform.

-gram default Will compute content and style loss in-place like the default Neural-Style

-gram mse Should compute content and style loss in-place like the default NeuralImageSynthesis iPython Notebook examples, but currently it does not work for a reason I cannot solve:

/home/ubuntu/torch/install/bin/luajit: /home/ubuntu/torch/install/share/lua/5.1/nn/Container.lua:67:
In 4 module of nn.Sequential:
/home/ubuntu/neural-style/LossLayers.lua:17: attempt to index field 'targets' (a number value)
stack traceback:
        /home/ubuntu/neural-style/LossLayers.lua:17: in function </home/ubuntu/neural-style/LossLayers.lua:15>
        [C]: in function 'xpcall'
        /home/ubuntu/torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors'
        /home/ubuntu/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'
        neural_style_gram.lua:205: in function 'main'
        neural_style_gram.lua:644: in main chunk
        [C]: in function 'dofile'
        ...untu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
        [C]: at 0x00405d50

WARNING: If you see a stack trace below, it doesn't point to the place where this error occurred. Please use only the one above.
stack traceback:
        [C]: in function 'error'
        /home/ubuntu/torch/install/share/lua/5.1/nn/Container.lua:67: in function 'rethrowErrors'
        /home/ubuntu/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'
        neural_style_gram.lua:205: in function 'main'
        neural_style_gram.lua:644: in main chunk
        [C]: in function 'dofile'
        ...untu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
        [C]: at 0x00405d50

I have been experimenting with the two loss functions with this command:
th neural_style_gram.lua -gram lin -num_iterations 500 -content_image bauerngarten.jpg -style_image vangogh_starry_rhone.jpg -output_image out1.png -image_size 512 -content_layers relu1_1,relu2_1,relu3_1,relu4_1,relu4_2,relu5_1 -style_layers relu1_1,relu2_1,relu3_1,relu4_1,relu4_2,relu5_1 -model_file models/VGG_ILSVRC_19_layers_conv.caffemodel -proto_file models/VGG_ILSVRC_19_layers_deploy.prototxt -original_colors 0 -tv_weight 0 -content_weight 1e6 -style_weight 2e8 -print_iter 50 -save_iter 50 -seed 876 -backend cudnn -cudnn_autotune -optimizer lbfgs

@ProGamerGov , saw in email about your attempt to add loss module that Gatys used. It looks like Gatys' code is based on original neural-style where the loss models get the targets as input. The new neural-style code (updated in December 2016 I think) works differently, the loss module must implement two modes:

'capture' in which it captures the target by its own
'loss' in which it evaluates loss

So what you were trying to do might with good luck work in the pre-12/2017 code.

@htoyryla Yea, I removed the comment because I doubted my modifications were even functioning correctly. But thanks for the tip. I'll take a look at older versions of Neural-Style, and then hopefully use that knowledge to port the features to the newer version.

@ProGamerGov, if you want to make your loss modules work in the new neural-style, you need to compare the old and new loss modules. Especially the part in the new modules where if self.mode = 'capture' and if self.mode = 'loss'.

Changed my code https://gist.github.com/htoyryla/9ee49c5ff38dda7d0907b6878c171974

allows using -histogram matchcolor -transfer lum -original_colors 1 to do color matching first, then a luminance-only style transfer and finally restore original colors

does color matching from content image to output image when -original_colors 2  (don't know if it is useful though)

Sample output using using -histogram matchcolor -transfer lum -original_colors 1

@VaKonS , one question as I am using your lua implementation of match_color. The comments say that CxWxH is expected, while torch.image uses CxHxW. I don't know if color matching is sensitive to the order of the dimensions. I can of course try swapping them.

@htoyryla, no need to swap them, it's simply me who didn't notice that dimensions are swapped in Torch. I only checked that intermediate results match arrays in Python.
Probably this is why I had to use "transpose" on "torch.potrf(Cx)", while "np.linalg.cholesky(Cx)" is not transposed in Python.
Still, resulting images in all 3 modes were exactly the same as Python output during tests.

@VaKonS, interestingly I got much better results with my own test material after I swapped the dimensions, but there may be some other reason for that.

@htoyryla, the image is presented as [3 channels * n pixels] array in line 46, and all processing is done on 3 "lines of bytes", until in line 79 they are converted back to "3 * y * x" format.
So, if I'm not mistaken, the function shouldn't depend on image dimensions. But I only tried to repeat the code, maybe changing some internal parameters can improve results.

I see... went through it manually in interactive th. It is possible that there was some other difference that I overlooked. I could make a better test., maybe. I don't, however, fully understand the more complex operations (such as the pca specific part, whether it makes a difference if the images are of different size, because in practice they almost always are unless one deliberately deforms either of them.

An interesting issue when trying to replicate Gatys' color matching match, is "bright spots".
My Neural-Style example is on the left, and Gatys' example is on the right:

I am not sure what is causing them, and how to limit them. Though Gatys seems to have figured that out.

I have also discovered that the match_color function, does not work with every single style image.
linear-color-transfer.py:

linear-color-transfer.py versus lum-transfer.py:

Full image: https://i.imgur.com/w4sTTNa.jpg, https://i.imgur.com/wDjExW2.jpg
You can see that some either really dark, or really bright parts of the style image create issues, with both the match_color function (linear-color-transfer.py), and luminance transfer (lum-transfer.py). Though when used together, the issue becomes better disguised.
Oddly enough, the first style image and content image were part of my attempt to replicate Gatys' images:
Color Matching:

Luminance-Only:

Full images, and the content/style image here: https://imgur.com/a/vjPjj

Areas of black pixels? Maybe when the levels are adjusted (like subtracting the mean) one gets some pixels with negative values which are then clamped to 0 = black?

@htoyryla That could be the case. I'll try adding some print statements to the code, so that I can see what the levels are, and check for negative values?

In principle yes, in torch I would print min() of image tensors in selected places in code looking for negative values. In python you (at least sometimes) have values in range 0 to 255 but the same should hold there too.

@htoyryla Yes, it appears there are negative values: https://gist.github.com/ProGamerGov/66cd6e662a5eb9fd9e88aa9810b60361
The line of code responsible for the negative values:
style_img -= style_img.mean(0).mean(0)
The line of code is found here on the unmodified lum-transfer.py script.
I used this command on the modified script:
python lum-transfer.py --content_image newyork_construction.jpg --style_image style_colored_pca.png --cp_mode lum --output_style_image output_lum_style_pca.png --output_content_image output_lum_content_pca.png --org_content newyork_construction.jpg 2>&1 | tee /media/ubuntu/Lexar/python.log

This was the modified script I used to output the values: https://gist.github.com/ProGamerGov/0ad73e5b235487806a35c6ef2fc86f54

Adding this to my, seems to have fixed the problem:
style_img [style_img < 0 ] = 0
style_img [style_img > 1 ] = 1

content_img [content_img < 0 ] = 0
content_img [content_img > 1 ] = 1

@htoyryla @VaKonS I ran a quick match_colors test in Python with:
source_img = skimage.transform.resize(source_img, target_img.shape)
There appears to be no relation between image dimensions, and the output.

@ProGamerGov it is natural that
style_img -= style_img.mean(0).mean(0)

causes negative values, what matters is after the following line:
style_img += content_img.mean(0).mean(0)

If the mean of the style image < the mean of the content image (style image is darker) then there may still be negative values.  Setting negative values to zero does not necessarily help either (0 means black in luminance).

Tested whether swapping h and w makes a difference in match_color():
require 'image'
require 'match_colors'

timg = image.load('helsinki000.png',3) ;
simg = image.load('/home/hannu/hannu512.png',3)
oimg = match_color(timg, simg)
oimg2 = match_color(timg:transpose(2,3), simg:transpose(2,3))
oimg2 = oimg2:transpose(2,3)
print(oimg:dist(oimg2))

Yes, there is, but not more than 7.6392223546888e-13 :)

I have identified the difference between Gatys' Style and content loss functions, versus Neural-Styles
Gatys' code does this:
self.loss = self.loss + self.strength * self.crit:forward(self.G, self.target):mul(self.strength)
While default Neural-Style does this:
self.loss = self.strength * self.crit:forward(self.G, self.target)
From this line of code: https://github.com/jcjohnson/neural-style/blob/master/neural_style.lua#L544
Though when I try to change it, I get this error:
neural_style_gatys.lua:546: attempt to index a number value
Full error: https://gist.github.com/ProGamerGov/5b4644494ea682bbf85250d84da07a09

@ProGamerGov, this error probably means that Torch is trying array operations on some part of expression, which is a number.
Try to replace the line with:
self.loss = self.loss + self.strength * self.crit:forward(self.G, self.target) * self.strength
Because :mul() is meant to work "in-place" on a Tensor, if I'm not mistaken.
Maybe result of "self.crit:forward(self.G, self.target)" can not be changed directly or not a tensor at all.

@ProGamerGov, by the way, note that Gatys multiplies by "self.strength" once, and you do it twice.

@htoyryla, speaking of "Stable and Controllable Neural Texture Synthesis and Style Transfer Using Histogram Losses" (https://github.com/jcjohnson/neural-style/issues/376#issuecomment-281581370) – I didn't quite understand what those "histogram losses" are, but results look very promising on textures, to say the least.

It seems to me that Gatys has self.loss = self.loss + ...
because his code is adding up losses using multiple targets within the loss module itself: https://github.com/leongatys/NeuralImageSynthesis/blob/master/LossLayers.lua#L74-L76
Seems to me too that Gatys is using the targets more flexibly than neural-style (isn't he setting the targets in the python notebook).

@VaKonS I just posted the link as the paper looked relevant, haven't read it carefully yet. But already "histogram loss" sounds interesting. I guess histograms could be used (in addition to gram matrices) to evaluate losses (distance between current histogram and target histogram), although a color image exists only at input, elsewhere we have the feature map activations so the meaning of histogram there is maybe a little vague.

From the error
/home/ubuntu/torch/install/bin/luajit: /home/ubuntu/torch/install/share/lua/5.1/nn/Container.lua:67:
In 4 module of nn.Sequential:
neural_style_gatys.lua:546: attempt to index a number value

it looks like the problem happens inside the self.crit when forward is called with the given input (pay attention to that the error happens within nn.Sequential). This could mean that there is something wrong with the inputs.

I have been experimenting with modifying the content and style loss functions in order to gain a better understanding of how to modify them appropriately.
This version of neural_style.lua containts the L2Penalty module from lines 498-539 in Gatys' code: https://gist.github.com/ProGamerGov/ea432324a09822a19af916fe1bfcfc01
To use the L2Penalty module, you specify a weight value with the new -l2_weight parameter.

@htoyryla The padding option I had implemented here on lines 121-136 in my pull request version of neural_style.lua did not seem to work correctly. I tried to fix it, but I am unsure if my modifications solved the problem correctly.

@ProGamerGov I can't see any problem immediately, can you tell in which way it is not working.
The code structure is not as clean as one would hope, but L123-136 add the extra padding layer and the convolution layer gets added on L147, with zero padding as set on L134-135.

Looking at the code of L2Penalty, I can't actually see that it is doing much. It says that it is "adding the
--[gradOutput] to the gradient of the L2 loss" but I cannot see the gradient of L2 loss being calculated anywhere inside this class. L2 loss itself is calculated and stored in self.loss which is not used anywhere. I get the impression that the class expects something of the code inside which it is to be used. I couldn't find Gatys using L2Penalty anywhere. The same code is found here too https://github.com/aleju/mario-ai/blob/master/layers/L2Penalty.lua .

@htoyryla For the padding code issue, the older version with the padding option looked like this: https://github.com/ProGamerGov/neural-style/blob/4bcf625b6e478d35b00901f1fa0f8b80f94a90cd/neural_style.lua#L121-L135
And resulted in a module error when I tried to run it. So I modified it to look like this in order to correct the issue: https://github.com/ProGamerGov/neural-style/blob/master/neural_style.lua#L121-L136

@ProGamerGov I see now... it was a scope problem again. Defining padlayer as a local variable inside the if-then-else block leaves padlayer undefined if the net:add(padlayer) is outside the block. Another solution would have been to declare padlayer before the if-then-else block.

Looked at @VaKonS 's new assortment of prepadding options. I have no problem with those... it is only that in my mind padding in convlayers has been a method to get the correct output size. Especially when the model contains one or more FC layers, there is really not any freedom to changes padding size, because that will change the output size from a layer and there will be a size mismatch somewhere. Here this is not a problem, as we have only convlayers which automatically adjust to the size of the input. Still, it is good to remember this.

@htoyryla, I'm just saving what I'm currently experimenting with, after closing pull request. :)
Opened request didn't allow me to change the file without adding more irrelevant commits to original project.

@htoyryla This is a bit off topic from the thread, but Say I want use the image package image.scale, image.translate, etc... to modify the current image between or during iterations, so that my changes stick. Like how Lua implementations here, and here work in regards to Deepdream.
Using the image package inside an updateOutput function, or a updateGradInput does not seem to work?
In the feval(x) function, I am not sure what if any variables inside the function to use the image library functions on. Though I did try a variety of values.
As I understand things, the feval(x) function is where neural_style.lua computes the gradient with respect to the loss, and thus is run every iteration? And the feval(x) function is the function that runs the loss functions?
Where should I be trying to perform these operations?

It seems to me that combining the approach of these deepdream scripts with neural-style is quite problematic. Neural-style sets up the model including the loss modules and the delegates the task of optimizing the image to the optimizer, l-bfgs or adam. In each iteration, the optimizer calls feval and expects it to return the loss and the correct gradients. Based on these (and the history it stores internally), the optimizer the modifies the image.
It should be technically possible to change the image in feval, but doing so will interfere with the optimizer as the pixels will no longer be in the same place as the gradients calculated from them.  I guess this is why the deepdream code examples handle the iterations directly, not through an optimizer.

@htoyryla, saving x input as an image inside feval(x) function seems to work, at least like this:
if x ~= nil then image.save("feval_x.png", deprocess(x:double())) end
(Note that "deprocess" takes DoubleTensor, and processing within model is done on floats, if I'm not mistaken, therefore :double() is needed).

I was aware of the possibility of saving as an image and loading back, but it feels terribly inefficient. There must be an easier way.  Also don't think de/preprocessing is needed if we only want to scale or transform the tensor which contains the image.
Furthermore, I am not sure if it is wise to modify x directly in the middle of an iteration. But it all depends how the optimizer works internally. Maybe x is simply to pointer to img. Like I said, changing the image outside the optimizer is a bad idea anyway, just wanted to test how bad. Like do we get a size mismatch if we change the size of the image. I think we will.

Back at my main computer...
This works
    image.save("f.png", img)

this does not
  img = image.scale(img, 511, 511, 'bilinear')

and to me it appears that it is because img, being a plain tensor and not an image created by loading from an image file, lacks .image attribute (which is supposed to contain pointers to functions like scale). The funny thing is that using th interpreter, a plain tensor can be manipulated by image.scale etc and the image attribute does contain those functions.
Anyway, this question is more of general interest to me (as I don't think manipulating the image both directly and by the optimizer makes sense). Moreover, having now seen @ProGamerGov's earlier post about implementing the octaves method inside DeepDreamLoss class, I am not at all sure whether the image which needs to be scaled is the image img which is evolving (which is problematic in neural-style), or some other image which would be used internally by the DeepDreamLoss class (which could make sense... I must admit I have not looked deeper into how this DeepDream through scaling is meant to work).

@htoyryla, strange that "image.scale" doesn't work – this, for example, works for me (inserted between lines 281 and 282):
    if x ~= nil then
      image.save("feval_x.png", deprocess(x:double()))
      local ti = image.scale(x, 511, 511, 'bilinear')
      image.save("feval_x_scaled.png", deprocess(ti:double()))
    end

Both "feval_x.png" and scaled "feval_x_scaled.png" are saved on every iteration.

@htoyryla, by the way, maybe that is because I'm running on CPU.
If you are using CUDA, then, I suppose, it should be:
local ti = image.scale(x:double(), 511, 511, 'bilinear')

@htoyryla Can the image library be substituted for modifying the tensor directly, like jcjohnson does?
On lines L109-L112 on this DeepDream lua code, clipping is performed like this:
if clip then
      bias = Normalization.mean/Normalization.std
      img:clamp(-bias,1/Normalization.std-bias)
   end

On Fast-Neural-Style's DeepDreamLoss:updateGradInput function, clamping appears to be done line this:
self.clipped:resizeAs(input):clamp(input, -self.max_grad, self.max_grad)

I am not sure what operations I should be doing on DeepDreamLoss:__init, DeepDreamLoss:updateOutput, DeepDreamLoss:updateGradInput, but I tried to figure it out below, using this modified neural_style.lua: https://gist.github.com/ProGamerGov/ea432324a09822a19af916fe1bfcfc01
I tried to copy DeepDream's octave related operations fromhere:
  local octaves = {}
   octaves[octave_n] = torch.add(base_img, -Normalization.mean):div(Normalization.std)
   local _,h,w = unpack(base_img:size():totable())

   for i=octave_n-1,1,-1 do
      octaves[i] = image.scale(octaves[i+1], math.ceil((1/octave_scale)*w), math.ceil((1/octave_scale)*h),'simple')
   end

With this in the DeepDreamLoss:updateGradInput:
     local octaves = {}
     octaves[self.octave_n] = self.gradInput:add(self.clipped, -self.max_grad):div(self.max_grad)
     for i=self.octave_n-1,1,-1 do
        octaves[i] = octaves[i+1]
        local c,h,w = unpack(self.base_img:size():totable())
        local octave_w = math.ceil((1/self.octave_scale)*w), octaves[i+1]
        local octave_h = math.ceil((1/self.octave_scale)*h)
        self.gradInput = self.gradInput:resize(1, c, 2, octave_h, 3, octave_w)
     end

Though I have yet figure out the  code below for octave, octave_base in pairs(octaves) do: https://github.com/bamos/dream-art/blob/master/deepdream.lua#L135-L156
And the function it uses here: https://github.com/bamos/dream-art/blob/master/deepdream.lua#L87-L114

@VaKonS cuda vs. double was the reason, it also explains why it worked in th. The error message however was misleading:
/home/hannu/torch/install/share/lua/5.1/image/init.lua:716: attempt to index field 'image' (a nil value)

and looking at the source code of image/init.lua I found that it expects to find an attribute image which was missing.
Using code like this
   x = image.scale(x:double(), x:size(2)-1, x:size(3)-1, 'bilinear'):cuda()
   print("x", x:size())

in different places it looks like

scaling (at the end of feval) x does not make permanent changes to size so x is probably a clone
scaling (at the beginning of feval) x does indeed crash l-bfgs
scaling img makes no permanent changes to size, so l-bfgs probably overwrites img at the end of each iteration

@ProGamerGov of course the tensor can be modified directly, but the example you give modifies the values in the tensor. Scaling an image is quite a different thing. Anyway, we know now that the tensor can be scaled, but we cannot make permanent changed because l-bfgs doesn't expose the master copy of the image to us. Also changing its size would crash l-bfgs anyway.

@VaKonS this demonstrates that a cuda tensor lacks image capabilities. The first print statement prints out the image capabilities, the second prints nil.
require 'image'
require 'cunn'

img = torch.randn(3,64,64):float():mul(0.001)
print(img.image)
img2 = img:cuda()
print(img2.image)

@htoyryla So self.clipped:resizeAs(input) in the DeepDreamLoss:updateGradInput function takes the empty tensor created by self.clipped = torch.Tensor() in the DeepDreamLoss:__init function, and resizes it to match the value of input which I assume is the input image in tensor form?
Then :clamp(input, -self.max_grad, self.max_grad) first fills the resized but still empty tensor with the input image, and then -self.max_grad and self.max_grad is used to normalize higher and lower values.

Anyway, we know now that the tensor can be scaled, but we cannot make permanent changed because l-bfgs doesn't expose the master copy of the image to us. Also changing its size, if we could do it, would crash l-bfgs anyway.

So performing changes to the image in the feval(x) does not work. Could one use code like you have shown above, to convert the input in either the DeepDreamLoss:updateGradInput or the DeepDreamLoss:updateOutput function, to a non tensor form so that image.scale, and image.translate can be used for the DeepDream operations, and then the resulting output can be reconverted to the tensor form?
Ex: dd_image = image.scale(input:double(), input:size(2)-1, input:size(3)-1, 'bilinear'):cuda()
And then dd_image would be fed into the two DeepDream functions?

On the subject of spaitial control, I have made progress in my understanding of the process:

On the subject of spaitial control, I have made progress in my understanding of the process.
In the style loss functions:
The spatially limited tensor is defined like this at the start: local input_chan = input:size()[1]
Then it is added the input value.
   if self.guidance then
        input = torch.cat(input, self.guidance, 1)
    end

The spatial targets are given with this in the python code:
                if guide.ndim==2:
                    guide = guide[:,:,None]
                else:
                    guide = guide[:,:,:1]

Where :,:,None means no guides/spatial targets are used, and :,:,:1 means 1 spatial target is used.
I am still don't understand content vs style guides in his code, and how the code knows which part of the guide image to use.

@ProGamerGov I'll just answer a few points now.

resizes it to match the value of input which I assume is the input image in tensor form?

The input is the input to the module and depends on where in the model it has been placed. Most nn classes are designed to be general purpose, so that one can stack conv layers, relus etc as long as one obeys certain principles. Also the StyleLoss and ContentLoss modules work this way. For them, the input is the output of the layer below, usually the ReLU of a convlayer.
TVLoss in neural-style is different in that is designed to be placed directly above an image at the input to the whole model. I guess your DeepDream module is intended to be used in the  same way, so that the input is really the image. But remember that you cannot change the image directly, only through the gradients you pass to the optimizer, and this can be tricky.

then -self.max_grad and self.max_grad is used to normalize higher and lower values.

This is not really normalization because there is no division involved, one is simply limiting the range.

Finally, self.gradInput:add(-self.strength, self.clipped) adds -self.strength to the self.clipped tensor.

Nope, I think it adds self.clipped (which is a tensor) multiplied by -self.strength (which is a scalar) to self.gradInput.

Could one use code like you have shown above, to convert the input in either the DeepDreamLoss:updateGradInput or the DeepDreamLoss:updateOutput function, to a non tensor form so that image.scale, and image.translate can be used for the DeepDream operations, and then the resulting output can be reconverted to the tensor form?
Ex: dd_image = image.scale(input:double(), input:size(2)-1, input:size(3)-1, 'bilinear'):cuda()

Your question raises many problems, really, but assuming your DeepDream layer is at the bottom of the model, then you can use the input (or a clone of it) as you would an image. There is no conversion into a non-tensor, the input is a tensor, and you can apply image operations to it if you first convert it to float or double. Afterwards, change it into whatever the dtype is. My test was using cuda() but you probably want your code to work on CPU as well.

And then dd_image would be fed into the two DeepDream functions?

Which two functions? If you are using input, it already happens inside either updateOutput or updateGradInput. If you want to influence the image, your code probably should be in updateGradInput.
OK, now I have commented your post, but I still cannot guarantee that anything will work. For me, these layers that sit directly on top of the image and are influenced through the gradient appear anything but clear. Good luck.

As to the spatial control, isn't the idea that masks are used to define parts of the image, so that each area can be processed using a different style. To me it looks like the user is responsible for preparing the masks and assigning the styles to them. The idea seems quite simple to me, use multiple gram matrices each behind its own mask.

This is what I have so far: https://gist.github.com/ProGamerGov/7b04fe82d44c3e50b5de195200d1bc0f
There are two errors I am currently facing.
I don't know the cause of this one: https://gist.github.com/ProGamerGov/dd5a703603a9211495652e8f14514d72
This is related to Cuda vs cpu: https://gist.github.com/ProGamerGov/35be22fb4ac452ef303e09590dde487a
This line (L709) when un-commented should work, but it does not. When it's un-commented, I get this error. It is meant be this line of code from the DeepDream.lua code.
This line (L695), also does not work. It is meant to do what this line of code does in the DeepDream.lua code.

@ProGamerGov I thought we were discussing about doing some image manipulation inside updateGradInput of a DeepReam layer, assuming that the layer is placed so it gets the image as an input. Instead I see you have copied a complete DeepDream implementation with its own model and everything. Sorry to say, but see no way this could work, makes no sense to me.

Inside it you are now loading another full neural network model there and trying to run a full cycle of iterations.

Yes, I see now that I am pointlessly duplicating a lot of code and that is causing my issues. The DeepDream.lua code is carrying out the same functions as neural_style.lua for a lot of the code, like how the areas My issues are because I was duplicating the feval(x) code, rather than using the existing neural_style.lua to handle things.
Please correct me if I am wrong, but inside the feval(x) function, first the net:forward(x) runs all of the :updateOutput(input) functions, and then the :updateGradInput(input, gradOutput) functions are run by this line in the feval(x) function: local grad = net:updateGradInput(x, dy). My code did not work because I was trying to do the work of both of the feval(x) lines.

Disregard my previous comment, there were a lot of errors with what I said.

@htoyryla From my current understanding, I have isolated the code that creates the "DeepDream" hallucinations, and added it into my DeepDream loss functions:
--The DeepDream Magic
      for i=1,iter_n do
        local forward = self.crit:forward(input, self.target)
        -- Set the output gradients at the outermost layer to be equal to the outputs (So they keep getting amplified)
        local forward_grad = forward
        local final = self.crit:updateGradInput(input, forward_grad)
         -- Gradient ascent
        input = input:add(final:mul(step_size/torch.abs(final):mean()))
      end

The code can be found in my modified script here: https://gist.github.com/ProGamerGov/ccc9c41375845f08c4b0f902f251b612#file-neural_style_dd-lua-L642-L649
The problem is however I get the following error:
MSECriterion.lua:27: attempt to index local 'target' (a number value)
Full error message: https://gist.github.com/ProGamerGov/aa08b5cdbb7d064f207013014ddebf93
I do not understand the cause of this

Modifying neural_style.lua's feval(x) loss function to feature the above function, results in:

The modified neural_style.lua I used: https://gist.github.com/ProGamerGov/47cbd692e7e6977b6044ef141b0bd81f

This last code seems again something simple and understandable, even if a bit unorthodox. I think the discussion steered off when you asked about how to scale the image permanently inside feval, which led into a series of misunderstandings.
Let's see what is happening here. Line 287 gets the gradient through the network as usual, but you ignore it. Then you run x ten more times through the net, each time modifying it
 x = x:add(grad:mul(step_size/torch.abs(grad):mean()))

Interesting. I think I'll have to try this and see how
step_size/torch.abs(grad):mean()
behaves.

@htoyryla After testing the various DeepDream lua implementations by disabling portions of the code and save images after every step, I can confirm the DeepDream function we are experimenting with is responsible for the DeepDream hallucinations.
I think your outputs are due to x = x:add(grad:mul(step_size/torch.abs(grad):mean())) not having an affect on the output. In my testing, that line is crucial for the DeepDream function to work properly.
By placing the DeepDream function into the layer setup process, it appears to work correctly, and creates this output:

Full image: https://i.imgur.com/JcENto9.png
This is the modified neural_style.lua I used to make the above image: https://gist.github.com/ProGamerGov/f666dedde46dc425e8c7714e578300c4
I am not sure at the moment how to go from here, for making the process occur simultaneously with the normal style transfer process.

Comparing the above image, with one that uses the octave code in addition to the DeepDream function, it appears as though octaves are for providing control over the prominence of the hallucination features:

Full image: https://i.imgur.com/JuJlnjO.png
A modified version of the above script with octaves: https://gist.github.com/ProGamerGov/0cd618a20cefcbb98dbc912e015fa483

@ProGamerGov it seems we are talking past each other. You are interested in achieving specific visual results, I was now trying to understand what was happening between your code and the optimizer.
When you write

I think your outputs are due to x = x:add(grad:mul(step_size/torch.abs(grad):mean())) not having an affect on the output. In my testing, that line is crucial for the DeepDream function to work properly.

so I was using exactly your code, unmodified. What interested me was whether the effect is

due to gradient, which is the expected way feval influences the optimizer indirectly
because changing x (if passed by reference) directly affects the optimizer (which is usually not a good practice and likely to have side effects)

That you say that your line is needed to have an effect proves nothing either way. My concern is whether the effect is through the gradient (note that changing x and recalculating also changes the gradient) or by directly messing with optimizer internals. By the way, an easy way to check whether 2) happens would be to use a clone of x in the loop.
That is my way of working forward, by trying the understand what is happening. Understand the processes and programs in order to modify them.
As to your goal of combining the two methods, I am rather skeptical by this time. Neural-style and the deepdream.lua implementations you are trying to import into it are both complete implementations. You have now managed to insert the deepdream code into neural-style so that it is run before the actual neural-style iterations begin. That is fine, you can get the deepdream image and then you might run neural-style to get another image, but it leaves the problem of combining the two processes.
I think the discussion of the past few days has been about this combining. Currently I don't think it is possible, at least not this way by playing around with code. The problem is not about code, it is how to duplicate the algorithm of deepdream.lua using a different iteration environment. Breaking up and rearranging the algorithm so that it fits the iterative process in neural-style.

You are interested in achieving specific visual results, I was now trying to understand what was happening between your code and the optimizer.

I think that what was occurring, was that the feval(x) net:forward(x) and local grad = net:updateGradInput(x, dy) lines were being run repeatedly. Which from what I understand, computing the gradient with respect to the loss is what allows style transfer to work. But instead of doing it the normal amount of time, my modifications were computing the gradient for the x value too many times, and then this x value is added to the grad, which then goes on to affect the output image. This is what you described in an earlier comment.

In addition, changing x might directly affect the image inside the optimizer, although I earlier concluded that x is a clone (that was on l-bfgs, could also be that adam and l-bfgs work differently).

My concern is whether the effect is through the gradient (note that changing x and recalculating also changes the gradient) or by directly messing with optimizer internals. By the way, an easy way to check whether 2) happens would be to use a clone of x in the loop.

I can't answer this question as I don't know how and what the grad variable is. The output of the feval(x) function seems to come from: return loss, grad:view(grad:nElement()). Though I may be wrong about the grad variable being important for your question.

I think the discussion of the past few days has been about the possibility of combining them. Currently I don't think it is possible, at least not this way by playing around with code.

In crowsonkb's style_transfer, simultaneous DeepDream and style transfer seems to work pretty well, so I know that it is at least possible to do.

My understand of "DeepDream" hallucinations is that they are created via performing gradient ascent repeatedly. This makes sense considering I've seen DeepDream described as the opposite of the normal gradient descent process. From what I understand, normally a layer is selected to be "amplified" for many DeepDream implementations.
With understanding the basics of what makes DeepDream work, JcJohnson's DeepDream code (the original DeepDream code from Fast-Neural-Style, I was messing with), has to also perform gradient ascent repeatedly in order to produce the basic hallucinations that it does.  I admit that I don't understand how Fast-Neural-Style's code performs gradient ascent, but I think that it's not doing so enough.
Modifying the DeepDream layer setup process like this (full script here):
if name == deepdream_layers[next_deepdream_idx] then
        print("Setting up Deepdream layer  ", i, ":", layer.name)
	local loss_module = nn.DeepDreamLoss(params.deepdream_weights):type(dtype)
        for i=1,params.deepdream_iter do
          net:add(loss_module)
        end
        table.insert(deepdream_losses, loss_module)
        next_deepdream_idx = next_deepdream_idx + 1
end

Results in what one would expect from repeated gradient ascent. Neural-Style's normal parameters also interact with the DeepDream parameters as one would expect. Though one of the problems is that the DeepDream hallucinations slowly become brighter and brighter. I am currently testing different ways of correcting this if possible.

If you're interested, I've added some more color transfer functions: "lms", "rgb", "lab", "xyz", "hsl": https://github.com/VaKonS/neural-style/commit/49302ae6f2c7236d8028eee24d55e3dc2a4060ce.
They make very different images, so it's hard to tell which is better.
By the way, @htoyryla, try to use torch.std() instead of torch.var() – in my tests it behaved probably better.

Disregard my last comment. Creating a loop in the layer setup process is not needed. I believe I have found a better way to get simultaneous DeepDream and style transfer working, by adding only 1 line of code.

The full output image and variations of the output image can be found here: https://imgur.com/a/WGqgd
The modified script: https://gist.github.com/ProGamerGov/ce36ce2693d2a464c7d6fec0b97cf926
In the script on line 614, I have added:
self.output = self.output:mul(1/self.output:max())

I am currently trying to figure out why/how adding this line works.
For testing, I recommend a  -deepdream_weights value of 0.05 for layers relu4_1 and up. For lower relu layers, one must use a lower -deepdream_weights value. Your chosen -style weight and -content_weight also will likely play a role in what -deepdream_weights value works best. Currently only 1 weight value is supported, but multiple -deepdream_layers values are supported.

Now I've definitely seen enough random changes, each morning something different. For me understanding the whole is essential. Trying out things is useful when it adds to this understanding. Neural-style works by inserting loss modules inside the network, then uses gradients in the backward pass, controlled by the optimizer, rippling down to the bottom and there modifying the image. Deepdream.lua implementations use the model as a whole, taking the gradient at the bottom and modifying the image through what they call normalized ascent, without any optimizer. Combining these approaches for me should start from an idea how these two can fruitfully coexist. I've done some  attempts of my own to make a deep dream layer inside the model, but these fail to have the effect that direct tampering with the image has. And conversely, direct tampering with the image kills the idea on which neural-style is built, that the optimizer develops the image based on the gradients from the loss modules. Good luck.

@htoyryla Do you happen to still have the modified prototxt file for using the FC layers, and/or the code relating feature names from what the model was seeing, in Neural-Style?
And as for the DeepDream, that was off topic and not within the scope of this thread. I'll try to keep future comments in this thread related to the features in Gatys' code, and the associated research paper.

Any VGG prototxt already contains the FC layers. Like I wrote in my blog, the trick was to avoid size mismatch between the convlayers (which expand according to input) and FC layers which have fixed size. I used adaptive max pooling which worked well enough for 512px image size, but could be inaccurate for much larger images.
Something like this, in the old neural-style code.
 for i = 1, #cnn do
    local layer = cnn:get(i)
    if (torch.type(layer) == "nn.View") then
        addlayer = nn.SpatialAdaptiveMaxPooling(7,7):float()
        if params.gpu >= 0 then
          if params.backend ~= 'clnn' then
            addlayer:cuda()
          else
            addlayer:cl()
          end
        end
	net:add(addlayer)
    end

I used the label names with places205 and places365, I believe that I found the list together with the model(s). At the beginning I read the labels into a table:
 --assuming we are using http://places.csail.mit.edu/downloadCNN.html
  f = io.open("/home/hannu/neural-style/models/places205_categories.txt")
  for line in f:lines() do
    table.insert(labels, line)
  end

and then later I can print them after doing a net:forward(cimage), where cimage is either the content_image preprocessed for caffe (to show labels for the content image right after net has been built) or the current image (to show labels for the current iteration inside maybe_print).
local p = net:forward(cimage)
  print("----------- seeing the features --------------")
  for i=1, #labels do
    if p[i] > 0.03 then
      print(string.format("%.4f   %s", p[i], labels[i]))
    end
  end

This assumes a SoftMax layer above fc8 so I added it (just after net has been built, before cleaning up cnn)
--vgg16 places lacks softmax at the end, so insert one
  local prob = nn.SoftMax():float()
  if params.gpu >= 0 then
            if params.backend ~= 'clnn' then
              prob:cuda()
            else
              prob:cl()
            end
  end
  net:add(prob)

This is basically what is needed to a) display labels b) use fc layers as content layers. Finding good balance between weights is not easy, especially when using both a conv layer to control spatial arrangement of content and an fc layer to control general content. It helps if one provides separate weight settings for these.
I have been thinking of publishing a complete working solution, maybe when I have put together a version which feels solid enough.

@ProGamerGov it is not a problem for me if a thread goes on to do further experiments. There is need for such discussion around neuraI-style, not limited to the existing implementation. It is only that I'd rather not participate if it gets too chaotic (as I perceive it).

@ProGamerGov this response of yours illustrates the problem in our discussion concerning deepdream

I can't answer this question as I don't know how and what the grad variable is. The output of the feval(x) function seems to come from: return loss, grad:view(grad:nElement()). Though I may be wrong about the grad variable being important for your question. I could also be widely off with my responses here due to my lack of understanding of what you are trying to figure out.

Here you confirm that you don't understand how neural-style works. I wrote

Neural-style works by inserting loss modules inside the network, then uses gradients in the backward pass, controlled by the optimizer, rippling down to the bottom and there modifying the image.

Grad is the gradient obtained from the backward pass, after all layers have contributed to it. If the layers have been implemented properly, it indicates how the image should be modified to minimize the loss. Neural-style gives this to the optimizer which then figures out the how much to change in the direction shown by the gradient.
The standalone deepdream implementions modify the image directly based on the gradient.
To me, solving how these two processes could be combined should be based on an understanding their inherent difference. Not just copying, pasting and modifying code. But I have already said this several times.

May I ask more information about spatial control?
I have seen your saying:

As to the spatial control, isn't the idea that masks are used to define parts of the image, so that each area can be processed using a different style. To me it looks like the user is responsible for preparing the masks and assigning the styles to them. The idea seems quite simple to me, use multiple gram matrices each behind its own mask. The idea of a mask is basically very simple, like that a b&w image consisting of black and white pixels can be used to block or pass parts of the input into the gram matrix. (Haven't checked Gatys' code, this is simply how I understood the idea when reading the paper... at least the basic principle).

After seeing your saying, I finally got where I did wrong. I used a masked input image for style loss computing. And I didn't get expected output image. I should use masks for feature maps to generate new Gram matrixes. So, if I want to used two different styles, one for the ground and another for sky in one picture, Should I define two Gram matrixes, which are generated by two kinds of masked feature maps?

@SSeanHsu I would proceed by having two Gram matrices, like you say, getting the same input but each through a different mask, but I haven't thought out the implementation in detail. I would place both gram matrices inside the same loss module, to ensure that the output will correctly match the input, and to get the gradInput by adding the gradients from the two Gram matrices filtered by the masks.
I have not attempted this myself yet... maybe later as an exercise. I enjoy using output from style transfer as material that I modify manually, like the background in this selfie.

@htoyryla Thank you for sharing the idea. I'll try it. If it succeed, I'll let you know. By the way, your selfie is awesome. I love this one better than the one you are using now.

@SSeanHsu one more thing. One also need to find a way to make the style modules to capture two style targets and associate each of them to the right mask/Gram-matrix.

I wrote "the masks need to be scaled so that they match the size of the output from the underlying convlayer" but Gatys' paper says this does not confine the style to the area correctly, because the receptive fields of neurons extend outside the mask boundary (each neuron is affected by a proportionally larger part of the input image when we go to a higher level). Instead they propose using smaller masks for the areas with a specific style and an additional style for the whole image.

The most obvious approach would be to down- sample Tr to the dimensions of each layer’s feature map. However, we often find that doing so fails to keep the de- sired separation of styles by region, e.g., ground texture still appears in the sky. This is because neurons near the boundaries of a guidance region can have large receptive fields that overlap into the other region. Instead we use an eroded version of the spatial guiding channels. We define spatial guidance channels only on the neurons whose receptive field is entirely inside the guidance region and add another global guidance channel that is constant over the entire image. In that way we only enforce spatial guidance for neurons that have a clear assignment to a region and use the unguided style loss for neurons whose receptive field overlaps with the boundary.

Here's now my neural-mirage code (referred to in http://liipetti.net/erratic/2016/04/20/getting-the-space-back/)
https://gist.github.com/htoyryla/806ca4d978f0528114282cd00022ad71
I have been using this with places-205 dataset from here http://places.csail.mit.edu/downloadCNN.html
Finding a good balance between generated content (fc layers, fc_weight), style and spatial content (relu layers, content_weight) is quite tricky and the weight depend heavily on the layers used.
Set content and fc layers using -content_layers. E.g.
-content_layers relu4_1,fc7,fc8

Set spatial content weight with content_weight, fc generated content weight with -fc_weight.
Looks like it can also produce deepdream-like output. Here I have used
-content_layers relu2_1,relu3_1,relu4_1,fc6
-content_weight 150 -fc_weight 8000 -style_weight 1e4

@htoyryla The first image from the FC layers in have what appears to be the same baseball diamond as the one in this DeepDream image I made many months ago:

Full Image: https://i.imgur.com/dAweBXs.jpg
And the structures in your image, look like the structures from this other DeepDream session of mine: https://imgur.com/a/8vWDB

There may be something wrong with my mirage code. Although it does interesting things, I have not been able to achieve some essential effects I have used the same methods earlier. Need to delve deeper into it.

@htoyryla I didn't succeed on spatial control. This is my method,

I define 2 kinds of Gram Matrix: GramSky and GramGround
The GramSky is generated by feature maps which are multiplied with sky mask. The GramGround uses the same steps, but using ground mask.
I place both gram matrices inside the StyleLoss module. As for the updateGradInput of StyleLoss, I derive the input in the loss function.
I will attach my code later.

@SSeanHsu it is not so simple at all how to implement the idea in torch/nn/neural-style.
The StyleLoss module gets the input from the underlying convlayer.
The input is branched into two parallel branches.
Each branch has first a mask and then a Gram matrix which is used to calculate losses and the gradient. Losses and gradients from both branches are added together (and with gradOutput) in updateGradInput.
updateOutput should just copy the unmasked input to output.
Then there is the question of setting style target, because now we need two targets. One could try using "captureSky" and "captureGround" for capturing two style targets targetSky and targetGround.
I haven't tried this but if I would, I would proceed like this.

I did so, and I also copy the unmasked input to output in updateOutput of both Gram matrixes. Do you mean I should define two modules ( "captureSky" and "captureGround" ) for capturing target? I capture both sky and ground targets  in the updateOutput of StyleLoss, one target is GramSky(style image), another is GramGround(style image).
Here is my content image:

And this is mask:

Style image:

So when I pass the style image into StyleLoss, it would pass the GramSky with sky mask and GramGround with ground mask ( I do mask with the feature maps, not the style image ). So I would be able to capture sky target and ground target. The 2 targets I captured are different. Because of the sky mask, GramSky(style image) will be able to capture the style of "the_scream". And because of the ground mask, GramGround(style image) will be able to capture the style of "starry_night".
Here I pick one feature map of style image when passing GramGround, and I copy the unmasked input to output in updateOutput
( before masked / after masked)

I did this way, but I did get expected result.

Now I see, you are using a single style image, combined of two styles. Fine if that works. I understood from your first post today that you couldn't get it to work when you wrote "I didn't succeed on spatial control".
My concept was different, using two (or more) style images, so that here https://github.com/jcjohnson/neural-style/blob/master/neural_style.lua#L164-L175 we would first set the mode="capture1" and forward the first style image, then set mode=capture2 and forward the second. Correspondingly, in StyleLoss module when mode = "capture1" I would capture target1 and so on. Then there would be no need to create a mixed style image, just use any image as style as before.
Also, like I wrote, I would not modify GramMatrix class at all. I would just create two gram matrix instances here https://github.com/jcjohnson/neural-style/blob/master/neural_style.lua#L526
  self.gram1 = nn.GramMatrix()
  self.gram2 = nn.GramMatrix()

and then I would apply the masks inside StyleLoss:updateOutput to input just before feeding it to the respective Gram matrices.  This makes it easy to add more masked areas and styles. And so on. I may try it out myself when I have time.

I did that, and it succeeded. I simply scale the masks to generate Tr. As you said,

The most obvious approach would be to down- sample Tr to the dimensions of each layer’s feature map. However, we often find that doing so fails to keep the de- sired separation of styles by region, e.g., ground texture still appears in the sky. This is because neurons near the boundaries of a guidance region can have large receptive fields that overlap into the other region. Instead we use an eroded version of the spatial guiding channels. We define spatial guidance channels only on the neurons whose receptive field is entirely inside the guidance region and add another global guidance channel that is constant over the entire image. In that way we only enforce spatial guidance for neurons that have a clear assignment to a region and use the unguided style loss for neurons whose receptive field overlaps with the boundary.

there is much work to do. And I am pretty enjoy this exercise :D
content image:

style image (sky part):

style image (ground part):

generated image (iterations 100 / 400)

@htoyryla How would I allow for different DeepDream layer weights in my modified neural_style.lua?
The modified neural_style.lua https://gist.github.com/ProGamerGov/39be3ceb669ffe8369580a0d3ba3d3a7
Currently -deepdream_layers accepts multiple input layers separated by a comma. However -deepdream_weights only accepts a single weight value. I know it requires something like local deepdream_weights = params.deepdream_weights:split(","), but I am not sure what else I need to do for this feature?

@ProGamerGov if you want to apply different weights to layers of the same type, have a look how I implemented style_layers_weights for style layers: https://github.com/htoyryla/neural-style/blob/master/neural_style.lua .

@htoyryla I was wondering if your solution for supporting FC layers, would also work (with some modifications) could be made to work with inception layers from a model like the Places365 GoogleNet caffemodel? Or if not, do you know where I would start for supporting GoogleNet inception layers in Neural-Style (Preferably without requiring .t7 models)?

@ProGamerGov my guess is that loadcaffe is the showstopper for using networks like GoogleNet, ResNet etc.  Loadcaffe is pretty limited as to what layer types it supports. Sometime last summer I spent some time training a fancy model with a special build of Caffe only to find out that loadcaffe would not load the caffemodel.
It is easy to check in th interpreter whether loadcaffe can load this GoogleNet caffemodel. You first require 'loadcaffe' and then cnn = loadcaffe(...) to see if it loads. My bet is negative.
The limitations of loadcaffe is probably why everybody is using t7 files today. With it one can get errors about missing nn classes, but that can be solved by adding the missing classes into to project.

@htoyryla Loadcaffe appears to be able to load the network, and in fact one can already use the conv2/relu_3x3 layer in the unmodified neural_style.lua:

Using the th interpreter, I get this:
th> require 'loadcaffe'
{
  load : function: 0x4061b9d0
  C : userdata: 0x406c9eb0
}
                                                                      [0.0356s]
th> model = loadcaffe.load('deploy_googlenet_places365.prototxt','googlenet_places365.caffemodel')
Successfully loaded googlenet_places365.caffemodel
warning: module 'data [type Data]' not found
warning: module 'label_data_1_split [type Split]' not found
warning: module 'pool2/3x3_s2_pool2/3x3_s2_0_split [type Split]' not found
warning: module 'inception_3a/output [type Concat]' not found
warning: module 'inception_3a/output_inception_3a/output_0_split [type Split]' not found
warning: module 'inception_3b/output [type Concat]' not found
warning: module 'pool3/3x3_s2_pool3/3x3_s2_0_split [type Split]' not found
warning: module 'inception_4a/output [type Concat]' not found
warning: module 'inception_4a/output_inception_4a/output_0_split [type Split]' not found
conv1/7x7_s2: 64 3 7 7
conv2/3x3_reduce: 64 64 1 1
conv2/3x3: 192 64 3 3
inception_3a/1x1: 64 192 1 1
inception_3a/3x3_reduce: 96 192 1 1
inception_3a/3x3: 128 96 3 3
inception_3a/5x5_reduce: 16 192 1 1
inception_3a/5x5: 32 16 5 5
inception_3a/pool_proj: 32 192 1 1
inception_3b/1x1: 128 256 1 1
inception_3b/3x3_reduce: 128 256 1 1
inception_3b/3x3: 192 128 3 3
inception_3b/5x5_reduce: 32 256 1 1
inception_3b/5x5: 96 32 5 5
inception_3b/pool_proj: 64 256 1 1
inception_4a/1x1: 192 480 1 1
inception_4a/3x3_reduce: 96 480 1 1
inception_4a/3x3: 208 96 3 3
inception_4a/5x5_reduce: 16 480 1 1
inception_4a/5x5: 48 16 5 5
inception_4a/pool_proj: 64 480 1 1
loss1/conv: 128 512 1 1
loss1/fc: 1 1 2048 1024
loss1/classifier: 1 1 1024 365
                                                                      [0.1000s]
th>

There are missing modules for some layers, but the model still seems to work with loadcaffe.

@ProGamerGov you and I then have a different meaning for "able to load the network". For me that implies loading the whole network as it is intended to work.
It looks like loadcaffe had loaded the layers it knows, resulting in a normal convolutional network, but has not loaded the split and concat layers that would implement the branches that would make the network GoogleNet. I now remember seeing this myself and also that it is possible to use the resulting model in neural-style but it is no longer GoogleNet.