Does Keras support using multiple GPUs?

Theano has supported multiple GPUs since v0.8.0.
(cf. Using multiple GPUs — Theano 0.8.0 documentation )
Does Keras also support using multiple GPUs?
For example, can I run the below task?

Learn a sequential model A on gpu0
Learn a sequential model B on gpu1
Merge A and B on gpu0

Yes, can run Keras models on multiple GPUs. This is only possible with the TensorFlow backend for the time being, because the Theano feature is still rather new. We are looking at adding support for multi-gpu in Theano in the near future (it should be fairly straightforward).
With the TensorFlow backend, you can achieve this the same way as you would in pure TensorFlow: by using the with tf.device(d) scope when defining Keras layers.

We are looking at adding support for multi-gpu in Theano in the near future (it should be fairly straightforward).

I'm looking forward to it 😃
Thank you.

tf.device() scope?
Can you expand on this?
I haven't seen it in the api

Any example to use multiple gpus with TF?

Hm. Theano has libgpuarray, which allows one to push shared variables to different devices. This will not do all the work for you of recombining weight matrices but with a little effort you could use multiple GPUs.

There is platoon, a project on top of Theano for data parallelism. Should
be easy to use. We currently focus more on days parallelism then model
parallelism in Theano. But both are possible.
Fred
Le 23 avr. 2016 17:24, "phalexo"<EMAIL_ADDRESS>a écrit :

Hm. Theano has libgpuarray, which allows one to push shared variables to
different devices. This will not do all the work for you of recombining
weight matrices but with a little effort you could use multiple GPUs.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/fchollet/keras/issues/2436#issuecomment-213834678

I have looked into Platoon and it seemed like it was pretty much compatible
with Keras out of the box except for a couple lines of code. Easy to adapt,
in any case...
On 25 April 2016 at 05:46, Frédéric Bastien<EMAIL_ADDRESS>wrote:

There is platoon, a project on top of Theano for data parallelism. Should
be easy to use. We currently focus more on days parallelism then model
parallelism in Theano. But both are possible.
Fred
Le 23 avr. 2016 17:24, "phalexo"<EMAIL_ADDRESS>a écrit :

Hm. Theano has libgpuarray, which allows one to push shared variables to
different devices. This will not do all the work for you of recombining
weight matrices but with a little effort you could use multiple GPUs.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/fchollet/keras/issues/2436#issuecomment-213834678

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
https://github.com/fchollet/keras/issues/2436#issuecomment-214299393

The way libgpuarray work is by mapping variables to different GPUs, and
then function automatically generates code to transfer data between GPUs as
needed.
On Apr 25, 2016 16:13, "François Chollet"<EMAIL_ADDRESS>wrote:

I have looked into Platoon and it seemed like it was pretty much compatible
with Keras out of the box except for a couple lines of code. Easy to adapt,
in any case...
On 25 April 2016 at 05:46, Frédéric Bastien<EMAIL_ADDRESS>wrote:

There is platoon, a project on top of Theano for data parallelism. Should
be easy to use. We currently focus more on days parallelism then model
parallelism in Theano. But both are possible.
Fred
Le 23 avr. 2016 17:24, "phalexo"<EMAIL_ADDRESS>a écrit :

Hm. Theano has libgpuarray, which allows one to push shared variables
to
different devices. This will not do all the work for you of recombining
weight matrices but with a little effort you could use multiple GPUs.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/fchollet/keras/issues/2436#issuecomment-213834678

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
https://github.com/fchollet/keras/issues/2436#issuecomment-214299393

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
https://github.com/fchollet/keras/issues/2436#issuecomment-214450940

I have looked into Platoon and it seemed like it was pretty much compatible
with Keras out of the box except for a couple lines of code. Easy to adapt,
in any case...

What's the priority of adding multi GPU support for the theano backend?

I think it would expand user base for Keras. I have several Titan X in the
same box. Please, take a look at libgpuarray as well.
On Jun 22, 2016 19:54, "themummy"<EMAIL_ADDRESS>wrote:

I have looked into Platoon and it seemed like it was pretty much compatible
with Keras out of the box except for a couple lines of code. Easy to adapt,
in any case...
What's the priority of adding multi GPU support for the theano backend?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/fchollet/keras/issues/2436#issuecomment-227911398,
or mute the thread
https://github.com/notifications/unsubscribe/AEY95aBPElrTcVv0ZzPFyDgcDKMaw-0iks5qOcsXgaJpZM4IMTcS
.

How does this actually work in tensorflow? There is a brief tutorial here: http://blog.keras.io/keras-as-a-simplified-interface-to-tensorflow-tutorial.html, I understand the concept of running the model replicas on seperate GPU devices & then merging the weights, but how do we actually run this? instead of model.fit do we call merged.fit on the result of the merged models?

@tetmin I have the same confusion as yours. Although the blog shows how to predict model in different GPUs, it is still unclear how to train the same model across different GPUs in a single machine, i.e. I need data parallelism and don't know how to implement it in Keras with TensorFlow as backend.

Agreed with @pengpaiSH and @tetmin . Hope there would be more details.

@rudaoshi Well, I know this would no be proper to suggest since we are in the Keras community, and personally I am a Big Big Big fan of Keras! We know TensorFlow could utilize Multi-GPUs by computing averaging gradients across different devices, however, I am expecting Keras could provide a simple and unified API (Keras's Style) to help me focus my big picture end hide those IO/Parallel Computing details. For the time being, in order to make good use of multiple GPUs, I am doing my deep learning programs with MXNET, which I only specify the GPU IDs and the lib will do everything it needs under the hood.

@fchollet I saw your blog with multi gpu training, thanks for pointing out the way doing multi gpu training, but I would really appreciate it if say model.fit() has a gpu=n option, I'll willing to implement my own version on that, may I ask for suggestions? or I'm willing to contribute on the multi gpu training within keras with more abstraction from end users. Thanks in advance!

@WenchenLi +1, gpus=0,1,2... is exactly what I need!

@WenchenLi did you create a PR for multigpu abstraction?

Hope someone can contribute on the multi gpu training within keras. Thanks in advance.
I have two gpus. I did not do anything to set which gpu would be used for training. But when I used the nvidia-smi to check memory. I found almost all of the memory in two gpus were in use. I thought only one gpu would be used.

@anewlearner apparently this is the intended functionality of TF.
Use export CUDA_VISIBLE_DEVICES="0".
See https://github.com/tensorflow/tensorflow/issues/5066 for details
Looking forward to a simplified version of mult-gpu  :)

For data parallelization in keras, you can use this approach:
import tensorflow as tf
from keras import backend as K
from keras.models import Model
from keras.layers import Input, merge
from keras.layers.core import Lambda
def slice_batch(x, n_gpus, part):
sh = K.shape(x)

L = sh[0] / n_gpus

if part == n_gpus - 1:

    return x[part*L:]

return x[part*L:(part+1)*L]

def to_multi_gpu(model, n_gpus=2):
with tf.device('/cpu:0'):

    x = Input(model.input_shape[1:], name=model.input_names[0])

towers = []

for g in range(n_gpus):

    with tf.device('/gpu:' + str(g)):

        slice_g = Lambda(slice_batch, lambda shape: shape,

arguments={'n_gpus':n_gpus, 'part':g})(x)
        towers.append(model(slice_g))

    with tf.device('/cpu:0'):

        merged = merge(towers, mode='concat', concat_axis=0)

return Model(input=[x], output=merged)

To use just take any model and set model = to_multi_gpu(model).
model.fit() and model.predict() should work without any change.
On Fri, Oct 21, 2016 at 6:13 PM, Alexander<EMAIL_ADDRESS>wrote:

@anewlearner https://github.com/anewlearner apparently this is the
intended functionality of TF.
Use export CUDA_VISIBLE_DEVICES="0".
See tensorflow/tensorflow#5066
https://github.com/tensorflow/tensorflow/issues/5066 for details
Looking forward to a simplified version of mult-gpu :)
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/fchollet/keras/issues/2436#issuecomment-255404169,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFdLCAoGsCYxA9CVIN7IIJqX7ddkxEurks5q2NaagaJpZM4IMTcS
.

@jonilaserson , looks great! Does this work with the Theano backend or only TF?

@jonilaserson Could you please provide more detailed comments for the codes?
For example, what's the purpose of slice_g? And what does tower do actually? Thank you !

I tested the code provided by @jonilaserson and got a error.
merged = merge(towers, mode='concat', concat_axis=0)
Exception: A Merge should only be applied to a list of layers with at least 2 elements. Found: [<keras.engine.training.Model object at 0x7f9c1c3123d0>]

@anewlearner Have you solved the problem that you met with before?

@carol
There was an indentation error in the code I posted.
The [with tf.device('/cpu:0')] paragraph should be outside the loop.
Here is a piece of code that should work:
import tensorflow as tf
from keras import backend as K
from keras.models import Model
from keras.layers import Input, merge
from keras.layers.core import Lambda
def slice_batch(x, n_gpus, part):
"""

Divide the input batch into [n_gpus] slices, and obtain slice no.

[part].
i.e. if len(x)=10, then slice_batch(x, 2, 1) will return x[5:].

"""

sh = K.shape(x)

L = sh[0] / n_gpus

if part == n_gpus - 1:

    return x[part*L:]

return x[part*L:(part+1)*L]

def to_multi_gpu(model, n_gpus=2):
"""Given a keras [model], return an equivalent model which parallelizes

the computation over [n_gpus] GPUs.

Each GPU gets a slice of the input batch, applies the model on that

slice
and later the outputs of the models are concatenated to a single

tensor,
hence the user sees a model that behaves the same as the original.

"""

with tf.device('/cpu:0'):

    x = Input(model.input_shape[1:], name=model.input_names[0])

towers = []

for g in range(n_gpus):

    with tf.device('/gpu:' + str(g)):

        slice_g = Lambda(slice_batch, lambda shape: shape,

arguments={'n_gpus':n_gpus, 'part':g})(x)
        towers.append(model(slice_g))

with tf.device('/cpu:0'):

    merged = merge(towers, mode='concat', concat_axis=0)

return Model(input=[x], output=merged)

To use just take any model and set model = to_multi_gpu(model).
model.fit() and model.predict() should work without any change.

Example:
from keras.layers.convolutional import Convolution2D
from keras.layers.core import Activation
import numpy as np
def get_model():
x = Input( (96,96,1), name="input1")
output = Convolution2D(64, 5, 5, border_mode='same', name="conv1")(x)
output = Activation('relu', name="relu1")(output)
[More layers...]
model = Model(input=x, output=output)
model.compile(optimizer='rmsprop', loss='mse')
return model
model = get_model()
model = to_multi_gpu(model)
x = np.random.rand(1000, 96, 96, 1)
y = model.predict(x, verbose=True)
On Mon, Oct 31, 2016 at 10:18 AM, Pai Peng<EMAIL_ADDRESS>wrote:

@anewlearner https://github.com/anewlearner Have you solved the problem
that you met with before?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fchollet/keras/issues/2436#issuecomment-257228011,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFdLCCOCSqV-6UR7QtUN7Gv8YGe73u03ks5q5ZZegaJpZM4IMTcS
.

@jonilaserson Thank you for you updating! Would you please comments on the code snippets ```
for g in range(n_gpus):
with tf.device('/gpu:' + str(g)):
slice_g = Lambda(slice_batch, lambda shape: shape,
arguments={'n_gpus':n_gpus, 'part':g})(x)
towers.append(model(slice_g))

@jonilaserson
Thanks for sharing your code. It works. :)
I tested code to compare the cost of time bettween a gpu and two gpus.
When I used two gpus(same type of gpus here), the speed was slower than expected. Does the switch bettween cpu and gpu affects the speed?
My test results is as follows.
Two gpus
97650/682307 [===>..........................] - ETA: 1933s - loss: 0.3320 - acc: 0.8320
188593/682307 [=======>......................] - ETA: 1654s - loss: 0.2354 - acc: 0.8904
279093/682307 [===========>..................] - ETA: 1348s - loss: 0.1936 - acc: 0.9140

One gpu
97650/682307 [===>..........................] - ETA: 2669s - loss: 0.3488 - acc: 0.8266
188593/682307 [=======>......................] - ETA: 2239s - loss: 0.2431 - acc: 0.8880
279093/682307 [===========>..................] - ETA: 1844s - loss: 0.2004 - acc: 0.9116

I think you should compile the model in case of a error: you must compile the model before training/testing.

But I got the error.
Traceback (most recent call last):
  File "mgpumain.py", line 329, in <module>
    main()
  File "mgpumain.py", line 165, in main
    train_val_test()
  File "mgpumain.py", line 315, in train_val_test
    train_model_runner(n)
  File "mgpumain.py", line 232, in train_model_runner
    model = to_multi_gpu(model,4)
  File "mgpumain.py", line 200, in to_multi_gpu
    slice_g = Lambda(slice_batch, lambda shape: shape, arguments={'n_gpus':n_gpus, 'part':g})(x)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 514, in __call__
    self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 572, in add_inbound_node
    Node.create_node(self, inbound_layers, node_indices, tensor_indices)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 149, in create_node
    output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
  File "/usr/local/lib/python2.7/dist-packages/keras/layers/core.py", line 556, in call
    return self.function(x, **arguments)
  File "mgpumain.py", line 186, in slice_batch
    return x[part*L:(part+1)*L]
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_ops.py", line 383, in _SliceHelper
    name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_ops.py", line 537, in strided_slice
    shrink_axis_mask=shrink_axis_mask)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 2750, in strided_slice
    shrink_axis_mask=shrink_axis_mask, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 573, in apply_op
    _Attr(op_def, input_arg.type_attr))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 60, in _SatisfiesTypeConstraint
    ", ".join(dtypes.as_dtype(x).name for x in allowed_list)))
TypeError: DataType float64 for attr 'Index' not in list of allowed values: int32, int64

@anewlearner
I received better speedups the larger the batches were, but yes, there must be some overhead concatenating the outputs on the CPU. I assume we can get additional speedup if the computation would be kept on the GPU as far as the loss function (using the CPU just to add the losses), but this would mean you will need to use the native tensorflow optimizer instead of the keras one, so more work and less backward compatibility.

@pengpaiSH
 for g in range(n_gpus):
        # Work on GPU number g.
        with tf.device('/gpu:' + str(g)):
            # Obtain the g-th slice of the batch.
            slice_g = Lambda(slice_batch, lambda shape: shape,
                            arguments={'n_gpus':n_gpus, 'part':g})(x)
            # Apply model on the batch slice.
            towers.append(model(slice_g))

@barrykui
It seems L is not an integer, perhaps the division in the line
L = sh[0] / n_gpus
resulted in a fraction. It does not suppose to happen if both n_gpus and sh[0] are int32, check if that's the case.

@themummy
Sorry, I'm using TF specific idioms (with tf.device()), no Theano.

@jonilaserson - My models train 4X slower with TF backend than Theano :\

@jonilaserson I get the same not an int error. When I look at that var sh it is actually None, as in the None that goes to indicate variable batch size.
For instance, I get that sh is (None, 40, 120).

What's TF version are you using?
On Wed, Nov 2, 2016 at 11:27 PM, Fernando Cucchietti <
<EMAIL_ADDRESS>wrote:

@jonilaserson https://github.com/jonilaserson I get the same not an int
error. When I look at that var sh it is actually None, as in the None that
goes to indicate variable batch size.
For instance, I get that sh is (None, 40, 120).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fchollet/keras/issues/2436#issuecomment-258004336,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFdLCFsFhlInrsFwQmi1-Y-LG_rKEDyjks5q6QBEgaJpZM4IMTcS
.

It seems that 0.11.0rc0.
It's a shared machine so upgrading might take a couple of days.

@jonilaserson Thanks for your reply. Could you please give more details what does the slice_g do and why it should be a Lambda layer if it attempts to get a piece of batch data? Besides, Why should we concatenate multiple results computed in different GPUs, instead of averaging them?

@jonilaserson you are right. The information shows blew. I really confused, and don't know how to solve this problem.
>>print(K.shape(x))
Tensor("Shape_3:0", shape=(3,), dtype=int32, device=/device:GPU:0)
>>print(sh[0])
Tensor("strided_slice_2:0", shape=(), dtype=int32, device=/device:GPU:0)
>>print(type(sh))
<class 'tensorflow.python.framework.ops.Tensor'>
>>print(L)
Tensor("truediv:0", shape=(), dtype=float64, device=/device:GPU:0)

@barrykui
Can you try doing floor division?
L = sh[0] // n_gpus

@jonilaserson Yeah! but still float64.
a = sh[0] // n_gpus
x = Tensor("input1_1:0", shape=(?, 100, 4), dtype=float64, device=/device:CPU:0)
n_gpus = 2
part = 0

@jonilaserson I confirm that with Tensorflow 0.11.0rc2 (latest version) still triggers the error.
Floor division won't work either because it starts as None.
>>print (K.int_shape(x))
(None, 40, 120)
>>print (K.shape(x))
Tensor("Shape_2:0", shape=(3,), dtype=int32, device=/device:GPU:0)
>>sh=K.shape(x)
>> print(sh)
Tensor("Shape_3:0", shape=(3,), dtype=int32, device=/device:GPU:0)

@fercook Yes! You got the right reason. I really doubt that @anewlearner can run it without error.

@themummy  try using data format (batchsize, height, width, channels) for tensorflow. it improves speed

@rollingstone thanks for the tip. I'll try that!

@anewlearne My experimental results are similar to yours. When I use one single GPU, the mnist_cnn.py takes 104 seconds for running 12 epochs while 77 seconds with 4 GPUs.

@pengpaiSH
The speed improvement is not desired. Maybe we can try tensorflow. More work, faster speed.
Besides, I have some doubts about the jonilaserson's code.
def slice_batch(x, n_gpus, part):
    """
    Divide the input batch into [n_gpus] slices, and obtain slice no.[part].
    i.e. if len(x)=10, then slice_batch(x, 2, 1) will return x[5:].
    """
    sh = K.shape(x)
    L = sh[0] / n_gpus
    if part == n_gpus - 1:
        return x[part*L:]
    return x[part*L:(part+1)*L]

Assume this is a 2D case, the shape of x is (dim1/rows, dim2/cols, channels/num_featuremap). In slice_batch, the dim1/rows is divided instead of channes/num_featuremap. It is strange.
I want to test the accuracy of the code. But all of my gpus are in use and I don't have time now. Can you compare the result of using multi-gpus(2 type: jonilaserson's code,  jonilaserson's code with my change below) or only using one gpu? If you have time to do this, try more epochs, because of the random initialization.
Thanks.
To divide the samples according to channels, try
# 2d case
def slice_batch(x, n_gpus, part):
    sh = K.shape(x)
    L = sh[-1] / n_gpus
    if part == n_gpus - 1:
        return x[:, :, part*L]
    return x[:, :, part*L:(part+1)*L]

@anewlearner I think you are mixing up the data shape of x which is (number of samples, rows, cols, channels) assuming you are in the backend of TensorFlow. Thus, It has nothing to do with 2D case or 3D case.

@pengpaiSH
 x = Input(model.input_shape[1:], name=model.input_names[0])

@anewlearner Er...Why  we should set x by skipping input_shape[0]?

@pengpaiSH
You are right :)

@anewlearner Totally confused now...

@anewlearner What if ```x = Input(model.input_shape, name=model.input_names[0])

@pengpaiSH
It should be x = Input(model.input_shape[1:], name=model.input_names[0]).
model.input_shape = (None, rows, cols, channels).
In the definition of Input,
def Input(shape=None, batch_shape=None,
          name=None, dtype=K.floatx(),
          tensor=None):
    # Arguments
        shape: a shape tuple (integer), not including the batch size.
            For instance, `shape=(32,)` indicates that the expected input
            will be batches of 32-dimensional vectors.
        batch_shape: a shape tuple (integer), including the batch size.
            For instance, `batch_shape=(10, 32)` indicates that
            the expected input will be batches of 10 32-dimensional vectors.
            `batch_shape=(None, 32)` indicates batches of an arbitrary number
            of 32-dimensional vectors.

@anewlearner Thanks for your explanation. Besides, would you please explain why a lambda layer could distribute the computation over different GPUs by splitting the batch data? ```slice_g = Lambda(slice_batch, lambda shape: shape, arguments={'n_gpus':n_gpus, 'part':g})(x)

@pengpaiSH Hi, peng, my wechat id is xukui1347. Could we make friends on wechat? I would benefit a lot from you. Thank you.

@barrykui Sure, done!

@pengpaiSH
It might be used to divided the input. Lamda is a layer in tensorflow.
In the definition of Lamda,
def __init__(self, function, output_shape=None, arguments={}, **kwargs):
function: The function to be evaluated.
'''
# Arguments
function: The function to be evaluated.
    Takes input tensor as first argument.
output_shape: Expected output shape from function.
    Can be a tuple or function.
    If a tuple, it only specifies the first dimension onward;
         sample dimension is assumed either the same as the input:
         `output_shape = (input_shape[0], ) + output_shape`
         or, the input is `None` and the sample dimension is also `None`:
         `output_shape = (None, ) + output_shape`
    If a function, it specifies the entire shape as a function of the
    input shape: `output_shape = f(input_shape)`
arguments: optional dictionary of keyword arguments to be passed
    to the function.
'''

@jonilaserson Thanks for posting the above code. Although I second the question of @pengpaiSH : why do you simply concatenate the results returned from the GPUs? Doesn't it make sense to average these results, which is the typical protocol in batching? See for example the following two multi-gpu tensorflow examples, where the returned gradients are indeed explicitly averaged:
https://github.com/tensorflow/tensorflow/blob/r0.11/tensorflow/models/image/cifar10/cifar10_multi_gpu_train.py#L137
https://github.com/tensorflow/models/blob/master/inception/inception/inception_train.py#L169

The results from the GPUs are not the loss values but the actual outputs
from the networks, so you can't average them. I don't know how to directly
average the losses and still work (and train) within the keras framework.
On Nov 14, 2016 8:29 PM, "samuelBB"<EMAIL_ADDRESS>wrote:

@jonilaserson https://github.com/jonilaserson Thanks for posting the
above code. Although I second the question of @pengpaiSH
https://github.com/pengpaiSH : why do you simply concatenate the
results returned from the GPUs? Doesn't it make sense to average these
results, which is the typical protocol in batching? See for example the
following two multi-gpu tensorflow examples, where the returned gradients
are indeed explicitly averaged:
https://github.com/tensorflow/tensorflow/blob/r0.11/
tensorflow/models/image/cifar10/cifar10_multi_gpu_train.py#L137
https://github.com/tensorflow/models/blob/master/inception/
inception/inception_train.py#L169
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fchollet/keras/issues/2436#issuecomment-260419140,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFdLCP6rJC8EAPXjMczLAx-ecp8XLglMks5q-KhugaJpZM4IMTcS
.

For the time being, in order to make good use of multiple GPUs, I am doing my deep learning programs with MXNET, which I only specify the GPU IDs and the lib will do everything it needs under the hood.

A simple example:
keras.backend.gpu_setup = ["gup0", "gpu1"...]
#Proceed to code in Keras as usual. All of the GPU complexity handled in the background.
Would hate to have to go to MXNET for this. Keras is truly an amazing library. This would make Keras the best DL library by far in my opinion.

For the time being, I found https://github.com/kuza55/keras-extras/blob/master/utils/multi_gpu.py to be a nice stand-alone solution to achieve data parallelism. Explained by the author here<EMAIL_ADDRESS>But yeah, I too am greatly looking forward to bonafide multi-GPU integration!

@pGit1 This is exactly what I think and I need!!! +1

@samuelBB I could not open the link URL that you provided<EMAIL_ADDRESS>

@pengpaiSH Maybe try the link without "#.1x6yd12n3" at the end. Or just try searching "Transparent Multi-GPU Training on TensorFlow with Keras" on google, and it should be one of the top hits (form medium.com). Let me know if you get it, I can post the example code if you need...

@samuelBB This function is great, but I get errors for any model that contains batch normalization.
ValueError: Variable batchnormalization_2_running_mean/biased already exists, disallowed. Did you mean to set reuse=True in VarScope? Originally defined at:
I can't figure out a workaround so that I can use batch normalization. But if I remove the batch normalization layers, I do get a 2x speedup using 2 gpus (bn removed for both trials)!

@tstandley Try setting mode=2 for the batch normalization: BatchNormalization(mode=2). This will be slightly different behavior than the default, as explained in the Keras docs. If that fixes your issue, you can actually get the default mode=0 to work by updating your Keras to the latest commit from github (specifically, sha 771010f), which will probably be available in the next pypi release.

@samuelBB Dude, you're awesome! I've been spending so much time trying in vain to fix that issue. Setting the mode to 2 works.
I don't really understand the documentation on the modes. What's the downside?

@samuelBB, the link you provided (from kuza55) is doing exactly what my
code is doing above.
On Thu, Dec 1, 2016 at 3:28 AM, samuelBB<EMAIL_ADDRESS>wrote:

For the time being, I found https://github.com/kuza55/
keras-extras/blob/master/utils/multi_gpu.py to be a nice stand-alone
solution to achieve data parallelism. Explained by the author here:
https://medium.com/@kuza55/transparent-multi-gpu-
training-on-tensorflow-with-keras-8b0016fd9012#.1x6yd12n3
But yeah, I too am greatly looking forward to bonafide multi-GPU
integration!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fchollet/keras/issues/2436#issuecomment-264051515,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFdLCCutkDalDybXfJF0Lt6wfsvtNeaVks5rDiK8gaJpZM4IMTcS
.

@jonilaserson Indeed, I noticed that its the same method. However, when I tried to integrate your code from above into my pipeline, it gave some unclear errors; kuza55's code worked smoothly for me without modification (hence I didn't bother to figure out the issue). Your example worked fine on my machine, but I am using more complex models (e.g. resnets).

@jonilaserson I used to train my model on a single GPU with a batch size of 20 without a problem. But now when I tried training with 4 GPUs with 4x the batch size, I got Out of Memory error. As I understood it correctly, for data parallelism, the batch will be splitted into 4 parts and loaded onto 4 GPUs respectively. Could you shed some light on it?

@samuelBB @KlaymenGC These methods implement data parallelism. With data parallelism, only neural activations can be spread across GPU's. Parameters must be copied to each GPU. Both parameters and activations take memory. For convolutional neural networks, the parameter cost is typically insignificant compared to the activations, so data parallelism works well for memory distribution so you can increase batch size almost linerly. On the other hand, parameters are the bulk of the memory usage for RNN's and densely connected layers. For stacked RNN's, model parallelism can sometimes be made to work well and can be used to store parameters on separate GPU's. Model parallelism is typically more difficult to deal with though, and you have to determine how to split your model manually. On the plus side, I think Keras already supports model parallelism. Just use with tf.device to determine which layers should run on which GPU's.
That said, it doesn't really make sense to wait for first party Keras support, because I don't think that will fix your OOM issue. When first party Keras support does land, it'll be in the form of data parallelism and or model parallelism and have the same problems you have now.

well if that is true that is just sad...and makes an mxnet backend that
much more appealing.
On Thu, Dec 15, 2016 at 4:37 PM, tstandley<EMAIL_ADDRESS>wrote:

@samuelBB https://github.com/samuelBB @KlaymenGC
https://github.com/KlaymenGC These methods implement data parallelism.
With data parallelism, only neural activations can be spread across GPU's.
Parameters must be copied to each GPU. Both parameters and activations take
memory. For convolutional neural networks, the parameter cost is typically
insignificant compared to the activations, so data parallelism works well
for memory distribution so you can increase batch size almost linerly. On
the other hand, parameters are the bulk of the memory usage for RNN's and
densely connected layers. For stacked RNN's, model parallelism can
sometimes be made to work well and can be used to store parameters on
separate GPU's. Model parallelism is typically more difficult to deal with
though, and you have to determine how to split your model manually. On the
plus side, I think Keras already supports model parallelism. Just use with
tf.device to determine which layers should run on which GPU's.
That said, it doesn't really make sense to wait for first party Keras
support, because I don't think that will fix your OOM issue. When first
party Keras support does land, it'll be in the form of data parallelism and
or model parallelism and have the same problems you have now.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fchollet/keras/issues/2436#issuecomment-267451474,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ANU-SiwctTAyk3ZyvLShEOQfk7jykhhOks5rIbMcgaJpZM4IMTcS
.

@pGit1 I think you misunderstand me. It is not a limitation of Keras. This is how deep learning works. Unless you want to get VERY researchy, you have to choose data parallelism or model parallelism (or a combination). No backend change could fix that. These two options have their various drawbacks, and those drawbacks will exist no matter what engine you are using. Like I said, as of right now, you can use both parallelization techniques with Keras (with the TF backend).
On a more practical note, you should simply use the multi_gpu.py code (which implements model parallelism) and change the batch size until it runs. The batch size will be no smaller than it would for a single GPU, but you will get an almost linear speedup with more GPU's.

Thank you for the clarification. This makes sense!
On Thu, Dec 15, 2016 at 11:13 PM, tstandley<EMAIL_ADDRESS>wrote:

@pGit1 https://github.com/pGit1 I think you misunderstand me. It is not
a limitation of Keras. This is how deep learning works. Unless you want to
get VERY researchy, you have to choose data parallelism or model
parallelism (or a combination). No backend change could fix that. These two
options have their various drawbacks, and those drawbacks will exist no
matter what engine you are using. Like I said, as of right now, you can use
both parallelization techniques with Keras (with the TF backend).
On a more practical note, you should simply use the multi_gpu.py code
(which implements model parallelism) and change the batch size until it
runs. The batch size will be no smaller than it would for a single GPU, but
you will get an almost linear speedup with more GPU's.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fchollet/keras/issues/2436#issuecomment-267513064,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ANU-SlXiRiksPfANkzrrjwGamGGF6Wr5ks5rIg_igaJpZM4IMTcS
.

@tstandley isn't multi_gpu.py using data parallelism? As well as this<EMAIL_ADDRESS>The model is replicated on several GPUs, and the input batch is sliced.

@KlaymenGC Thanks for spotting my typo. I've fixed it. For the record, multi_gpu.py does indeed use data parallelism!

+1 to everything tstandley said about the batch size and data parallelization.
One comment though is that it could be better to do join the different GPU towers at the loss function level (instead of the model output level) since that means that the loss function computation happens in the CPU and not the GPU and might slow things down. @tstandley, what do you think?

I wonder why multi_gpu.py tells the CPU to do the concatenation. Not sure that affects where the loss calculation happens, but the bigger issue is where the gradient update happens. I'm not sure about any of these. I think a built-in Keras implementation could avoid the concatenation step entirely by computing the loss for each portion of the batch on its GPU and just averaging them, possibly on GPU0.
I also don't understand what happens to the weights in normal training/parallel training. If the update happens in the GPU, then for multi-gpu training, the result would have to be re-propagated to the rest of the GPU's, which is a special step. It doesn't seem efficient to transfer possibly hundreds of megabytes of weights for every mini-batch. Maybe someone who knows Keras better could comment.

@jonilaserson @tstandley I don't know if you have encountered such strange problem, but for me, the weights file that I saved from the multiple-GPU-training are always about 50KB smaller than the weights that are from 1-GPU-training, the accuracy and loss looked normal during both training processes but the prediction result of multiple-GPU-training is a total mess... I tried to find the cause but until now still no success, could you share some ideas on this one?

I updated to the most recent version of keras and multi_gpu.py is no longer working for me. I get an error:
File "/home/tstand/Dropbox/image2weight/volume_model/multi_gpu.py", line 31, in make_parallel
outputs = model(inputs)
File "/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py", line 569, in call
self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
File "/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py", line 632, in add_inbound_node
Node.create_node(self, inbound_layers, node_indices, tensor_indices)
File "/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py", line 164, in create_node
output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
File "/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py", line 2235, in call
output_tensors, output_masks, output_shapes = self.run_internal_graph(inputs, masks)
File "/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py", line 2378, in run_internal_graph
computed_mask))
File "/usr/local/lib/python3.5/dist-packages/keras/layers/normalization.py", line 130, in call
broadcast_running_mean = K.reshape(self.running_mean, broadcast_shape)
File "/usr/local/lib/python3.5/dist-packages/keras/backend/tensorflow_backend.py", line 1228, in reshape
return tf.reshape(x, shape)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 2448, in reshape
name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 493, in apply_op
raise err
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 490, in apply_op
preferred_dtype=default_dtype)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 669, in convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py", line 176, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py", line 165, in constant
tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape, verify_shape=verify_shape))
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/tensor_util.py", line 441, in make_tensor_proto
tensor_proto.string_val.extend([compat.as_bytes(x) for x in proto_values])
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/tensor_util.py", line 441, in
tensor_proto.string_val.extend([compat.as_bytes(x) for x in proto_values])
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/compat.py", line 65, in as_bytes
(bytes_or_text,))
TypeError: Expected binary or unicode string, got 1
Any idea what's going on?

Hi,
I recently updated to Keras 2.0 and the code in multi_gpu.py broke. Below is the error trace. Nothing changed in the way I was using this code. Any thoughts?

ValueError                                Traceback (most recent call last)
 in ()
2
3 # this uses the TensorFlow backend to spread computation on multiple GPUs
----> 4 model_gpu = make_parallel(model, GPUS)
/home/adalbert/nbserver/urban-environments/keras-utils/multi_gpu.pyc in make_parallel(model, gpu_list)
36                 for x in model.inputs:
37                     input_shape = tuple(x.get_shape().as_list())[1:]
---> 38                     slice_n = Lambda(get_slice, output_shape=input_shape, arguments={'idx':i,'parts':gpu_count})(x)
39                     inputs.append(slice_n)
40
/usr/local/lib/python2.7/dist-packages/keras/engine/topology.pyc in call(self, inputs, **kwargs)
552
553             # Actually call the layer, collecting output(s), mask(s), and shape(s).
--> 554             output = self.call(inputs, **kwargs)
555             output_mask = self.compute_mask(inputs, previous_mask)
556
/usr/local/lib/python2.7/dist-packages/keras/layers/core.pyc in call(self, inputs, mask)
657         if 'mask' in arg_spec.args:
658             arguments['mask'] = mask
--> 659         return self.function(inputs, **arguments)
660
661     def compute_mask(self, inputs, mask=None):
/home/adalbert/nbserver/urban-environments/keras-utils/multi_gpu.pyc in get_slice(data, idx, parts)
13         print shape
14         print shape[:1] // parts, shape[:1]
---> 15         size = tf.concat(0, [ shape[:1] // parts, shape[1:] ])
16         stride = tf.concat(0, [ shape[:1] // parts, shape[1:]*0 ])
17         start = stride * idx
/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_ops.pyc in concat(values, axis, name)
1027       ops.convert_to_tensor(axis,
1028                             name="concat_dim",
-> 1029                             dtype=dtypes.int32).get_shape(
1030                             ).assert_is_compatible_with(tensor_shape.scalar())
1031       return identity(values[0], name=scope)
/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.pyc in convert_to_tensor(value, dtype, name, preferred_dtype)
635       name=name,
636       preferred_dtype=preferred_dtype,
--> 637       as_ref=False)
638
639
/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.pyc in internal_convert_to_tensor(value, dtype, name, as_ref, preferred_dtype)
700
701         if ret is None:
--> 702           ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
703
704         if ret is NotImplemented:
/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_ops.pyc in _autopacking_conversion_function(v, dtype, name, as_ref)
903   if dtype is not None and dtype != inferred_dtype:
904     return NotImplemented
--> 905   return _autopacking_helper(v, inferred_dtype, name or "packed")
906 # pylint: enable=invalid-name
907
/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_ops.pyc in _autopacking_helper(list_or_tuple, dtype, name)
866           elems_as_tensors.append(
867               constant_op.constant(elem, dtype=dtype, name=str(i)))
--> 868       return gen_array_ops._pack(elems_as_tensors, name=scope)
869     else:
870       return converted_elems
/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_ops.pyc in _pack(values, axis, name)
2039     A Tensor. Has the same type as values. The packed tensor.
2040   """
-> 2041   result = _op_def_lib.apply_op("Pack", values=values, axis=axis, name=name)
2042   return result
2043
/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.pyc in apply_op(self, op_type_name, name, **keywords)
761         op = g.create_op(op_type_name, inputs, output_types, name=scope,
762                          input_types=input_types, attrs=attr_protos,
--> 763                          op_def=op_def)
764         if output_structure:
765           outputs = op.outputs
/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.pyc in create_op(self, op_type, inputs, dtypes, input_types, name, attrs, op_def, compute_shapes, compute_device)
2327                     original_op=self._default_original_op, op_def=op_def)
2328     if compute_shapes:
-> 2329       set_shapes_for_outputs(ret)
2330     self._add_op(ret)
2331     self._record_op_seen_by_control_dependencies(ret)
/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.pyc in set_shapes_for_outputs(op)
1715       shape_func = _call_cpp_shape_fn_and_require_op
1716
-> 1717   shapes = shape_func(op)
1718   if shapes is None:
1719     raise RuntimeError(
/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.pyc in call_with_requiring(op)
1665
1666   def call_with_requiring(op):
-> 1667     return call_cpp_shape_fn(op, require_shape_fn=True)
1668
1669   _call_cpp_shape_fn_and_require_op = call_with_requiring
/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/common_shapes.pyc in call_cpp_shape_fn(op, input_tensors_needed, input_tensors_as_shapes_needed, debug_python_shape_fn, require_shape_fn)
608     res = _call_cpp_shape_fn_impl(op, input_tensors_needed,
609                                   input_tensors_as_shapes_needed,
--> 610                                   debug_python_shape_fn, require_shape_fn)
611     if not isinstance(res, dict):
612       # Handles the case where _call_cpp_shape_fn_impl calls unknown_shape(op).
/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/common_shapes.pyc in _call_cpp_shape_fn_impl(op, input_tensors_needed, input_tensors_as_shapes_needed, debug_python_shape_fn, require_shape_fn)
674       missing_shape_fn = True
675     else:
--> 676       raise ValueError(err.message)
677
678   if missing_shape_fn:
ValueError: Dimension 0 in both shapes must be equal, but are 1 and 3
From merging shape 0 with other shapes. for 'tower_0/lambda_1/concat/concat_dim' (op: 'Pack') with input shapes: [1], [3].

@adrianalbert
Change the code in multi_gpu from:
size = tf.concat(0, [shape[:1] // parts, shape[1:]])
stride = tf.concat(0, [ shape[:1] // parts, shape[1:]*0 ])
to:
size = tf.concat([shape[:1] // parts, shape[1:]], 0)
stride = tf.concat([shape[:1] // parts, shape[1:] * 0], 0)

@avolkov1 Are you able to run the multi_gpu.py now? Some people like me are stuck with the an "Incompatible shapes error" described here: https://github.com/kuza55/keras-extras/issues/7

@Eric2333 Yes, I'm able to run it. I modified it slightly. I don't know if this multi-gpu parallelism is correct to begin with, but it seems to run on multiple GPUs with the fixes I mentioned above. Here's the whole file code with modifications I made to get rid of warnings for Keras 2.0:
`# ref: https://raw.githubusercontent.com/kuza55/keras-extras/master/utils/multi_gpu.py @IgnorePep8
from keras import backend as K
from keras.layers.core import Lambda
from keras.models import Model
from keras.layers import merge
from keras.layers.merge import concatenate
if K.backend() == 'tensorflow':
import tensorflow as tf  # @UnresolvedImport
all = ('make_parallel',)
def make_parallel(model, gdev_list):
'''
:param gdev_list: List of gpu devices i.e. ['/gpu:0', '/gpu:1', ...]
'''
gpu_count = len(gdev_list)
def get_slice(data, idx=0, parts=0):
    shape = tf.shape(data)
    size = tf.concat([shape[:1] // parts, shape[1:]], 0)
    stride = tf.concat([shape[:1] // parts, shape[1:] * 0], 0)
    start = stride * idx
    return tf.slice(data, start, size)

outputs_all = [[] for _ in range(len(model.outputs))]

# Place a copy of the model on each GPU, each getting a slice of the batch
for idev, gdev in enumerate(gdev_list):  # range(gpu_count):
    with tf.device(gdev):  # tf.device('/gpu:%d' % i):
        with tf.name_scope('tower_%d' % idev) as _:  # as scope
            inputs = []
            # Slice each input into a piece for processing on this GPU
            for x in model.inputs:
                input_shape = tuple(x.get_shape().as_list())[1:]
                slice_n = Lambda(
                    get_slice, output_shape=input_shape,
                    arguments={'idx': idev, 'parts': gpu_count})(x)
                inputs.append(slice_n)

            outputs = model(inputs)

            if not isinstance(outputs, list):
                outputs = [outputs]

            # Save all the outputs for merging back together later
            for l in range(len(outputs)):
                outputs_all[l].append(outputs[l])

# merge outputs on CPU
with tf.device('/cpu:0'):
    merged = []
    for outputs in outputs_all:
        # merged.append(merge(outputs, mode='concat', concat_axis=0))
        merged.append(concatenate(outputs, 0))

    return Model(inputs=model.inputs, outputs=merged)

`

@avolkov1 What's your tensorflow version? It seems that my error comes from the callbacks function.

@Eric2333
Version 1.0.1. I am using a docker container for tensorflow:
tensorflow/tensorflow:1.0.1-devel-gpu
Taken from docker hub: https://hub.docker.com/r/tensorflow/tensorflow/tags/
I use nvidia-docker to run the container.

Here is an update version of to_multi_gpu() that is compatible with Keras 2.0:
from keras import backend as K
from keras.models import Model
from keras.layers import Input
from keras.layers.core import Lambda
from keras.layers.merge import Concatenate

def slice_batch(x, n_gpus, part):
    """
    Divide the input batch into [n_gpus] slices, and obtain slice no. [part].
    i.e. if len(x)=10, then slice_batch(x, 2, 1) will return x[5:].
    """
    sh = K.shape(x)
    L = sh[0] / n_gpus
    if part == n_gpus - 1:
        return x[part*L:]
    return x[part*L:(part+1)*L]

def to_multi_gpu(model, n_gpus=2):
    """Given a keras [model], return an equivalent model which parallelizes
    the computation over [n_gpus] GPUs.

    Each GPU gets a slice of the input batch, applies the model on that slice
    and later the outputs of the models are concatenated to a single tensor,
    hence the user sees a model that behaves the same as the original.
    """
    with tf.device('/cpu:0'):
        x = Input(model.input_shape[1:], name=model.input_names[0])

    towers = []
    for g in range(n_gpus):
        with tf.device('/gpu:' + str(g)):
            slice_g = Lambda(slice_batch, lambda shape: shape, arguments={'n_gpus':n_gpus, 'part':g})(x)
            towers.append(model(slice_g))

    with tf.device('/cpu:0'):
        merged = Concatenate(axis=0)(towers)

    return Model(inputs=[x], outputs=[merged])

Thanks @jonilaserson ! The I can run three GPUs now :) However, I have to compile my model before and after I call "to_multi_gpu". Any ideas?
model=Sequential()
model.add(Dense(32,activation='relu',input_dim=(28 * 28)))
model.add(Dense(16,activation='relu'))
model.add(Dense(10,activation='softmax'))

model.compile(optimizer=RMSprop(lr=0.001), loss='categorical_crossentropy', metrics=['accuracy'])
model = multigpu.to_multi_gpu(model, n_gpus=3)
model.compile(optimizer=RMSprop(lr=0.001), loss='categorical_crossentropy', metrics=['accuracy'])
model.fit(train_images, train_labels, validation_split = 0.05, epochs=25, batch_size=64)
...
works!

model=Sequential()
model.add(Dense(32,activation='relu',input_dim=(28 * 28)))
model.add(Dense(16,activation='relu'))
model.add(Dense(10,activation='softmax'))

model.compile(optimizer=RMSprop(lr=0.001), loss='categorical_crossentropy', metrics=['accuracy'])
model = multigpu.to_multi_gpu(model, n_gpus=3)
...
RuntimeError: You must compile a model before training/testing. Use `model.compile(optimizer, loss)`.

model=Sequential()
model.add(Dense(32,activation='relu',input_dim=(28 * 28)))
model.add(Dense(16,activation='relu'))
model.add(Dense(10,activation='softmax'))

model = multigpu.to_multi_gpu(model, n_gpus=3)
model.compile(optimizer=RMSprop(lr=0.001), loss='categorical_crossentropy', metrics=['accuracy'])
...
AttributeError: 'Sequential' object has no attribute 'input_names'

@abrahamrhoffman
If you want to run anyway, change this line
x = Input(model.input_shape[1:], name=model.input_names[0])

to
x = Input(model.input_shape[1:], name="test_name")

and call multigpu.to_multi_gpu(model, n_gpus=3) before model.compile.

@palloc @avolkov1 @jonilaserson I can load the model no problem using model to json and from json. My problem is loading the weigths of multi-gpu model. I saved them with ModelCheckpoint .h5 extension. Any help or suggestion aprreaciated.

@abnera ModelCheckpoint calls your model's save function. Because you have a multi-gpu model, you would like it to call the underlying model's save function instead. I wrote this code to monkeypatch the underlying model's save method onto the multi-gpu model:
new_model = Model(input=model.inputs, output=merged)
funcType = type(model.save)
monkeypatch the save to save just the underlying model
def new_save(self_,filepath, overwrite=True):
model.save(filepath, overwrite)
new_model.save=funcType(new_save, new_model)
return new_model

@abnera For me when I load the saved weights from the multi-GPU model, the model
won't give me the correct prediction. I have to save the model before data parallelism to make it work.

@tstandley thanks for the quick reply, are you saying that by using this type(model.save) ModepCheckpoint knows about this function and saves it in the original (single_gpu_model). So all I need to do is create this function and ModelCheckpoint knows about it?

@abnera You need to replace the save function on the multi-gpu model with the save function of the single gpu model. The code I posted does that (or did several months ago).

@tstandley Yeah. dummy me. I simply created a local ModelCheckpoint Callback where I always save the original model instead of the gpu version.
Thanks!

For those of you who are curious I posted my experimentation with different data-parallelism strategies.
https://github.com/avolkov1/keras_experiments
python package/library: https://github.com/avolkov1/keras_experiments/tree/master/keras_exp
More specifically take a look at:
https://github.com/avolkov1/keras_experiments/blob/master/keras_exp/multigpu/_multigpu.py
https://github.com/avolkov1/keras_experiments/blob/master/keras_exp/multigpu/optimizers.py
Example cifar10:
https://github.com/avolkov1/keras_experiments/blob/master/examples/cifar10_cnn_mgpu.py
In the optimizers.py I implemented an experimental approach OptimizerMultiGPUMixin trying to use NCCL to average the gradients. This approach did not seem to yield any speedup. Perhaps someone smarter than me can tinker with this code and get something working.

@jonilaserson thanks for the update for keras 2.0. I wonder can this multi-gpu code can be used for fit_generator function? If so, how can I use it?  Thanks.

If you use batchnormalization layer in multi-GPU I would suggest to compile to the latest master (not included in 1.1.0 yet) due to this: https://github.com/tensorflow/tensorflow/pull/8906

Does anyone use this to_multi_gpu on keras2 succesfully on more than 2 gpus. my environment is a remote centos server which has 4 k80 cards(8 gpus), when i try to call on more than 2 gpus, the server crash, and I lost the connection ......

The to_multi_gpu() is not working in my environmnet (Python 3.6, TF 1.1 compiled three days ago).
however, make_parallel() from avolkov1 works fine. thanks!
BTW: for the training and validation set:  len(train_set) % number_of_gpu  ==   0
This is the error message I get from to_multi_gpu():

Traceback (most recent call last):
File "main.py", line 273, in
sys.exit(main())
File "main.py", line 99, in main
testmodel.doc2vec_train()
File "/home/../NLP/tweetmodel.py", line 261, in doc2vec_train
self.tweetmodel.doc2VecModel_train_keras(self.parameter)
File "/home/.../NLP/tweetcluster.py", line 1490, in doc2VecModel_train_keras
model = to_multi_gpu(model, 2)
File "/home/.../NLP/tweetmodel.py", line 921, in to_multi_gpu
slice_g = Lambda(slice_batch, lambda shape: shape, arguments={'n_gpus': n_gpus, 'part': g})(x)
File "/home/.../develop/anaconda3/lib/python3.6/site-packages/keras/engine/topology.py", line 585, in call
output = self.call(inputs, **kwargs)
File "/home/../develop/anaconda3/lib/python3.6/site-packages/keras/layers/core.py", line 659, in call
return self.function(inputs, **arguments)
File "/home/.../NLP/tweetmodel.py", line 905, in slice_batch
return x[part * L:(part + 1) * L]
File "/home/../develop/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 497, in _SliceHelper
name=name)
File "/home/.../develop/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 655, in strided_slice
shrink_axis_mask=shrink_axis_mask)
File "/home/.../develop/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3568, in strided_slice
shrink_axis_mask=shrink_axis_mask, name=name)
File "/home/.../develop/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 590, in apply_op
param_name=input_name)
File "/home/.../develop/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 61, in _SatisfiesTypeConstraint
", ".join(dtypes.as_dtype(x).name for x in allowed_list)))
TypeError: Value passed to parameter 'begin' has DataType float32 not in list of allowed values: int32, int64

which version of keras do you use？
On May 8, 2017 11:05 AM, "JoeSchnorcher"<EMAIL_ADDRESS>wrote:
The to_multi_gpu() is not working in my environmnet (Python 3.6, TF 1.1
compiled three days ago).
however, make_parallel() from avolkov1 works fine. thanks!
BTW: for the training and validation set: len(train_set) % number_of_gpu ==
0
This is the error message I get from to_multi_gpu():
Traceback (most recent call last):
File "main.py", line 273, in
sys.exit(main())
File "main.py", line 99, in main
testmodel.doc2vec_train()
File "/home/../NLP/tweetmodel.py", line 261, in doc2vec_train
self.tweetmodel.doc2VecModel_train_keras(self.parameter)
File "/home/.../NLP/tweetcluster.py", line 1490, in doc2VecModel_train_keras
model = to_multi_gpu(model, 2)
File "/home/.../NLP/tweetmodel.py", line 921, in to_multi_gpu
slice_g = Lambda(slice_batch, lambda shape: shape, arguments={'n_gpus':
n_gpus, 'part': g})(x)
File "/home/.../develop/anaconda3/lib/python3.6/site-packages/keras/engine/topology.py",
line 585, in call
output = self.call(inputs, **kwargs)
File "/home/../develop/anaconda3/lib/python3.6/site-packages/keras/layers/core.py",
line 659, in call
return self.function(inputs, **arguments)
File "/home/.../NLP/tweetmodel.py", line 905, in slice_batch
return x[part * L:(part + 1) * L]
File "/home/../develop/anaconda3/lib/python3.6/site-packages/
tensorflow/python/ops/array_ops.py", line 497, in _SliceHelper
name=name)
File "/home/.../develop/anaconda3/lib/python3.6/site-packages/
tensorflow/python/ops/array_ops.py", line 655, in strided_slice
shrink_axis_mask=shrink_axis_mask)
File "/home/.../develop/anaconda3/lib/python3.6/site-packages/
tensorflow/python/ops/gen_array_ops.py", line 3568, in strided_slice
shrink_axis_mask=shrink_axis_mask, name=name)
File "/home/.../develop/anaconda3/lib/python3.6/site-packages/
tensorflow/python/framework/op_def_library.py", line 590, in apply_op
param_name=input_name)
File "/home/.../develop/anaconda3/lib/python3.6/site-packages/
tensorflow/python/framework/op_def_library.py", line 61, in
_SatisfiesTypeConstraint
", ".join(dtypes.as_dtype(x).name for x in allowed_list)))
TypeError: Value passed to parameter 'begin' has DataType float32 not in
list of allowed values: int32, int64
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/fchollet/keras/issues/2436#issuecomment-299893024, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AIBBircVMTKwyHx7AxagFSgf8Co8jqIKks5r3y9JgaJpZM4IMTcS
.

I use this version of Keras:
pytest$ python -c "import keras; print(keras.version)"
Using TensorFlow backend.
2.0.4

@JoeSchnorcher @avolkov1 The error you mention is related to the batch_size not being a multiple of the number of gpus. Therefore, when you divide an input batch into number of gpus you get a floating value instead of a integer number and that breaks the code.
Error: TypeError: Value passed to parameter 'begin' has DataType float32 not in list of allowed values: int32, int64
Solution:
In method: def slice_batch(x, n_gpus, part)
Replace : L = sh[0] / n_gpus
With: L = sh[0] // n_gpus
The double slash will ensure you truncate the value into an integer example: 11/2 = 5 instead of 5.5
That should fix your code.
I also cced @avolkov1 so that he could fix it in his repository.

Thanks @abnera, this solved the error. Interesting, the batch_size was 128 and the number of GPUs is 2......

@JoeSchnorcher it could also be that your very last batch is smaller than 128 and it crashes at the last batch on an epoch. An example number of samples = 131 first batch no problem but on the second batch 3/2 gives you an error, assuming your exhausting all the samples in each epoch. Anyways, glad to help.

@abnera Thanks. I'll post my latest code soon. I added various features I was experimenting with such as contrib.nccl, and data_flow_ops.StagingArea. None of it made the code any faster though and sometimes slowed it down. But perhaps someone else will have better luck.
I want to experiment with Dataflow implementation of Tensorpack. I am not able to saturate the GPU utilization with this multi-gpu approach so I'm experimenting with different methods of staging data. My goal is to have a simple and peformant mini-framework/library. I'm using Keras for the Model layer, and use something like Dataflow for the data layer. Tensorpack mostly does what I'm trying to achieve, but it's very complicated with too many classes. I'm hoping to just re-use the Dataflow portion.

At this point, I really think multi_gpu.py should be integrated into the keras repo. This way people can do pull requests instead of trying to keep track of tweaks posted in an issue.

I think it should be integrated with keras also, having to copy all that data (I'm doing dense predictions) onto CPU instead of being able to calculate loss on each gpu then only transfer that kills performance.

I think Keras should simple have an interface that simply lets you specify
GPU id's to be used for training and testing time. This multi-gpu thing
with all its tweaks and nuances is a huge annoyance.
The contributions are exrtemely valuable and should be standard in the
Keras library for sure.
On Tue, May 9, 2017 at 7:18 PM, Joe Yearsley<EMAIL_ADDRESS>wrote:

I think it should be integrated with keras also, having to copy all that
data (I'm doing dense predictions) onto CPU instead of being able to
calculate loss on each gpu then only transfer that kills performance.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fchollet/keras/issues/2436#issuecomment-300328602,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ANU-SrcJQyVQO878A5ZFeLnKmUgfJwufks5r4PQ7gaJpZM4IMTcS
.

@avolkov1  does doing
 with tf.device(gdev):  # tf.device('/gpu:%d' % i):
    with tf.name_scope('tower_%d' % idev) as _:  # as scope
        inputs = []
        # Slice each input into a piece for processing on this GPU
        for x in model.inputs:
            input_shape = tuple(x.get_shape().as_list())[1:]
            slice_n = Lambda(
                get_slice, output_shape=input_shape,
                arguments={'idx': idev, 'parts': gpu_count})(x)
            inputs.append(slice_n)

requires all the data to be passed to each GPU while they will only use a slice of it? Or are things working just fine ?

@gsabran Yes, I think it might. So that's wasteful of GPU memory. I just pushed my latest tweaks for this multi-gpu stuff. Refer to my implementation here:
https://github.com/avolkov1/keras_experiments
And here for the actual implementation details:
https://github.com/avolkov1/keras_experiments/blob/master/keras_exp/multigpu/_multigpu.py
I create the slice under cpu context and apply the model under the dev context thus avoiding this copy I think.
# snippet
        for idev, dev in enumerate(gdev_list):
            # TODO: The last slice could cause a gradient calculation outlier
            # when averaging gradients. Maybe insure ahead of time that the
            # batch_size is evenly divisible by number of GPUs, or maybe don't
            # use the last slice.
            with tf.device('/cpu:0'):
                slices = []  # multi-input case
                for ix, x in enumerate(model.inputs):
                    slice_g = Lambda(
                        slice_batch,  # lambda shape: shape,
                        lambda shape: x.shape.as_list(),
                        name='stage_cpuSliceIn{}_Dev{}'.format(ix, idev),
                        arguments={'ngpus': ngpus, 'part': idev,
                                   'dev': dev})(x)
                    slices.append(slice_g)
                    # print('SLICE_G: {}'.format(slice_g))  # DEBUG
                # print('SLICES: {}'.format(slices))  # DEBUG

            with tf.device(dev), \
                    tf.variable_scope(global_scope, reuse=idev > 0), \
                    tf.name_scope('tower_%i' % idev):
                modeltower = model_(slices)
                towers.append(modeltower)
# ...

Test it and let me know. I was able to use huge batch sizes in cifar example that wouldn't fit on a single GPU, but seemed to run fine with multiple-GPUs. The huge batch size messes up the learning convergence, but for testing I just wanted to verify that I could overcome the memory limitation of a single GPU.
With gradients averaging I ran out of memory, but I was still able to max out slightly bigger batch sizes than single GPU. Don't use gradients averaging if you are memory limited. It's not necessary anyways, because TF implicitly adds gradients (some kind of magic via TF parameter server stuff I think and Keras shared layers). When averaging gradients the multi-GPU performance is not any faster compared to single-gpu, nor better convergence compared to not averaging gradients at least in my testing.

@avolkov1 Glad you're doing this! Keep up the good work.
A few of points:

This was written for python 2 right? It almost works with python3. I had to change
from cStringIO import StringIO
to
from io import StringIO

the part that overrides the compile method assumes your optimizer is explicit. I was using a string 'Adadelta' in my call to compile which doesn't have the usenccl property (because it's a string). I wound up explicitly creating the optimizer instead, and that worked, but it would be nice if it worked with strings

Finally:
Sadly I couldn't get this to work for my model It just kept printing warnings:
WARNING:tensorflow:Tried to colocate gradients/tower_0/model_2/model_1/block12_sepconv3_bn/moments/sufficient_statistics/count_grad/

@tstandley Yea, I primarily work with Python 2.7 but I'll add the fix to make it compatible with Python 3. These features are experimental. Maybe I'll add support for passing a string to be compatible with Model interface, but besides slight convenience it's actually in bad style to have completely different expected types for a parameter (string is not related to Optimizer derived class object).
Are you getting the warning when specifying to use NCCL? Or regardless you're always getting the warning when parallelizing your model for multi-GPUs?

@avolkov1 How does your method compare to make_parallel() in terms of speed? Have you compared the two?

Hey guys,
I'm trying to run to_multi_gpu(model, 1) and getting the error:
TypeError: () takes 1 positional argument but 2 were given
Here is the exact code I'm using:
model = to_multi_gpu(model, n_gpus=1)
model.compile(loss='categorical_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
And (updated with the updates earlier in this thread):
import tensorflow as tf
from tensorflow.contrib.keras.python.keras import backend as K
from tensorflow.contrib.keras.python.keras.models import Model
from tensorflow.contrib.keras.python.keras.layers import Input
from tensorflow.contrib.keras.python.keras.layers.core import Lambda
from tensorflow.contrib.keras.python.keras.layers.merge import Concatenate
def slice_batch(x, n_gpus, part):
"""
Divide the input batch into [n_gpus] slices, and obtain slice no. [part].
i.e. if len(x)=10, then slice_batch(x, 2, 1) will return x[5:].
"""
sh = K.shape(x)
L = sh[0] // n_gpus
if part == n_gpus - 1:
return x[partL:]
return x[partL:(part+1)*L]
def to_multi_gpu(model, n_gpus=2):
"""Given a keras [model], return an equivalent model which parallelizes
the computation over [n_gpus] GPUs.
Each GPU gets a slice of the input batch, applies the model on that slice
and later the outputs of the models are concatenated to a single tensor,
hence the user sees a model that behaves the same as the original.
"""
with tf.device('/cpu:0'):
    x = Input(model.input_shape[1:], name=model.input_names[0])

towers = []
for g in range(n_gpus):
    with tf.device('/gpu:' + str(g)):
        slice_g = Lambda(slice_batch, lambda shape: shape, arguments={'n_gpus':n_gpus, 'part':g})(x)
        towers.append(model(slice_g))

with tf.device('/cpu:0'):
    merged = Concatenate(axis=0)(towers)

return Model(inputs=[x], outputs=[merged])

Thanks!

@avolkov1
How will your model save the weight after each epoch? I tried the way provided by @jonilaserson but I could not save the model because Keras seemed to try saving all the weights from all the duplicate gpus (it should not).
From your snippet, the approach is nearly the same so does your also behave like that?

@vqdang
It works for me. The model weights and biases are shared between GPUs via the cpu (internals of TF via parameter server). The saving needs to happen using the non-mgpu model instance, ie the model passed to make parallel function. The mgpu model augmentation injects data slicing layers at input, and concat layer at output. It's not a typical Keras model, that's why I wrote a new class ModelMGPU to handle details like saving/loading model via reference to non-mgpu model.

@jonilaserson I'm getting the error: ValueError: Concatenate layer should be called on a list of inputs from to_multi_gpu.
I'm using the tensorflow.contrib version of Keras.
The call is: merged = Concatenate(axis=0)(towers) where 'towers' is a list of models.
Is this correct?
Any help greatly appreciated.
Thank you,
Dylan

Just use Pytorch. This is such an annoying issue with so many nuances that
make it almost impossible to keep up with. This issue is one of Keras
biggest downfalls. The ideal state is to feed gpu_ids as a list through
some API that contains a parameter "Optimize_for" that expects one of the
following ['Speed','Memory', 'Balanced'].
On Mon, Jun 12, 2017 at 11:22 AM, dylanrandle<EMAIL_ADDRESS>wrote:

@jonilaserson https://github.com/jonilaserson I'm getting the error:
ValueError: Concatenate layer should be called on a list of inputs from
to_multi_gpu.
I'm using the tensorflow.contrib version of Keras.
The call is: merged = Concatenate(axis=0)(towers) where 'towers' is a list
of models.
Is this correct?
Any help greatly appreciated.
Thank you,
Dylan
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fchollet/keras/issues/2436#issuecomment-307822698,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ANU-Smi558PEsIcuXO2yA6_PzR-vKTR8ks5sDVejgaJpZM4IMTcS
.

@avolkov1
@jonilaserson
I am experiencing a bizarre problem with both of you implementations: If I expand from 1 to 2 GPUs I can pretty much double the batch size without getting OOM but if I go to 3 or 4 GPUs I get OOM on the batch size that is only slightly bigger than the 2 GPU one.
Something like this: 1 GPU (batch 800) to 2 GPUs (batch 1600) is okay by 1 GPU (batch 800) to 4 GPUs (batch 2000) I get OOM).

@Slanothy
Don't use the multi gpu optimizers that I defined here:
https://github.com/avolkov1/keras_experiments/blob/master/keras_exp/multigpu/optimizers.py
Using those specialized optimizers batch size doesn't scale in regards to memory. Use regular keras optimizers: from keras_exp.multigpu.optimizers ... with an instance of class ModelMGPU (either by calling make_parallel or instantiating ModelMGPU directly).
If you're getting OOM with regular optimizers let me know. I would need a reproducible example.

@avolkov1
I am using the native optimizers as per your suggestion here https://github.com/avolkov1/keras_experiments/issues/1 and still getting this error...
All of the flags on make_parallel are kept at default. I will try to come up with a good reproducible example but it might be challenging due to differences in the GPUs we use (unless you have a box with 4 16GB P100s). I can't share the data I use unfortunately but I use relatively long sequences for inputs (750 timesteps - 4 numbers each), embed them and  concatenate into the LSTM layer of size 300...
Let me know if you would like a skeleton of the code or if I could play around with flags...
Testing specific numbers: batch of 1000 fits on 1 GPU but only 1600 fits on 2 and 2000 to 2400 on 4 (with training OOMing after a few batches)

@Slanothy
Yea, I have access to the hardware. Share whatever is shareable and I'll take a look. Send it to me at<EMAIL_ADDRESS>if you'd like. If you are using a container maybe try these flags: --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 when starting up the container.

@avolkov1 I am getting Invalid argument "name" passed to K.function with Tensorflow backend (in make_train_function), when using the latest version of your code. the previous one with no monkey patches worked...

@abnera
See this issue https://github.com/avolkov1/keras_experiments/issues/1 . You can add the name argument yourself for now or just comment out the monkey patch. It works for me

@abnera
@tRosenflanz
Sorry about that. I pushed the latest update for the monkey patch. Pull the latest changes. The monkey patched Function now has the following signature:
class Function(object):
    """Runs a computation graph.

    # Arguments
        inputs: Feed placeholders to the computation graph.
        outputs: Output tensors to fetch.
        updates: Additional update ops to be run at function call.
        name: a name to help users identify what this function does.
        enqueue_ops: List of ops to be run at funtion call for enqueue'ing.
    """

    def __init__(self, inputs, outputs, updates=None, name=None,
                 enqueue_ops=None, **session_kwargs):

I did this for the enqueue option in make_parallel function. If you're not using that enqueue option in the make_parallel function, then comment out the monkey patching in keras_exp/multigpu/_multigpu.py lines 36 and 37. Eventually I'll probably remove that monkey patching code because it's fragile. In this one case I'm modifying things under the hood instead of adhering to the public Keras API.

@avolkov1  et al.
I'm able to run keras_exp/multipgpu out of the box without problems on 4 Titan Xp with latest Keras/TF. Similar to what some describe above, occasionally I'll get a CUDA error that crashes training immediately after the first epoch, but seems to resolve with tweaks to the batch number (it happens even with batch numbers that are divisible by 4).
tensorflow/stream_executor/cuda/cuda_dnn.cc:427] could not set cudnn tensor descriptor: CUDNN_STATUS_BAD_PARAM
tensorflow/stream_executor/cuda/cuda_dnn.cc:427] could not set cudnn tensor descriptor: CUDNN_STATUS_BAD_PARAM
Aborted (core dumped)

On other situations I get this warning after first epoch— but training goes on without problems: can anyone clarify what it means?
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:247]
PoolAllocator: After 3788 get requests, put_count=3769 evicted_count=1000
eviction_rate=0.265322 and unsatisfied allocation rate=0.295407
I tensorflow/core/common_runtime/gpu/pool_allocator.cc:259]
Raising pool_size_limit_ from 100 to 110

@mptorr Hmm, based on some googling it seems like the CUDNN_STATUS_BAD_PARAM might be due to an empty array being processed by Tensorflow session. If the bug is not due to your logic of how you are reading/feeding the data into TF/Keras, it might be that the multigpu is not slicing the data properly causing this (in which case I would like to identify this corner case and fix it). I found a few references to that bug on internet search: google the bug
The other warning with pool_allocator I think is benign although maybe that's a symptom of some issue. I've seen a use case where out-of-memory (OOM) error was occurring. The fix was to instantiate the initial serial model under the tf.device('/cpu:0') context i.e. pin to CPU prior to parallelizing for multi-gpu. I suggest doing that in the docstring of make_parallel function if an OOM error occurs.
If you can share a code snippet to reproduce the issue I can dig into it some more. Please file an issue https://github.com/avolkov1/keras_experiments/issues with a reproducible code snippet.

thanks so much @avolkov1, yes I had seen the thread on TF  and had thoroughly checked my workflow for issues, therefore I was wondering if keras_exp could be the source for CUDNN_STATUS_BAD_PARAM under specific circumstances (i.e., certain batch sizes).
I'll work on posting a reproducible code snippet error #2: interestingly, the pool_allocator warning also seems to be dependent on batch size.

I have tried to_multi_gpu function but I got 2x slower result than when I use single GPU. I have a 10 Nvidia Titan Xp GPUs environment. Anybody knows what is the problem?

@dasolyn , Does your server enable full connectivity between the 10 GPUs?

@barrykui Yes, I think so, maybe. Here is output of tensorflow when I train it.

@dasolyn I got similar results (multiple GPUs slower than 1). So far the problem seems to be the gradients are not computed in parallel, but on the parameter server device. See issue #7515 for more details and work-in-progress implementation of gradient averaging.

@barrykui Thanks. Just from looking at the code it seems it's just another version of the split/concat code evolving around. There's not parallel gradient computation and gradient averaging.
According to my measurement of this code on 1. 2 or 4 GTX 1070 GPUs, the multi-GPU variant is slower and not scaling well. Time per epoch is taken cca after epoch 2-3, when the times stabilize.
1x GTX 1070
$ CUDA_VISIBLE_DEVICES=2 time python mnist_cnn.py
1:34.58 min
7s/epoch

2x GTX 1070
$ CUDA_VISIBLE_DEVICES=2,3 time python mnist_cnn.py
10s/epoch

4x GTX 1070
$ CUDA_VISIBLE_DEVICES=0,1,2,3 time python mnist_cnn.py
21s/epoch

Yes. Keras doesn't support multi-gpu. If you want to utilize your
environment properly (with near linear scaling) use MXNET or pyTorch.
On Tue, Aug 8, 2017 at 1:40 AM, dasolyn<EMAIL_ADDRESS>wrote:

I have tried to_multi_gpu function but I got 2x slower result than when I
use single GPU. I have a 10 Nvidia Titan Xp GPUs environment. Anybody knows
what is the problem?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fchollet/keras/issues/2436#issuecomment-320855751,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ANU-StD8dBhUVaCPlhIgSN-dyg4085RXks5sV_TkgaJpZM4IMTcS
.

@pGit1 This is simply not true. Keras does support multiple GPUs with tensorflow as this issue discussion shows. multi_gpu.py implements data parallelism. Data parallelism isn't useful for all architectures, so that might be the problem for @bzamecnik. That said, data parallelism is probably what most people want for most things, so it's also likely that @bzamecnik is actually encountering some kind of error. We'd need more details to diagnose.
For many architectures, you can use model parallelism with Keras and tensorflow as well. Pointers are in the code above.

When using this Code https://github.com/kuza55/keras-extras/blob/master/utils/multi_gpu.py  it seems, that there is an error in the regularizers like described in this issue https://github.com/kuza55/keras-extras/issues/22. I cant tell if this is caused by the method, the model is copied and merged or if there is a flaw in the regularizers of keras. This is in tensorflow 1.12.
def` conv2d_bn(x, nb_filter, nb_row, nb_col, padding='same', strides=(1, 1), bias=False):

 """
    Utility function to apply conv + BN.
    (Slightly modified from https://github.com/fchollet/keras/blob/master/keras/applications/inception_v3.py)
    """
    if K.image_data_format() == "channels_first":
        channel_axis = 1
    else:
        channel_axis = -1
    x = Convolution2D(nb_filter, (nb_row, nb_col),
                      strides=strides,
                      padding=padding,
                      use_bias=bias,
                      kernel_regularizer=regularizers.l2(0.00004), ##<---- causes error because no _loss
                      kernel_initializer=initializers.VarianceScaling(scale=2.0, mode='fan_in', distribution='normal',
                                                                      seed=None))(x)
    x = BatchNormalization(axis=channel_axis, momentum=0.9997, scale=False)(x)
    x = Activation('relu')(x)
    return x

I get the error:
 „AttributeError: 'Model' object has no attribute '_losses'„
caused by outputs = model (inputs) that merges the outputs of the different splits in one model.

Dont use Keras for Multi-GPU training. That is the solution. Use native
Tesnorflow, MXNET, or some other reasonably multi-gpu friendly library.
On Tue, Aug 22, 2017 at 12:17 AM, CeadeS<EMAIL_ADDRESS>wrote:

When using this Code https://github.com/kuza55/keras-extras/blob/master/
utils/multi_gpu.py it seems, that there is an error in the regularizers
like described in this issue kuza55/keras-extras#22
https://github.com/kuza55/keras-extras/issues/22. I cant tell if this
is caused by the method, the model is copied and merged or if there is a
flaw in the regularizers of keras. This is in tensorflow 1.12.
def` conv2d_bn(x, nb_filter, nb_row, nb_col, padding='same', strides=(1, 1), bias=False):
"""
Utility function to apply conv + BN.
(Slightly modified from https://github.com/fchollet/keras/blob/master/keras/applications/inception_v3.py)
"""
if K.image_data_format() == "channels_first":
channel_axis = 1
else:
channel_axis = -1
x = Convolution2D(nb_filter, (nb_row, nb_col),
strides=strides,
padding=padding,
use_bias=bias,
kernel_regularizer=regularizers.l2(0.00004), ##<---- causes error because no _loss
kernel_initializer=initializers.VarianceScaling(scale=2.0, mode='fan_in', distribution='normal',
seed=None))(x)
x = BatchNormalization(axis=channel_axis, momentum=0.9997, scale=False)(x)
x = Activation('relu')(x)
return x
I get the error:
„AttributeError: 'Model' object has no attribute '_losses'„
caused by outputs = model (inputs) that merges the outputs of the
different splits in one model.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fchollet/keras/issues/2436#issuecomment-323914735,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ANU-Soh_82c7QOgf3s5O6shUUNNmj7yhks5salZKgaJpZM4IMTcS
.

@pGit1 it has already been pointed out that Keras can support multi-gpu training so this kind of defeatist attitude is not helpful and is quite misguiding for a lot of people trying to find a solution... I personally prefer https://github.com/avolkov1/keras_experiments/ for now for it's ease of use

@tRosenflanz Yes, also I came around https://github.com/avolkov1/keras_experiments/ recently and have been testing that. So far it seems it's indeed working well. Also a problem with my previous measurements was a wrongly set up machine. On a standard cloud instance with multiple GPUs I can observe a speed up, especially with avolkov1 code. Anyway I'm writing a summary article with a lot of measurements, so stay tuned.

FYI - we just added an example of data-parallel distributed training with Keras using Horovod - https://github.com/uber/horovod/blob/master/examples/keras_mnist.py. It works both for multiple GPUs within the server, and across servers. Hope it helps.

I used the code of @jonilaserson. And it works.  However, it seems that multi-gpu converged slower compared to single gpu. Anyone else observed the same?

@michelleowen you typically want to adjust learning rate to total # of GPUs across all the servers - here's an example for very simple scaling.  Facebook published a paper with a more sophisticated strategy that works for a large number of GPUs.

@alsrgv, thank you. This is very helpful. I will do some experiments to see how it works in my case.

I think the function previously mentioned by @avolkov1 is finally coming into Keras:
(https://github.com/fchollet/keras/blob/master/keras/utils/training_utils.py)

@fernandoandreotti Yes and no. It's a cleaned-up variant of function from kuza55. It has nice documentation and grabs list of devices via device_lib instead of CUDA_VISIBLE_DEVICES. On the other hand it's missing some stuff from avolkov1: slicing on CPU, save/load of parameters of original serial model. Since there's no wrapper class, so the latter is not necessary, but at least might be documented.

Keras v2.0.9 now includes it. Despite the improvements that can be done, I guess this issue should be closed.

Any example of how to use this in the docs?
On Fri, Nov 3, 2017 at 8:55 AM, Fernando Andreotti<EMAIL_ADDRESS>
wrote:

Keras v2.0.9 now includes it. Despite the improvements that can be done, I
guess this issue should be closed.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fchollet/keras/issues/2436#issuecomment-341695139,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ANU-StrjyNRZPJPvmUVjvJLg-EBD08eCks5syw0pgaJpZM4IMTcS
.

Yes: https://keras.io/utils/#multi_gpu_model
You can also check out Horovod, which seems nice.

Is there any intention for making it work with CNTK too?

@avolkov1 @jonilaserson I see another issue with fit_generator.
It is failing for a particular callback, and that is for : checkpoint = ModelCheckpoint(filepath=filepath, monitor='val_acc',verbose=1,
save_best_only=True)
Now If you use fit_generator, and in to that, if you pass ModelCheckpoint callback, It will fail after the first epoch. I removed ModelCheckpoint callback and its working fine with other multiple callbacks! So there seems to be an issue while using fit_generator and ModelCheckpoint callback.
Looking forward to your feedback!

@avolkov1,  I tried your approach and hit an issue with tensorflow keras 1.11  (tf.keras.utils.multi_gpu_model) while loading the model saved through this approach, because __init__()  was missing required positional arguments, see a similar issue. After I failed to load the entire model, I tried to load the weights only and I still wasn't able to solve the problem, so I assume defining  **kwargs in the __init__() in the custom class of multi_gpu_model is necessary.
However, I found this solution, instead of customizing the multi_gpu_model this author customized ModelCheckPoint, where **kwargs** is passed in to the __init()__ of the class, and passed it in the Callback during the training model.fit(), I find this solution pretty neat. I tested it and I had no problem loading the model trained using this ModelCheckPoint with multiple GPUs.
Just for the people who might hit the same problem as I did.

@avolkov1 , note that one thing which is missing from the proposed solution is optimizer state - this will lead to the optimizer state not being saved, as the inner model does not hold the optimizer.
To fix this, I propose to add the following method to the compile method to the class:
`class ModelMGPU(Model):
def init(self, ser_model, gpus):
pmodel = multi_gpu_model(ser_model, gpus)
self.dict.update(pmodel.dict)
self._smodel = ser_model
def __getattribute__(self, attrname):
    """Override load and save methods to be used from the serial-model. The
    serial-model holds references to the weights in the multi-gpu model.
    """
    if attrname in ['to_json', 'load', 'load_weights', 'save', 'save_weights']:
        return getattr(self._smodel, attrname)

    return super(ModelMGPU, self).__getattribute__(attrname)

def compile(self, optimizer,
            loss=None,
            metrics=None,
            loss_weights=None,
            sample_weight_mode=None,
            weighted_metrics=None,
            target_tensors=None,
            **kwargs):
    Model.compile(self, optimizer,
            loss=loss,
            metrics=metrics,
            loss_weights=loss_weights,
            sample_weight_mode=sample_weight_mode,
            weighted_metrics=weighted_metrics,
            target_tensors=target_tensors,
            **kwargs)
    self._smodel.optimizer = self.optimizer
    self._smodel.loss = self.loss
    self._smodel.metrics = self.metrics
    self._smodel.sample_weight_mode = self.sample_weight_mode
    self._smodel.loss_weights = self.loss_weights

`


Column possible values get shuffled when action in is imported

The order of the possible values in a column get shuffled when imported.

Fixed in version B.2.8.0


book-store: Implement exercise

Why?
Ruby Exercism was missing this exercise.
What?

How Has This Been Tested?

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] Enhancement (non-breaking change which adds functionality)
[ ] Breaking change (fix or enhancement that would cause existing functionality to change)
[ ] Gardening (code and/or documentation cleanup)

Checklist:

[ ] I have read the CONTRIBUTING document.
[ ] My change relies on a pending issue/pull request
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

Thanks @jgr let us know if you need any help.
Out of interest: Which documentation have you referred to for making this PR?

Howdy @Insti! I'm following this guide to adding an exercism exercise to another language. I regret that I've lost the context of how I ended up on that document! Is it out of date?

Thanks. That is the most up to date document.
I want to remove the section about creating an empty WIP pull request. so I was interested if that was the document you'd gotten the idea from.

Have you made any progress on this @jgr ?

Even partial commits are helpful, so we can tell you you're on the right track..
We should probably write a guide to creating new exercises and generators.

@Insti I'd love a review of this PR now, I think I have all the bits required of a new exercise.  Here's a list of the files in the PR and how they were created, I look forward to your feedback!

config.json I added a section to this file, per this document.
exercises/book-store/.meta/generator/book_store_case.rb I wrote this, following the directions here.
exercises/book-store/.meta/solutions/.version I added this manually, because I saw it referenced in the README a few places.  It isn't clear to me how it is used, or if I was supposed to add it manually or expect it to be generated somehow.
exercises/book-store/.meta/solutions/book_store.rb This is my sample solution.
exercises/book-store/README.md I generated this using the Configlet tool.  It wasn't clear to me if I was supposed to do this myself or not.
exercises/book-store/book_store_test.rb I generated this using ./bin/generate book-store.

Great, thanks. I'm a bit busy a t the moment, I'll check this out later in the week.

I had a quick look, and this all looks good 👍
Just the test data that needs updating to the latest version.

Thanks @Insti! I went and and pushed an update with updated tests.

Thank you @jgr!


[TASK] Reformat code to PSR-2 + normalize file header comment

:+1:
When this one is merged we can configure StyleCI

Beside one missing space looks good for me :+1:


DHT Vote

Does anyone know how utorrent implements the "vote" system over the DHT?
I looked online but there is next to nothing about it.
Some information:
http://lists.ibiblio.org/pipermail/bittorrent/2011-January/002413.html
http://arxiv.org/ftp/arxiv/papers/1305/1305.0711.pdf
You get a message something like this:
{ a:
   { id: '907050177fb05376db71ce066663819503eb0a9a',
     target: '546d6d0e5aacfae66299f26a300a428c497b23b5',
     token: '5b3f3de3d4f5ba42721dac14b2076dc64bc829da',
     vote: 0 },
  q: 'vote',
  t: '+=\u0000\u0000',
  v: 'UT??',
  y: 'q' }

The only thing I dont get it the target They say its sha1(info-hash + "rating") but i dont know. So the rating 1-5 and 0 been the query for votes. So if a infohash has 5 votes ranging from 1 to 5 would you receive 5 responses?
Has anyone played around with this?

It seems that the peers can cast their vote at sha1(info# + {0,1,2,3,4}). Although, it's not been bep'd so I think only utorrent would handle these requests.
And an user willing to check the ratings for a given infohash would DHT.get at sha1(info# + 0), sha1(info# + 1), sha1(info# + 2), sha1(info# + 3) and sha1(info# + 4). So 5 DHT.get, to receive the number of votes received for each rating.
That paper states that 1% of the total DHT traffic is related to votes (mostly 0 stars), and that seems to be like an awful lot. I'll try to hack something around bittorent-dht to see if I can reproduce that figure.

@pldubouilh Thank you, that is really helpful :+1:

So I played around with the idea.
Here is the code:
var DHT = require('bittorrent-dht');
var crypto = require('crypto');
function sha1(str) {
	return crypto.createHash('sha1').update(str).digest('hex')
}

var dht = new DHT();

dht.listen(8000, function() {
	console.log('now listening');
});

function lookup(infoHash) {
	console.log('lookup(' + infoHash + ')');

	function cb(err, result) {
		if (result)
			console.log(err, result);
	}

	for (var i = 0,
	    j = 5; i < j; i++) {
		dht.get(sha1(infoHash + i), cb);
	};
}

dht.on('ready', function() {
	console.log('now ready');
});
dht.on('error', function(err) {
	console.log(err);
});
dht.on('warning', function(warning) {
	console.log(warning);
});
dht.on('vote', function(query, peer) {
	var infoHash = query.a.target.toString('hex');
	lookup(infoHash);
});

dht.on('announce', function(peer, infoHash) {
	lookup(infoHash.toString('hex'));
});

It is interesting that nodes send out a vote query. I dont think they use the get query but the vote. From what I understand the vote: 0 is the query for the votes from 1-5 so then vote: 1 would be a vote of one and so on.
Using the put method to keep track of how many votes would not work. The number of votes would keep getting overwritten.

I think they are only using the vote even to broadcast the fact that their is new vote to query
So you are not query the i+1 hash all the time

Right, so I did some tests as well - the query is, indeed, vote and not get, so you'll have to change a few lines in client.js otherwise the message is considered as unexpected.
Also, for the record, it seems that the paper's correct. I receive a good amount of votes, and they are mostly 0...

@pldubouilh would you mind sharing your code?

There you go. It's nothing really fancy, I just added a case: 'vote', so I can see what's happening instead of treating it as unexpected.


no load_dataset.py、concatenate_datasets.py、dataset_dict.py

Hi, I'm interested in your work, but When running the command :
python baddiffusion.py --project default --mode train+measure --dataset CIFAR10 --batch 128 --epoch 50 --poison_rate 0.1 --trigger BOX_14 --target HAT --ckpt DDPM-CIFAR10-32 --fclip o -o --gpu 5
I meet the error :

I have run insatll.sh successfully, but I can't find these thress python files.

Again, these three files are not in the datasets folder.

Hi, thank you for your interest. I think this issue is caused by the missing package, datasests of HuggingFace. As a result, please use the command, pip install datasets to install the package. Let me know if there is any issues. Thank you so much.


Avoid unnecessary allocations in resize! by using views.

When looking into https://github.com/DynareJulia/FastLapackInterface.jl/issues/39 I noticed that for a QRWYWs the T matrix is re-allocated quite often, even when computing the QR factorization of a smaller matrix then specified at instantiation.
This can be avoided by using views.
This is a proof-of-concept re-implementation based on what is currently in FastLapackInterface.
import LinearAlgebra: BlasInt, BlasFloat, require_one_based_indexing, chkstride1
import LinearAlgebra.BLAS: @blasfunc
import LinearAlgebra.LAPACK: chklapackerror, liblapack

mutable struct QRWYWs{R<:Number,MT<:StridedMatrix{R}}
    work::Vector{R}
    T::MT
end

function QRWYWs(A::StridedMatrix{T}; kwargs...) where {T <: BlasFloat}
    resize!(QRWYWs(T[], Matrix{T}(undef, 0, 0)), A; kwargs...)
end

function Base.resize!(ws::QRWYWs, A::StridedMatrix; blocksize=36, work=true)
    require_one_based_indexing(A)
    chkstride1(A)
    m, n = BlasInt.(size(A))
    @assert n >= 0 ArgumentError("Not a Matrix")
    m1 = min(m, n)
    nb = min(m1, blocksize)
    ws.T = similar(ws.T, nb, m1)
    if work
        resize!(ws.work, nb*n)
    end
    return ws
end

function geqrt!(ws::QRWYWs, A::AbstractMatrix{Float64}; resize=true, blocksize=36)
    m, n = size(A)
    minmn = min(m, n)
    nb = min(minmn, blocksize)
    t1 = size(ws.T, 1)
    if t1 < nb
        if resize
            resize!(ws, A, work = true)
        else
            #throw(WorkspaceSizeError(nb, minmn))
            error("Cannot resize.")
        end
    end
    T = @view(ws.T[1:nb, 1:minmn])
    if nb > 0
        lda = max(1, stride(A, 2))
        work = ws.work
        info = Ref{BlasInt}()
        ccall(
            (@blasfunc(dgeqrt_), liblapack), Cvoid,
            (Ref{BlasInt}, Ref{BlasInt}, Ref{BlasInt}, Ptr{Float64},
            Ref{BlasInt}, Ptr{Float64}, Ref{BlasInt}, Ptr{Float64},
            Ptr{BlasInt}),
            m, n, nb, A,
            lda, T, max(1, stride(ws.T, 2)), work,
            info
        )
        chklapackerror(info[])
    end
    return A, T
end

At the moment, this is for Float64 only and I have not yet thought about the kwargs (blocksize).
But it seems to save allocations:
import FastLapackInterface as FLA
let n=50;
   A = rand(n, n);
   ws1 = QRWYWs(A);
   ws2 = FLA.QRWYWs(A);
   for j in (0, 5, 10, 25, 50, 60)
     B = rand(n, j)
     @show size(B)
     println("new:")
     @time geqrt!(ws1, B);
     println("old:")
     @time geqrt!(ws2, B);
   end
 end

gives
size(B) = (50, 0)
new:
  0.000001 seconds
old:
  0.000003 seconds (1 allocation: 64 bytes)
size(B) = (50, 5)
new:
  0.000049 seconds
old:
  0.000006 seconds (1 allocation: 256 bytes)
size(B) = (50, 10)
new:
  0.000016 seconds
old:
  0.000015 seconds (1 allocation: 896 bytes)
size(B) = (50, 25)
new:
  0.000048 seconds
old:
  0.000038 seconds (1 allocation: 5.062 KiB)
size(B) = (50, 50)
new:
  0.000098 seconds
old:
  0.000085 seconds (1 allocation: 14.188 KiB)
size(B) = (50, 60)
new:
  0.000112 seconds
old:
  0.000090 seconds (2 allocations: 46.125 KiB)

Is this worth a PR?

Yes @manuelbb-upb , please prepare a PR. Best


This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.

I'm having this same thing happen: https://github.com/Microsoft/vscode/issues/46255#issue-307227468
Originally posted by @vscodebot in https://github.com/Microsoft/vscode/issues/46255#issuecomment-376936452

I haven't added any extensions, so I'm reasonably sure that it isn't caused by an extension.

Can you please follow the steps in https://github.com/Microsoft/vscode/wiki/Native-Crash-Issues to get at more details around the crash and attach the result here? Thanks!