Repeated "Already exists: Resource" errors when I increase my model size

System information

Custom model built by creating convolutional network tf graph with SGD optimizer and running
Windows 10
TensorFlow installed from pip
TensorFlow version (use command below): 1.12
Python version: 3.6
Running on cpu

I've created a convolutional network by making a custom graph for feed forward and using a tensorflow optimizer to train it (I've tried with SGD and Adam).
`sess = tf.Session()
optimizer = tf.train.GradientDescentOptimizer(learning_rate=.01)

input_size = 16
input_matrix = tf.placeholder(tf.float32, shape=[1,input_size,input_size])
label_ph = tf.placeholder(tf.float32, shape=[28*28])
model_out = Model.feed_forward(input_matrix)

init = tf.global_variables_initializer()
sess.run(init)

loss = tf.losses.mean_squared_error([tf.convert_to_tensor(label_ph, tf.float32)], [model_out])
train = optimizer.minimize(loss)
sess.graph.finalize()`

This is my graph definition. I know Model.feed_forward is opaque but it's basically a series of functions that get variable weights and feed input through layers. Just to give you an idea here's an example of a convolutional feed forward:
`def feed_input(input_tensors, filters, biases, stride, zero_pad=True):
        if zero_pad:
            pad_size = int((int(filters[0].shape[0]) + 1)/2 - 1)
            input_tensors = tf.map_fn(lambda inp:
                tf.pad(inp, [[pad_size, pad_size], [pad_size, pad_size]]), input_tensors)

        _, shape_x, _ = input_tensors.shape

        unflattened_output = tf.map_fn(lambda input_tensor: tf.convert_to_tensor([[[tf.add(tf.tensordot(curr_filter,
            ConvLayer.__get_input_slice__(input_tensor, x, y, tf.shape(curr_filter)[0]), 2), biases[filter_ind])
         for x in range(0, shape_x - curr_filter.shape[0] + 1, stride)]
         for y in range(0, shape_x - curr_filter.shape[0] + 1, stride)]
         for filter_ind, curr_filter in enumerate(filters)], dtype=tf.float32), input_tensors)

        shape_1, shape_2, shape_3, shape_4 = unflattened_output.shape

        return tf.reshape(unflattened_output, [shape_1 * shape_2, shape_3, shape_4])`

After defining my graph I am running a loop that gets some input and runs sess.run(train, feed_dict={input_matrix:img, label_ph:label_image}). If I make input_size smaller (e.g. 4) or if I reduce the number of layers in my network, everything runs fine and it does exactly what I expect it to do. However, when my network grows in the ways I just mentioned it consistently spits out these errors during sess.run(train, feed_dict={input_matrix:img, label_ph:label_image}):

2018-11-15 00:46:31.069913: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.069914: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.069914: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.069927: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.070138: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.078467: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.080434: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.081575: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.083125: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.085370: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.089281: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.090932: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.091080: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.091398: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.091916: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.092735: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.098503: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.094920: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.099036: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.101236: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.101988: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.101966: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.103264: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.102945: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.106092: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.106858: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.108494: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.109495: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.109987: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.112326: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.112525: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.113595: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.113831: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.115342: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.115418: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.120685: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.122569: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.122708: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.123904: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.124832: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.126926: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.127336: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.129575: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.128498: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.115477: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.130180: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.130601: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.131469: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.135938: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.132834: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.139111: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_20/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.139728: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.140827: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.141429: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.142202: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.140004: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.142621: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.144039: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.147757: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.145043: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.149502: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.146723: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.148971: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.145861: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.149850: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.151642: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.152947: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.154227: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.155666: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.157249: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.157877: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.159295: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_24/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.160204: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.163481: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.163750: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot_256/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.164518: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.166533: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/strided_slice_1/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.172400: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.174768: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2018-11-15 00:46:31.174793: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_4/gradients/map_22/while/Tensordot/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
Traceback (most recent call last):
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1334, in _do_call
return fn(*args)
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.AlreadyExistsError: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
[[{{node gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var}} = TemporaryVariabledtype=DT_FLOAT, shape=[5,5], var_name="gradients/...dd/tmp_var", _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/Aidan/Documents/PycharmProjects/SketchIt/src/network/FeedImages.py", line 57, in
sess.run(train, feed_dict={input_matrix:img, label_ph:label_image})
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 929, in run
run_metadata_ptr)
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1328, in _do_run
run_metadata)
File "C:\Users\Aidan\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.AlreadyExistsError: Resource __per_step_4/gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
[[{{node gradients/map_24/while/Tensordot_512/transpose/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var}} = TemporaryVariabledtype=DT_FLOAT, shape=[5,5], var_name="gradients/...dd/tmp_var", _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Anyone know what might be the cause? Maybe some concurrency issue since its telling me things that shouldn't exist have already been created? Help would be much appreciated, I'm close to making this thing work! Thanks for your time.

What is "AddOpsRewrite_Add/tmp_var/struct" in your graph? Why are you using temporary variables? They are not compatible with while loops, which is what you're seeing in the error message.
(I thought temporary variables had been removed as part of tf's API a long time ago)

I st

Those are created by sess.run(train) where train is GradientDescentOptimizer.minimize. So tensorflow is generating them, not me.
Also I should update that I rolled back to tensorflow 1.8 (haven't tried 9, 10, or 11 just due to lack of time) and the error no longer comes up.

Issue close was accidental

What is ArithmeticOptimizer in your code? These ops don't look like they were added by TF's backward pass.

Hm this might come from grappler.

@rmlarsen I see that the memory optimizer adds temporaryvariable. This is not safe to do inside a while loop because the temporaryvariable op has weird issues with the resource manager.

ArithmeticOptimizer comes from running GradientDescentOptimizer.minimize as well. I'm running train in a few nested python for loops. I figured that's super typical, but should I be making a list of inputs and feeding them in with one call instead?
If anyone's interested you can see exactly what I'm running here. I already uploaded the network on a different account. https://github.com/awoods12/Sketch_It

Could it be an issue about tf.map_fn rather than optimizer?
I met the same errors and the core part of my code was like
y = tf.map_fn(some_function, x)

I found it worked when x had shape [1], but got errors otherwise.
And then I found another way without map_fn and everything works well now.

The bug seems to be in a Grappler optimization.
On Mon, Dec 3, 2018 at 7:07 AM yangyangxcf<EMAIL_ADDRESS>wrote:

Could it be an issue about tf.map_fn rather than optimizer?
I met the same errors and the core part of my code was like
y = tf.map_fn(some_function, x)
I found it worked when x had shape [1], but got errors otherwise.
And then I found another way without map_fn and everything works well now.
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/23780#issuecomment-443740562,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAATxY7VtRUpvdBNn-8g8JG3V6oETeCJks5u1T42gaJpZM4Yg0tH
.

--

Alex

I have the same problem on TF 1.12 + Ubuntu 16.04 and my custom code. Weirdly enough, even though I share the vact_scope code in different modules, the problem does not happen always (It might be due to calling tf.nn.dynamic_rnn in the same scope twice with scope.reuse_variables() between the calls)
@ezhulenev, any news on the problem? Any workarounds? Maybe there's a way to disable the ArithmeticOptimizer? (or whole Grappler? What is the expected performance hit then?)
2019-02-07 17:26:05.004296: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.030579: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.031806: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.048779: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.052690: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.054059: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.056958: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.062997: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.074335: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.093691: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.106734: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.128811: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.145847: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.147453: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.164530: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.168118: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.174878: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.181530: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.242049: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.287269: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.375503: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.737421: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.737421: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.756781: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn1/while/vact_scope/vact_iteration/prior_halting_logits/dense/BiasAdd/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19Te
mporaryVariableOp6TmpVarE
2019-02-07 17:26:05.757632: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE
2019-02-07 17:26:05.915909: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE

Weirdly, I see strange nested scopes (__per_step_186/OptimizeLoss/gradients/network/rnn2/while/vact_scope/vact_iteration/vact_iteration/output_projection_wrapper/BiasAdd_10/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/
N10tensorflow19TemporaryVariableOp6TmpVarE), which I do not create in the code (the GLOBAL_VARIABLES collection does not contain any variables with vact_iteration/vact_iteration in the name)

You can disable all optimizations in arithmetic optimizer with something like:
    from google3.third_party.tensorflow.core.protobuf import rewriter_config_pb2

    config_proto = tf.ConfigProto()

    off = rewriter_config_pb2.RewriterConfig.OFF
    config_proto.graph_options.rewrite_options.arithmetic_optimization = off

    session = tf.Session(config=config_proto)

another example in https://github.com/tensorflow/tensorflow/blob/b17d53c0cd07bbae46a55dc9c12d3f48a1c19604/tensorflow/python/debug/lib/debug_graph_reconstruction_test.py#L45-L51
But I suspect that the real problem is in memory optimizer, you can disable it with this option: https://github.com/tensorflow/tensorflow/blob/4e19402174593095a0837d6e7a8ca3f686b67543/tensorflow/core/protobuf/rewriter_config.proto#L122
Can you try if disabling one of them helps.

Either arithmetic_optimization = off or memory_optimization = off does the job, thanks!

@artsobolev Out of curiosity, what was the issue with the model size?
I was running into the same issue as described but was able to 'fix' it by disabling arithmetic optimizations via
    from google3.third_party.tensorflow.core.protobuf import rewriter_config_pb2

    config_proto = tf.ConfigProto()

    off = rewriter_config_pb2.RewriterConfig.OFF
    config_proto.graph_options.rewrite_options.arithmetic_optimization = off

    session = tf.Session(config=config_proto)

without having to disable memory optimization.
Also, I was wondering if anyone might have any additional insight into this issue and/or any ideas about possibly fixing it instead of just disabling arithmetic optimizations?

Do you have minimal reproducible example? I have a few theories, but small example would help.

@saforem2, disabling optimizations worked for me as well, but so did decreasing the hidden layer size in the LSTM from 512 to 128. Apparently, the optimizer behaves differently in these cases.
@ezhulenev, unfortunately, I do not have such example.

@ezhulenev Unfortunately I don't have a minimal reproducible example either.
If you really think it would help, I can work on writing up a gist or walking through the steps to reproduce the error, but I think it would require cloning a (moderately sized) repository of mine.

I'll try to work on a fix, I'll let you know when it will be in master, so you could test it.

Hello,
I am facing a similar issue but I can't seem to find this google3 library to import the off  constant mentioned by the comments above.
Can anyone tell me where to get google3 ?
Thank you.

I also meet the same issue with tf.nn.conv1d, by using
    config_proto = tf.ConfigProto(log_device_placement=False, allow_soft_placement=True)
    off = rewriter_config_pb2.RewriterConfig.OFF
    config_proto.graph_options.rewrite_options.arithmetic_optimization = off

    sess = tf.Session(config=config_proto)

the issue is fixed. So i think maybe tf1.13 has some inner bug.

I had the same problem about import rewriter_config_pb2 from google3
from tensorflow.core.protobuf import rewriter_config_pb2 config_proto = tf.ConfigProto(log_device_placement=False, allow_soft_placement=True) off = rewriter_config_pb2.RewriterConfig.OFF config_proto.graph_options.rewrite_options.arithmetic_optimization = off      session = tf.Session(config=config_proto)

I had the same problem about import rewriter_config_pb2 from google3
from tensorflow.core.protobuf import rewriter_config_pb2

config_proto = tf.ConfigProto(log_device_placement=False, allow_soft_placement=True)

off = rewriter_config_pb2.RewriterConfig.OFF

config_proto.graph_options.rewrite_options.arithmetic_optimization = off

session = tf.Session(config=config_proto)

Just remove "google3" from the example and it should work.

Just remove "google3" from the example and it should work.

But I had another problem I used python3.6 and I countered this problem "AlreadyExists error"

I tried to disable the arithmetic optimizer but I don't know where does it ? in  version python3.7, TF1.13.1

2019-05-10 23:23:48.280599: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.280882: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.281480: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.281866: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.282864: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.283515: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.284000: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.284436: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.284976: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.285390: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.285765: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.286219: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.286526: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.287017: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.287346: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.287845: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.288143: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.288607: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.289050: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.289331: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.289795: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.290269: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.290532: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.291102: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.292775: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.293485: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.293981: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.294523: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.295292: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.296044: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
2019-05-10 23:23:48.296854: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.AlreadyExistsError: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
	 [[{{node training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "Generator2.py", line 232, in <module>
    estimator_model.train(input_fn=customTrain, steps=100)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 358, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1124, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1158, in _train_model_default
    saving_listeners)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1407, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 676, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1171, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1270, in run
    raise six.reraise(*original_exc_info)
  File "/usr/local/lib/python3.7/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1255, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1327, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1091, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.AlreadyExistsError: Resource __per_step_12/training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var/N10tensorflow19TemporaryVariableOp6TmpVarE
	 [[{{node training/TFOptimizer/gradients/lstm/while/ReadVariableOp_8/Enter_grad/ArithmeticOptimizer/AddOpsRewrite_Add/tmp_var}}]]
'''

If you are using the estimators API, try something like this:
import tensorflow as tf
from tensorflow.core.protobuf import rewriter_config_pb2
session_config = tf.ConfigProto()
off = rewriter_config_pb2.RewriterConfig.OFF
session_config.graph_options.rewrite_options.arithmetic_optimization = off
run_config = tf.estimator.RunConfig(session_config=session_config)
...
estimator = tf.estimator.Estimator(model_fn, model_dir, run_config)

Note to developers: this issue is not fixed. Theses are just workarounds as far as I can understand. Could you please re-open the issue?

Thank so much. I will try then If it's still existing. I will re-open issue @183amir

Perfect solution. I ran my model on non GPU machine and ended up with this error, on GPU machine it works totally fine.. Does presence/absence of GPU anything to do with arithmetic optimization?

Same problem here. Workaround is not a fix.
Reopen it.

Thank you for the report. We will fix the corresponding rewrite in Grappler. It looks like it generates non-unique node names.

I have this exact same issue in tf 2.0 beta0.  Is there a workaround for that?

I'm having the same issue and I can't get the workaround to work. Do I do something with the tf.Session instance after creating it or will it automatically be used?
For now I just try different sequence lengths in my model, some work, some don't.

tensorflow.keras.backend.set_session worked. using TF 1.13

For me, either reducing the graph size or memory_optimization  = off helped. TF 1.14
As follows:
from tensorflow.core.protobuf import rewriter_config_pb2

config_proto = tf.ConfigProto()
off = rewriter_config_pb2.RewriterConfig.OFFrite_options.arithmetic_optimization = off
config_proto.graph_options.rewrite_options.memory_optimization  = off

Here is the corrected code example:
from tensorflow.core.protobuf import rewriter_config_pb2
config = tf.ConfigProto()
off = rewriter_config_pb2.RewriterConfig.OFF
config.graph_options.rewrite_options.memory_optimization  = off
return tf.Session(config=config)

For me making the second parameter of Embedding layer in LSTM worked.

@aidanw95
It looks like you are using an older Version of Tensorflow. Many bugs have been fixed in the latest version. Can you please execute your code using Latest Version (2.6.0) and let us know if the issue still persists? Thanks!

from tensorflow.core.protobuf import rewriter_config_pb2
config_proto = tf.ConfigProto()
off = rewriter_config_pb2.RewriterConfig.OFFrite_options.arithmetic_optimization = off
config_proto.graph_options.rewrite_options.memory_optimization  = off

thank you my friend, god bless you!


config: remove configs

All files move into https://github.com/tidb-community-bots/configs.

/assign @Mini256
PTAL~ Thanks!

Codecov Report

Merging #164 (4d36ae4) into master (34498df) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #164   +/-   ##
=======================================
  Coverage   85.72%   85.72%
=======================================
  Files          11       11
  Lines         946      946
=======================================
  Hits          811      811
  Misses         76       76
  Partials       59       59

Flag
Coverage Δ

unittests
85.72% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 34498df...4d36ae4. Read the comment docs.

Why split the configuration file to a new repo?

/can-merge

/merge

/lgtm

/hold

/merge

/hold cancel

Why split the configuration file to a new repo?

Because we have too many configuration files, and I plan to develop a new program for org configuration to merge configuration files, I plan to separate it for independent management and development.


grpc->grpc-js

NOTE: this is a draft just so it's easier to have a discussion about the issue. I know I'm not doing the commits properly.
Which problem is this PR solving?

opentelemetry-js uses grpc, even though this has been deprecated in favor of grpc-js
i'm personally seeing build times for my services instrumented with otel going WAY up as we compile all this C++ (eww). this is bad enough now, and going to get worse as we apply otel instrumentation to all our other microservices.
previously , we wanted to upgrade to nodejs 15.x.x, but ran into bugs with node-gyp during build & couldn't do so. I know technically you guys don't concern yourself with non-lts node versions, and we were able to fix this by using node 14.x.x which is fine for us, but wouldn't you rather not have these kinds of dependency issues?

Short description of the changes

replacing grpc with grpc-js
modifying the tests so they still work with grpc-js (this is where the difficulty comes in)

As you can see, despite the claims, grpc-js is not really a drop-in replacement for grpc. I'm sure the client is a drop-in replacement, but the server isn't, and I don't know enough about this test framework & how the server is supposed to work normally to fix it.
For reference: https://www.npmjs.com/package/@grpc/grpc-js

Please also make sure you sign the CLA or we cannot accept your contribution.

@dyladan as I say, I didn't particularly expect this version to be accepted since I didn't bother doing it right (i used --no-verify). But I'll sign the CLA just in case.

I started looking into this error and it's a tricky one. The failure occurs somewhere deep in the internals of grpc-js and it's hard to see what's happening

@dyladan assuming you're referring to
Uncaught TypeError: Cannot read property 'interceptors' of undefined

Is there reason to believe this is actually a problem with the grpc-js client & thus that it's actually not compatible? Or is it just a test problem?

@dyladan assuming you're referring to
Uncaught TypeError: Cannot read property 'interceptors' of undefined

Is there reason to believe this is actually a problem with the grpc-js client & thus that it's actually not compatible? Or is it just a test problem?

Yes that's my feeling. I considered that it might be a bug with the client, but in order to prove that i would want to create a much more minimal reproduction and I don't really have time for that at the moment.

@dyladan fortunately, I have time. It's amazing how efficiently you can work when you remove the requirement of "not writing code that is absolute, utter shit."
anyway, pulled what i think is the equivalent of what we do in the test out into just a normal script and it works fine. I even got into it with the debugger (which I had originally tried to set up to debug inside of mocha) and the callProperties.callOptions looked fine (it was {}, but at least you could read its .interceptors).
That's the annoying part btw; I know how to set a breakpoint in grpc-js's client.ts and run a jest test with the debugger so i break in the test, but doing the same steps with mocha failed. And I definitely don't have time to learn how to do weird things with mocha.

@dyladan fortunately, I have time. It's amazing how efficiently you can work when you remove the requirement of "not writing code that is absolute, utter shit."
anyway, pulled what i think is the equivalent of what we do in the test out into just a normal script and it works fine. I even got into it with the debugger (which I had originally tried to set up to debug inside of mocha) and the callProperties.callOptions looked fine (it was {}, but at least you could read its .interceptors).
That's the annoying part btw; I know how to set a breakpoint in grpc-js's client.ts and run a jest test with the debugger so i break in the test, but doing the same steps with mocha failed. And I definitely don't have time to learn how to do weird things with mocha.

You can run tests with debugging enabled by doing npm test -- --inspect-brk and connecting to it with vs code's "Attach to process" command. I typically also do npm test -- -b --inspect-brk so the tests will stop running when a failure is encountered.

So it turns out that the problem was that the options object was actually secretly a function returning
CollectorExporterNodeBase.ts? [sm]:68 Uncaught ReferenceError: Cannot access 'promise' before initialization
    at _onFinish (/home/dmr/job/repo/bukkitz/vendor/opentelemetry-js/packages/opentelemetry-exporter-collector-grpc/src/CollectorExporterNodeBase.ts:18:1158)
    at _onSuccess (/home/dmr/job/repo/bukkitz/vendor/opentelemetry-js/packages/opentelemetry-exporter-collector-grpc/src/CollectorExporterNodeBase.ts:18:820)
    at Object.options (/home/dmr/job/repo/bukkitz/vendor/opentelemetry-js/packages/opentelemetry-exporter-collector-grpc/src/util.ts:15:4)
    at eval (eval at makeUnaryRequest (/home/dmr/job/repo/bukkitz/vendor/grpc-node/packages/grpc-js/src/client.ts:292:5), <anonymous>:1:1)
    at ServiceClientImpl.makeUnaryRequest (/home/dmr/job/repo/bukkitz/vendor/grpc-node/packages/grpc-js/src/client.ts:292:5)
    at ServiceClientImpl.<anonymous> (/home/dmr/job/repo/bukkitz/vendor/grpc-node/packages/grpc-js/src/make-client.ts:177:15)
    at CollectorTraceExporter.send [as _send] (/home/dmr/job/repo/bukkitz/vendor/opentelemetry-js/packages/opentelemetry-exporter-collector-grpc/src/util.ts:15:4)
    at promise (/home/dmr/job/repo/bukkitz/vendor/opentelemetry-js/packages/opentelemetry-exporter-collector-grpc/src/CollectorExporterNodeBase.ts:18:1263)
    at new Promise (<anonymous>)
    at CollectorTraceExporter._sendPromise (/home/dmr/job/repo/bukkitz/vendor/opentelemetry-js/packages/opentelemetry-exporter-collector-grpc/src/CollectorExporterNodeBase.ts:18:641)
    at CollectorTraceExporter.send (/home/dmr/job/repo/bukkitz/vendor/opentelemetry-js/packages/opentelemetry-exporter-collector-grpc/src/CollectorExporterNodeBase.ts:45:55)
    at /home/dmr/job/repo/bukkitz/vendor/opentelemetry-js/packages/opentelemetry-exporter-collector-grpc/src/util.ts:15:4
    at Array.forEach (<anonymous>)
    at /home/dmr/job/repo/bukkitz/vendor/opentelemetry-js/packages/opentelemetry-exporter-collector-grpc/src/util.ts:15:4```

which was subsequently converted by `checkOptionalUnaryResponseArguments<ResponseType>` to `undefined` (this is probably a bad thing for it to do).

In any case, my suspicion all along was that something in how the async code was done in this test is causing strange things to happen.

So it turns out that the problem was that the options object was actually secretly a function returning
CollectorExporterNodeBase.ts? [sm]:68 Uncaught ReferenceError: Cannot access 'promise' before initialization
    at _onFinish (/home/dmr/job/repo/bukkitz/vendor/opentelemetry-js/packages/opentelemetry-exporter-collector-grpc/src/CollectorExporterNodeBase.ts:18:1158)
    at _onSuccess (/home/dmr/job/repo/bukkitz/vendor/opentelemetry-js/packages/opentelemetry-exporter-collector-grpc/src/CollectorExporterNodeBase.ts:18:820)
    at Object.options (/home/dmr/job/repo/bukkitz/vendor/opentelemetry-js/packages/opentelemetry-exporter-collector-grpc/src/util.ts:15:4)
    at eval (eval at makeUnaryRequest (/home/dmr/job/repo/bukkitz/vendor/grpc-node/packages/grpc-js/src/client.ts:292:5), <anonymous>:1:1)
    at ServiceClientImpl.makeUnaryRequest (/home/dmr/job/repo/bukkitz/vendor/grpc-node/packages/grpc-js/src/client.ts:292:5)
    at ServiceClientImpl.<anonymous> (/home/dmr/job/repo/bukkitz/vendor/grpc-node/packages/grpc-js/src/make-client.ts:177:15)
    at CollectorTraceExporter.send [as _send] (/home/dmr/job/repo/bukkitz/vendor/opentelemetry-js/packages/opentelemetry-exporter-collector-grpc/src/util.ts:15:4)
    at promise (/home/dmr/job/repo/bukkitz/vendor/opentelemetry-js/packages/opentelemetry-exporter-collector-grpc/src/CollectorExporterNodeBase.ts:18:1263)
    at new Promise (<anonymous>)
    at CollectorTraceExporter._sendPromise (/home/dmr/job/repo/bukkitz/vendor/opentelemetry-js/packages/opentelemetry-exporter-collector-grpc/src/CollectorExporterNodeBase.ts:18:641)
    at CollectorTraceExporter.send (/home/dmr/job/repo/bukkitz/vendor/opentelemetry-js/packages/opentelemetry-exporter-collector-grpc/src/CollectorExporterNodeBase.ts:45:55)
    at /home/dmr/job/repo/bukkitz/vendor/opentelemetry-js/packages/opentelemetry-exporter-collector-grpc/src/util.ts:15:4
    at Array.forEach (<anonymous>)
    at /home/dmr/job/repo/bukkitz/vendor/opentelemetry-js/packages/opentelemetry-exporter-collector-grpc/src/util.ts:15:4

which was subsequently converted by checkOptionalUnaryResponseArguments<ResponseType> to undefined (this is probably a bad thing for it to do).
In any case, my suspicion all along was that something in how the async code was done in this test is causing strange things to happen.

Does that mean you discovered a solution?

@dyladan I do not. I mean that I believe this is a bug in the test's async code & not a sign that grpc-js is incompatible with the exporter code.
In other words, I think that if you were to rewrite the test more carefully to set up in the proper order (probably with async/await), it would work fine.

Closing since this has been done with https://github.com/open-telemetry/opentelemetry-js/pull/2092


JS API generation fails due to jsdossier exception, illegal char

I'm trying to figure out how to adapt OpenSphere for my usage but I can't seem to find an API for the JS. Does such a thing exist?
I tried using JSDoc to generate one but it just gives me an unstructured list of several hundred classes, which is more useful than digging through the files, but still pretty hard to use.
Is there a certain editor I can use? Another JS doc generation library that does better?

From the README.md:
API documentation can be built locally by running npm run apidoc, which outputs to dist/apidoc.

However that probably isn't the best place to start. I recommend https://opensphere.readthedocs.io/en/latest/guides/plugin_guide.html instead. Reading the code (in VS Code) is also good...
If you have something specific you need help with, I've had awesome results asking on the Slack channel - the core team is really response.
HTH.

Oh, I thought that was to rebuild the handwritten guides for some reason.
I tried it in git bash but it doesn't seem to work...
Processing js-dossier configuration file...
Adding 1912 files to js-dossier sources...
Writing JSDoc configuration to: E:\git\opensphere-yarn-workspace\workspace\opensphere\.build\dossier.conf.json
Exception in thread "main"
java.nio.file.InvalidPathException: Illegal char <*> at index 8: externs/*.js
        at sun.nio.fs.WindowsPathParser.normalize(Unknown Source)
        at sun.nio.fs.WindowsPathParser.parse(Unknown Source)
        at sun.nio.fs.WindowsPathParser.parse(Unknown Source)
        at sun.nio.fs.WindowsPath.parse(Unknown Source)
        at sun.nio.fs.WindowsFileSystem.getPath(Unknown Source)
        at sun.nio.fs.AbstractPath.resolve(Unknown Source)
        at com.github.jsdossier.Config$PathSpec.resolve(Config.java:773)
        at com.github.jsdossier.Config$ImmutablePathSetDeserializer.deserialize(Config.java:981)
        at com.github.jsdossier.Config$ConfigMarshaller.deserialize(Config.java:883)
        at com.github.jsdossier.Config$ConfigMarshaller.deserialize(Config.java:814)
        at com.google.gson.internal.bind.TreeTypeAdapter.read(TreeTypeAdapter.java:69)
        at com.google.gson.Gson.fromJson(Gson.java:927)
        at com.google.gson.Gson.fromJson(Gson.java:865)
        at com.github.jsdossier.Config.fromJson(Config.java:610)
        at com.github.jsdossier.Config.fromFlags(Config.java:602)
        at com.github.jsdossier.Main.run(Main.java:96)
        at com.github.jsdossier.Main.main(Main.java:217)

I just tried it in Linux, and while I don't see that problem, I do see a different problem in jsdossier (later than this, I think).

@schmidtk I think this can be closed now jsdossier is removed.

Thanks @bradh. We will look into reviving our API docs after we migrate all of our source to ES modules, but that will take some time.


WIP - Modified readme and created some examples.

This is meant to address #14. @thespain - please review. In particular, please have a look a the limitations section. Also, do you have any other domain join scripts besides the two I included?

In addition to fixing what the linter found I also want to write tests that cover the example code to ensure its valid.

These changes have been merged into master.


Need a design doc for onboarding services on VM to ambient

Flat network may already been working, multi-network could have similar solution as #42334. Prob is also outlined in https://github.com/istio/istio/issues/41375
Also need to think through how services on VM can be onboard to ambient, does it still require our existing VM onboarding process by creating service account, bootstrap vm and create WE & SE.

not stale

I think @stevenctl is working on a doc


PRetty printer for -l

Print -l columns in a pretty way

:tada: This issue has been resolved in version 1.1.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Bot response sometimes includes end markers

First off, thanks for your work on this bot. It's awesome to be able to plug ooba into discord.
I've noticed that for certain models, my bot's response sometimes includes EOF style markers. Specifically, ### End of Transcript ### or <|endoftext|> or both.
Is there a way to filter these out?

Thanks for letting me know, esp. with the specific tokens.  A few questions:

when they appear, are they on separate lines, or included on another line?
which models produce them?

I have some hacks in there to filter out other immersion-breaking messages, would be happy to add those.

They are always on their own line. Either as:
### End of Transcript ###<|endoftext|>
or
`<|endoftext|>'
Thank you.

Fixed in https://github.com/chrisrude/oobabot/commit/2389b73ace0886444c487b6b949cbac221671ef1, to be included in next release.
The list of markers can be set in the new config.yml file, in case of future trouble.
It will only look for exact matches, so each marker needs to be on its own line to match.  We might want to loosen this in the future, but for now I'd rather err on the side of not accidentally canceling valid text.

is there anyway to interrupt the API call, when stop marker is reached? the API generates until the end of the generation limit currently

We can stop reading from the input whenever we want.  I find having a shorter gen limit makes it generate more realistic responses.
The diff above stops reading any further data when it notices a stop marker.


Question min max value

Hello have a question,in library u didn't implement for some min value of stepper? i should make custom then?
//**
I made custom class extending your SG thx a lot for library :)

Hi @matijagorsek ,
Could you share your custom class ?
Thanks a lot !

https://pastebin.com/y07BzGBC here you go @Yaninho


[S3] Skip using temp files/folders when writing to S3

S3 supports an eventual consistency model, which does not work like a regular file system. This means that many operations currently in the s3 file system crash now and then because the conditions it expects are not satisfied by S3. The following changes tries to address such crashes.

Skip usage of temp file when writing to S3:
When writing a file to S3, the file system currently writes a temp file locally and syncs at the end by creating a file on S3. This coupled with the fact that S3 provides atomic uploads of files means that we will never see incomplete data uploaded for a file. In this scenario, the current usage of temp files when writing anything to the s3 file system (such as for Checkpoints, SavedModels) are unnecessary. I've removed such usage of temp files for S3 as they not just are unnecessary but cause issues with S3's consistency model. (It may so happen that after writing to a temp file, the file is not visible while moving to final location because changes haven't propagated yet. )

Skip usage of temp directory when saving Checkpoints and SavedModels to When saving Checkpoints and SavedModels, there is also the use of a temp directory where all files are uploaded and then moved to the final location. This is even more problematic given S3's eventual consistency model. This sequence of operations involves the writing to temp location, listing of files in temp location, copying each file from this temp location to final location, and deleting the temp location. There are multiple issues users have experienced because of this sequence. When listing files in temp directory, in some cases not all files show up in the list. This causes some files to not be copied. While deleting directory, there is another listing of all files in the directory. Again this list may be inconsistent and the deleting of directory fails. So the usage of such temp directory has been bypassed for S3 file system.

The above two changes have been implemented by introducing a NeedsTempLocation method for the filesystem and exposing it to python through gfile similar to how IsDirectory works. This method needs to be used multiple places both in python and CPP. By default for any file system other than S3, it returns False, which allows us to skip the usage of temp locations. The above issues for Checkpointing and SavedModels have been addressed and verified for both code paths of Estimator based training, Session based training as well as Keras.

@karmel I've updated my code. Please review again. Thanks!

Can someone please help me with this test failure? I don't understand what's going on in the test, and how my changes are related to the failure.
https://source.cloud.google.com/results/invocations/df79ab6c-104d-41be-9fb8-1a3402851511/targets/%2F%2Ftensorflow%2Fpython%2Fsaved_model:save_test/log

It's because control flow (tf.cond) creates reference cycles and we currently have a check that tf.saved_model.save doesn't create reference cycles. Unfortunate, but the easy thing is just to delete this test: https://github.com/tensorflow/tensorflow/blob/8aa4fe59eb98147609dc690c1244dde315312cd4/tensorflow/python/saved_model/save_test.py#L675

Thanks @allenlavoie

`FAIL: Found 1 non-whitelisted pylint errors:
tensorflow/python/saved_model/save_test.py:22: [W0611(unused-import), ] Unused import sys
=== Sanity check step 3 of 15: do_check_futures_test (Check that python files have certain future imports) ===
=== Sanity check step 4 of 15: do_buildifier (buildifier check) ===
Running do_buildifier on 413 files
buildifier took 1 s
PASS: No buildifier errors or warnings were found
=== Sanity check step 5 of 15: do_bazel_nobuild (bazel nobuild) ===`

can you please fix above error ?

@rthadur Just fixed it. Please rerun the build.

@rthadur Sorry, looks like I missed these two files somehow. Just fixed them. Please rerun the build.

Sanity build failure is unrelated. Will retrigger a new build once it gets fixed.

@mihaimaruseac Have this and the llvm build issue been fixed now?

I think so, it just needs to pass internal tests now.

@MihailSalnikov It looks like the test which failed was rolledback yesterday, as seen from the commits here https://github.com/tensorflow/tensorflow/commits/master/tensorflow/python/keras/layers/preprocessing/index_lookup_test.py.
I've rebased it on current master. Could you please retry the build again?

Here is the internal error ,
Traceback (most recent call last):   File "/build/work/1cf1065dd0f8aeb94d1dc40db4f35efc3c7d/google3/runfiles/google3/third_party/py/absl/third_party/unittest3_backport/case.py", line 37, in testPartExecutor     yield   File "/build/work/1cf1065dd0f8aeb94d1dc40db4f35efc3c7d/google3/runfiles/google3/third_party/py/absl/third_party/unittest3_backport/case.py", line 162, in run     testMethod()   File "/build/work/1cf1065dd0f8aeb94d1dc40db4f35efc3c7d/google3/runfiles/google3/learning/brain/contrib/hub/tools/make_module/make_module_test.py", line 58, in testKeepTags     module_path = self._make_module(model_path, config_proto)   File "/build/work/1cf1065dd0f8aeb94d1dc40db4f35efc3c7d/google3/runfiles/google3/learning/brain/contrib/hub/tools/make_module/make_module_test.py", line 43, in _make_module     make_module.make_module(meta_graphs, checkpoint_path, export_path, config)   File "/build/work/1cf1065dd0f8aeb94d1dc40db4f35efc3c7d/google3/runfiles/google3/learning/brain/contrib/hub/tools/make_module/make_module.py", line 52, in make_module     _modify_collections(meta_graph, meta_graph_config)   File "/build/work/1cf1065dd0f8aeb94d1dc40db4f35efc3c7d/google3/runfiles/google3/learning/brain/contrib/hub/tools/make_module/make_module.py", line 97, in _modify_collections     "the config." % list(undef_set)) RuntimeError: Unspecified actions for collections: [u'cond_context']. Please add them to keep_collection or drop_collection field in the config.

@rthadur could it be a failure due to something other than this PR? I don't see my changes being relevant to that.
Also, the two windows failures are failures I saw in different PRs unrelated to these changes. I rebased changes from master today. Can we retry?

This fails internally:
  File ".../third_party/tensorflow/python/lib/io/file_io_test.py", line 620, in testHasAtomicMove
    self.assertFalse(file_io.has_atomic_move("s3://x/y"))
  File "<embedded stdlib>/unittest/case.py", line 676, in assertFalse
    raise self.failureException(msg)
AssertionError: True is not false

Ref: https://github.com/tensorflow/tensorflow/pull/36388#discussion_r382307430
Is there anything else different internally that might be relevant for this PR?

Maybe we just don't have S3 registered, so it's falling back to the default? In which case maybe we should skip the test if s3 is not registered. Or if that's only a google thing, we could skip that one internally somehow.

I think it's because internally s3 does not exist. I'd suggest not having the test at all.

Ah that makes sense.
Okay removing the test since there's some s3 tests in cpp. I'll just leave the non s3 one in python just to make sure the python API works.

Looks merged in https://github.com/tensorflow/tensorflow/commit/eacf534690aa2c582e41556b4f92b4adde1d6c8d
There were some minor lint changes to appease tooling. But let me know if anything is objectionable and we can follow up. Thank you for your patience.

Thanks @allenlavoie and @mihaimaruseac for your support!


⚠ Bofrost.at has degraded performance

In 107f64e, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 966 ms

Resolved: Bofrost.at performance has improved in 44d4f35.


[fix] remove .interpreted-text role

We are using markdown now instead of pure rst, no need for specific role.

This close #108


RFC-0015 Supply Chain selection with traits

I think it makes sense to start with characteristics approach. I heard others in the community meeting that agreed.

The features option sounds a bit like the Switch through fall-back strategy in RFC 9, but falling back on whole supply-chain matches rather than on template matches.

This may be my history in web development speaking, but I lean hard in favor of the second option. Feature detection is almost always preferable to manual special casing. Doubly so when combined with progressive enhancement. If the Workload specifies source build it; if it specifies an image, skip the build; if it specifies a test pipeline, run it.


Reserve capacity before extending Punctuated

While extending Punctuated from an iterator, use size_hint to reserve capacity before pushing to avoid repeated allocations.
I did also avoid unneeded allocations of a new Box.
There are also small optimizations like removing the check of nomore in the while loop and only checking if we did receive an End.
I did mention in a TODO comment that the behavior of extend from an iterator of pairs is not consistent with the behavior of extend from an iterator of items. You can also see it in the added test. I would recommend not adding a punctuation if the iterator is empty, but since that could be a braking change, I did not do it.
This is my first contribution and I did not find a CONTRIBUTING file. Therefore, please tell me if something is odd :)
Sorry for the typo in the branch name :/

I have done the following benchmark:
cargo b -r && hyperfine -N "BINARY":
for _ in 0..10_000 {
    std::hint::black_box(Punctuated::<u64, syn::token::Comma>::from_iter(0..4096));
    std::hint::black_box(Punctuated::<u64, syn::token::Comma>::from_iter(
        (0..4096).map(|i| syn::punctuated::Pair::Punctuated(i, Default::default())),
    ));
}

Results:
master branch:
Time (mean ± σ):     630.4 ms ±   9.4 ms    [User: 628.7 ms, System: 0.7 ms]
Range (min … max):   617.5 ms … 642.1 ms    10 runs

This branch:
Time (mean ± σ):     257.0 ms ±   4.3 ms    [User: 256.0 ms, System: 0.5 ms]
Range (min … max):   253.3 ms … 265.7 ms    11 runs

The speed up for this benchmark is about 2.5x :D

I think that the assertions in push_punct and push_value should be debug_assert! instead of assert!. People using them are opting in for better performance, otherwise they would just use push.

While waiting for a review, I did just see that I could recycle Box in push too!
Here is a benchmark:
for _ in 0..1_000_000 {
    let mut p: Punctuated<_, syn::token::Comma> = Punctuated::new();
    for i in 0..16 {
        p.push(i);
    }
}

master branch:
Time (mean ± σ):     240.9 ms ±   1.7 ms    [User: 240.0 ms, System: 0.4 ms]
Range (min … max):   239.2 ms … 245.3 ms    12 runs

This branch:
Time (mean ± σ):     124.2 ms ±   2.2 ms    [User: 123.5 ms, System: 0.4 ms]
Range (min … max):   121.1 ms … 127.1 ms    23 runs

Speedup about 1.9x

Actually, why is last boxed in the first place?

@dtolnay Hey, is there something wrong with the PR?
I know that it ended up a bit messy. I could create a new one, but I need some feedback first :)

@dtolnay I did read some of syn's source code while working on typed-builder and did find out that it can be optimized to avoid allocations.
I did just remove some of unsafe to reduce the complexity. I did declare that function unsafe because I wanted to use unwrap_unchecked, but this is not compatible with the current MSRV.
Currently, unsafe is only used once to avoid the comparison of the length and capacity before pushing because we do the allocation. I can remove it if you prefer that and just use Vec::push.
More complexity could be avoided if last is not boxed. Could you please explain to me why it is boxed? Probably to avoid having huge objects in the stack, but is this really the case here?
I think that 2.5x speedup is worth it, but this is only my opinion.
I wanted to add reserve and with_capacity later to further optimize pushing. But that would be another PR.

As a heavy user of typed-builder and also the author of fp-bindgen, both of which depend heavily on syn, I am very much interested in work that speeds up compilation time of syn-using macros.
Not sure how much this PR is going to help on our overall build times of course, but I might give it a shot soon to see if this PR improves build times for us.

I tried recompiling our crates that heavily use both typed-builder and fp-bindgen, both with syn 2.0.20 and with this branch, but I have to admit the results appear to be a wash. Sometimes one was faster, sometimes the other. In any case they were within margin of error from each other.

As you can probably tell, I am not surprised to hear that. A couple dozen memory allocations max per macro call is not a perceptible amount. If I had to guess, you are looking at an effect on the order of 0.0001% (1 microsecond per second spent on compilation). Any positive effect is reversed by 2 orders of magnitude by putting this amount of extra code into syn. The number of extra memory allocations from this new code during compiling syn would be 2 orders of magnitude bigger than it could possibly save downstream.

@dtolnay Alright, thank you a lot for this explanation.
I am still interested in knowing why last is boxed, just out of curiosity.
Would it be fine for you if I would at least add with_capacity and reserve? They don't make the code more complex, just 2 simple additional methods. I am fine with it if you say that you don't want it.


spring.cloud.kubernetes.reload.monitoring-secrets not working

When enabling monitoring-secrets within the bootstrap.yaml the changes to secrets doesn't get reflected in the spring boot app. Here is a link to a repository trying to utilize it.

Having the same problem. Any workaround?

Having the same problem. Any workaround?

I have yet to find a solution.

Can you try with the latest Hoxton RELEASE (SR3)?

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.


Ember 2.3 deprecations

Bunch of changes here... almost entirely around the fact that the container APIs are deprecated... should refer to owner only from here on out.
Got all the tests passing. Let me know if you have questions about this stuff.

LGTM


Poor results with tensorflow DNNClassifier and cross_val_score

I am using python 3.5, tensorflow 0.10 and its DNNClassifier. If I perform a single training and testing stage, as below, the test result is decent: accuracy = 0.9333
import tensorflow as tf
from tensorflow.contrib import learn
from sklearn.cross_validation import cross_val_score, ShuffleSplit, train_test_split
from sklearn.metrics import accuracy_score
import numpy as np
from sklearn.metrics import accuracy_score
from sklearn import datasets, cross_validation
iris = datasets.load_iris()
feature_columns = learn.infer_real_valued_columns_from_input(iris.data)
x_train, x_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.20, random_state = 20)
model = learn.DNNClassifier(hidden_units=[5],
n_classes=3,
feature_columns=feature_columns,
)
model.fit(x_train, y_train, steps=1000)
predicted = model.predict(x_test)
print('Accuracy on test set: %f' % accuracy_score(y_test, predicted))
If I use sklearn's cross_val_score, then the final results is much poorer, about 0.33 accuracy:
model = learn.DNNClassifier(hidden_units=[5],
n_classes=3,
feature_columns=feature_columns,
)
scores = cross_val_score(estimator=model,
X=iris.data,
y=iris.target,
scoring = 'accuracy',
cv=5,
fit_params={'steps': 1000},
)
print(scores)
print(np.mean(scores))
The scores ad their mean are:
[ 0.          0.33333333  1.          0.33333333  0.        ]
0.333333333333
What's wrong with my code in cross validation estimation?

This is a general usage question more suitable for StackOverflow. Please can you re-ask it there.

Thanks for suggestion, done.


Group config bug?

I just created a new first group. The groups.yaml file in user/config has permission 644, it should be 664. The latter is the standard for all Grav files. The reason is that I tend to set myself as the owner of Grav files, and have the unix group set to www-data so that the Appache/NGINX software can also alter it.
The setup where I noticed this is a Synology NAS. It is a linux based, but proprietary, OS. It may be an OS issue that the file permission has to be set on creation.
I am filing another issue regarding the setting of user passwords. I am not sure if these problems are related.

Closed due to inactivity


Update lift and shift branch dependencies

This PR merges all security dependency updates made as part of https://github.com/epimorphics/hmlr-linked-data/issues/127 into the lift and shift branch

I will close this and resolve conflicts on a separate branch, resolving them in dev will break the legacy app


Market Resolution API

Resolves #559
Notes:

I added additional validation to resolve-market to prevent some invalid states, like resolving to a FR outcome that doesn't exist, or resolving MULTI with percentages that don't add up to 100%.
Currently, we're using contractId for market resolution. Eventually, we should move to slug.

I just went ahead and refactored it so that I liked it better. Basically, try to take advantage of Typescript more -- if you correctly use type narrowing then you shouldn't have to be doing stuff like checking 'resolutions in freeResponseParams` all over the place.
Anyway this seems to work great so I am merging and deploying.