Can't restore previous models

I installed Tensoflow from source. I can't restore previous models saved with precompiled Tensorflow (installed via pip). I get lots of these:
Caused by op u'save/RestoreV2_56', defined at:
  File "grid-blstm.py", line 144, in <module>
    loadModel(session, './'+saveDir+'/')
  File "/home/user/Testing/DeepLearning/debug/grid-blstm/utils.py", line 267, in loadModel
    saver = tf.train.Saver()
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1067, in __init__
    self.build()
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1097, in build
    restore_sequentially=self._restore_sequentially)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 691, in build
    restore_sequentially, reshape)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 247, in restore_op
    [spec.tensor.dtype])[0])
  File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 669, in restore_v2
    dtypes=dtypes, name=name)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
    op_def=op_def)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2402, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1264, in __init__
    self._traceback = _extract_stack()

NotFoundError (see above for traceback): Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/GridRNNCell/recurrent/cell_0/lstm_cell/w_i_diag not found in checkpoint.

What if you use metagraph to load/save models? I expect Saver approach to not work between TensorFlow version changes because node names can change

I use the following functions to load and save models:
def loadModel(session, path):
    saver = tf.train.Saver()
    ckpt = tf.train.get_checkpoint_state(path)
    if ckpt and ckpt.model_checkpoint_path:
        saver.restore(session, ckpt.model_checkpoint_path)
        print('Checkpoint restored')
    else:
        print('No checkpoint found')
        exit()

def saveModel(session, filename, epoch):
    saver = tf.train.Saver()
    saver.save(session, filename, global_step=epoch+1)

Meta files are created already.  Thanks.

Don't remember off the top of my head, but there's metagraph howto somewhere

@petewarden or @concretevitamin, could you give a pointer.

Automatically closing due to lack of recent activity. Since this issue is old at this point, please reopen the issue if it still occurs when tried with the latest version of Tensorflow. Thank you.

Any idea how to properly import and export my trained model? Back then, I kept using Tensorflow 1.0. But now I upgraded to 1.3 and I'm facing this issue. Is it possible to fix my previously saved models in order for them to work on version 1.3? Thanks.

Can you recap a few points:

Are you exporting and importing with TF 1.3, or between 1.0 and 1.3?
What's the code you're using to export and import models, and the error?

(I realize you already went over some of this, but just wanna make sure we have all the up-to-date info.)

Thanks for your reply.
1- On version 1.3, I'm trying to import models exported with version 1.0.
2- I'm using the following code:
def loadModel(session, path):
    saver = tf.train.Saver()
    ckpt = tf.train.get_checkpoint_state(path)

    if ckpt and ckpt.model_checkpoint_path:
        saver.restore(session, ckpt.model_checkpoint_path)
        print('Checkpoint restored')
    else:
        print('No checkpoint found')
        exit()

def saveModel(session, filename, epoch):
    saver = tf.train.Saver()
    saver.save(session, filename, global_step=epoch+1)

When I try to load the model it gives an error that some keys are not found in the checkpoint.
The code below does not give an error, but it does not load the weights (because I get high loss, as if I'm training my first epoch).
    saver = tf.train.import_meta_graph('./model/' + modelName + '-' + str(startingEpoch) + '.meta')
    saver.restore(session, tf.train.latest_checkpoint('./model'))

Can you provide the exact error message? This will help locate where the problem is happening. Thanks.

Sure
2017-09-28 01:54:29.029070: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
2017-09-28 01:54:29.029086: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam_1 not found in checkpoint
2017-09-28 01:54:29.029071: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/kernel/Adam_1 not found in checkpoint
2017-09-28 01:54:29.029103: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/w_o_diag/Adam_1 not found in checkpoint
2017-09-28 01:54:29.029187: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias not found in checkpoint
2017-09-28 01:54:29.029070: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/kernel not found in checkpoint
2017-09-28 01:54:29.029355: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/w_f_diag not found in checkpoint
2017-09-28 01:54:29.029441: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/kernel/Adam not found in checkpoint
2017-09-28 01:54:29.029827: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/w_f_diag/Adam_1 not found in checkpoint
2017-09-28 01:54:29.029968: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/w_f_diag/Adam not found in checkpoint
2017-09-28 01:54:29.030036: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/w_i_diag not found in checkpoint
2017-09-28 01:54:29.030168: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/w_o_diag/Adam not found in checkpoint
2017-09-28 01:54:29.030347: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/w_i_diag/Adam not found in checkpoint
2017-09-28 01:54:29.030549: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/w_o_diag/Adam not found in checkpoint
2017-09-28 01:54:29.030658: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/w_i_diag/Adam_1 not found in checkpoint
2017-09-28 01:54:29.030878: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/w_o_diag not found in checkpoint
2017-09-28 01:54:29.031069: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/w_o_diag/Adam_1 not found in checkpoint
2017-09-28 01:54:29.031205: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/w_i_diag/Adam_1 not found in checkpoint
2017-09-28 01:54:29.031220: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.031230: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.031316: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.031405: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.031770: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/w_o_diag not found in checkpoint
2017-09-28 01:54:29.031814: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.031966: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_10/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.032042: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_1/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.032045: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/w_i_diag/Adam not found in checkpoint
2017-09-28 01:54:29.032208: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_1/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.032411: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_2/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.032535: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_10/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.032620: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_2/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.032715: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_11/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.032789: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_3/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.032897: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_3/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.033057: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_4/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.033299: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_5/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.033319: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_4/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.033423: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_5/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.033608: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_12/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.033631: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/w_i_diag not found in checkpoint
2017-09-28 01:54:29.033718: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_6/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.033745: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_11/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.033776: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_12/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.034128: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_7/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.034171: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_6/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.034232: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_8/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.034384: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_9/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.034511: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_7/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.034698: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/w_f_diag/Adam_1 not found in checkpoint
2017-09-28 01:54:29.034757: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.034875: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.034965: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.035100: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_8/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.035174: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_9/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-09-28 01:54:29.035223: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/w_f_diag/Adam not found in checkpoint
2017-09-28 01:54:29.035304: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.035457: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.035502: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.035847: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/w_f_diag not found in checkpoint
2017-09-28 01:54:29.036123: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.036185: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.036291: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.036334: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.036373: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.036409: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.036450: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.036630: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/kernel/Adam_1 not found in checkpoint
2017-09-28 01:54:29.037031: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.037076: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.037109: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.037242: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.037413: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.037599: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.037619: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/kernel/Adam not found in checkpoint
2017-09-28 01:54:29.037672: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.037781: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.037830: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.038099: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.038155: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.038198: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.038261: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.038566: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.038587: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.038596: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/kernel not found in checkpoint
2017-09-28 01:54:29.038717: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.038884: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.038913: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.039043: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.039149: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.039182: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.039241: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.039341: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/bias/Adam_1 not found in checkpoint
2017-09-28 01:54:29.039428: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.039972: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.040255: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.040302: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.040314: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.040345: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.040457: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/bias/Adam not found in checkpoint
2017-09-28 01:54:29.040500: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.040649: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.040977: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/bias not found in checkpoint
2017-09-28 01:54:29.041094: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.041273: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.041350: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.041550: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.041791: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.041948: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/w_o_diag/Adam_1 not found in checkpoint
2017-09-28 01:54:29.042010: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.042019: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.042132: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.042246: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.042476: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.042577: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.042744: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/w_o_diag/Adam not found in checkpoint
2017-09-28 01:54:29.042873: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.043038: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.043110: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.043244: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.043444: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.043687: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.043882: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.044070: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/w_o_diag not found in checkpoint
2017-09-28 01:54:29.044124: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.044165: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.044182: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.044418: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.044611: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.044781: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.046131: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.046812: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.047403: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.047954: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.047954: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.047967: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.048079: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.048309: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.048642: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/w_i_diag/Adam_1 not found in checkpoint
2017-09-28 01:54:29.048816: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.049102: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.049235: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.049358: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/w_i_diag/Adam not found in checkpoint
2017-09-28 01:54:29.049416: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.050520: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.050715: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.052098: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.057755: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/w_i_diag not found in checkpoint
2017-09-28 01:54:29.058297: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/w_f_diag/Adam_1 not found in checkpoint
2017-09-28 01:54:29.058717: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/w_f_diag/Adam not found in checkpoint
2017-09-28 01:54:29.059062: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/w_f_diag not found in checkpoint
2017-09-28 01:54:29.081236: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/kernel/Adam_1 not found in checkpoint
2017-09-28 01:54:29.081806: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/kernel/Adam not found in checkpoint
2017-09-28 01:54:29.082183: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/kernel not found in checkpoint
2017-09-28 01:54:29.082518: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/bias/Adam_1 not found in checkpoint
2017-09-28 01:54:29.082869: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/bias/Adam not found in checkpoint
2017-09-28 01:54:29.083226: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
2017-09-28 01:54:29.084649: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/w_o_diag/Adam_1 not found in checkpoint
2017-09-28 01:54:29.084987: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/w_o_diag/Adam not found in checkpoint
2017-09-28 01:54:29.085319: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/w_o_diag not found in checkpoint
2017-09-28 01:54:29.085618: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/w_i_diag/Adam_1 not found in checkpoint
2017-09-28 01:54:29.085921: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/w_i_diag/Adam not found in checkpoint
2017-09-28 01:54:29.086207: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/w_i_diag not found in checkpoint
2017-09-28 01:54:29.086494: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/w_f_diag/Adam_1 not found in checkpoint
2017-09-28 01:54:29.086779: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/w_f_diag/Adam not found in checkpoint
2017-09-28 01:54:29.087061: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/w_f_diag not found in checkpoint
2017-09-28 01:54:29.087343: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/kernel/Adam_1 not found in checkpoint
2017-09-28 01:54:29.087655: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/kernel/Adam not found in checkpoint
2017-09-28 01:54:29.087966: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/kernel not found in checkpoint
2017-09-28 01:54:29.088252: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam_1 not found in checkpoint
2017-09-28 01:54:29.088543: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
2017-09-28 01:54:29.088849: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/bias not found in checkpoint
2017-09-28 01:54:29.089145: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/w_o_diag/Adam_1 not found in checkpoint
2017-09-28 01:54:29.089426: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/w_o_diag/Adam not found in checkpoint
2017-09-28 01:54:29.089723: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/w_o_diag not found in checkpoint
2017-09-28 01:54:29.090001: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/w_i_diag/Adam_1 not found in checkpoint
2017-09-28 01:54:29.090306: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/w_i_diag/Adam not found in checkpoint
2017-09-28 01:54:29.090592: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/w_i_diag not found in checkpoint
2017-09-28 01:54:29.090884: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/w_f_diag/Adam_1 not found in checkpoint
2017-09-28 01:54:29.091170: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/w_f_diag/Adam not found in checkpoint
2017-09-28 01:54:29.091452: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/w_f_diag not found in checkpoint
2017-09-28 01:54:29.091790: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/kernel/Adam_1 not found in checkpoint
2017-09-28 01:54:29.092118: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/kernel/Adam not found in checkpoint
2017-09-28 01:54:29.092415: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/kernel not found in checkpoint
2017-09-28 01:54:29.092733: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/bias/Adam_1 not found in checkpoint
2017-09-28 01:54:29.093026: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/bias/Adam not found in checkpoint
2017-09-28 01:54:29.093322: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/bias not found in checkpoint
2017-09-28 01:54:29.093611: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/w_o_diag/Adam_1 not found in checkpoint
2017-09-28 01:54:29.093926: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/w_o_diag/Adam not found in checkpoint
2017-09-28 01:54:29.094215: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/w_o_diag not found in checkpoint
2017-09-28 01:54:29.094510: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/w_i_diag/Adam_1 not found in checkpoint
2017-09-28 01:54:29.096658: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/w_i_diag/Adam not found in checkpoint
2017-09-28 01:54:29.096946: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/w_i_diag not found in checkpoint
2017-09-28 01:54:29.097230: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/w_f_diag/Adam_1 not found in checkpoint
2017-09-28 01:54:29.097588: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/w_f_diag/Adam not found in checkpoint
2017-09-28 01:54:29.097906: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/w_f_diag not found in checkpoint
2017-09-28 01:54:29.098216: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/kernel/Adam_1 not found in checkpoint
2017-09-28 01:54:29.098619: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/kernel/Adam not found in checkpoint
2017-09-28 01:54:29.099400: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/kernel not found in checkpoint
2017-09-28 01:54:29.101016: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias/Adam_1 not found in checkpoint
2017-09-28 01:54:29.101777: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias/Adam not found in checkpoint
2017-09-28 01:54:29.102768: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
2017-09-28 01:54:29.103182: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.103550: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.133696: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.186340: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.190424: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.193140: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.195105: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.198566: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.202022: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.205228: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.205455: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.208992: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.213061: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.216810: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.218523: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.221706: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.224300: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.224887: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.227726: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.231690: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.235380: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.243888: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.246548: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.247331: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.250411: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.271893: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.274740: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.275471: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.285288: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.288315: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.292186: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.312444: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.332352: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.351708: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.352671: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.360767: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.384700: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.408684: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.410683: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.417299: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
2017-09-28 01:54:29.420543: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
Traceback (most recent call last):
  File "base.py", line 259, in <module>
    if startingEpoch != 0: loadModel(session, './'+saveDir+'/')
  File "/home/x454447415244/Testing/DeepLearning/Databases/Rodrigo/Systems/test/util.py", line 122, in loadModel
    saver.restore(session, ckpt.model_checkpoint_path)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1560, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 895, in run
    run_metadata_ptr)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1124, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
    options, run_metadata)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
	 [[Node: save/RestoreV2_39/_405 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_888_save/RestoreV2_39", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

Caused by op u'save/RestoreV2_123', defined at:
  File "base.py", line 259, in <module>
    if startingEpoch != 0: loadModel(session, './'+saveDir+'/')
  File "/home/x454447415244/Testing/DeepLearning/Databases/Rodrigo/Systems/test/util.py", line 118, in loadModel
    saver = tf.train.Saver()
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1140, in __init__
    self.build()
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1172, in build
    filename=self._filename)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 688, in build
    restore_sequentially, reshape)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 247, in restore_op
    [spec.tensor.dtype])[0])
  File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 663, in restore_v2
    dtypes=dtypes, name=name)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1204, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

NotFoundError (see above for traceback): Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias/Adam not found in checkpoint
	 [[Node: save/RestoreV2_123 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_123/tensor_names, save/RestoreV2_123/shape_and_slices)]]
	 [[Node: save/RestoreV2_39/_405 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_888_save/RestoreV2_39", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

Hi, sorry for the delay. I thought I'd have time to look at this but apparently I do not! @martinwicke do you know who can look this? (This = a checkpoint from 1.0 can't be loaded from 1.3, for more details start reading here: https://github.com/tensorflow/tensorflow/issues/7490#issuecomment-332373089)

This is not super surprising -- I think the bidirectional rnn code that was used to write this checkpoint was in contrib at 1.0, and the internal structure of that code has changed since, changing some variable names, and thus making the checkpoint unreadable.
@ebrevdo can you confirm or refute this? The only thing that makes me wonder about the errors is that it's complaining about Adam state, but I still blame a scope name change for now.

We have a tool to help rewrite old checkpoints here.  The top of the script has instructions on how to convert your old RNN checkpoints to a format that will work with TF 1.3+.

Let me know if this does not work. I'll close the issue for now.

@ebrevdo @martinwicke I got the following errors after I converted the model using the tool you provided.
2017-10-07 01:31:01.998649: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
2017-10-07 01:31:01.999185: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/w_i_diag not found in checkpoint
2017-10-07 01:31:01.999241: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/kernel not found in checkpoint
2017-10-07 01:31:01.999209: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/w_o_diag not found in checkpoint
2017-10-07 01:31:01.999382: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/w_f_diag not found in checkpoint
2017-10-07 01:31:01.999976: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.000057: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/w_f_diag not found in checkpoint
2017-10-07 01:31:02.000126: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/bias not found in checkpoint
2017-10-07 01:31:02.000174: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/kernel not found in checkpoint
2017-10-07 01:31:02.000347: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.000493: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/bias not found in checkpoint
2017-10-07 01:31:02.000627: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/w_i_diag not found in checkpoint
2017-10-07 01:31:02.000684: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.000958: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/w_o_diag not found in checkpoint
2017-10-07 01:31:02.001080: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/kernel not found in checkpoint
2017-10-07 01:31:02.001225: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.001331: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/w_o_diag not found in checkpoint
2017-10-07 01:31:02.001471: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/w_i_diag not found in checkpoint
2017-10-07 01:31:02.001547: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
2017-10-07 01:31:02.001783: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/w_f_diag not found in checkpoint
2017-10-07 01:31:02.001941: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/w_o_diag not found in checkpoint
2017-10-07 01:31:02.001970: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/kernel not found in checkpoint
2017-10-07 01:31:02.002007: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/w_f_diag not found in checkpoint
2017-10-07 01:31:02.002113: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.002186: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/w_i_diag not found in checkpoint
2017-10-07 01:31:02.002268: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.002501: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/bias not found in checkpoint
2017-10-07 01:31:02.002531: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/w_i_diag not found in checkpoint
2017-10-07 01:31:02.002680: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/w_o_diag not found in checkpoint
2017-10-07 01:31:02.002860: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.002890: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/kernel not found in checkpoint
2017-10-07 01:31:02.002921: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/w_f_diag not found in checkpoint
2017-10-07 01:31:02.003058: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias not found in checkpoint
2017-10-07 01:31:02.003102: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/w_o_diag not found in checkpoint
2017-10-07 01:31:02.003244: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/w_i_diag not found in checkpoint
2017-10-07 01:31:02.003336: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.003448: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.003514: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.003172: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/w_f_diag not found in checkpoint
2017-10-07 01:31:02.003109: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/kernel not found in checkpoint
2017-10-07 01:31:02.003813: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_1/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.003963: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.003974: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_1/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.003992: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.003965: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_2/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.004291: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_4/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.004668: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_4/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.004769: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_2/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.004808: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_5/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.004832: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_5/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.004855: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_3/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.004870: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_3/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.004933: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.005211: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.005292: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_6/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.005501: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.005529: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.005581: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_6/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
Traceback (most recent call last):
  File "logits.py", line 199, in <module>
    loadModel(session, './'+saveDir+'/')
  File "/home/x454447415244/Testing/DeepLearning/Databases/Bozen/WFST/Systems2/16-3L-G-MS-BN/util.py", line 112, in loadModel
    saver.restore(session, ckpt.model_checkpoint_path)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1560, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 895, in run
    run_metadata_ptr)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1124, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
    options, run_metadata)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
	 [[Node: save/RestoreV2_52/_79 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_216_save/RestoreV2_52", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

Caused by op u'save/RestoreV2_10', defined at:
  File "logits.py", line 199, in <module>
    loadModel(session, './'+saveDir+'/')
  File "/home/x454447415244/Testing/DeepLearning/Databases/Bozen/WFST/Systems2/16-3L-G-MS-BN/util.py", line 109, in loadModel
    saver = tf.train.Saver()
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1140, in __init__
    self.build()
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1172, in build
    filename=self._filename)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 688, in build
    restore_sequentially, reshape)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 247, in restore_op
    [spec.tensor.dtype])[0])
  File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 663, in restore_v2
    dtypes=dtypes, name=name)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1204, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

NotFoundError (see above for traceback): Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
	 [[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
	 [[Node: save/RestoreV2_52/_79 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_216_save/RestoreV2_52", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

Can you print the variable names created by your old model code, and the
ones expected from the new model code?  We may not have covered all cases
in the checkpoint rewriter.
On Fri, Oct 6, 2017 at 3:35 PM, Edgard Chammas<EMAIL_ADDRESS>wrote:

@ebrevdo https://github.com/ebrevdo @martinwicke
https://github.com/martinwicke I got the following errors after I
converted the model using the tool you provided.
2017-10-07 01:31:01.998649: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
2017-10-07 01:31:01.999185: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/w_i_diag not found in checkpoint
2017-10-07 01:31:01.999241: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/kernel not found in checkpoint
2017-10-07 01:31:01.999209: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/w_o_diag not found in checkpoint
2017-10-07 01:31:01.999382: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/w_f_diag not found in checkpoint
2017-10-07 01:31:01.999976: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
[[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.000057: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/w_f_diag not found in checkpoint
2017-10-07 01:31:02.000126: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/bias not found in checkpoint
2017-10-07 01:31:02.000174: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/kernel not found in checkpoint
2017-10-07 01:31:02.000347: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
[[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.000493: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/bias not found in checkpoint
2017-10-07 01:31:02.000627: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/w_i_diag not found in checkpoint
2017-10-07 01:31:02.000684: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
[[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.000958: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_1/lstm_cell/w_o_diag not found in checkpoint
2017-10-07 01:31:02.001080: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/kernel not found in checkpoint
2017-10-07 01:31:02.001225: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
[[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.001331: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/w_o_diag not found in checkpoint
2017-10-07 01:31:02.001471: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/w_i_diag not found in checkpoint
2017-10-07 01:31:02.001547: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
2017-10-07 01:31:02.001783: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_2/lstm_cell/w_f_diag not found in checkpoint
2017-10-07 01:31:02.001941: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/w_o_diag not found in checkpoint
2017-10-07 01:31:02.001970: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/kernel not found in checkpoint
2017-10-07 01:31:02.002007: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/w_f_diag not found in checkpoint
2017-10-07 01:31:02.002113: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
[[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.002186: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_0/lstm_cell/w_i_diag not found in checkpoint
2017-10-07 01:31:02.002268: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
[[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.002501: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/bias not found in checkpoint
2017-10-07 01:31:02.002531: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/w_i_diag not found in checkpoint
2017-10-07 01:31:02.002680: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/w_o_diag not found in checkpoint
2017-10-07 01:31:02.002860: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
[[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.002890: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/kernel not found in checkpoint
2017-10-07 01:31:02.002921: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_1/lstm_cell/w_f_diag not found in checkpoint
2017-10-07 01:31:02.003058: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/bias not found in checkpoint
2017-10-07 01:31:02.003102: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/w_o_diag not found in checkpoint
2017-10-07 01:31:02.003244: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/w_i_diag not found in checkpoint
2017-10-07 01:31:02.003336: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
[[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.003448: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.003514: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
[[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.003172: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/w_f_diag not found in checkpoint
2017-10-07 01:31:02.003109: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/fw/multi_rnn_cell/cell_2/lstm_cell/kernel not found in checkpoint
2017-10-07 01:31:02.003813: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_1/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.003963: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
[[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.003974: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_1/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.003992: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.003965: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_2/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.004291: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_4/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.004668: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_4/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.004769: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_2/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.004808: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_5/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.004832: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_5/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.004855: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_3/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.004870: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_3/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.004933: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
[[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.005211: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
[[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.005292: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_6/moments/Squeeze/ExponentialMovingAverage not found in checkpoint
2017-10-07 01:31:02.005501: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
[[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.005529: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
[[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
2017-10-07 01:31:02.005581: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key bn/bn_6/moments/Squeeze_1/ExponentialMovingAverage not found in checkpoint
Traceback (most recent call last):
File "logits.py", line 199, in
loadModel(session, './'+saveDir+'/')
File "/home/x454447415244/Testing/DeepLearning/Databases/Bozen/WFST/Systems2/16-3L-G-MS-BN/util.py", line 112, in loadModel
saver.restore(session, ckpt.model_checkpoint_path)
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1560, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 895, in run
run_metadata_ptr)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1124, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
options, run_metadata)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
[[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
[[Node: save/RestoreV2_52/_79 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_216_save/RestoreV2_52", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]
Caused by op u'save/RestoreV2_10', defined at:
File "logits.py", line 199, in
loadModel(session, './'+saveDir+'/')
File "/home/x454447415244/Testing/DeepLearning/Databases/Bozen/WFST/Systems2/16-3L-G-MS-BN/util.py", line 109, in loadModel
saver = tf.train.Saver()
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1140, in init
self.build()
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1172, in build
filename=self._filename)
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 688, in build
restore_sequentially, reshape)
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps
tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 247, in restore_op
[spec.tensor.dtype])[0])
File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 663, in restore_v2
dtypes=dtypes, name=name)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1204, in init
self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access
NotFoundError (see above for traceback): Key bidirectional_rnn/bw/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint
[[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]
[[Node: save/RestoreV2_52/_79 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_216_save/RestoreV2_52", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/7490#issuecomment-334885816,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABtimz_w_UKAl2HuVIqhiz_fa3wBTiwXks5spqsygaJpZM4MAEN0
.

Does this gives all the variables?
[x454447415244@love OldModel]$ strings model-bozen.ckpt-16.index | sort | uniq
_1/beta
1/LSTMCell/B
2/beta
2/cond/bn_2/moments/moments_1/mean/ExponentialMovingAverage/biased
2/LSTMCell/B
2_power
32.+
3/beta
3/cond/bn_3/moments/moments_1/mean/ExponentialMovingAverage/biased
4/beta
4/cond/bn_4/moments/moments_1/mean/ExponentialMovingAverage/biased
 5 ?&
 5A>
5abC
5/beta
5ca1y
5Hn/{
 5vR8
5vYU
5W6s
 5z%8
65ws
6/beta
6/cond/bn_6/moments/moments_1/mean/ExponentialMovingAverage/biased
8FT2
!/Adam
/Adam
$/Adam
#/Adam
%/Adam
bDT2
beta1_power
BiRNN/BW/MultiRNNCell/Cell0/LSTMCell/B
BiRNN/BW/MultiRNNCell/Cell1/LSTMCell/B
BiRNN/BW/MultiRNNCell/Cell2/LSTMCell/B/Adam
BiRNN/FW/MultiRNNCell/Cell0/LSTMCell/B/Adam_1
BiRNN/FW/MultiRNNCell/Cell1/LSTMCell/W_0
BiRNN/FW/MultiRNNCell/Cell2/LSTMCell/W_0/Adam
bn_1/cond/bn_1/moments/moments_1/mean/ExponentialMovingAverage
bn_3/cond/bn_3/moments/moments_1/mean/ExponentialMovingAverage
bn_5/cond/bn_5/moments/moments_1/mean/ExponentialMovingAverage
bn/bn_1/cond/bn_1/moments/moments_1/mean/ExponentialMovingAverage/biased
bn/bn_5/cond/bn_5/moments/moments_1/mean/ExponentialMovingAverage/biased
cond/bn_2/moments/moments_1/mean/ExponentialMovingAverage
cond/bn_4/moments/moments_1/mean/ExponentialMovingAverage
cond/bn_6/moments/moments_1/mean/ExponentialMovingAverage
cond/bn/moments/moments_1/mean/ExponentialMovingAverage
F_diag
FW/MultiRNNCell/Cell0/LSTMCell/B
gamma
?I/%
I_diag
local_step
n/beta
n/cond/bn/moments/moments_1/mean/ExponentialMovingAverage/biased
O_diag
phase_train
]PXF
!Variable
#Variable_1/Adam_1
%Variable_7/Adam_1
variance/ExponentialMovingAverage
variance/ExponentialMovingAverage/biased
Ve|2
xSX!!
xSXB

You should be able to use this checkpoint inspector
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/tools/inspect_checkpoint.py
.
On Fri, Oct 6, 2017 at 4:18 PM, Edgard Chammas<EMAIL_ADDRESS>wrote:

Does this gives all the variables?
[x454447415244@love OldModel]$ strings model-bozen.ckpt-16.index | sort | uniq
_1/beta
1/LSTMCell/B
2/beta
2/cond/bn_2/moments/moments_1/mean/ExponentialMovingAverage/biased
2/LSTMCell/B
2_power
32.+
3/beta
3/cond/bn_3/moments/moments_1/mean/ExponentialMovingAverage/biased
4/beta
4/cond/bn_4/moments/moments_1/mean/ExponentialMovingAverage/biased
5 ?&
5A>
5abC
5/beta
5ca1y
5Hn/{
5vR8
5vYU
5W6s
5z%8
65ws
6/beta
6/cond/bn_6/moments/moments_1/mean/ExponentialMovingAverage/biased
8FT2
!/Adam
/Adam
$/Adam
#/Adam
%/Adam
bDT2
beta1_power
BiRNN/BW/MultiRNNCell/Cell0/LSTMCell/B
BiRNN/BW/MultiRNNCell/Cell1/LSTMCell/B
BiRNN/BW/MultiRNNCell/Cell2/LSTMCell/B/Adam
BiRNN/FW/MultiRNNCell/Cell0/LSTMCell/B/Adam_1
BiRNN/FW/MultiRNNCell/Cell1/LSTMCell/W_0
BiRNN/FW/MultiRNNCell/Cell2/LSTMCell/W_0/Adam
bn_1/cond/bn_1/moments/moments_1/mean/ExponentialMovingAverage
bn_3/cond/bn_3/moments/moments_1/mean/ExponentialMovingAverage
bn_5/cond/bn_5/moments/moments_1/mean/ExponentialMovingAverage
bn/bn_1/cond/bn_1/moments/moments_1/mean/ExponentialMovingAverage/biased
bn/bn_5/cond/bn_5/moments/moments_1/mean/ExponentialMovingAverage/biased
cond/bn_2/moments/moments_1/mean/ExponentialMovingAverage
cond/bn_4/moments/moments_1/mean/ExponentialMovingAverage
cond/bn_6/moments/moments_1/mean/ExponentialMovingAverage
cond/bn/moments/moments_1/mean/ExponentialMovingAverage
F_diag
FW/MultiRNNCell/Cell0/LSTMCell/B
gamma
?I/%
I_diag
local_step
n/beta
n/cond/bn/moments/moments_1/mean/ExponentialMovingAverage/biased
O_diag
phase_train
]PXF
!Variable
#Variable_1/Adam_1
%Variable_7/Adam_1
variance/ExponentialMovingAverage
variance/ExponentialMovingAverage/biased
Ve|2
xSX!!
xSXB
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/7490#issuecomment-334891305,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABtim69MzMy2g06-yZX-hjHT5Kr-x1rUks5sprVWgaJpZM4MAEN0
.

@ebrevdo Thank you.
On Sun, Oct 8, 2017 at 9:44 AM, Edgard Chammas<EMAIL_ADDRESS>wrote:

@ebrevdo https://github.com/ebrevdo Thank you.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/7490#issuecomment-335019639,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABtim72RrjKKHH_gPgEkJZp-8H2I1EuXks5sqPvngaJpZM4MAEN0
.

Some batch-norm related changes in the variable names:
-bn/bn_1/cond/bn_1/moments/moments_1/mean/ExponentialMovingAverage/biased
(DT_FLOAT) [128]
-bn/bn_1/cond/bn_1/moments/moments_1/mean/ExponentialMovingAverage/local_step
(DT_FLOAT) []
+bn/bn_1/moments/Squeeze/ExponentialMovingAverage (DT_FLOAT) [128]
+bn/bn_1/moments/Squeeze_1/ExponentialMovingAverage (DT_FLOAT) [128]
etc.
Martin, any ideas when we had batchnorm changes that weren't backwards
compatible?
On Sun, Oct 8, 2017 at 10:20 PM, ebrevdo<EMAIL_ADDRESS>wrote:

Looks like there are more problems than just with the RNN cells; for
example, the batch norm parameters are different and even the variable
shapes are different:
-Variable (DT_FLOAT) [3,3,3,64]
-Variable/Adam (DT_FLOAT) [3,3,3,64]
-Variable/Adam_1 (DT_FLOAT) [3,3,3,64]
+Variable (DT_FLOAT) [3,3,1,64]
+Variable/Adam (DT_FLOAT) [3,3,1,64]
+Variable/Adam_1 (DT_FLOAT) [3,3,1,64]
It doesn't look like the parameters correspond to logically the same things
anymore.
On Sun, Oct 8, 2017 at 9:44 AM, Edgard Chammas<EMAIL_ADDRESS>wrote:

@ebrevdo https://github.com/ebrevdo Thank you.
Reply to this email directly, view it on GitHub
<https://github.com/tensorflow/tensorflow/issues/
7490#issuecomment-335019639>,
or mute the thread
<https://github.com/notifications/unsubscribe-
auth/ABtim72RrjKKHH_gPgEkJZp-8H2I1EuXks5sqPvngaJpZM4MAEN0>
.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/7490#issuecomment-335068356,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABtim_jq7Qku_UpsqdlzuBaGt3-4fU10ks5sqa0jgaJpZM4MAEN0
.

I don't recall any such changes. That last one looks particularly weird,
did we eliminate some cond?
I remember a change we did not accept about simplifying the scope names for
moving averages, but again, we did not accept it (and it would have been
too recent to show up here).

Thanks for your reply.
1- On version 1.3, I'm trying to import models exported with version 1.0.
2- I'm using the following code:
def loadModel(session, path):
    saver = tf.train.Saver()
    ckpt = tf.train.get_checkpoint_state(path)

    if ckpt and ckpt.model_checkpoint_path:
        saver.restore(session, ckpt.model_checkpoint_path)
        print('Checkpoint restored')
    else:
        print('No checkpoint found')
        exit()

def saveModel(session, filename, epoch):
    saver = tf.train.Saver()
    saver.save(session, filename, global_step=epoch+1)

When I try to load the model it gives an error that some keys are not found in the checkpoint.
The code below does not give an error, but it does not load the weights (because I get high loss, as if I'm training my first epoch).
saver = tf.train.import_meta_graph('./model/' + modelName + '-' + str(startingEpoch) + '.meta')
saver.restore(session, tf.train.latest_checkpoint('./model'))

I meet the same question with you. Can you tell me how to solve this problem?

@MCCCSunny I didn't bother to solve it. I had to retrain all my models with the new version of Tensorflow.

@MCCCSunny I didn't bother to solve it. I had to retrain all my models with the new version of Tensorflow.

Thanks for your reply. And I meet a question which has bothered me a long time. And I asked it on the stackoverflow (https://stackoverflow.com/questions/56535990/change-the-optimizer-of-the-model-and-restore-the-original-model-from-the-checkp). So can you help me to solve it if you can?  Many thanks!