Hej Guys !
I am trying to run locally the cat example and I got stuck on the trainning step. I get this very long error. Could someone help me out to understand what is wrong ?
Thanks in advance.
The command: bertalan@mbqs:~/tensorflow/models$ python object_detection/train.py --logtostderr --pipeline_config_path=/home/bertalan/tensorflow/models/object_detection/samples/configs/Myfaster_rcnn_resnet101_pets.config --train_dir=TrainCat
Here the output with error:
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Summary name Learning Rate is illegal; using Learning_Rate instead.
INFO:tensorflow:Summary name /clone_loss is illegal; using clone_loss instead.
/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gradients_impl.py:93: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
2017-07-26 11:27:18.416172: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-26 11:27:18.416220: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-26 11:27:18.416248: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-07-26 11:27:20.437921: I tensorflow/core/common_runtime/simple_placer.cc:669] Ignoring device specification /device:GPU:0 for node 'prefetch_queue_Dequeue' because the input edge from 'prefetch_queue' is a reference connection and already has a device field set to /device:CPU:0
INFO:tensorflow:Restoring parameters from /home/bertalan/tensorflow/models/faster_rcnn_inception_resnet_v2_atrous_coco_11_06_2017/model.ckpt
2017-07-26 11:27:25.179612: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv3/BatchNorm/beta not found in checkpoint
2017-07-26 11:27:25.179639: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv2/weights not found in checkpoint
2017-07-26 11:27:25.179673: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv3/BatchNorm/moving_mean not found in checkpoint
2017-07-26 11:27:25.179612: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv3/BatchNorm/gamma not found in checkpoint
2017-07-26 11:27:25.185127: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv3/BatchNorm/moving_variance not found in checkpoint
2017-07-26 11:27:25.187191: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv3/weights not found in checkpoint
2017-07-26 11:27:25.187614: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/gamma not found in checkpoint
2017-07-26 11:27:25.188036: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/beta not found in checkpoint
2017-07-26 11:27:25.188324: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv2/BatchNorm/moving_variance not found in checkpoint
2017-07-26 11:27:25.189131: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/shortcut/weights not found in checkpoint
2017-07-26 11:27:25.190319: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_variance not found in checkpoint
2017-07-26 11:27:25.190613: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/shortcut/BatchNorm/moving_mean not found in checkpoint
2017-07-26 11:27:25.190923: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv2/BatchNorm/moving_mean not found in checkpoint
2017-07-26 11:27:25.191949: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv1/BatchNorm/moving_mean not found in checkpoint
2017-07-26 11:27:25.192728: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv1/BatchNorm/gamma not found in checkpoint
2017-07-26 11:27:25.193354: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_2/bottleneck_v1/conv1/BatchNorm/moving_variance not found in checkpoint
2017-07-26 11:27:25.194102: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key
...
2017-07-26 11:27:25.204869: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/conv1/BatchNorm/gamma not found in checkpoint
2017-07-26 11:27:25.205198: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/conv1/BatchNorm/moving_mean not found in checkpoint
2017-07-26 11:27:25.205799: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv1/BatchNorm/moving_mean not found in checkpoint
2017-07-26 11:27:25.205853: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/conv1/BatchNorm/moving_variance not found in checkpoint
2017-07-26 11:27:25.209234: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/conv2/BatchNorm/beta not found in checkpoint
2017-07-26 11:27:25.210446: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/conv1/weights not found in checkpoint
2017-07-26 11:27:25.210829: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv1/BatchNorm/gamma not found in checkpoint
2017-07-26 11:27:25.212274: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/conv2/BatchNorm/gamma not found in checkpoint
2017-07-26 11:27:25.212305: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_3/bottleneck_v1/conv2/BatchNorm/moving_mean not found in checkpoint Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_20/bottleneck_v1/conv2/BatchNorm/moving_variance not found in checkpoint
2017-07-26 11:27:25.613441: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_20/bottleneck_v1/conv3/BatchNorm/beta not found in checkpoint
2017-07-26 11:27:25.613721: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_20/bottleneck_v1/conv3/BatchNorm/gamma not found in checkpoint
2017-07-26 11:27:25.615790: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_20/bottleneck_v1/conv3/weights not found in checkpoint
2017-07-26 11:27:25.615937: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_20/bottleneck_v1/conv3/BatchNorm/moving_mean not found in checkpoint
2017-07-26 11:27:25.616601: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_20/bottleneck_v1/conv3/BatchNorm/moving_variance not found in checkpoint
2017-07-26 11:27:25.616872: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_21/bottleneck_v1/conv1/BatchNorm/beta not found in checkpoint
2017-07-26 11:27:25.617185: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_22/bottleneck_v1/conv2/BatchNorm/moving_variance not found in checkpoint
2017-07-26 11:27:25.617505: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_21/bottleneck_v1/conv1/BatchNorm/gamma not found in checkpoint
2017-07-26 11:27:25.618701: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_21/bottleneck_v1/conv1/BatchNorm/moving_mean not found in checkpoint
2017-07-26 11:27:25.618781: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_22/bottleneck_v1/conv2/BatchNorm/moving_mean not found in checkpoint
2017-07-26 11:27:25.620022: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_21/bottleneck_v1/conv1/weights not found in checkpoint
2017-07-26 11:27:25.621149: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_21/bottleneck_v1/conv2/BatchNorm/beta not found in checkpoint
2017-07-26 11:27:25.621225: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_22/bottleneck_v1/conv2/BatchNorm/gamma not found in checkpoint
2017-07-26 11:27:25.621225: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_21/bottleneck_v1/conv1/BatchNorm/moving_variance not found in checkpoint
2017-07-26 11:27:25.623092: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_21/bottleneck_v1/conv2/BatchNorm/gamma not found in checkpoint
2017-07-26 11:27:25.624135: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_21/bottleneck_v1/conv2/BatchNorm/moving_variance not found in checkpoint
2017-07-26 11:27:25.627327: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_21/bottleneck_v1/conv2/BatchNorm/moving_mean not found in checkpoint
2017-07-26 11:27:25.627572: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_21/bottleneck_v1/conv2/weights not found in checkpoint
2017-07-26 11:27:25.628414: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_21/bottleneck_v1/conv3/BatchNorm/gamma not found in checkpoint
2017-07-26 11:27:25.628844: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_21/bottleneck_v1/conv3/BatchNorm/moving_mean not found in checkpoint
2017-07-26 11:27:25.629118: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_22/bottleneck_v1/conv2/BatchNorm/beta not found in checkpoint
2017-07-26 11:27:25.629480: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_21/bottleneck_v1/conv3/BatchNorm/beta not found in checkpoint
2017-07-26 11:27:25.629624: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_21/bottleneck_v1/conv3/BatchNorm/moving_variance not found in checkpoint
2017-07-26 11:27:25.630848: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_22/bottleneck_v1/conv1/weights not found in checkpoint
2017-07-26 11:27:25.631122: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_22/bottleneck_v1/conv1/BatchNorm/beta not found in checkpoint
2017-07-26 11:27:25.631167: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_21/bottleneck_v1/conv3/weights not found in checkpoint
2017-07-26 11:27:25.632471: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_22/bottleneck_v1/conv1/BatchNorm/gamma not found in checkpoint
2017-07-26 11:27:25.633056: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_22/bottleneck_v1/conv1/BatchNorm/moving_variance not found in checkpoint
2017-07-26 11:27:25.633295: W tensorflow/core/framework/op_kernel.cc:1152] Not found: Key FirstStageFeatureExtractor/resnet_v1_101/block3/unit_22/bottleneck_v1/conv1/BatchNorm/moving_mean not found in checkpoint
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.NotFoundError'>, Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv3/BatchNorm/beta not found in checkpoint
[[Node: save/RestoreV2_475 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_475/tensor_names, save/RestoreV2_475/shape_and_slices)]]
Caused by op u'save/RestoreV2_475', defined at:
File "object_detection/train.py", line 198, in <module>
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "object_detection/train.py", line 194, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "/home/bertalan/tensorflow/models/object_detection/trainer.py", line 216, in train
from_detection_checkpoint=train_config.from_detection_checkpoint)
File "/home/bertalan/tensorflow/models/object_detection/meta_architectures/faster_rcnn_meta_arch.py", line 1447, in restore_fn
saver = tf.train.Saver(first_stage_variables)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1056, in __init__
self.build()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1086, in build
restore_sequentially=self._restore_sequentially)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 691, in build
restore_sequentially, reshape)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps
tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 247, in restore_op
[spec.tensor.dtype])[0])
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_io_ops.py", line 669, in restore_v2
dtypes=dtypes, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2336, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1228, in __init__
self._traceback = _extract_stack()
NotFoundError (see above for traceback): Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv3/BatchNorm/beta not found in checkpoint
[[Node: save/RestoreV2_475 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_475/tensor_names, save/RestoreV2_475/shape_and_slices)]]
Traceback (most recent call last):
File "object_detection/train.py", line 198, in <module>
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "object_detection/train.py", line 194, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "/home/bertalan/tensorflow/models/object_detection/trainer.py", line 290, in train
saver=saver)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/learning.py", line 725, in train
master, start_standard_services=False, config=session_config) as sess:
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 960, in managed_session
self.stop(close_summary_writer=close_summary_writer)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 788, in stop
stop_grace_period_secs=self._stop_grace_secs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 949, in managed_session
start_standard_services=start_standard_services)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 706, in prepare_or_wait_for_session
init_feed_dict=self._init_feed_dict, init_fn=self._init_fn)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/session_manager.py", line 264, in prepare_session
init_fn(sess)
File "/home/bertalan/tensorflow/models/object_detection/meta_architectures/faster_rcnn_meta_arch.py", line 1450, in restore
saver.restore(sess, checkpoint_path)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1457, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 778, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 982, in _run
feed_dict_string, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1032, in _do_run
target_list, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1052, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv3/BatchNorm/beta not found in checkpoint
[[Node: save/RestoreV2_475 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_475/tensor_names, save/RestoreV2_475/shape_and_slices)]]
Caused by op u'save/RestoreV2_475', defined at:
File "object_detection/train.py", line 198, in <module>
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "object_detection/train.py", line 194, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "/home/bertalan/tensorflow/models/object_detection/trainer.py", line 216, in train
from_detection_checkpoint=train_config.from_detection_checkpoint)
File "/home/bertalan/tensorflow/models/object_detection/meta_architectures/faster_rcnn_meta_arch.py", line 1447, in restore_fn
saver = tf.train.Saver(first_stage_variables)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1056, in __init__
self.build()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1086, in build
restore_sequentially=self._restore_sequentially)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 691, in build
restore_sequentially, reshape)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps
tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 247, in restore_op
[spec.tensor.dtype])[0])
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_io_ops.py", line 669, in restore_v2
dtypes=dtypes, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2336, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1228, in __init__
self._traceback = _extract_stack()
NotFoundError (see above for traceback): Key FirstStageFeatureExtractor/resnet_v1_101/block4/unit_1/bottleneck_v1/conv3/BatchNorm/beta not found in checkpoint
[[Node: save/RestoreV2_475 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_475/tensor_names, save/RestoreV2_475/shape_and_slices)]]
By the way, here is my config file:
# Faster R-CNN with Resnet-101 (v1) configured for the Oxford-IIIT Pet Dataset.
# Users should configure the fine_tune_checkpoint field in the train config as
# well as the label_map_path and input_path fields in the train_input_reader and
# eval_input_reader. Search for "PATH_TO_BE_CONFIGURED" to find the fields that
# should be configured.
model {
faster_rcnn {
num_classes: 37
image_resizer {
keep_aspect_ratio_resizer {
min_dimension: 600
max_dimension: 1024
}
}
feature_extractor {
type: 'faster_rcnn_resnet101'
first_stage_features_stride: 16
}
first_stage_anchor_generator {
grid_anchor_generator {
scales: [0.25, 0.5, 1.0, 2.0]
aspect_ratios: [0.5, 1.0, 2.0]
height_stride: 16
width_stride: 16
}
}
first_stage_box_predictor_conv_hyperparams {
op: CONV
regularizer {
l2_regularizer {
weight: 0.0
}
}
initializer {
truncated_normal_initializer {
stddev: 0.01
}
}
}
first_stage_nms_score_threshold: 0.0
first_stage_nms_iou_threshold: 0.7
first_stage_max_proposals: 300
first_stage_localization_loss_weight: 2.0
first_stage_objectness_loss_weight: 1.0
initial_crop_size: 14
maxpool_kernel_size: 2
maxpool_stride: 2
second_stage_box_predictor {
mask_rcnn_box_predictor {
use_dropout: false
dropout_keep_probability: 1.0
fc_hyperparams {
op: FC
regularizer {
l2_regularizer {
weight: 0.0
}
}
initializer {
variance_scaling_initializer {
factor: 1.0
uniform: true
mode: FAN_AVG
}
}
}
}
}
second_stage_post_processing {
batch_non_max_suppression {
score_threshold: 0.0
iou_threshold: 0.6
max_detections_per_class: 100
max_total_detections: 300
}
score_converter: SOFTMAX
}
second_stage_localization_loss_weight: 2.0
second_stage_classification_loss_weight: 1.0
}
}
train_config: {
batch_size: 1
optimizer {
momentum_optimizer: {
learning_rate: {
manual_step_learning_rate {
initial_learning_rate: 0.0003
schedule {
step: 0
learning_rate: .0003
}
schedule {
step: 900000
learning_rate: .00003
}
schedule {
step: 1200000
learning_rate: .000003
}
}
}
momentum_optimizer_value: 0.9
}
use_moving_average: false
}
gradient_clipping_by_norm: 10.0
fine_tune_checkpoint: "/home/bertalan/tensorflow/models/faster_rcnn_inception_resnet_v2_atrous_coco_11_06_2017/model.ckpt"
from_detection_checkpoint: true
data_augmentation_options {
random_horizontal_flip {
}
}
}
train_input_reader: {
tf_record_input_reader {
input_path: "python /home/bertalan/tensorflow/models/pet_train.record"
}
label_map_path: "/home/bertalan/tensorflow/models/object_detection/data/pet_label_map.pbtxt"
}
eval_config: {
num_examples: 2000
}
eval_input_reader: {
tf_record_input_reader {
input_path: "/home/bertalan/tensorflow/models/pet_val.record"
}
label_map_path: "/home/bertalan/tensorflow/models/object_detection/data/pet_label_map.pbtxt"
shuffle: false
num_readers: 1
}
