TensorFlow 2 Object Detection API Model Evaluation

Question

I am using TensorFlow 2.x object detection API. I have trained a deep learning model from the model zoo on my dataset. I am using Google Colab. After training now I want to evaluate my model. I am using coco detection metrics. I used the following script to evaluate my model,

!python3 model_main_tf2.py \ 
    --model_dir = path/to/model directory \
    --pipeline_config_path = path/to/pipeline config file \
    --checkpoint_dir = path/to/checkpoint directory

After running the above code I get the mean average precision (mAP) and average recall (AR) for the latest checkpoint on my test set. But for academic purposes, I want to get these metrics on all the checkpoints to get a graph of how my model has improved over time. Is there a possible way to that? or is it possible to train and evaluate at the same time in TensorFlow 2 object detection API? I am a beginner in this field so kindly help me out with this issue. Thank you.

Thanks, @TFer for your help. I visited this website you mentioned, but the script they have used for evaluation is the same as I have mentioned in my question, and they haven't mentioned how a model can be evaluated at all checkpoints or how training and evaluation can be done in parallel. — Zafar Iqbal
– Zafar Iqbal, Commented May 16, 2021 at 19:26

Dennis Chicaiza · Accepted Answer · 2022-06-22 17:18:36Z

2

I am facing the same problem. So I had an idea. We can run the model_main_tf2.py you mentioned to eval the model but changing the current checkpoint (first line) to evaluate in the checkpoint file

model_checkpoint_path: "ckpt-1"
then
model_checkpoint_path: "ckpt-2"
then
model_checkpoint_path: "ckpt-3"
. . .

For each checkpoint you will get a .tfevent so then you open TensorBoard pointing to the directory that contains all the .tfevent and you can see how the model improves over time.

I just saved the last 3 checkpoints in my computer so I can't see the progress from the beginning (my fault) but if you have all the checkpoints try to do what I suggest.

See my graph evaluating the last 3 checkpoints.

answered Jun 22, 2022 at 17:18

Dennis Chicaiza

317 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user2685079 Over a year ago

Moreover, that is the only line that is required. I handled a similar setup my creating checkpoint file with model_checkpoint_path: ckpt_{i} in the eval script

Buur · Accepted Answer · 2022-04-01 06:28:15Z

0

You should have an eval directory including an events.out.tfevents file under your model directory. You can run !tensorboard --logdir=path/to/eval/directory to access the graphs. You can run training with the same snipped you have except without the checkpoint_dirand can open another terminal to run evaluation like you're currently doing.

answered Apr 1, 2022 at 6:28

Buur

13 bronze badges

Collectives™ on Stack Overflow

TensorFlow 2 Object Detection API Model Evaluation

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related