Evaluate quality of TensorFlow model

[Maxim Abalenkov]

Dear all,

I hope all is well with you. I'm looking for help evaluating a custom object detection model based on EfficientDet D1 640x640 from the TensorFlow 2 Detection Model Zoo. I posted my question on StackOverflow (https://tinyurl.com/n4k3azyv), but so far there are no replies. If possible, would you please help me or direct me to another (more appropriate) venue?

In short, I would like to train and evaluate a neural network using a pre-trained TensorFlow2 model called "efficientdet_d1_coco17_tpu-32". It is a sandbox example aimed at object detection. The training data set consists of 130 images and testing dataset has 15. Currently I split the work into two phases: (i) training and (ii) evaluation. To train I use the following command:

python ./model_main_tf2.py \

    --pipeline_config_path=${PIPELINE_CONFIG_PATH} \

    --model_dir=${MODEL_DIR} \

    --num_train_steps=${NUM_TRAIN_STEPS} \

    --num_eval_steps=${NUM_EVAL_STEPS} \

    --sample_1_of_n_eval_examples=${SAMPLE_1_OF_N_EVAL_EXAMPLES} \

    --alsologtostderr

Once the training is complete (ran for 25000 iterations) I launch the evaluation with a similar command:

python ./model_main_tf2.py \

    --pipeline_config_path=${PIPELINE_CONFIG_PATH} \

    --model_dir=${MODEL_DIR} \

    --checkpoint_dir=${CHECKPOINT_DIR} \

    --alsologtostderr

However, when I view the training and evaluation results on TensorBoard I see only one evaluation point on my graphs. Would you please direct me, how to make an evaluation plot based on multiple points? I assumed it was the --num_eval_steps parameter, but apparently it's not that. Any guidance would be appreciated.

According to a similar question evaluation uses the data from the final checkpoint. Is it possible to instruct TensorFlow/TensorBoard to use all the checkpoints available? Also I noticed that the checkpoint files are gradually deleted as the training progresses. Would it be possible to preserve all of them?

--

Best wishes,

Maxim