faster R-CNN training loss was small but performance very poor

Hi, I am training an faster R-CNN detector on my own dataset, the training loss can below 0.1 when training come to an end. But APs on test dataset are about 0.5, and them seems to be worser when I add epoch. I am wondering what's wrong? And when should I stop training? The training dataset has 3500 images and 5 classes.

[Madhav Thakker]

Do the images in test set follow the same distribution in training set? It is unlikely that the model trained on some images would work well on different distribution of images.

If the images are of similar distribution, check that you are not overfitting to the training set.