0

I am doing an experiment. The following image is an example of the annotation I do. There are 2 classes: 1) sun, 2) moon. Red boundary box labels the moon, and the green boundary box labels the sun. I would like the model to learn that: "if the background is dark blue, it is the moon. If it is light blue, it is the sun"

I intentionally make the boundary box exclude the surrounding (the blue background), so to test whether an algorithm can distinguish the same object as different classes only based on different surroundings.

This would be useful, for example, to detect a toy car vs a real car. Assuming the toy car and real car looks very similar, the object detection algorithm have to be aware of its surrounding.

Do you think popular algorithm such as FRCNN can achieve that? If not, what algorithm is available to solve this problem?

enter image description here

Raymond Pang
  • 103
  • 2

2 Answers2

2

Yes this is actually a common problem with modern object detectors, that detections change with the background, which means that the detector looks beyond the bounding box and into the background, which can have positive or negative impacts.

Take the example below from the paper The Elephant in the room.

Figure from paper The Elephant in the room, showing how detections change with background

You can see in the figure how the cat/zebra detections change with different backgrounds, showing how the context/background affects object detectors.

In your case, you want this to happen, so you can use Faster R-CNN, SSD, or Mask R-CNN, that are known to have this "problem" according to the elephant in the room paper. Note that this is not an exhaustive list, other detectors might show the same issue.

Dr. Snoopy
  • 1,355
  • 7
  • 12
1

Sure, this can happen. If you have toy cars that in your training data were always photographed in toy shops and then during test you provide images of toy cars in a different environment, you may have situations where they will be classified incorrectly. As long as the neural network sees the context around the object during training, it will also capture information about the context along with information about the object. And if the conext is biased towards a specific environment (i.e. a toy shop), that will also affect the decision significantly.

Otniel-Bogdan Mercea
  • 149
  • 5