Object detection - background labeling

[Pablo Ramón Soria]

I am training a SSD network for detecting tools. I achieved an acceptable performance with a dataset of 700 images but it has some false positives in random places that should be background. 

I am planning to extend the dataset to try to improve the performance but I have a doubt.  By now all the images I used for the training contained at least one object of the 3 categories I have. The question is, is it good to feed the network with images without any instance of object in there?. I assume that the SSD-framework will consider the whole region as background. That would make it easy to extend the dataset with just background images taking random pictures from internet. 

Thanks in advance.