There are both positives and negatives when it comes to image sizes. If the image is smaller than the recommended size, the processing time and bandwidth usage will decrease but the accuracy of the prediction will be lower. If the image is larger than the recommended size, the processing time and bandwidth usage will increase but the accuracy will be higher.
The recommended sizes provided are the middle ground where they take the best options for processing time, bandwidth usage and accuracy of the prediction. If your image is smaller than the recommended size, it would be best to upscale the image while maintaining the aspect ratio. Similarly, if your image was larger than the recommended size, it would be optimal to downscale the image.
As to whether you should use padding or resizing, adding unnecessary padding will increase processing time and will not increase accuracy therefore it is not recommended, where as resizing (while maintaining original aspect ratio) to meet the documented resolutions is recommended.