Hello Adrita,
Using C++ might indeed improve classification performance, but I wouldn't recommend doing it yet if you are a beginner with Caffe.
As a starting point, try compiling caffe with OpenBLAS as the BLAS library. Ideally, you should compile OpenBLAS from sources, to get the latest optimizations.