Hi Saugat,
Caffe installed with 'brew' or similar commands is very unlikely to work, because these pre-compiled versions of Caffe are outdated and not compatible with current systems. Caffe unfortunately no longer receives real support, hence you have to adapt your system and compile it yourself.
Do you have a Linux-based operating system? Else I won't be able to help you much.
Regards,
Tamas