This really depends on a lot of factors. OMNeT++ comes with a pre-packaged mingw toolchain, so the best approach is to use openSSL also from the MinGW distribution (mingw has a different ABI than the normal viusal C compiled libraries (however pure C functions should be compatible). In short, you need the header and lib file for the openSSL libarary. You need to place them somewhere in your project and add the include file path and the library path in your makefile or in 'src/makefrags' (probably a better option as other thrid-party libs are also added there. You can check how). In short:
For example NSC is added like this:
INCLUDE_PATH += -I../3rdparty/nsc-$(NSC_VERSION)/sim
LIBS += -Wl,-rpath,$(abspath ../3rdparty/nsc-$(NSC_VERSION))
As a side note, I'm not sure actual cryptography is needed ina simulation library. Why not just simulate the overhead added to the protocol and have a bit signalling, that "this is encrypted data, you cannot read". Unless you want to make network emulation. and communicate with external devices, I see no point of actual encryption.