You probably need to set
https://github.com/openresty/lua-nginx-module#lua_ssl_trusted_certificate to your system CA bundle (or the specific root CA certs for your target server).
You might need to tweak the lua_ssl_verify_depth option too.
Otherwise you are calling ssl_handshake() yourself and just need to set the option there.
It would help if you posted a full example of how you're using resty-http though.
Hamish