Issue understanding how to properly stream to Cloud Speech API in C++ with grpc

[balad...@gmail.com]

Hello everyone!

I've written some code in attempt to stream mic audio input from a separate thread to Cloud Speech API in C++ but I keep getting TCP errors.  I believe this is because I am not quite grasping the whole flow of how to use the API as in when to call what and how often.  My thought after looking at the documentation and source code for some time was to call StreamingRecognize, send some initial config data, send audio data as I get it and check for responses from the server along the way, then wait for the final message from the server.  Hopefully someone can tell me the silly thing(s) I am doing wrong!  Oh also to note, I put the path of my key.json in an environment variable.  grpc should be able to pick up on that right?

So here is my implementation:

// grpc (v1.0.x) and protobuf (v3.0.0): Release, x64, VS2015, Runtime Library: Multi-threaded DLL (/MD)

// libs: gpr.lib, grpc_unsecure.lib, grpc++_unsecure.lib, libprotobuf.lib, z.lib

// UE4: 4.12.5

#include "grpc++.h"

#include "google/cloud/speech/v1beta1/cloud_speech.pb.h"

#include "google/cloud/speech/v1beta1/cloud_speech.grpc.pb.h"

#include "protobuf/Includes/google/protobuf/text_format.h"

#include "protobuf/Includes/google/protobuf/arenastring.h"

#define GoogleSpeechAPI google::cloud::speech::v1beta1

bool FNetworkWorker::HandleReadAppend(std::unique_ptr<grpc::ClientReaderWriter<GoogleSpeechAPI::StreamingRecognizeRequest, GoogleSpeechAPI::StreamingRecognizeResponse>>& rw, GoogleSpeechAPI::StreamingRecognizeResponse& responce, FString& strToAppend) {

WLOG("Checking for message from google");

if (rw->Read(&responce)) {

if (responce.has_error()) {

WLOG("grpc error! %i", responce.error().message().c_str());

APlayerCharacter::GetInstance()->Shutdown();

return false;

}

const auto& result = responce.results(responce.results_size());

WLOG("Result: %s", result.SerializeAsString().c_str());

if (result.is_final())

strToAppend += result.alternatives(0).transcript().c_str();

}

return true;

}

uint32 FNetworkWorker::Run() {

GoogleSpeechAPI::RecognitionConfig* recConf = new GoogleSpeechAPI::RecognitionConfig;

GoogleSpeechAPI::StreamingRecognitionConfig* streamConf = new GoogleSpeechAPI::StreamingRecognitionConfig;

GoogleSpeechAPI::StreamingRecognizeRequest request;

GoogleSpeechAPI::StreamingRecognizeResponse responce;

recConf->set_sample_rate(22050);

recConf->set_encoding(GoogleSpeechAPI::RecognitionConfig::LINEAR16);

streamConf->set_allocated_config(recConf);

streamConf->set_single_utterance(true);

request.set_allocated_streaming_config(streamConf);

auto stub = GoogleSpeechAPI::Speech::NewStub(grpc::CreateChannel("speech.googleapis.com:443", grpc::InsecureChannelCredentials()));

grpc::ClientContext context;

auto rw = stub->StreamingRecognize(&context);

if (!rw->Write(request)) {

WLOG("inital write failed!");

std::vector<grpc::string> errors;

request.FindInitializationErrors(&errors);

for (grpc::string error : errors) {

WLOG("error: %s", error.c_str());

}

}

request.clear_streaming_config();

message.Empty(); // message is a string

char dataToSend[1024];

while (dataIsStreaming) {// false when voice recording thread is done 

WLOG("data is streaming");

// check if there is a message from google and append to message if is_final

// With the following "if(!HandleReadAppend)":

//{code_=INTERNAL (13) details_="{\"created\":\"@1473050344.233000000\",\"description\":\"OS Error\",\"file\":\"D:\\grpc\\vsprojects\\..\\src\\core\\lib\\iomgr\\tcp_windows.c\",\"file_line\":320,\"os_error\":\"An established connection was aborted by the software in your host machine. \n\r syscall:"WSASend", "wsa_error":10053 }

// Without:

//"{\"created\":\"@1473050181.891000000\",\"description\":\"End of TCP stream\",\"file\":\"D:\\grpc\\vsprojects\\..\\src\\core\\lib\\iomgr\\tcp_windows.c\",\"file_line\":180,\"grpc_status\":14}"

if(!HandleReadAppend(rw, responce, message)) 

return responce.error().code(); // return the read error

// check if we have voice data from the voice thread for us to send

if (!voiceBufferQueue.empty()) {

WLOG("Sending data");

// get data from voice thread and place it in dataToSend

BufferData<MY_TYPE>& data = voiceBufferQueue.front();

memcpy((void*)dataToSend, data.readptr, 1024);

free(data.readptr);

// attempt to send the data

request.set_audio_content(dataToSend);

if (!rw->Write(request)) {

WLOG("write failed!");

std::vector<grpc::string> errors;

request.FindInitializationErrors(&errors);

for (grpc::string error : errors) {

WLOG("error: %s", error.c_str()); // This has never logged so far.

}

grpc::Status stat = rw->Finish();

if (!stat.ok()) {

WLOG("stat error code: %i", (int32)stat.error_code());

const grpc::string errorMessage = stat.error_message();

WLOG("stat error message: %s", errorMessage.c_str());

APlayerCharacter::GetInstance()->Shutdown();

return (int32)stat.error_code(); // either 13 or 14 depending on whether the "if(!HandleReadAppend)" is commented out or not.

}

}

voiceBufferQueue.pop();

}

FPlatformProcess::Sleep(0.1f);

}

while (responce.endpointer_type() != GoogleSpeechAPI::StreamingRecognizeResponse_EndpointerType::StreamingRecognizeResponse_EndpointerType_END_OF_AUDIO) {

WLOG("Waiting for END_OF_AUDIO");

if(rw->Read(&responce) && !HandleReadAppend(rw, responce, message))

return responce.error().code();

FPlatformProcess::Sleep(0.1f);

}

WLOG("end");

message.ParseIntoArray(FNetworkWorker::Manager->Manager->spokenWords, TEXT(" "), false);

return 0;

}

[Yang Gao]

You are talking to port 443 and using Insecure credentials (clear text). Should you use ssl credentials at least?

[balad...@gmail.com]

When I remove :443 from the url I get the same thing.  I haven't taken the time to try and compile openssl for Windows 64bit using MD as the runtime so that boringssl can work with grpc.  I think you might be correct.  I will attempt that next.

[balad...@gmail.com]

I finally got around to getting openssl working in Unreal and happy.  So now I am using grpc::GoogleDefaultCredentials() and talking over 443.  Now I get error 13 (internal) "Failed to create security connector."  when I have the HandleReadAppend call uncommented and error 14 (unavailable) with nothing in stat.details_ when I have it commented.  I feel like that's progress?

[muzah...@gmail.com]

were u able to make it work with ue4?

[Trevor Berninger]

Unfortunately not in C++. I ended up writing the GoogleSheech stuff in Python and had Unreal communicate with that instead. That was a while ago now though. I did this while it was all still beta. Might work in c++ now but I'm not sure.

--
You received this message because you are subscribed to a topic in the Google Groups "grpc.io" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/grpc-io/YjoUtQk08z8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/041701d2-728d-4c04-ae06-e0829cc571ce%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ned...@gmail.com]

I had somewhat negative experience with Windows C++ grpc to GoogeSpeach myself.

Google Speech c++ set of samples (https://github.com/GoogleCloudPlatform/cpp-docs-samples/tree/master/speech/api) works perfectly well under Ubuntu. But fails under grpc 1.8.0 Windows. Fails in a sense that I never receive the reply and get a DEADLINE_EXCEEDED with I specify a timeout.

Oddly enough, C# version on Windows works just fine.