Speech to Text

[leste...@googlemail.com]

Hi there,

i'm looking for a way to implement Speech to Text in my CodenameOne Project.

And i have no idea how to do it, since i cannot use Intents as sugested S2T Tutorial

any help?
 I am not really familiar with the native Android framework and XML files. This was my reason to use CodenameOne

thank you

[Shai Almog]

Hi,
you can use intents using native code. See this tutorial on writing native code: http://www.codenameone.com/how-do-i---access-native-device-functionality-invoke-native-interfaces.html
You can also look at the Dr. Sbaitso demo: https://code.google.com/p/codenameone/source/browse/trunk/Demos/
It includes the reverse Text To Speech which might be an interesting starting point.

[leste...@gmail.com]

ok i've created the interface und generated the impl file. Filled the one for android with dummy functionality, and changed the isSupported to true.

But the nativelookup returns null.

the package names are "userclasses" without any further hierarchy. i tried adding the build hint android.xapplication = <activity android:name="userclasses.MyNativeStuff" />

but still the nativelookup returns null.

Any help?

[Shai Almog]

Does it return null on the device?
Do not add that build hint!

Did you make the class package private or add a non-default constructor?
In the constructor are you doing something that might trigger an exception?

[leste...@gmail.com]

the returned null was on device.
Stripped from all functionality it worked... so i figured that my implementation is bad.

After another hour of not-seeing-my-mistake i got it. Atleast i know what i did wrong.
to use an Intent as in the tutorial i mentioned i extended my impl class. which could
add a new constructor and messes all the project. then i tried your implementation
in the DrSbaitso project and get stuck on a new problem:

import com.codename1.impl.android.AndroidNativeUtil;

in my codename1 lib there is NO com.impl.android 
what did i miss? where can i get my Activity?

[Shai Almog]

If you import that in the native code it will be available when you build in the server.

[leste...@gmail.com]

Yay this problem is fixed!!

but i got stuck on the next one:
on device
speechrecognizer should be used only from the applications main thread

stackoverflow suggests to get any view and create a new Runnable.
 but how can this(or something better/equal) be done on CodeName1?

[Shai Almog]

Try:

        Activity activity = AndroidNativeUtil.getActivity();
        activity.runOnUiThread(new Runnable() {
            @Override
            public void run() {
                 // your code here
            }
        });

[leste...@gmail.com]

Okay running it on the device lets it make the sounds as if does the speech recognition. but after starting the getTextFromSpeech() Method the app get stuck and doesn't react to anything any more. Should i do a yield after startlistening or do the stoplistening after a thread.sleep() ? how do i get the results, because the sounds of my device suggests that recognition is finished.

package userclasses;

import android.app.Activity;
import android.content.Intent;
import android.os.Bundle;
import android.os.Handler;
import android.os.Looper;
import android.speech.RecognitionListener;
import android.speech.RecognitionService;
import android.speech.RecognizerIntent;
import android.speech.SpeechRecognizer;
import android.util.Log;
import java.util.ArrayList;
import java.util.Locale;
import com.codename1.impl.android.AndroidNativeUtil;
import com.codename1.ui.Dialog;
import com.codename1.ui.Label;
import com.codename1.ui.geom.Rectangle;
import generated.StateMachineBase;

public class MyNativeStuffImpl {

    String speech = "";
    private SpeechRecognizer sr;
    private static final String TAG = "MyStt3Activity";

    class listener implements RecognitionListener {

        public void onReadyForSpeech(Bundle params) {
            Log.d(TAG, "onReadyForSpeech");
        }

        public void onBeginningOfSpeech() {
            Log.d(TAG, "onBeginningOfSpeech");
        }

        public void onRmsChanged(float rmsdB) {
            Log.d(TAG, "onRmsChanged");
        }

        public void onBufferReceived(byte[] buffer) {
            Log.d(TAG, "onBufferReceived");
        }

        public void onEndOfSpeech() {
            Log.d(TAG, "onEndofSpeech");
            new Dialog("endOfSpeech").showPopupDialog(new Rectangle());
        }

        public void onError(int error) {
            Log.d(TAG, "error " + error);
        }

        public void onResults(Bundle results) {
            String str = new String();
            Log.d(TAG, "onResults " + results);
            ArrayList data = results.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION);
            for (int i = 0; i < data.size(); i++) {
                Log.d(TAG, "result " + data.get(i));
                str += data.get(i);
            }
            speech = str;
            lbl.setText("i got: "+speech);
        }

        public void onPartialResults(Bundle partialResults) {
            Log.d(TAG, "onPartialResults");
            new Dialog("partialResults").showPopupDialog(new Rectangle());
        }

        public void onEvent(int eventType, Bundle params) {
            Log.d(TAG, "onEvent " + eventType);
        }
    }

    public String getTextFromSpeech() {
        final Intent intent = new Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH);
        intent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL,
                RecognizerIntent.LANGUAGE_MODEL_FREE_FORM);
        intent.putExtra(RecognizerIntent.EXTRA_LANGUAGE, Locale.getDefault());
        intent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_PREFERENCE, "en");
        intent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL,
                RecognizerIntent.LANGUAGE_MODEL_WEB_SEARCH);
        intent.putExtra(RecognizerIntent.EXTRA_MAX_RESULTS, 3);

        Activity activity = AndroidNativeUtil.getActivity();
        activity.runOnUiThread(new Runnable() {
            @Override
            public void run() {

                sr = SpeechRecognizer.createSpeechRecognizer(AndroidNativeUtil.getActivity());
                sr.setRecognitionListener(new listener());
                sr.startListening(intent);
            }
        });
        return "waiting for input\n" + speech;
    }

    public String getText() {
        sr.stopListening();
        return "listened: " + speech;
    }

    public boolean isSupported() {
        return true;
    }

}

[Shai Almog]

You can fix the code to be synchronous and it will work as you designed it.
First you must make sure to invoke getTextFromSpeech from a thread that isn't the main thread (EDT). Assuming you did that instead of returning just loop and wait until the listener updates the value and once it does that you can return the actual speech result.

[leste...@gmail.com]

YES!!

I finally got it. Thank you very much for your help!
Now i will polish the prototype and post it here, so
other programmers don't go all that trouble again.

[leste...@gmail.com]

StateMachine:

/**
 * Your application code goes here
 */
package userclasses;

import android.os.Looper;
import com.codename1.system.NativeLookup;
import com.codename1.ui.*;
import com.codename1.ui.events.*;
import com.codename1.ui.geom.Rectangle;
import com.codename1.ui.util.Resources;
import generated.StateMachineBase;

/**
 *
 * @author Your name here
 */
public class StateMachine extends StateMachineBase {

    private MyNativeStuff s2t;
    int first = 0;

    public StateMachine(String resFile) {
        super(resFile);
        // do not modify, write code in initVars and initialize class members there,
        // the constructor might be invoked too late due to race conditions that might occur
    }

    /**
     * this method should be used to initialize variables instead of the
     * constructor/class scope to avoid race conditions
     *
     * @param res
     */
    @Override
    protected void initVars(Resources res) {
    }

    @Override
    protected void onMain_ButtonAction(Component c, ActionEvent event) {
        //create Class
        s2t = (MyNativeStuff) NativeLookup.create(MyNativeStuff.class);

        //2 Functions on one Button
        if (first == 1) {
            //Show result
            findLabel().setText(s2t.getText());
        } else if (s2t != null && s2t.isSupported()) {
            //invoke Speechrecognition
            s2t.getTextFromSpeech();
            first = 1;
            findLabel().setText("waiting");
        }
    }
}

MyNativeStuffImpl:

package userclasses;

import android.app.Activity;
import android.content.Intent;
import android.os.Bundle;

import android.speech.RecognitionListener;

import android.speech.RecognizerIntent;
import android.speech.SpeechRecognizer;
import android.util.Log;
import java.util.ArrayList;
import java.util.Locale;
import com.codename1.impl.android.AndroidNativeUtil;

public class MyNativeStuffImpl {

    //maybe use a getter for message to get the last event
    static String message = "";
    static String speech = "";
    private SpeechRecognizer sr;

    class listener implements RecognitionListener {

        public void onReadyForSpeech(Bundle params) {
            message = "onReadyForSpeech";
        }

        public void onBeginningOfSpeech() {
            message =  "onBeginningOfSpeech";
        }

        public void onRmsChanged(float rmsdB) {
            message =  "onRmsChanged";
        }

        public void onBufferReceived(byte[] buffer) {
            message =  "onBufferReceived";
        }

        public void onEndOfSpeech() {
            message =  "onEndofSpeech";
        }

        public void onError(int error) {
            message =  "error " + error;
        }

        public void onResults(Bundle results) {
            //here you have what google understand from the speech
            //maybe only save the first guess, which would have the highest
            //possibility
            String str = "";

            ArrayList data = results.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION);
            for (int i = 0; i < data.size(); i++) {

                str += data.get(i)+";";
            }
            speech = str;
        }

        public void onPartialResults(Bundle partialResults) {
            message =  "onPartialResults";

        }

        public void onEvent(int eventType, Bundle params) {

            message =  "onEvent " + eventType;
        }
    }

    public void getTextFromSpeech() {
        //initialize the Intent

        final Intent intent = new Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH);
        intent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL,
                RecognizerIntent.LANGUAGE_MODEL_FREE_FORM);
        intent.putExtra(RecognizerIntent.EXTRA_LANGUAGE, Locale.getDefault());

        intent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL,
                RecognizerIntent.LANGUAGE_MODEL_WEB_SEARCH);
        //number of guesses
        intent.putExtra(RecognizerIntent.EXTRA_MAX_RESULTS, 3);
        //Speechrecognition must be run on main Thread

        Activity activity = AndroidNativeUtil.getActivity();
        activity.runOnUiThread(new Runnable() {
            @Override
            public void run() {
                sr = SpeechRecognizer.createSpeechRecognizer(AndroidNativeUtil.getActivity());
                sr.setRecognitionListener(new listener());
                sr.startListening(intent);
            }
        });
    }

    public String getText() {
        return speech;

    }

    public boolean isSupported() {
        return true;
    }

}

On second click on the Button the label shows the text from Speech

[armando...@gmail.com]

Hello, 

I have several compilation problems using code that you offered. 

You 'can get a simple NetBeans project that uses your existing sources.  

Thanks so much. 

Armando

[Ismael Baum]

Thanks! You saved me hours. For newer versions of Android we get Error 9, which means we don't have microphone permissions. This can be resolved requesting runtime permissions, or adding the permission to the build hints: https://www.codenameone.com/manual/advanced-topics.html