blockly-android and event driven model

[Tim Harvey]

Greetings,

I'm working on an Android app using blockly-android.

I want to use the event driven model where I have several starting blocks (those that accept functions and have no 'prev' block connector) and I wish these blocks to generate js functions that I can call asynchronously from Java event handlers.

I've run into a couple of issues I could use some help resolving with blockly-android:

1. how do you specify blocks to appear on the workspace by default (ie forced blocks like 'Start') and mark them as un-deletable?

2. I find that javascript code passed to onFinishCodeGeneration() is not organized with functions at the top and that blocks that are not encapsulated by start blocks appear - can I control the filtering of this? I would like to strip out all the blocks that are not within start blocks.

3. how is it suggested that we can 'execute' the javascript code within Android? The examples all use WebView's which make sense to execute javascript in, but in my case I'm using the code to drive a physical robot around. So far I found that using the open-source AndroidJSContext works, but I'm not sure if that's the easiest or even necessary. It seems like android-blockly is using javascript for code-generation so they must be interpreting it?

4. how do I enable functions in the toolbox?

Thanks for any ideas!

Tim 

[Andrew n marshall]

Hey Tim,

Thanks for trying out Blockly for Android.  Let's see if I can help:

 

1. how do you specify blocks to appear on the workspace by default (ie forced blocks like 'Start') and mark them as un-deletable?

I would suggest constructing a XML for your initial workspace, and loading that file inside your activity's onLoadInitialWorkspace() or onInitBlankWorkspace().

Within that XML, you can specify deletable="false" on a block, but we haven't actually implemented this (#305). This should be a quick fix, if you have the time (and I don't get to it first).  However, movable="false" is implemented, and will prevent users from dragging the block, including dragging it to the trash.  A mild but effective hack.

 

2. I find that javascript code passed to onFinishCodeGeneration() is not organized with functions at the top and that blocks that are not encapsulated by start blocks appear - can I control the filtering of this? I would like to strip out all the blocks that are not within start blocks.

In web blockly, the easiest way prevent the generation of code unconnected to event trigger blocks is to set Blockly.Events.disableOrphans=true.  Unfortunately, this is another case where Blockly for Android doesn't support this yet (#333).  This should be feasible once all the basic event code is implement (my current project).

Both problems (ordering and skipping code generation) are solvable by modifying the JavaScript generator to your needs.  You can find the source of the JavaScript generator (and others), here, on the main Blockly repo.  Blockly for Android is compatible with all of them. Replace references to javascript_compressed.js with the file name of your own generator code (#99, add an API for this).

 

3. how is it suggested that we can 'execute' the javascript code within Android? The examples all use WebView's which make sense to execute javascript in, but in my case I'm using the code to drive a physical robot around. So far I found that using the open-source AndroidJSContext works, but I'm not sure if that's the easiest or even necessary. It seems like android-blockly is using javascript for code-generation so they must be interpreting it?

Blockly does use a limited amount of off-screen WebViews to execute generator JavaScript, but we are agnostic regarding how apps should run their code.

In our own robots (such as those seen at Google IO), we upload the generated code into the robot and run it there.  We like this strategy, because it means the robot can run independent of the tablet and the wifi connection.  In that case, depending on your robot, JavaScript might not be the right language, but it is pretty easy to use your own language generator.

 

4. how do I enable functions in the toolbox?

Blockly for Android does not support functions yet.  We still exploring appropriate touch/mobile UIs equivalents for the "mutator" window used in function argument configuration and other blocks.

It's great to hear your needs, and hopefully we can prioritize a few of these small ones.

[Erik Pasternak]

One other note, we are tracking support for Event blocks (start blocks that wrap the code below them) on the web side: https://github.com/google/blockly/issues/384

[Tim Harvey]

On Monday, August 29, 2016 at 3:23:21 PM UTC-7, Andrew n marshall wrote:

Hey Tim,

Thanks for trying out Blockly for Android.  Let's see if I can help:

 

1. how do you specify blocks to appear on the workspace by default (ie forced blocks like 'Start') and mark them as un-deletable?

I would suggest constructing a XML for your initial workspace, and loading that file inside your activity's onLoadInitialWorkspace() or onInitBlankWorkspace().

Within that XML, you can specify deletable="false" on a block, but we haven't actually implemented this (#305). This should be a quick fix, if you have the time (and I don't get to it first).  However, movable="false" is implemented, and will prevent users from dragging the block, including dragging it to the trash.  A mild but effective hack.

Hi Andrew,

Great - I didn't know about movable="false". This sounds like a good workaround for adding a 'start' block which outputs 'start() { <funcs> }' text.

 

 

2. I find that javascript code passed to onFinishCodeGeneration() is not organized with functions at the top and that blocks that are not encapsulated by start blocks appear - can I control the filtering of this? I would like to strip out all the blocks that are not within start blocks.

In web blockly, the easiest way prevent the generation of code unconnected to event trigger blocks is to set Blockly.Events.disableOrphans=true.  Unfortunately, this is another case where Blockly for Android doesn't support this yet (#333).  This should be feasible once all the basic event code is implement (my current project).

What is your best guess at timeframe on getting to that?

I'm thinking if I require a 'start' block then I could likely filter out code that is not within functions as orphaned blocks.

Both problems (ordering and skipping code generation) are solvable by modifying the JavaScript generator to your needs.  You can find the source of the JavaScript generator (and others), here, on the main Blockly repo.  Blockly for Android is compatible with all of them. Replace references to javascript_compressed.js with the file name of your own generator code (#99, add an API for this).

I haven't yet dug into the android-blockly code. Are you saying it is referencing the main blockly repo and using those code generators or are you saying I could use those to override what is built into android-blockly at the moment?

 

 

3. how is it suggested that we can 'execute' the javascript code within Android? The examples all use WebView's which make sense to execute javascript in, but in my case I'm using the code to drive a physical robot around. So far I found that using the open-source AndroidJSContext works, but I'm not sure if that's the easiest or even necessary. It seems like android-blockly is using javascript for code-generation so they must be interpreting it?

Blockly does use a limited amount of off-screen WebViews to execute generator JavaScript, but we are agnostic regarding how apps should run their code.

Do you have an example of using an offscreen WebView you can point me to? I have to admit I'm very new to Android programming and even my Java is about 10 years rusty. It sounds like using an off-screen WebView may eliminate the need for AndroidJSContext but I would need to understand how to bind javascript functions back to Java.

I should be able to create event handlers by using AndroidJSContext to invoke javascript bindings on events, I just haven't quite figured out how to do that yet. My first use case is to have a voiceCommand background service capturing simple voice commands and issuing callbacks for those. I haven't yet figured out what the best course of action for this is.  

 

In our own robots (such as those seen at Google IO), we upload the generated code into the robot and run it there.  We like this strategy, because it means the robot can run independent of the tablet and the wifi connection.  In that case, depending on your robot, JavaScript might not be the right language, but it is pretty easy to use your own language generator.

 

4. how do I enable functions in the toolbox?

Blockly for Android does not support functions yet.  We still exploring appropriate touch/mobile UIs equivalents for the "mutator" window used in function argument configuration and other blocks.

It's great to hear your needs, and hopefully we can prioritize a few of these small ones.

It would probably help to explain more about what i'm doing. In my use case the robot's are simple home-made robots using 4 servo's controlled by an Arduino. An Android phone acts as the robot head and currently is where the Blockly code is designed and executed. Thus I need to hook the javascript functions (or objects) that my blocks call out to interpret them into simple strings sent to the Arduino over bluetooth. My goal is to eventually allow the user to code in javascript directly as well and eventually allow syncing with javascript created by a web-based blockly/javascript editor (kind of like MIT's App Inventor uses a companion app to keep data in sync with their web based IDE).

This is for a grade school robotics class - See http://github.com/tharvey/BlocklyBot and http://robotstem.wordpress.com for details.

Thanks for the great suggestions!

Tim

[Andrew n marshall]

On Tue, Aug 30, 2016 at 1:13 AM, Tim Harvey <harve...@gmail.com> wrote:

On Monday, August 29, 2016 at 3:23:21 PM UTC-7, Andrew n marshall wrote:

2. I find that javascript code passed to onFinishCodeGeneration() is not organized with functions at the top and that blocks that are not encapsulated by start blocks appear - can I control the filtering of this? I would like to strip out all the blocks that are not within start blocks.

In web blockly, the easiest way prevent the generation of code unconnected to event trigger blocks is to set Blockly.Events.disableOrphans=true.  Unfortunately, this is another case where Blockly for Android doesn't support this yet (#333).  This should be feasible once all the basic event code is implement (my current project).

What is your best guess at timeframe on getting to that?

I'm not going to pin a time on that right now, because too many other things are coming up in the coming weeks.  Apologies.

Both problems (ordering and skipping code generation) are solvable by modifying the JavaScript generator to your needs.  You can find the source of the JavaScript generator (and others), here, on the main Blockly repo.  Blockly for Android is compatible with all of them. Replace references to javascript_compressed.js with the file name of your own generator code (#99, add an API for this).

I haven't yet dug into the android-blockly code. Are you saying it is referencing the main blockly repo and using those code generators or are you saying I could use those to override what is built into android-blockly at the moment?

I'm suggestion you could create your own generator, based off the original sources, that filters out blocks that aren't connected to event triggers.  It would also allow you to expose the event triggers as functions.

 

 

 

3. how is it suggested that we can 'execute' the javascript code within Android? The examples all use WebView's which make sense to execute javascript in, but in my case I'm using the code to drive a physical robot around. So far I found that using the open-source AndroidJSContext works, but I'm not sure if that's the easiest or even necessary. It seems like android-blockly is using javascript for code-generation so they must be interpreting it?

Blockly does use a limited amount of off-screen WebViews to execute generator JavaScript, but we are agnostic regarding how apps should run their code.

Do you have an example of using an offscreen WebView you can point me to? I have to admit I'm very new to Android programming and even my Java is about 10 years rusty. It sounds like using an off-screen WebView may eliminate the need for AndroidJSContext but I would need to understand how to bind javascript functions back to Java.

I should be able to create event handlers by using AndroidJSContext to invoke javascript bindings on events, I just haven't quite figured out how to do that yet. My first use case is to have a voiceCommand background service capturing simple voice commands and issuing callbacks for those. I haven't yet figured out what the best course of action for this is.

I'm assuming you mean AndroidJSCore, since I couldn't find AndroidJSContext.  I'm not very familiar with it, but I see this:

context.evaluateScript("var f = factorial(10);")

Seems like you replace factorial() with your own function name.

 

It would probably help to explain more about what i'm doing. In my use case the robot's are simple home-made robots using 4 servo's controlled by an Arduino. An Android phone acts as the robot head and currently is where the Blockly code is designed and executed. Thus I need to hook the javascript functions (or objects) that my blocks call out to interpret them into simple strings sent to the Arduino over bluetooth. My goal is to eventually allow the user to code in javascript directly as well and eventually allow syncing with javascript created by a web-based blockly/javascript editor (kind of like MIT's App Inventor uses a companion app to keep data in sync with their web based IDE).

This is for a grade school robotics class - See http://github.com/tharvey/BlocklyBot and http://robotstem.wordpress.com for details.

Thanks for the great suggestions!

That looks awesome!  Really happy to see Blockly for Android in use and helping kids.

It seems your setup isn't too different than our own, but we use a different chip. Synchronization is easier when pushing commands one-by-one to the app, and we don't have that solved right now, pushing the entire program down to the robot.

[Erik Pasternak]

A few more details:

• The easiest way to make events wrap code and ignore code not wrapped by an event would be to modify generator.workspaceToCode. You can then build your own version of blockly_compressed and javascript_compressed to replace the ones in the Android library.
• Take a look at CodeGeneratorService to see how we're running a WebView in a background thread. You should be able to do something similar with a custom interface for calls to the robot. As Andrew mentioned, there are also some third party libraries for executing JS directly (WebView assumes it's in the UI, so you have to trick it a bit to use it in the background.

--
You received this message because you are subscribed to a topic in the Google Groups "Blockly" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/blockly/cMaqarnW3kw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to blockly+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Erik Pasternak | Master of the House | epas...@google.com     

[Tim Harvey]

On Monday, August 29, 2016 at 3:23:21 PM UTC-7, Andrew n marshall wrote:

Hey Tim,

Thanks for trying out Blockly for Android.  Let's see if I can help:

 

1. how do you specify blocks to appear on the workspace by default (ie forced blocks like 'Start') and mark them as un-deletable?

I would suggest constructing a XML for your initial workspace, and loading that file inside your activity's onLoadInitialWorkspace() or onInitBlankWorkspace().

Within that XML, you can specify deletable="false" on a block, but we haven't actually implemented this (#305). This should be a quick fix, if you have the time (and I don't get to it first).  However, movable="false" is implemented, and will prevent users from dragging the block, including dragging it to the trash.  A mild but effective hack.

It looks like all the code is in place for get/setDeletable - it just isn't checking this when deleting a block. The following seems to work fine for me: https://github.com/tharvey/BlocklyBot/commit/c8569e074971f6c21e83b1affd1c95f3313afa2a 

 

 

2. I find that javascript code passed to onFinishCodeGeneration() is not organized with functions at the top and that blocks that are not encapsulated by start blocks appear - can I control the filtering of this? I would like to strip out all the blocks that are not within start blocks.

In web blockly, the easiest way prevent the generation of code unconnected to event trigger blocks is to set Blockly.Events.disableOrphans=true.  Unfortunately, this is another case where Blockly for Android doesn't support this yet (#333).  This should be feasible once all the basic event code is implement (my current project).

Both problems (ordering and skipping code generation) are solvable by modifying the JavaScript generator to your needs.  You can find the source of the JavaScript generator (and others), here, on the main Blockly repo.  Blockly for Android is compatible with all of them. Replace references to javascript_compressed.js with the file name of your own generator code (#99, add an API for this).

sounds good - currently I have another reason that I need to 'parse' the javascript, so I'm just fixing it up there by deleting code that's not within the 'start' blocks that I created.

 

3. how is it suggested that we can 'execute' the javascript code within Android? The examples all use WebView's which make sense to execute javascript in, but in my case I'm using the code to drive a physical robot around. So far I found that using the open-source AndroidJSContext works, but I'm not sure if that's the easiest or even necessary. It seems like android-blockly is using javascript for code-generation so they must be interpreting it?

Blockly does use a limited amount of off-screen WebViews to execute generator JavaScript, but we are agnostic regarding how apps should run their code.

In our own robots (such as those seen at Google IO), we upload the generated code into the robot and run it there.  We like this strategy, because it means the robot can run independent of the tablet and the wifi connection.  In that case, depending on your robot, JavaScript might not be the right language, but it is pretty easy to use your own language generator.

I like this strategy as well, but I'm not aware of a javascript interpreter in android that you can 'hook' into. I'm currently using AndroidJSCore as an interpreter and its working well. I'm able to hook the function calls that my blocks create.

I added some 'Listen' blocks that use voice recognition (I found Android's VoiceRecognizer to not be fast enough (which may be me not tuning it proplery) and thus am using CMU's PocketSphinx which is working well for regognizing phrases. This way I can have a 'Listen' block with a text field where the user selects the phrase they want to trigger off of and I have an event handler that calls the function that is comprised of the function input to the block. Once I had 'listen' blocks working as event handlers it made sense to add a 'start' block and filter out orphaned root blocks.

 

 

4. how do I enable functions in the toolbox?

Blockly for Android does not support functions yet.  We still exploring appropriate touch/mobile UIs equivalents for the "mutator" window used in function argument configuration and other blocks.

It's great to hear your needs, and hopefully we can prioritize a few of these small ones.

props to you and Erik for the tips!

Tim

 

[Andrew n marshall]

[Replying inline.]

On Sat, Sep 10, 2016 at 7:01 PM, Tim Harvey <harve...@gmail.com> wrote:

It looks like all the code is in place for get/setDeletable - it just isn't checking this when deleting a block. The following seems to work fine for me: https://github.com/tharvey/BlocklyBot/commit/c8569e074971f6c21e83b1affd1c95f3313afa2a 

That's definitely part of it.  We'd also want reconnect the block to its location at the beginning of the drag.  This requires reverting all the events during the drag.

 

I like this strategy as well, but I'm not aware of a javascript interpreter in android that you can 'hook' into. I'm currently using AndroidJSCore as an interpreter and its working well. I'm able to hook the function calls that my blocks create.

We're doing this in a WebView.  See Binding JavaScript and the @JavaScriptInterface annotation.  Or maybe you mean the more strict definition of interpreter.

 

I added some 'Listen' blocks that use voice recognition (I found Android's VoiceRecognizer to not be fast enough (which may be me not tuning it proplery) and thus am using CMU's PocketSphinx which is working well for recognizing phrases. This way I can have a 'Listen' block with a text field where the user selects the phrase they want to trigger off of and I have an event handler that calls the function that is comprised of the function input to the block. Once I had 'listen' blocks working as event handlers it made sense to add a 'start' block and filter out orphaned root blocks.

Cool.  We explored some speech synthesis and recognition blocks over the summer, though this was web APIs.  I expect we'll share with the world soon.

When you say "Android's VoiceRecognizer", I assume you are referring to SpeechRecognizer.

--

Andrew, Blockly for Android