Dragging Toolbox blocks with virtual pointers?

[Oz Ramos]

Hi! I created a web based Face Pointer library called Handsfree.js, and now I'm using it to build a Blockly plugin that lets you code via face gestures:

For the dragging in the clip above, I'm using the BlockDragger class, but it only seems to work with blocks that are already in your workspace. How can I use the BlockDragger to drag over a block from the Toolbox into the Workspace, or is there a workaround?

I noticed that block paths inside Toolboxes have a [data-id], but new Blockly.BlockDragger returns null on that ID, maybe because it's not actually in the workspace yet?

Thanks!

[Oz Ramos]

Whoops image didn't come through:

[Oz Ramos]

Hi! I still haven't made any progress on this :( Basically I can determine which Toolbox SVG path was clicked, but I don't know how to determine which block it was because getToolbox() doesn't return a Toolbox class with a Blockly.Toolbox.getToolboxItemById method

My current psuedocode is:

- When virtual pointer clicks inside toolbox, traverse up DOM until closest .blocklyDraggable element is found

- Extract the [data-id] attribute of the SVG path that represents the block

- I think I should pass this ID into Blockly.Toolbox.getToolboxItemById  , but when I do getToolbox() on my Blockly workspace (which I get with Blockly.inject) this method is not available. Neither is getToolboxItems 

- An alternative, would be to count which index this block is among other blocks and then use selectItemByPosition if ID doesn't work

If I can somehow determine which block was picked from the Toolbox, then my next plan is to:

- Add block onto Workspace with newBlock

- Immediately startDrag so that it seems as if the user "picked" it up from toolbox

- Drag it around as in GIF above

Sorry for so many words. Btw, new Blockly docs are amazing!

[Abby Schmiedt]

Hello!

It sounds like there might be a few things at play here: 

1. The new toolbox that has getToolboxItemById came out in the latest release (3.20200924.0 and up). If you update you should see the getToolboxItemById. However, I'm not sure this is exactly what you are looking for.

2. I think what you actually want is the flyout. The toolbox encompasses only the categories. In your case this is Events, Event Modifiers etc. The flyout holds the actual blocks and is made up of a Blockly workspace. You can get this workspace by: 

   yourWorkspace.getToolbox().getFlyout().getWorkspace() 

OR

   yourWorkspace.getToolbox().flyout_.getWorkspace() (If you are on an older version of Blockly)

From what I understand once you have the flyout workspace you can do basically the same thing you were doing on the main workspace.

I'm not entirely sure here, but I think you might also want to check out createBlock on the flyout instead of newBlock on the workspace.

Hope this helps! And let me know if you are still having problems or if I have misunderstood something : ) 

Abby

 

[Oz Ramos]

That worked perfectly thank you!!

Here's a GIF, and a longer video to demonstrate it working: https://www.youtube.com/watch?v=bcDOv09-5RI

[Beka Westberg]

Wow that is so cool! It's awesome that you got the implementation working so quickly :D

I know this is kind of side tracking this thread, but after watching your video it got me wondering what else could be done to make Blockly work better with handsfree.js, or other tools for motor-impared people.

For example, how do you feel about the drop targets for connections? Do you think that making connections have a bigger "connection range" would be helpful?

I also know there's a feature request for allowing the workspace to automatically scroll when a block hits the edge of the workspace. It seems to me like that would be really helpful for something like this! What do you think?

Best wishes,

Beka

--
You received this message because you are subscribed to the Google Groups "Blockly" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blockly+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/blockly/7b377f59-7785-4724-9155-d951519d500bn%40googlegroups.com.

[Abby Schmiedt]

Hello!

Just wanted to check in on this! I think this is really awesome and would love to see this as some kind of plugin! There is some new documentation on creating plugins that we have just added.

Let me know if you have any questions or if there is anything we can help with!

Cheers, 

Abby

[Oz Ramos]

Hi, thanks for the links!

Yes I should have a plugin in the next week or two, I meant to respond sooner but I kept thinking "I'll respond after I finish this afternoon" haha. The plugin itself is maybe 80% ready, the last major bit is to add a virtual keyboard so that people can type into fields, and also apply some of the suggestions Beka mentioned. Here's what I mean by virtual keyboard:

It works directly with the Blockly API and also fires native browser events (eg native mouse clicks) so it should work with any Blockly-based app out of the box, regardless of the framework it's embedded in (Vue, React, Electron App, etc)

I plan on creating a separate post to announce my app and the plugin together, and I'll share documentation, examples, boilerplates, etc then!

Oz

[Oz Ramos]

Hello! I'm still working on this but I wanted to share a preview of a new update: hand tracking! The idea is to use pinch gesture to click and drag blocks around. However this one is less about assistive tech and more about exploring.

Here's the high quality video if the GIF below doesn't work (sorry about the loud music haha I didn't realize that was being recorded): https://vimeo.com/485180395

[Beka Westberg]

Hello,

That is so cool! I love seeing the progress updates on this :D

I like the gestures you picked. The pinching for clicking seems really intuitive. It also makes me curious about the challenges of implementing other common mouse "gestures" into hand gestures. For example, I imagine it would be hard to do double clicking -- regardless of the gesture you pick -- because you'd have to gesture fast enough for it to count as "double" but slow enough for the computer to pick it up lol. Not that that is a problem you'll have to deal with with Blockly :P This project just always gets me thinking about interesting challenges :D

Thank you again for sharing! I really enjoy the updates =)

--Beka

To view this discussion on the web visit https://groups.google.com/d/msgid/blockly/79dd9597-a495-4963-b6e1-a9fde161210bn%40googlegroups.com.

[Oz Ramos]

Thanks!! Oh interesting I hadn't thought about double clicking yet (or right clicking for that matter) these will be interesting to explore. Maybe pinching with different fingers can do different things, like pinching with middle finger does a double click and the pinky does a right click.

The Blockly plugin will basically be a wrapper around Handsfree.js but with specific helpers for Blockly (for like dragging blocks and the grid). I just get so anxious releasing things so I'm trying to make sure everything works right haha

Anyways thank you so much for the response, it's very encouraging!

Oz

[Beka Westberg]

Hi Oz,

Sorry it's been so long since I messaged :/ I was just wondering: how is it going?

Best wishes,

Beka

[Oz Ramos]

Hi Beka no worries at all, it's really cool of you to check in!! I still plan on releasing the plugin but probably not for another month or two. I ended up rewriting the Handsfree.js library from scratch so that it could better integrate with Blockly, and the work I did so far helped me land a 4 month art/tech residency at Carnegie Mellon which I'm in right now 🤯

The idea so far is to put my Blockly app into a Chrome Extension so that people can map gestures and create automations to help them browse the web handsfree. It's kind of like a Blockly version of TamperMonkey (a userscript manager, which lets you add custom code to any website). Here are some GIFs of the handsfree browser extension...these are hardcoded with JavaScript but the idea is to use Blockly instead:

Here is what it looks like to scroll pages, you can even use 2 hands (actually it supports 2 people, so up to 4 hands at once!)

Here is playing desktop games from the browser (this prototype does use Blockly, and I dragged those blocks together with my face):

The challenge is that different websites require different modes of inputs, and people have all kinds of abilities so it would be literally impossible for even a team to do all this for every possible condition which is my goal. The idea is that Blockly would allow people who don't code or who can't even use a keyboard to create and share their own mappings. I think the idea will really take off once people can see that they can play games (especially Google Stadia) handsfree...I still lose sleep some nights thinking about it haha

[Beka Westberg]

> I ended up rewriting the Handsfree.js library from scratch so that it could better integrate with Blockly, and the work I did so far helped me land a 4 month art/tech residency at Carnegie Mellon which I'm in right now 🤯

Wow dude that's awesome! It's so cool that you just jumped in and decided to rewrite it. I'm sure it was a huge undertaking, but from your gifs it looks like it was definitely worth it! And congratulations on the residency, you definitely deserve it :D

> The idea so far is to put my Blockly app into a Chrome Extension so that people can map gestures and create automations to help them browse the web handsfree. It's kind of like a Blockly version of TamperMonkey (a userscript manager, which lets you add custom code to any website)... The challenge is that different websites require different modes of inputs, and people have all kinds of abilities so it would be literally impossible for even a team to do all this for every possible condition which is my goal. The idea is that Blockly would allow people who don't code or who can't even use a keyboard to create and share their own mappings.

That's a really clever idea!

I never thought about it in the way you've framed it before, but as you said, accessibility creates a really difficult problem, because optimally, every person & application would get a customized solution. To solve this you could:

1) Create a bunch of toggles. But with the amount of customizability people need, this would easily get overwhelming.

2) Use machine learning on it somehow? But machine learning will always be "buggy" and cause frustration.

3) Create a language (in this case a visual programming language!) so that people with accessibility needs can talk to their computer.

I don't think most people think about option 3 hehe. But obviously you have, and it really seems like the best solution out of all the options! I'm so excited to see what happens with this hehe.

> I think the idea will really take off once people can see that they can play games (especially Google Stadia) handsfree...I still lose sleep some nights thinking about it haha

Oh my gosh yessss haha. I can't wait to see that!

Thank you so much for this update! I love this project so much :D

Best wishes,

--Beka

--
You received this message because you are subscribed to the Google Groups "Blockly" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blockly+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/blockly/208ffe95-b8e0-47cc-a6a2-68ca2dbab354n%40googlegroups.com.