Blockly

[Eric Den Hollander]

Since I'm not a real good programmer, writing functions is real troublesome for me. Is it possible to incorporate Blockly in a Function block?

https://developers.google.com/blockly/

Kind regards,

Eric

[Dave C-J]

Hi Eric

sorry - we have no plans for this in the core project. Of course being an open project others may wish to step in.

We would like the think the function block we have is a fairly "safe" sandbox in which to try messing about and that you can't really do anything too "bad" in there. But I grant you it may not be the absolutely easiest place to learn javascript.

[Eric Den Hollander]

Thank you Dave,

i'ts not that I want to become a real programmer like the most of you here, but it's more a necessity to make things work. The way "Blockly" works is more understandable for me, it keeps the syntax hidden and I can thus focus on the logic. I'm sure there are lots of people like me, playing around with sensors and stuff (e.g. Jeenode's). And since "coding" is a hot item worldwide, especially for kids, such an addition would make a wonderful system like Node-Red is (Respect!) available to a much larger group of tinkerers like me...

I hope someone picks this idea up!

Kind regards,

Eric

Op vrijdag 12 december 2014 15:07:42 UTC+1 schreef Eric Den Hollander:

[Dave C-J]

Hi Eric

I hear you. We like to think that Node-RED also keeps the syntax hidden but at a slightly higher level - that allows us to add things like Twitter nodes, web requests, database access and lots of things that need configuring, but yes you do often need to reach for a function node maybe too quickly - certainly something we should look at. Are there capabilities or nodes in particular that you think are missing that would avoid you having to write a function ?

In the meantime  - if you like blockly and devices - then can I recommend the Espruino to you. It's an arduino like board - but more flexible (IMHO) and can be programmed directly via both javascript and blockly. Gordon is doing a great job over at http://www.espruino.com/

[Julian Knight]

As a newcomer myself, one of the things that surprised me is to find all the nodes, code, etc. stuffed into a single JSON file.

That doesn't quite feel the Node way to me? If, for example, function nodes kept their code in a separate .js file, there would be all sorts of opportunities for external apps to be able to take over the editing. All NR would need to do would be to fs.watch the folders and reload on change.

Then it would probably be quite easy to incorporate things like Blockly which I think can compile to JS directly. We could also all use our favourite IDE's and editors without interminable copying and pasting.

Julian.

[Dave C-J]

One of the main features of Node-RED is the ability to cut/paste/email etc, flows or parts of flows around to both share and deploy. Keeping it all in one file means you get a complete (hopefully) working blob every time. Node-RED is trying to hide some of the underlying mess and trees of code that node.js sits on. Doing that as a single file (or http post) is much simpler than trawling around to build a deploy bundle, sending, and unpacking.

It is intended to be used mainly graphically - while allowing simple functions and extra nodes to be customised easily. If you are ending up writing loads of code in an editor then either we have missed some key functions or maybe Node-RED isn't the answer.

No doubt over time the built in function editor will get revisited/enhanced - and likewise I'm sure blockly could be incorporated - but I do feel that would rather be mixing graphical metaphors, and potentially more confusing than just sticking with text. 

[Julian Knight]

Hi Dave, many thanks for the reply.

It is perfectly possible I'm not using NR correctly :)

I can also see that having everything in one file makes the underlying manipulation easier & personally I agree that Blockly would create quite a clashing interface if integrated directly. However, it is clear that we all spend a fair bit of time actually writing functions. I have to say that I find the code editor built in to NR horribly slow on my PC, not sure why that is. (it might improve next week when I get my new i7 MS Surface Pro 3!). It is also very limiting compared to Adobe Brackets which is my normal editor (and about the only thing from Adobe I will touch!).

Using Blockly or some other editor as an external editor however might well make sense since, if we are already writing Function code, we are already outside of flow programming.

I do find many of the nodes very limiting which is why I'm ending up with so much code (isn't it possible to add data to an existing MongoDB document?). Most of my code however, is simply reworking (transforming) JSON data. I have data coming in from numerous sources (different sensor platforms and weather API's) and I'm trying to harmonise them. I don't think you will ever get away from code for that.

Having said that, I really benefit from the flow style for linking together the disparate elements. Triggering web API calls, listening for different sensors, transforming the data and then pushing the transformed data to a combination of MQTT (between devices and to do event handling), Web Sockets (to do smart web pages) and MongoDB for persistence.

I don't think that is a particularly unusual use case is it?

So I think there are a lot of nodes missing and some that are not nearly flexible enough. It might be helpful to compare to other graphical programming tools. Scratch comes to mind though obviously it isn't flow-based. But perhaps more interesting would be to look at some of the ETL tools available. Many have flow-based workflows.

I haven't spent long enough with NR yet to really dig into what might be missing & I want to continue to build using what is already here. Hopefully, with a bit more experience, I can make some sensible suggestions and even create some myself.

In case I sound negative, please don't think that, I'm loving NR as a tool and it has really kick-started my home automation work. It is working well on both my Raspberry Pi and my Synology NAS and I'm really pleasantly surprised at how stable it is. I know that's not easy to achieve.

Forgive the ramblings, regards, Julian.

[Dave C-J]

Hi Julian,

don't forget we are a browser based tool - 95% of all those ETL tools are native clients/applications. Even brackets is not designed to fully function in a browser (though there has been some work to make it do so - it is not part of the main branch). Trying to call out to a non-browser tool for editing functions is fraught with possibilities for errors etc, which we would rather avoid. Would the editor be local ? on the server (also served up ?) - opens up too many dependencies and installation fun and games that we don't want to get into right now. 

The Orion embedded editor we use could be enhanced - but again it is in the browser so we need to be mindful of size before going too full-on.

There may well be a better compromise and we're happy to hear all the ideas.

You mention the mongodb node - I thought that either save  or update should allow modification of existing data - in both cases you would need to supply the existing document _id or a new one gets allocated.

You also mention other nodes not being flexible enough... happy to hear your thoughts on those - or even bullet points for discussion, and likewise any suggestions for missing nodes - we're not looking for an extensive audit/list - just some clues for big holes to start with :-)

Ramble on !

[Julian Knight]

Cheers Dave.

Agree that there is a disjoint between the browser and other client tools. There are, of course, recognised ways around this issue. I get that a design decision has been made to keep everything in one file, I'll have to live with that. If I get too annoyed by it, I can always write a quick AutoHotkey script to automate the select-all/copy/paste tasks :)  I do still think that it would have been nice to have separate files - there really wouldn't be any need for NR to do anything other than have a watch on the folder as with tools such as forever to do a reload. It would be up to individuals then to have their own workflow. For example, someone could use Github to manage their code and either edit online or edit locally and push. In theory, we could do this with the JSON file too but it would require a duplicate setup of NR on the local machine (which isn't possible with some of the Pi nodes) and wouldn't allow online edits.  All this isn't that big a deal - just would have been nice to have.

In terms of browser-based editors, there does seem to have been an explosion of them recently though I don't know which are suitable for embedding in other apps, some are web only of course. It would be really helpful to have some additional features such as code folding.

For Mongo, I don't know if I'm doing it wrong, the docs are not very clear. I've tried passing the _id through to an update node, id does an update but it actually replaces everything! So what I wanted to do was to have a combined weather record from both Weather Underground and the BBC. These are two different API calls, I clean up the data and then want to combine it. I thought I should be able to do 2 separate updates with the same _id (the date as I only want to keep 1 per day) and different keys. What happens is that each update deletes the existing data.

I guess I need to do a query first to get the existing data, combine the new data and then do a save. I just haven't had time to do that yet.

I've also abandoned the idea of using the built-in web serving capability of NR. Partly due to the edit issues but also because ultimately, I want the web pages to be available over the Internet and I can't currently confirm that all of the comms channels will be secure. The use of ws: for example would seem to imply that non-secure web sockets will be used though I don't know enough about direct web socket access to know. I've always used Socket.IO which takes care of the details for you. So I may end up creating a separate node.js server which I will use to listen/send ws messages to NR and secured with the standard Node security features I'm familiar with.

Regards, Julian.

[Luis Montes]

Take a look at https://github.com/node-red/node-red/blob/master/public/red/comms.js#L25

If you run node-red on https, the UI will establish a socket connection using wss:// instead of ws://

Socket.IO is great, but it actually just uses the ws lib under the covers.  There is nothing inherently more secure about it.

--
http://nodered.org
---
You received this message because you are subscribed to the Google Groups "Node-RED" group.
To unsubscribe from this group and stop receiving emails from it, send an email to node-red+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Julian Knight]

Ahh, thanks Luis for clarifying that. makes me a bit happier!

I was aware of how Socket.IO works. In fact it is able to use other channels too I think if ws is not available which makes it a good choice when using over the internet.

[Luis Montes]

I'm a bit biased in that websockets are available anywhere I'd like use node-red:  http://caniuse.com/#search=websocket

So I'm not interested in socket.io's fallbacks anymore.  And as of 1.0, socket.io/engine.io starts with the fallbacks(ajax) and then climbs up to websockets.  Pre-1.0 it would try sockets first, fail then got to a fallback. This was a slower for older browsers, but its now slower for newer browsers :)

I'd actually like to see all of the ajax communication with node-red happen purely over websockets with an rpc pattern, but I think I'm still alone in there.

[Julian Knight]

Sure, WS is available in all current browsers and that is fine if you are in a fully enclosed environment where you control all the elements. What happens though if you try to access, for example, your nice home control and monitoring system from your corporate network where you are separated from the Internet by a raft of Data Loss & Intrusion Protection, proxies, firewalls and routers!

In my line of work at least, this is very commonplace. So the fact that Socket.IO makes a nice, standard, web call first is a great bonus and I'm happy to live with the overhead. It means that it will most probably succeed in getting some kind of upgraded connection.

Then there is another issue that I think I'm already hitting with mobile clients at least. It looks to me (not had the chance to really investigate yet) that when I look at my home monitor page in the morning then go to work and check it again when I come back, the socket is disconnected and doesn't connect until I reload the page. This might not be that big a deal but it isn't how I think most people (e.g. not us techies!) would expect things to behave. I don't think that happens when you use Socket.IO though I've not confirmed it yet. Maybe because Socket.IO has the smarts to reconnect? Not sure.

So I can see your point certainly about the speed though I'm not convinced it would be noticed in real life. But sometimes resilience and reliability are much more important.

So, unless I can be convinced otherwise I think I wouldn't really support the suggestion of WS/RPC only patterns.

Interesting conversation BTW, thanks for prompting me to check some details out!

Regards, Julian.

[Luis Montes]

My motivation with the suggestion was only partly based on performance gains.  And only partly based on my desire to explicitly desupport old browser and old companies that are scared of things like websockets on their network :)

The cool thing that happens when you abstract the network communication into a generic rpc framework is that you can have implementations on top of things like window.postMessage where another DOM in an iframe can function as a browserifed node-red backend without a major rewrite.  It changes the lifecycle of node-red and removes the nodejs dependency.

Anyway probably kicking a dead horse, and a fork to do what I want isn't all that hard to maintain.  There are other bigger wins that I'd like to help out with when I get some time.

[Dave C-J]

"It's not dead yet" :-) - http://youtu.be/Sh8mNjeuyV4

[Julian Knight]

Thanks for taking the time to explain Luis.

I can't claim to be an expert in ws or rpc so I've no doubt there are advantages in what you are suggesting. My background is mainly in big IT rather than development so my take is based on the experience I see day-to-day of fighting constant battles with big infrastructure. It takes a long time, lots of money and a fair bit of dedication (not to mention "Magic Pixy Dust"!) to upgrade such infrastructure so big organisations are always struggling with outdated hardware, software and concepts. So I generally err on the side of safety when it comes to connectivity. It is also one of the reasons I'm keen to push security concepts to IoT & why I hate it when people assume that IoT will only ever be implemented on "trusted" networks when I know there is no such thing.