Just getting started, need advice and help acclimating to the "Blockly Paradigm"

[Chris Rowan]

Noob warning! Be gentle!

Relatively accomplished full stack programmer (html, js, php, sql, c#, etc.), completely new to Blockly.

Very excited by the promise that Blockly shows!

Been through some of the tutorials and it all makes sense, but because it's new and fresh, still struggling to fit all the pieces together.

I am trying to accomplish something that would be quite difficult without Blockly.

I have JSON inputs that I acquire from a REST api call. That JSON needs to be mapped and transformed to post to a different REST API based on a Swagger 2.0 schema.

The source data has some data sets expressed as arrays that need to map to objects in the output schema.

Some mappings are simple single source field to single target field mappings, but may require some transformations of the values.

There are certain boolean values in the source which indicate which source data should be mapped to the target schema.

Some of the source data fields need to be split into multiple target fields.

Other source data fields need to be combined into single target data fields.

Some source data fields need to be type or format converted to fit target data field needs.

In some cases source data field values need to be "enum mapped" to target fields.

When source data is not provided for fields in the target schema, a user defined default value needs be submitted.

All of this would be relatively trivial to hard code if the source data were static, but that's not an option. Variability in the source data requires that this all be done based on mapping rules, mapped by a person interactively.

Blockly seems perfect for mapping the various transformations and conditional mappings.

Some questions:

1. High level, what is the best way to accomplish this?
- Choose source and target fields in a div outside the Blockly Workspace, and feed those to blocks?
- Somehow feed blocks source and target field options and choose those within the blocks?
2. How can I save block data to outside shared storage (not browser storage)
3. Generally, should I do this all programmatically using Blockly just to map the transformations, or should I somehow use code generation?
4. What is the best paradigm for linking complex code with the blocks? Can logic be programmed into the blocks, or is code generation "the way?"

[Ronald Bourret]

Welcome to Blockly!

Data transformation is a well-established field, so I was wondering if you have looked around to see if such a tool already exists. ETL (Extract, Transform, Load) software is one possibility. It is usually used to populate data warehouses, but I would imagine it supports JSON to JSON transformations as well. It often provides a GUI for defining mappings.

You could also look for JSON to JSON transformers or REST API mapping/chaining tools.

I'll let somebody else address the suitability of Blockly for this purpose :)

Ronald Bourret

Technical Writer (Provided by Synergis)

rbou...@xwf.google.com

--
You received this message because you are subscribed to the Google Groups "Blockly" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blockly+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/blockly/3eb8f8a7-6dd2-460a-a921-1b11eafebf71n%40googlegroups.com.

[Mark Friedman]

Hi, Chris!  At its core, Blockly assumes that you want to generate code.  There are apps out there (notably Scratch) which (sort-of) interpret the block data structures directly, but I think that you'll get better support (from the documentation and this group) if you have your blocks generate code.  Given that, I think that a good way to proceed is to think about the various mapping and transformation functions/operators you want for your system and then what are the functions/operators that combine the core mapping and transformation operators.  Defining those things will make it clearer what the code that you need to generate will look like.  Then, of course, you'll need to actually define the blocks which generate that code and you'll need to implement those functions/operators that will appear in that generated code.  You'll also need to figure out how to process the generated code, i.e. how to interpret or compile it.  As you work through all this, feel free to ask more questions!

-Mark

On Mon, Nov 11, 2024 at 7:55 AM Chris Rowan <cmln...@gmail.com> wrote:

--

[Chris Rowan]

Thank you Ronald! We need a very end-user friendly mapping tool. Most of the mappings are simple point and click 1:1 relationships. This source to that target. But there are some complex mappings that need logic. We're trying to enable an everyday user to do it. ETL scripting languages are both too complex, and potentially (surprisingly) can't handle some of our required transformations. Go figure. After evaluating several ETL packages, we're still on the Blockly path!

[Chris Rowan]

Thanks Mark!

Based on the custom generator codelab, we've been able to create almost all the transformations we need to perform. It fun and almost magical. 

The next steps are getting the Blockly workspace to communicate with the enclosing webpage, and stacking the transformations. 

In pseudo: webform input field --> click button --> open blockly workspace and inject webform input field into first block which passes that value to next block in stack, stack custom transformation logic blocks to perform multiple transformations, click button to return the final transformation result to the enclosing webpage as the value for the target json object. 

I realize none of this should be difficult, just being a few days new to Blockly, having trouble wrapping my head around how all the pieces fit together, how values are passed, and once I'm done with the Blockly transformations, how to get that value back to the webpage so it can be submitted to a REST API. That's prototype stage one. Once that works, the idea is to serialize and save the Blockly logic so we can pass multiple values through it in an automated manner.

Any pointers, ideas or samples would be welcome!

Thanks.

[Mark Friedman]

Chris,

The next steps are getting the Blockly workspace to communicate with the enclosing webpage, and stacking the transformations.

I'm not sure that this is what you want.  If you go the code generation route (which is recommended) then, if I understand you correctly, it's the execution of the generated code which needs to communicate with the enclosing webpage.  This doc page might help you in figuring out how to do that.

-Mark

To view this discussion visit https://groups.google.com/d/msgid/blockly/eb44d562-0f59-4614-aad7-47684d33f5cfn%40googlegroups.com.

[Chris Rowan]

In the meantime, before reading your very helpful response, I created the prototype below which accomplishes exactly what I want (it will obviously get waaaay more complex than this after developed).

Is this heresy? Am I misusing Blockly? Will the team arrive at my door with torches and pitchforks? 🤣

(you can just drop this in an index.html file and load it in a browser)

<!DOCTYPE html>

<html>

<head>

    <title>Field Mapper</title>

    <script src="https://unpkg.com/blockly/blockly.min.js"></script>

    <style>

        .field-container {

            margin: 20px;

        }

        #blocklyDiv {

            height: 400px;

            width: 600px;

            display: none;

        }

    </style>

</head>

<body>

    <div class="field-container">

        <label>Source Field Name:</label>

        <input type="text" id="sourceFieldName" value="sourceProductCode" readonly>

        <br>

        <label>Source Field Value:</label>

        <input type="text" id="sourceFieldValue" value="5839">

        <br>

        <label>Target Field Name:</label>

        <input type="text" id="targetFieldName" value="targetProductCode">

        <br>

        <label>Target Field Value:</label>

        <input type="text" id="targetFieldValue" readonly>

        <br>

        <button onclick="showBlockly()">Complex Map</button>

    </div>

    <div id="blocklyDiv"></div>

    <button id="confirmButton" onclick="confirmAndClose()" style="display:none">Confirm & Close</button>

    <xml id="toolbox" style="display: none">

        <block type="source_block"></block>

        <block type="transform_block"></block>

        <block type="result_block"></block>

    </xml>

    <script>

        let workspace;

        Blockly.Blocks['source_block'] = {

            init: function() {

                this.appendDummyInput()

                    .appendField(document.getElementById('sourceFieldName').value)

                    .appendField(new Blockly.FieldTextInput(document.getElementById('sourceFieldValue').value), "SOURCE_VALUE");

                this.setNextStatement(true);

                this.setColour(120);

            }

        };

        Blockly.Blocks['transform_block'] = {

            init: function() {

                this.appendDummyInput()

                    .appendField("Prepend PC-")

                    .appendField("Current result [")

                    .appendField(new Blockly.FieldTextInput(""), "TRANSFORM_RESULT")

                    .appendField("]");

                this.setPreviousStatement(true);

                this.setNextStatement(true);

                this.setColour(230);

            }

        };

        Blockly.Blocks['result_block'] = {

            init: function() {

                this.appendDummyInput()

                    .appendField(document.getElementById('targetFieldName').value)

                    .appendField(new Blockly.FieldTextInput(""), "RESULT_VALUE");

                this.setPreviousStatement(true); // Only connect from above

                this.setColour(0);

            }

        };

        function showBlockly() {

            document.getElementById('blocklyDiv').style.display = 'block';

            document.getElementById('confirmButton').style.display = 'block';

           

            workspace = Blockly.inject('blocklyDiv', {

                toolbox: document.getElementById('toolbox'),

                trashcan: true

            });

            // Add change listener

            workspace.addChangeListener((event) => {

                if (event.type === Blockly.Events.BLOCK_MOVE) {

                    const blocks = workspace.getAllBlocks();

                    let currentValue = '';

                   

                    blocks.forEach(block => {

                        if (block.type === 'source_block') {

                            currentValue = block.getFieldValue('SOURCE_VALUE');

                        }

                        if (block.type === 'transform_block') {

                            const transformedValue = 'PC-' + currentValue;

                            block.setFieldValue(transformedValue, 'TRANSFORM_RESULT');

                            currentValue = transformedValue;

                        }

                        if (block.type === 'result_block') {

                            block.setFieldValue(currentValue, 'RESULT_VALUE');

                        }

                    });

                }

            });

        }

        function confirmAndClose() {

            const blocks = workspace.getAllBlocks();

            let resultValue = '';

           

            blocks.forEach(block => {

                if (block.type === 'source_block') {

                    resultValue = block.getFieldValue('SOURCE_VALUE');

                }

                if (block.type === 'transform_block') {

                    resultValue = 'PC-' + resultValue;

                }

                if (block.type === 'result_block') {

                    block.setFieldValue(resultValue, 'RESULT_VALUE');

                }

            });

            document.getElementById('targetFieldValue').value = resultValue;

            document.getElementById('blocklyDiv').style.display = 'none';

            document.getElementById('confirmButton').style.display = 'none';

        }

    </script>

</body>

</html>

[Mark Friedman]

Hey, Chris, it's a cute idea and if this is really all you need and it works for you then that's fine!  We're a judgement free zone here ;-). That said, I'll point out a couple of potential gotchas.  They don't affect your basic idea, though.

In a couple of places, you're depending on the ordering of the blocks that are returned by getAllBlocks.  However, there is no guarantee that that order will be maintained by the Blockly implementation.  You can do a little better if you call getAllBlocks(true) instead (see here). That will, I believe, be ok if you only have one block stack.  If you have multiple block stacks then I'm not sure even getAllBlocks(true) will work for you.  A somewhat safer way to do it, and still be true to your idea, would be to call Block.getDescendants(true) on each of the top-level blocks, using Workspace.getTopBlocks().

Note, though, that if you ever add inputs (in the Blockly sense) or more complex programming logic (e.g. conditionals), your approach will need to get more complex and you'll probably need to start walking the blocks data structure in a way that Blockly's code generation support might help. In your case the code that is generated could potentially simply be evaluated to yield your resultValue.

-Mark

To view this discussion visit https://groups.google.com/d/msgid/blockly/e20bb9e7-2ee2-412b-9e61-701e57b3e0f1n%40googlegroups.com.

[Chris Rowan]

Mark.

This application will only require a single stack, but logic could become MUCH more complex and conditionals are possible. I am strongly allergic to hard coded cul-de-sac solutions, so you've given me some excellent food for thought! Sounds like it's time to peel back another layer or three? Thanks again, and I hope this response isn't too chatty for this forum. I suspect I'll be back soon as I shed my noob armor of ignorance!

To view this discussion visit https://groups.google.com/d/msgid/blockly/CAOk7GcRbuDUgy6A_fot0UVfZ4BcUhpzJ7bZYTtEoEgYBLj0bGA%40mail.gmail.com.

[Ronald Bourret]

Hi, Chris,

This is definitely a reasonable use of Blockly. The only heresy is that it sounds like you're planning to use the *result* of the transformation ("click button to return the final transformation result to the enclosing webpage") instead of the transformation itself. Editors built with Blockly are generally design-time tools that generate code, which is then run elsewhere.

Here's a possible way of doing this, along with suggested UI:

1) The UI has three panes: a JSON file showing a sample input, a JSON file showing a sample output, and a Blockly editor. The values in the JSON file are not important -- they're acting as schemas by example.

2) The user loads input and output files.

3) The user chooses an input field and an output field and clicks OK.

4) The application constructs a Blockly editor with three categories: Input fields, Output fields, and Transformation operations.

5) The user builds a transformation in a manner shown in your example and clicks Done.

6) The application generates the code for a transformation function from the top-level block. The function accepts an input object and an output object. It looks up the chosen field on the input object, transforms it as directed, and stores it on the output object.

6) The application stores the generated code in a transformation object whose structure matches the the structure of the input file.

7) Return to step 3 until all desired fields are mapped. The result of this is an object containing strings that represent transformation functions.

For example, if my input object is:

{

  a: "foo",

  b: 1

}

and my output object is:

{

  c: 3,

  d: "foobar"

}

My transformation object would be:

{

  a: 'function (input, output) {output.d = input.a + "bar";}',

  b: 'function (input, output) {output.c = input.b + 2;}'

}

8) The application then serializes this as a .js file, but with real functions, not strings:

const transformer = {

  a: function (input, output) {output.d = input.a + "bar";},

  b: function (input, output) {output.c = input.b + 2;}

};

Your code that transforms API calls uses this as follows:

1) Import the .js file from (8) using a dynamic import.

2) Load the input JSON and convert it to a JavaScript object.

3) Create an empty output object.

4) Traverse the input JS object. For each field, pass the input and output objects to the corresponding transformation function. This builds an output object one field at a time.

5) Convert the final output object to JSON and pass it to the outgoing API call.

Notes:

1) The input and output field blocks need to store the path of the field they represent (e.g. input.a) so they can be used in code. These could be stored in a separate field on the block using an extension.

2) The sample input and output JSON files need to show all possible fields, even if a particular combination of fields isn't allowed in the real world. This is so the user can map everything.

3) To combine and split fields, the user selects multiple input and/or output fields. The transformation function is stored in the first field; when the processor encounters the second field, there is nothing there, so the transformation isn't run a second time.

4) This doesn't play well with heterogenous arrays or array-level operations, such as sorting or reversing. I can't think of how the UI would work.

This is just one possible architecture, but if gives you an idea of how Blockly could be used.

Ronald Bourret

Technical Writer (Provided by Synergis)

rbou...@xwf.google.com

To view this discussion visit https://groups.google.com/d/msgid/blockly/CALt%3DRxvUU8HQ2oa6FDmr8btTMBHLaTuo0wM%2B861k1%2BaGP8OQXw%40mail.gmail.com.

[Chris Rowan]

Ronald and everyone,

I have seen the light over the past couple days and have prototypes generating code. I've also realized that it's ultimately the only truly effective way to do it anyway. So, I'm over that hump!

But wow Ron. Thanks so much for taking the time to lay out a solution design. It's a brilliant approach and you saved me a ton of brain racking and from wasted time going down non-viable design paths. The way you lay it out and describe it is so concise and eloquent. It's almost as if you write on technical subjects professionally... 😉🤣

Still being a noob and not knowing what I don't know, I'm sure I'll be back with more questions, but I feel like I am on a glide path now thanks to everyone's help!

Thanks again!

Chris

To view this discussion visit https://groups.google.com/d/msgid/blockly/CAFRdcLYEWXahgrBxznFRZFJbAyHZ5uKxbtPDVbznk96Kn4URYg%40mail.gmail.com.