Adding new functionality to ewd-document-store

[Josef Zvonicek]

Hi

I would like to do some changes in  git ewd-document-store repository.

This changes will allow use my stream data saved in M globals.

This stream data are stored similarly to array but with one important difference.

There is not array item counter in parent node.

^array=5
^array(1)="one"
^array(2)="two"
^array(3)="three"
^array(4)="four"
^array(5)="five"

^stream(1)="<b>Hello"
^stream(2)=" "
^stream(3)="world"
^stream(4)="!</b>"

Is very important to project these globals differently to/from JSON.

Is expected this projection:

{
   "array":[
      "one",
      "two",
      "three",
      "four",
      "five"
   ],
   "stream":"<b>Hello world!</b>"
}

I didn't find a way to achieve such functionality with your implementation of ewd-document-store.I was extend your implementation of getDocumen.js and setDocument.js (I hope with backward compatibility. :-)

There is added parameter useStream which should turn on the described functionality Is possible to apply my override of these files ?Or Is there any other way or chance how to use your repository with my  functionality .

Thanks Josef.

Original getDocument.js

Modified getDocument.js

Original setDocument

Modified setDocument

[rtweed]

Josef

A key thing you need to understand about ewd-document-store / QEWD-JSdb is that, conceptually, it's a JavScript-centric perspective on how M globals could be used to represent persistent JSON structures.

What it most definitely isn't intended to be is an M-centric perspective of how M globals could be used to map back into JavaScript objects.  The problem with such an M-centric perspective is that M developers have dreamt up all manner of ways of using M global structures to represent data formats.  This is clearly exemplified by your example where you've arbitrarily (though not unreasonably) decided to represent a stream of data as a set of global nodes with an integer subscript for each stream element.  Your starting point is a set of M Global structures that you've designed to represent certain data structures, and you're wanting ways of manipulate them equivalently in JavaScript.  As such, you're coming at QEWD-JSdb/ewd-document-store from the wrong end.

What you're suggesting is therefore not something I'm prepared to do - ie modify ewd-document-store to cater for a specific use-case that is relevant to just your use of global structures.  It doesn't provide something that is generically useful for a JavaScript-centric developer wishing to map JavaScript objects to and from persistent storage provided by M globals.

That having been said, QEWD-JSdb / ewd-document-store nevertheless provides all the APIs needed to specifically do what you want - you'll just have to write your own methods using them to handle your M global structures in a meaningful way in JavaScript.

So your first data structure is handled automatically, ie starting with your M global:

^array=5
^array(1)="one"
^array(2)="two"
^array(3)="three"
^array(4)="four"
^array(5)="five"

let doc = this.db.use('array');

let arr = doc.getDocument(true, 1);  // the second argument tells getDocument that your numbering starts with 1 rather than the default of 0

will return arr =  ["one", "two", "three", "four", "five"]

Note that ^array=5 is ignored by getDocument().

To convert the other way to how you're representing the array in your global structure:

let arr =  ["one", "two", "three", "four", "five"];

let doc = this.db.use('array');

doc.delete();

doc.setDocument(arr, 1);

doc.value = arr.length;

Note that last step to manually add the length into your global - this can be done without upsetting any of the lower nodes in the global tree.

So now your stream global.  To convert it to what you want in JavaScript, starting with your global:

^stream(1)="<b>Hello"
^stream(2)=" "
^stream(3)="world"
^stream(4)="!</b>"

let doc = this.db.use('stream');

let arr = doc.getDocument(true, 1);

let stream = arr.join('');

Going the other way around, you'll need to chop your stream into an array of strings, then simply:

let doc = this.db.use('stream);

doc.delete();

doc.setDocument(arrayOfStrings, 1);

In summary, your use-case is specific to your own personal, specific M-centric requirements.  It can be easily handled in ewd-document-store, and isn't something that warrants or requires further changes or enhancements to ewd-document-store.

[Josef Zvonicek]

Hi Rob,

thank you for your answer.  I understand all what you wrote.

 But I'm sorry my example was very simple to be understand my situation.

We have  very very complex structure of globals.  There is a little complex global.

TREE>zw ^arrstream

^arrstream("arr")=5

^arrstream("arr",1)="one"

^arrstream("arr",2)="two"

^arrstream("arr",3)="three"

^arrstream("arr",4)="four"

^arrstream("arr",5)="five"

^arrstream("arr1")=2

^arrstream("arr1",1,"arr")=5

^arrstream("arr1",1,"arr",1)="one"

^arrstream("arr1",1,"arr",2)="two"

^arrstream("arr1",1,"arr",3)="three"

^arrstream("arr1",1,"arr",4)="four"

^arrstream("arr1",1,"arr",5)="five"

^arrstream("arr1",1,"stream",1)="<b>Hello"

^arrstream("arr1",1,"stream",2)=" "

^arrstream("arr1",1,"stream",3)="world"

^arrstream("arr1",1,"stream",4)="!</b>"

^arrstream("arr1",2,"item")="OK!"

^arrstream("stream",1)="<b>Hello"

^arrstream("stream",2)=" "

^arrstream("stream",3)="world"

^arrstream("stream",4)="!</b>"

I don't see any way how can i serialize this global to JS object by getDocument method as i suppose. There is no limitation of stream size . It can be >10MB  (some picture, PDFs, ...)

{

   "arr":["one", "two", "three", "four", "five" ],

   "arr1":[ 

      {

         "arr":[ "one", "two", "three", "four", "five" ],

         "stream":"<b>Hello world!</b>"

      },

      {

         "item":"OK!"

      }

   ],

   "stream":"<b>Hello world!</b>"

}

When i use 

let doc = this.db.use('arrstream);

let data = doc.getDocument(true, 1)

the data variable is JS object and i don't know if array object is really  array or if it is stream.

To explain we have been working with M technologii for 30 let yet. Our application is based on our object model. These object instances contain complex data types, array, lists with this syntax as i wrote. I don't able to devide serialize to many steps. I need to serialize one global with array and stream nodes to JS in one step. I don't have any sugestion how do it by getDocument method with current functionality .

For me, the only solution is to create a client-side JS data model  for which  JS object from ewd  will be source  and where each property will have its own data type and getter and setter  that will be able to convert any array using the join/split method for stream data type.

If you don't have any idea  we can close this issue. 

Thank you for help.

Josef.

Dne středa 21. října 2020 v 14:09:32 UTC+2 uživatel rtweed napsal:

[Rob Tweed]

Josef

I can think of ways to achieve what you want using ewd-document-store APIs.  I guess the key thing is to identify any global leaf nodes that have an integer value and whose first child node is a leaf node with a last subscript of 1 - these must be handled as arrays.  Otherwise, a leaf node with a subscript of 1 (ie without a parent node that has an integer value) denotes an element of a stream and therefore needs joining in the corresponding JavaScript property.

To be honest, IMO it's not a particularly convenient/computable global representation to have used for the eventual JavaScript structure you want to recreate from it, but I'll put together how I'd parse it using ewd-document-store in the next few days

Rob

[rtweed]

OK Here's a solution:

function isInt(value) {

  return !isNaN(value) && 

    parseInt(Number(value)) == value && 

    !isNaN(parseInt(value, 10));

}

var doc = this.db.use('arrstream');

var json = doc.getDocument(true, 1);

doc.forEachLeafNode(function(value, node) {

  if (parseInt(node.name) === 1) {

    let parentNode = node.parent;

    if (!parentNode.hasValue) {

      let obj = json;

      let lastIx = parentNode.path.length - 1;

      parentNode.path.forEach(function(ix, index) {

        if (isInt(ix)) ix--;

        if (index === lastIx) {

          obj[ix] = obj[ix].join('');

        }

        else {

          obj = obj[ix];

        }

      });

    }

  }

});

console.log(JSON.stringify(json, null, 2));

[Josef Zvonicek]

Rob thanks for inspiration. 

It works.

Josef.

Dne neděle 25. října 2020 v 15:18:26 UTC+1 uživatel rtweed napsal: