mg-dbx performance

[K.S. Bhaskar]

Chris –

In a Hardhats thread (https://groups.google.com/d/msg/hardhats/kYQFoMgjdXE/af1pMEdJBAAJ) Rob said that mg-dbx was an impressive 4-5× faster than Nodem. I asked what the workload was, and he said to ask in this group.

Also, what exactly was being measured. e.g, latency, updates/sec, etc.?

Thank you very much.

Regards

– Bhaskar

[Chris Munt]

Hi Bhaskar,

The performance measurements we have done are actually quite simple in that we're measuring things like the rate of global sets and gets.  Developers can relate to these sorts of benchmarks and they are easy for others to replicate.

As I understand it, Nodem is essentially an implementation of the InterSystems cache.node interface for YottaDB.  They both work to the same Node.js/V8 API (there is only one) and, as such, I believe (though I haven't done the experiment myself) the performance of these two solutions is in the same ballpark.  No surprises there.

mg-dbx is works differently to Nodem and cache.node and in designing this newer solution I spent a lot of time analyzing the performance at the Node.js/V8 API, particularly with a view to identifying areas where the performance was below par.  The mg-dbx solution seeks to limit use of the least performant parts of the V8 API while leaning more heavily on the parts that do perform well.  You can use mg-dbx in the same way as you would otherwise use cache.node (or Nodem) and, if you do, you will get only a marginal performance advantage over the older solutions.  However, if you use the newer mg-dbx APIs that make extensive use of embedded V8 classes then you can get a significant performance boost as, behind the scenes, mg-dbx is using the fastest parts of the V8 API, while, as much as possible, limiting reliance on the parts that I've found to be less optimal.

Regards,

Chris.

[K.S. Bhaskar]

Thanks for the explanation, Chris. What you say makes sense. Does mg-dbx support transaction processing?

Regards

– Bhaskar

[Chris Munt]

Well, the databases it connects to both support transaction processing so, by implication, mg-dbx supports it.  The DB environment is where Node.js developers will expect to find all the infrastructure for securely managing transactions.  And these are the developers that we are trying to reach.

Chris.

[K.S. Bhaskar]

Thanks for the answers, Chris. A further question: As YottaDB uses optimistic concurrency control, the API requires that you pass in a function that YottaDB calls to actually execute transaction logic. Since this is somewhat different from most databases (the low level API being https://docs.yottadb.com/MultiLangProgGuide/cprogram.html#ydb-tp-s-ydb-tp-st), I am wondering what the transaction abstraction is in the mg-dbx API and how it is the same for all supported databases. [I know that Caché does not implement Consistency and Isolation in its transactions, but that should not matter to the API, as far as I can imagine it.]

Regards

– Bhaskar

[chris....@gmail.com]

I appreciate that you have done a lot of good work in the areas of concurrency control and transaction management in YottaDB and mg-dbx does not as yet make use of the extended functionality you describe, I’m afraid.  However, mg-dbx is capable of calling functionality written in M code and presumably you can access all YottaDB’s good functionality there.

 

There’s an important point here.  Code managing concurrency and transactions will for obvious reasons contain some blocking operations and such operations are anathema to Node.js development.  Because of this, you really don’t want to handle concurrency and transactions directly in Node.js code but rather fire the code managing these operations asynchronously (i.e. by asynchronously calling an M function) and receive notification of the success (or otherwise) of the operation when the DB has done its work.

 

Regards,

 

Chris.

--
You received this message because you are subscribed to the Google Groups "Enterprise Web Developer Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to enterprise-web-develope...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/enterprise-web-developer-community/2d7aa459-fad5-40c9-afce-6f60ce383ff7o%40googlegroups.com.

[Rob Tweed]

The points Chris raises regarding blocking etc are important, and it's one of the key issues that QEWD's queue/worker process architecture is designed to solve.  

In QEWD, your JavaScript logic executes in a Node.js Worker Process within which it has exclusive access - ie unlike a "normal" Node.js single-threaded use-case, the QEWD Worker Process only runs a single use module and therefore issues of concurrency and problems caused by blocking are no longer relevant.  

It is the QEWD Worker Processes that actually use mg-dbx to connect to YottaDB (or IRIS/Cache). Your handler logic retains exclusive access to the assigned Worker Process until it decides it has finished, at which point it invokes a finished() function which returns your handler's results to the QEWD master process and releases the Worker Process back to the available pool for use by the next incoming request,

This means that QEWD + mg-dbx allows two things to safely happen, despite this being a Node.js environment:

- the mg-dbx APIs used to access and manipulate YottaDB globals can be synchronous (and therefore chainable).  The QEWD-JSdb abstraction and its associated APIs have been possible to build because of this;

- LOCK/UNLOCK and any M system-specific TP commands can be used from within JavaScript against the connected YottaDB database, because the user's code at that point has exclusive access to the attached M process for as long as it decides it needs that access.  Provided the LOCK and UNLOCK or the Transaction Start and End commands occur within the same handler module, then they can be used cleanly and without any impact on any other Node.js activity.

So, mg-dbx is just one piece of the puzzle - it's in combination with QEWD that JavaScript/Node.js becomes a serious substitute for M code, allowing anything you can do in M to also be do-able in JavaScript - something the many examples and repos I've been pushing out have been designed to demonstrate.

Rob

To view this discussion on the web, visit https://groups.google.com/d/msgid/enterprise-web-developer-community/00ee01d6ae0c%248c7d70c0%24a5785240%24%40gmail.com.

--

Rob Tweed
Director, M/Gateway Developments Ltd
http://www.mgateway.com

[Rob Tweed]

Question for Bhaskar: are the YottaDB-specific TP commands directly accessible via the YottaDB High-speed API interface?  If so, then in theory mg-dbx could be extended to include them in its API set.

And on that note, is there any chance of the MERGE command being added to the API?  As I understand it, it's not yet implemented, but would be a handy command to have direct access to from within Node.js/JS

[K.S. Bhaskar]

Thanks for the explanation Chris, and tutorial, Rob.

https://docs.yottadb.com/MultiLangProgGuide/cprogram.html#ydb-tp-s-ydb-tp-st describes the C API for transaction processing.

An API for Merge command is definitely on our wish list (https://gitlab.com/YottaDB/DB/YDB/-/issues/204 – there's even a proposed prototype). It's just waiting till we can spring loose a resource or one of our users springs loose a few dollars.

Regards

– Bhaskar

To unsubscribe from this group and stop receiving emails from it, send an email to enterprise-web-developer-community+unsubscribe@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/enterprise-web-developer-community/2d7aa459-fad5-40c9-afce-6f60ce383ff7o%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Enterprise Web Developer Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to enterprise-web-developer-community+unsubscribe@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/enterprise-web-developer-community/00ee01d6ae0c%248c7d70c0%24a5785240%24%40gmail.com.

--

Rob Tweed
Director, M/Gateway Developments Ltd
http://www.mgateway.com