Kx Dashboads with TorQ gateway

[Alex Carson]

Hi,

Is anyone using Kx Dashboards and querying the TorQ gateway? The dashboard wraps the users query to handle the returned data and errors etc.. It doesn't seem to play well with the gateway model (sync or async). Perhaps it's user error on my part so wondered if anyone could confirm they have it working. Obviously it all works fine hitting an rdb or hdb directly but I'd like to leverage the functionality of the gateway.

Thanks,

Alex

Disclaimer: The information contained in this email, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorised review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. The views or opinions expressed are the author's own and may not reflect the views or opinions of Epoch Capital Pty Limited or associated companies.

[Chad AquaQ]

Hi there Alex,

We've had experience connecting kx dashboards and the TorQ gateway in our TorQ-Air project. The most simple way to get the two connected is to use a "relay" process that you can select as your data source in dashboards. From there you can open a handle to the gateway and query it as normal. The relay process here is simply a fresh q process on the same machine as the TorQ stack. 

This approach has some downsides as you can't use load balancing, but it's fine for testing whilst you build out your dashboard setup. We are working on adapting the gateway to make it suitable for use in production, and we'll get back to you shortly with updates. 

I've added an example below. So in our case here, kx dashboards, the relay and the TorQ stack are all running on the same machine, with the TorQ gateway on port 22247.

If you have any further questions, do let us know

- Chad

[Alex Carson]

Ok great, thanks for the example Chad, that makes sense. I'll go with this approach for now and look out for your updates in the future.

Thanks

[JTennyson]

Hi Alex,

We were looking into this and have added some code to the gateway which allows async queries to be sent while connected to the gateway via dashboards, removing the need to connect to a different process and opening a connection handle to the gateway. The changes can be seen here

https://github.com/AquaQAnalytics/TorQ/pull/371

The format of the queries should take the form of

.kxdash.dashexec["select from t";`rdb`hdb;raze]

(i.e. .kxdash.dashexec[thing to run; processes to run against; join function])

A few things to note:

a) This is not fully tested; it has been tried with the TorQ-Financial-Starter-Pack and appeared to be working as expected (as shown in the screenshots). Testing is still a work in progress. Examples of the functions used were

  .kxdash.dashexec["select from trade";`rdb`hdb;raze]

  .kxdash.dashexec["select sum size by sym from trade";`rdb`hdb;{sum x}]

  In both cases, dashboards is directly connected to the gateway port of the running TorQ stack.

b)if Kx change the format of the request, or if there are other ways that the dashboard queries data then we will have to adapt/extend for that

Hopefully this helps, if you run into any problems let us know.

Thanks,

James

[Alex Carson]

Thanks a lot for the quick turnaround, that looks perfect.

[Alex Carson]

I've been playing around with this and wondered if it could be extended to support parameterised queries? Often the query depends on some view state parameter(s) from the dashboard, you might have a query such as:

{[s] .kxdash.dashexec["select from trade where sym=`",s;`rdb`hdb;raze]}

Thanks,

Alex

[JTennyson]

Hey Alex,

The use of paramaterised queries and viewstates should be working. I think in the query you sent through, the viewstate is a symbol and therefore to work in that query, the viewstate would either need to be of type string or to do

{[s] .kxdash.dashexec["select from trade where sym=`",string s;`rdb`hdb;raze]}

where the viewstate s is of type symbol.

However, the best practice would be to avoid using strings when sending the queries and to send the query as a function if possible, in order to avoid the problems that may arise with conversions between strings and different datatypes. So, for example, the previous query if sent using a function rather than a string, would be

{[s] .kxdash.dashexec[({select from trade where sym in x};s);`rdb`hdb;raze]}

which saves having to go between data types.

Side note: I have noticed that querying through the gateway in this way can be a problem if the max rows set in kx dashboards is greater than the number of records available to be returned, so we will try to look into that.

Thanks

James

[Alex Carson]

Thanks James. That works.

[Alex Carson]

Hi,

I have a couple of follow up questions on this one:

1) In the latest TorQ release 4.1.1 kxdash.q sets .z.ps:{x@y;.kxdash.dashps y}@[value;`.z.ps;{{value x}}]. In your original prototype it was simply .z.ps:.kxdash.dashps. When we upgraded to 4.1.1 our dashboard stopped working until I reverted the .z.ps definition back to the previous one. I can't quite figure out why the new definition causes an issue, it looks sensible...

2) My second issue may well be resolved by the answer to the first. If I enable permissioning via the .pm functionality everything works as expected except access via the dashboard.

Thanks,

Alex

[Alex Carson]

To clarify my second question. When permissioning is enabled the dashboard stops working entirely, similar to the behaviour mentioned in (1). No errors in log nor the dashboard, just no data.

[Robert Coen]

We're aware of the issue with kxdash.q on the latest version of TorQ. The issue is caused by .z.ps on the gateway not being changed to .kxdash.dashps as required for the dashboards. We're working to resolve this issue but in the meantime I would recommend using the old definition for .z.ps until this has been fixed and tested.

Please let us know if there are any other issues you are having using kxdash with TorQ

Robert

[Alex Carson]

Ok thanks Robert, I'll look out for the fix in the future. If you could also check kxdash when permissioning is enabled that would be great. I'm confident permissioning is working as expected when hitting the gateway directly, just not via the kxdash.

On a slightly different topic I have been using the new data access API you've released, it's really helpful. It would be great if it could be added to kxdash too.

Thanks,

Alex

[Robert Coen]

Thank you for the suggestion, we're working on incorporating the data access API into our kxdash functions at the same time. I'm hoping by solving the first issue, we should solve the second issue with regards to permission but we will test this to make sure.

All the best

Robert

[JerLucid]

Hi Robert, just to revive this thread. Has the issue setting .z.ps to .kxdash.dashps been resolved now