retrieving triples from the triple store

[George Lilly]

with help and advice from Conor, I've got an RPC coded to do some simple retrieval of triples from the triple store.

GTM>d rpctrip^C0XGET1(.G,"gpl:lex/757-15159 ?p ?o")

gpl:lex/757-15159^^

s:23130 p: o:

d do2(.triprtn,zt,zi,nsub)

GTM>zwr G

G(1)="gpl:lex/757-15159^skos:relatedMatch^gpl:icd9/519_3"

G(2)="gpl:lex/757-15159^skos:altLabel^Mediastinal Disease"

G(3)="gpl:lex/757-15159^rdf:type^skos:Concept"

G(4)="gpl:lex/757-15159^skos:broader^gpl:lex/757_12-47"

G(5)="gpl:lex/757-15159^skos:prefLabel^Mediastinal Diseases"

GTM>d rpctrip^C0XGET1(.G,"?s skos:relatedMatch gpl:icd9/519_3")

^skos:relatedMatch^gpl:icd9/519_3

s: p:673 o:16610

d do1(.triprtn,zt,zi,npred,nobj)

GTM>zwr G

G(1)="gpl:lex/757-17854^skos:relatedMatch^gpl:icd9/519_3"

G(2)="gpl:lex/757-7439^skos:relatedMatch^gpl:icd9/519_3"

G(3)="gpl:lex/757-15159^skos:relatedMatch^gpl:icd9/519_3"

GTM>d rpctrip^C0XGET1(.G,"gpl:lex/757-17854 ?p ?o")

gpl:lex/757-17854^^

s:16611 p: o:

d do2(.triprtn,zt,zi,nsub)

GTM>zwr G

G(1)="gpl:lex/757-17854^skos:relatedMatch^gpl:icd9/519_3"

G(2)="gpl:lex/757-17854^rdf:type^skos:Concept"

G(3)="gpl:lex/757-17854^skos:broader^gpl:lex/757_12-47"

G(4)="gpl:lex/757-17854^skos:prefLabel^Other diseases of mediastinum"

GTM>d rpctrip^C0XGET1(.G,"gpl:lex/757-7439 ?p ?o")

gpl:lex/757-7439^^

s:18631 p: o:

d do2(.triprtn,zt,zi,nsub)

GTM>zwr G

G(1)="gpl:lex/757-7439^skos:relatedMatch^gpl:icd9/519_3"

G(2)="gpl:lex/757-7439^rdf:type^skos:Concept"

G(3)="gpl:lex/757-7439^skos:broader^gpl:lex/757_12-47"

G(4)="gpl:lex/757-7439^skos:prefLabel^Disease, mediastinum NEC"

for now, only one "?s ?p ?o" and no extra whitespace allowed, just a single space separating.

The call is this:

rpctrip(rtn,query,limit,offset) ; rpc to access triples with a query

 ;

and it accesses a routine called triples:

triples(triplertn,sub,pred,obj,graph,fary) ; returns triples

 

I'm testing the limit and offset functionality now for queries with a large number of triples returned..

the code is on the trac server.

gpl

[glilly]

... looks like limit and offset work...

GTM>d rpctrip^C0XGET1(.G,"?s rdf:type skos:Concept",10,0)

^rdf:type^skos:Concept
s: p:23 o:644
d do1(.triprtn,zt,zi,npred,nobj)
GTM>ZWR G
G(1)="gpl:lex/757_12-34^rdf:type^skos:Concept"
G(2)="gpl:lex/757_12-48^rdf:type^skos:Concept"
G(3)="gpl:lex/757_12-1000^rdf:type^skos:Concept"
G(4)="gpl:lex/757_12-37^rdf:type^skos:Concept"
G(5)="gpl:lex/757_11-6^rdf:type^skos:Concept"
G(6)="gpl:lex/757_12-184^rdf:type^skos:Concept"
G(7)="gpl:lex/757_12-33^rdf:type^skos:Concept"
G(8)="gpl:lex/757_12-47^rdf:type^skos:Concept"
G(9)="gpl:lex/757_12-46^rdf:type^skos:Concept"
G(10)="gpl:lex/757_12-50^rdf:type^skos:Concept"


GTM>d rpctrip^C0XGET1(.G,"?s rdf:type skos:Concept",10,10)

^rdf:type^skos:Concept
s: p:23 o:644
d do1(.triprtn,zt,zi,npred,nobj)
GTM>ZWR G
G(11)="gpl:lex/757-134187^rdf:type^skos:Concept"
G(12)="gpl:lex/757-134800^rdf:type^skos:Concept"
G(13)="gpl:lex/757-17931^rdf:type^skos:Concept"
G(14)="gpl:lex/757-129570^rdf:type^skos:Concept"
G(15)="gpl:lex/757-130999^rdf:type^skos:Concept"
G(16)="gpl:lex/757-11501^rdf:type^skos:Concept"
G(17)="gpl:lex/757-139550^rdf:type^skos:Concept"
G(18)="gpl:lex/757-21398^rdf:type^skos:Concept"
G(19)="gpl:lex/757-17486^rdf:type^skos:Concept"
G(20)="gpl:lex/757-154105^rdf:type^skos:Concept"


GTM>d rpctrip^C0XGET1(.G,"?s rdf:type skos:Concept",10,20)

^rdf:type^skos:Concept

s: p:23 o:644
d do1(.triprtn,zt,zi,npred,nobj)

GTM>ZWR G
G(21)="gpl:lex/757-24200^rdf:type^skos:Concept"
G(22)="gpl:lex/757-136935^rdf:type^skos:Concept"
G(23)="gpl:lex/757-136992^rdf:type^skos:Concept"
G(24)="gpl:lex/757-3870^rdf:type^skos:Concept"
G(25)="gpl:lex/757-138689^rdf:type^skos:Concept"
G(26)="gpl:lex/757-267^rdf:type^skos:Concept"
G(27)="gpl:lex/757-129126^rdf:type^skos:Concept"
G(28)="gpl:lex/757-127724^rdf:type^skos:Concept"
G(29)="gpl:lex/757-138024^rdf:type^skos:Concept"
G(30)="gpl:lex/757-154665^rdf:type^skos:Concept"

it executes the full query each time. I'm thinking of persisting the
results if limit and offset are passed to speed up subsequent
identical queries with a different offset...

gpl

[glilly]

I've added the RPC to the Remote Procedure file on vistaewd.net

OUTPUT FROM WHAT FILE: REMOTE PROCEDURE//
Select REMOTE PROCEDURE NAME: C0XTRIPLES
ANOTHER ONE:
STANDARD CAPTIONED OUTPUT? Yes// (Yes)
Include COMPUTED fields: (N/Y/R/B): NO// BOTH Computed Fields and
Record Number
(IEN)

NUMBER: 2412 NAME: C0XTRIPLES
TAG: rpctrip ROUTINE: C0XGET1
RETURN VALUE TYPE: ARRAY
INPUT PARAMETER: QUERY PARAMETER TYPE: LITERAL
MAXIMUM DATA LENGTH: 32000 REQUIRED: YES
SEQUENCE NUMBER: 1
INPUT PARAMETER: LIMIT PARAMETER TYPE: LITERAL
MAXIMUM DATA LENGTH: 20 REQUIRED: NO
SEQUENCE NUMBER: 2
INPUT PARAMETER: OFFSET PARAMETER TYPE: LITERAL
MAXIMUM DATA LENGTH: 20 REQUIRED: NO
SEQUENCE NUMBER: 3

[conor dowling]

George, does the RPC have an option to go with it so that a user can
get permission to invoke it? I'm pretty sure your CCR RPC had one (and
FMQL does too). I'd just have to add it as a secondary memory option
and then I could invoke the RPC from FMQLUSER (or any user).

The way here is the same as the way we discussed invoking the CCR RPC
from Python. Remember I sent you that code oh so long ago (I can't
attach in this post??) but I found the file on my old box and the
calls were ...

# ##### 1 invoke RPC with permission (access/verify for user
with the correct option)
ccrRPCConnection = RPCConnection(args[0], int(args[1]), args[2],
args[3], "C0C CCR RPC", -1, CCRRPCLogger())

# ##### 2. ADD ARGS
patientId = str(args[4])
ccrArgs = [patientId]
# SEE C0CPARMS FOR A COMPLETE LIST OF SUPPORTED PARAMETERS
# FORM "PARM1:VALUE1^PARM2:VALUE2"
if len(args) > 5:
ccrArgs.append(args[5])
print "There are C0CPARMS: %s" % args[5]
if len(args) > 6:
ccrArgs.append(args[6])
print "There are C0CPARTs: %s" % args[6]

# ##### 3. Invoke
print "Invoking CCR RPC for patient %s ..." % patientId
reply = ccrRPCConnection.invokeRPC("C0C CCR RPC", ccrArgs, False,
False)

Sam used a variation of this when playing with his scheduler from
Python and FMQL has it too. No need to fork. All its needs is a
different config.

First I think I need that option.

[George Lilly]

Conor:

You're right. I forgot that part. I created the Option you can use:

You may need to add it to your secondary menu as you say.

Let me know if it works.

gpl

Select OPTION to edit: C0XTRIPLES       C0X TRIPLE STORE TRIPLES QUERY

NAME: C0XTRIPLES//

MENU TEXT: C0X TRIPLE STORE TRIPLES QUERY  Replace

PACKAGE:

OUT OF ORDER MESSAGE:

LOCK:

REVERSE/NEGATIVE LOCK:

DESCRIPTION:

  No existing text

  Edit? NO//

TYPE: Broker (Client/Server)//

HEADER:

ENTRY ACTION:

EXIT ACTION:

Select RPC: C0XTRIPLES//

  RPC: C0XTRIPLES//

  RPCKEY:

  RULES:

Select RPC:

Select ITEM:

CREATOR: WVEHR,PATCH INSTALLER//

HELP FRAME:

[conor dowling]

George, I'm going to try this quickly now. A couple of small things on
the RPC but they won't effect the first test:

1) I notice FMQL's return is a GLOBAL ARRAY, not an ARRAY. I think
that was because I needed to use TMP when building JSON - it can get
really big and Cache hit limits in some cases if I tried to use an in-
memory structure (I forget precisely but I think JSON in TMP and
Global Array were mixed up together)

NOTE: Dave Whitten mentioned XTmp too.

2) I think it best to have four parameters (besides limit offset), one
each for graph, subject, predicate, object and let all be optional.
"subject predicate object" as a sequence isn't really a "query" - it's
more of a template to match. I think as an RPC distinct arguments
makes more sense.

On Web calling format:
I think query args that match this RPC (and any others that come
along) ?op=triples&&graph=X&&subject=Y&&object=Z etc. ie/ use query
args. Again that's easy and intuitive for REST users. "op" is the
operation. It's best to use this now so that you could add operations
like "subjects", "objects", "value" etc if you want to later.

Finally on making replies:
In FMQL's reply implementation, I used a "Builder Pattern" which
separates reply building from query processing. In theory the same
walker could be reused for different output formats.

I know this may be too OO and not MUMPsy enough but as an approach it
means it should be easy to support all of the output forms with the
same walker. I'd need to abstract it some more but the idea would be

ASSERT^REPLY(OUTPUT,PRED,VALUE,TYPE)

leads to OUTPUT with either

PRED: {"value": VALUE, "type": ...} ... JSON
or
<PRED> VALUE ... XML
or
...

depending on the output formatter/serializer chosen - MUMPS seems to
love indirection so it's probably easy to make "REPLY" dynamic? ie/ to
make the exact routine used to serialize dynamic or is that not
possible?

Conor

> <conor-dowl...@caregraf.com>wrote:

[conor dowling]

George,

just FYI, I browsed the triples files in FMQL and they show up in the
FOIA vs WorldVistA audit file ( http://www.caregraf.org/fmql/reports/filesFOIAvsWVAudit.html#WORLDVISTA
):

172.101 C0X TRIPLE 1831 95974 [MISMATCH]
172.201 C0X STRING 632 67091 [MISMATCH]

FileMan's count for these files is way off their actual content? 1831
vs 95974? Any reason for that?

[conor dowling]

the basic call seems to work ...

# VERY BASIC GEORGE TEST
connection = VistARPCConnection(args[0], int(args[1]),
args[2], args[3], "C0XTRIPLES", RPCLogger())
reply = connection.invokeRPC("C0XTRIPLES", ["?s rdf:type
skos:Concept"])
print reply
return

It's in /home/conor/testTriplesRPC.py on vistaewd.net ... type: python
testTriplesRPC.py

If you run it, you'll see:

a) the first example: all things with types, takes quite a while
though it is a big list. This needs profiling.
b) the second is only a bit faster - limit 20 - but it gives back the
list of 20 twice
c) the last example ... fails (BROKERRPC -- RPCConnection Forced to
reconnect connection -1 after reply failed (empty reply)). It's
probably bad syntax on my part - how do you pass a literal?

It just needs profiling and test code and I do think it best to break
the arguments down like you did in the CCR RPC. That way, it is easy
to leave out "subject" or "object" in the template sent,

Conor

[George Lilly]

Conor:

I've reworked the RPC to separate out the parameters. They are all optional. It should be easier to pass strings with blanks in them now. Please give it a try.

OUTPUT FROM WHAT FILE: REMOTE PROCEDURE//

Select REMOTE PROCEDURE NAME:    C0XTRIPLES

ANOTHER ONE:

STANDARD CAPTIONED OUTPUT? Yes//   (Yes)

Include COMPUTED fields:  (N/Y/R/B): NO// BOTH Computed Fields and Record Number

 (IEN)

NUMBER: 2412                            NAME: C0XTRIPLES

  TAG: rpctrip                          ROUTINE: C0XGET1

  RETURN VALUE TYPE: ARRAY

INPUT PARAMETER: LIMIT                  PARAMETER TYPE: LITERAL

  MAXIMUM DATA LENGTH: 20               REQUIRED: NO

  SEQUENCE NUMBER: 5

INPUT PARAMETER: OFFSET                 PARAMETER TYPE: LITERAL

  MAXIMUM DATA LENGTH: 20               REQUIRED: NO

  SEQUENCE NUMBER: 6

INPUT PARAMETER: GRAPH                  PARAMETER TYPE: LITERAL

  MAXIMUM DATA LENGTH: 240              REQUIRED: NO

  SEQUENCE NUMBER: 1

INPUT PARAMETER: SUBJECT                PARAMETER TYPE: LITERAL

  MAXIMUM DATA LENGTH: 240              REQUIRED: NO

  SEQUENCE NUMBER: 2

[George Lilly]

oops forgot the rest of it:

OUTPUT FROM WHAT FILE: REMOTE PROCEDURE//

Select REMOTE PROCEDURE NAME:    C0XTRIPLES

ANOTHER ONE:

STANDARD CAPTIONED OUTPUT? Yes//   (Yes)

Include COMPUTED fields:  (N/Y/R/B): NO// BOTH Computed Fields and Record Number

 (IEN)

NUMBER: 2412                            NAME: C0XTRIPLES

  TAG: rpctrip                          ROUTINE: C0XGET1

  RETURN VALUE TYPE: ARRAY

INPUT PARAMETER: LIMIT                  PARAMETER TYPE: LITERAL

  MAXIMUM DATA LENGTH: 20               REQUIRED: NO

  SEQUENCE NUMBER: 5

INPUT PARAMETER: OFFSET                 PARAMETER TYPE: LITERAL

  MAXIMUM DATA LENGTH: 20               REQUIRED: NO

  SEQUENCE NUMBER: 6

INPUT PARAMETER: GRAPH                  PARAMETER TYPE: LITERAL

  MAXIMUM DATA LENGTH: 240              REQUIRED: NO

  SEQUENCE NUMBER: 1

INPUT PARAMETER: SUBJECT                PARAMETER TYPE: LITERAL

  MAXIMUM DATA LENGTH: 240              REQUIRED: NO

  SEQUENCE NUMBER: 2

INPUT PARAMETER: PREDICATE              PARAMETER TYPE: LITERAL

  MAXIMUM DATA LENGTH: 240              REQUIRED: NO

  SEQUENCE NUMBER: 3

INPUT PARAMETER: OBJECT                 PARAMETER TYPE: LITERAL

  MAXIMUM DATA LENGTH: 240              REQUIRED: NO

  SEQUENCE NUMBER: 4

INPUT PARAMETER: STORE                  PARAMETER TYPE: LITERAL

  MAXIMUM DATA LENGTH: 240              REQUIRED: NO

  SEQUENCE NUMBER: 7

(parameter STORE will all you to specify which triple store to use, for when we have more than one... for now has no effect.)

[George Lilly]

The mismatch looks like a bug in my code.. I'll look into it.

gpl

[George Lilly]

I copied the python test routine and modified it to try out the new parameter structure. It sort of works, but I think I'm missing something. It's in /home/glilly/testTriplesRPC-gpl.py

It appears to be doing each request twice ... "reconnecting"

=== First 20 SKOS:Concept ===

gpl:lex/757_12-34^rdf:type^skos:Concept

gpl:lex/757_12-48^rdf:type^skos:Concept

gpl:lex/757_12-1000^rdf:type^skos:Concept

gpl:lex/757_12-37^rdf:type^skos:Concept

gpl:lex/757_11-6^rdf:type^skos:Concept

gpl:lex/757_12-184^rdf:type^skos:Concept

gpl:lex/757_12-33^rdf:type^skos:Concept

gpl:lex/757_12-47^rdf:type^skos:Concept

gpl:lex/757_12-46^rdf:type^skos:Concept

gpl:lex/757_12-50^rdf:type^skos:Concept

gpl:lex/757-134187^rdf:type^skos:Concept

gpl:lex/757-134800^rdf:type^skos:Concept

gpl:lex/757-17931^rdf:type^skos:Concept

gpl:lex/757-129570^rdf:type^skos:Concept

gpl:lex/757-130999^rdf:type^skos:Concept

gpl:lex/757-11501^rdf:type^skos:Concept

gpl:lex/757-139550^rdf:type^skos:Concept

gpl:lex/757-21398^rdf:type^skos:Concept

gpl:lex/757-17486^rdf:type^skos:Concept

gpl:lex/757-154105^rdf:type^skos:Concept

BROKERRPC -- RPCConnection Message of length 828 received in 4 chunks on connec

tion -1

gpl:lex/757_12-34^rdf:type^skos:Concept

gpl:lex/757_12-48^rdf:type^skos:Concept

gpl:lex/757_12-1000^rdf:type^skos:Concept

gpl:lex/757_12-37^rdf:type^skos:Concept

gpl:lex/757_11-6^rdf:type^skos:Concept

gpl:lex/757_12-184^rdf:type^skos:Concept

gpl:lex/757_12-33^rdf:type^skos:Concept

gpl:lex/757_12-47^rdf:type^skos:Concept

gpl:lex/757_12-46^rdf:type^skos:Concept

gpl:lex/757_12-50^rdf:type^skos:Concept

gpl:lex/757-134187^rdf:type^skos:Concept

gpl:lex/757-134800^rdf:type^skos:Concept

gpl:lex/757-17931^rdf:type^skos:Concept

gpl:lex/757-129570^rdf:type^skos:Concept

gpl:lex/757-130999^rdf:type^skos:Concept

gpl:lex/757-11501^rdf:type^skos:Concept

gpl:lex/757-139550^rdf:type^skos:Concept

gpl:lex/757-21398^rdf:type^skos:Concept

gpl:lex/757-17486^rdf:type^skos:Concept

gpl:lex/757-154105^rdf:type^skos:Concept

=== FAILS - Subject 'Mediastinal Diseases' ===

gpl:lex/757-15159^skos:prefLabel^Mediastinal Diseases

BROKERRPC -- RPCConnection Message of length 55 received in 1 chunks on connect

ion -1

gpl:lex/757-15159^skos:prefLabel^Mediastinal Diseases

[glilly]

Conor:

Where are you getting your fileman numbers for this analysis? Here are
the zero nodes on the globals:

GTM>zwr ^C0X(:,0)
^C0X(101,0)="C0X TRIPLE^172.101I^95986^95986"
^C0X(201,0)="C0X STRING^172.201^67096^67096"

On Jan 28, 10:15 pm, conor dowling <conor-dowl...@caregraf.com> wrote:
> George,
>

> just FYI, I browsed the triples files in FMQL and they show up in the

> FOIA vs WorldVistA audit file (http://www.caregraf.org/fmql/reports/filesFOIAvsWVAudit.html#WORLDVISTA

[glilly]

Conor:

On web calling format... can we use these as examples in addition to
the url parameters you are suggesting? I'd like to be able to just
change the port number on these calls and to talking to our
triplestore (well maybe change a little more than that but you get the
idea..)
the openrdf workbench is implemented all in javascript ... well
actually all in xlst... and should not be hard to support..

http://vistaewd.net:8980/openrdf-workbench/repositories/gpltest/explore?resource=%3Chttp%3A%2F%2Fsmart%2Fcole-susan.rdf%3E

http://vistaewd.net:8980/openrdf-workbench/repositories/gpltest/explore?resource=%3Chttp%3A%2F%2Fsandbox-
api.smartplatforms.org%2Frecords%2F1551992%2Fdemographics%3E

Here's a sample of what's returned: - pretty straightforward stuff

<?xml version='1.0' encoding='utf-8'?>
<?xml-stylesheet type='text/xsl' href='/openrdf-workbench/
transformations/explore.xsl'?>
<sparql xmlns='http://www.w3.org/2005/sparql-results#' xmlns:q='http://
www.openrdf.org/schema/qname#'>
<head>
<variable name='subject'/>
<variable name='predicate'/>
<variable name='object'/>
<variable name='context'/>
<link href='info'/>
</head>
<results ordered='false' distinct='false'>
<result>
<binding name='subject'>
<uri>http://sandbox-api.smartplatforms.org/records/1551992/
demographics</uri>
</binding>
<binding name='predicate'>
<uri q:qname='rdf:type'>http://www.w3.org/1999/02/22-rdf-
syntax-ns#type</uri>
</binding>
<binding name='object'>
<uri>http://xmlns.com/foaf/0.1/Person</uri>
</binding>
<binding name='context'>
<uri>http://smart/cole-susan.rdf</uri>
</binding>
</result>
<result>
<binding name='subject'>
<uri>http://sandbox-api.smartplatforms.org/records/1551992/
demographics</uri>
</binding>
<binding name='predicate'>
<uri q:qname='sp:zipcode'>http://smartplatforms.org/
terms#zipcode</uri>
</binding>
<binding name='object'>
<literal>74126</literal>
</binding>
<binding name='context'>
<uri>http://smart/cole-susan.rdf</uri>
</binding>
</result>

[glilly]

Conor:

What do you think of moving the graph parameter (also called "context"
by some) to the fourth instead the first parameter. so it would be
triples(subject,predicate,object,graph,limit,offset,store).. ?

gpl