OntoServere LOINC Binary Index Issue

[prest...@prestonlee.com]

Jim/Michael,

FYI it looks like there is an issue with the upstream binary index of the LOINC 2.61 binary index .zip reference [1] in the alternate CSIRO syndication XML feed [2]. It 400's on every call. Is this a syndication/XML issue, or am I referencing the URI and version incorrectly? ("http://loinc.org" and "2.61", respectively)

[1] https://ontoserver.csiro.au/synd/getBinaryIndexZip?codeSystemId=http%3A%2F%2Floinc.org&codeSystemVersion=2.61&indexVersion=1.5.0
[2] https://ontoserver.csiro.au/synd/syndication.xml

Preston

The exception from the app logs is:
[snip]

The code system CodeSystemReference [id=LOINC-2.61, url=http://loinc.org, version=2.61] could not be indexed. 
au.csiro.ontoserver.exceptions.CodeSystemNotIndexableException: [c3f7e04e-acb7-40c3-8b24-3f1b73a187ce]: The code system CodeSystemReference [id=LOINC-2.61, url=http://loinc.org, version=2.61] could not be indexed. 
at au.csiro.ontoserver.OntoserverApi.indexCodeSystem(OntoserverApi.java:289) 
at au.csiro.ontoserver.web.OntoserverController.indexCodeSystem(OntoserverController.java:285)

[snip]

[Jim Steel]

Those parameter values are correct. We've got a little bit of flakiness in our syndication feed at the moment. I'm working on fixing the reasons behind it, but for the time being I've patched it up so it should be working.

Jim.

[prest...@prestonlee.com]

Jim,

It still doesn't seem to be working right. I can git the .zip manually, but when requested by the the v5.0.1 server, it seems to hang at the download step, here:

    2017-09-27 18:55:11.033Z INFO 1 --- [nio-8080-exec-2] au.csiro.ontoserver.atom.AtomManager : Attempting download from https://ontoserver.csiro.au/synd/getBinaryIndexZip?codeSystemId=http%3A%2F%2Floinc.org&codeSystemVersion=2.61&indexVersion=1.5.0

Preston 

[michael lawley]

Hi Preston,

What HTTP response do you get?

I've just tested the following:

curl -X POST 'http://localhost:8090/api/indexCodeSystem?codeSystemId=http%3A%2F%2Floinc.org&codeSystemVersion=2.61&validate=true'

And I get a 200 response with the body:

{

    "resourceType": "OperationOutcome",

    "issue": [

        {

            "severity": "information",

            "code": "informational",

            "diagnostics": "Code System http://loinc.org (2.61) is indexed"

        }

    ]

}

However, I don't see any additional (INFO level) log entries after the "Attempting download..." one.

michael

[Preston Lee]

curl works fine. It’s the OntoServer thread that hangs when trying to download from the CSIRO server. I have no visibility into what is happening though, so not sure if it’s an OntoServer bug or other problem.

-P

--
You received this message because you are subscribed to the Google Groups "HSPC Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform+u...@hspconsortium.org.
To post to this group, send email to plat...@hspconsortium.org.
To view this discussion on the web visit https://groups.google.com/a/hspconsortium.org/d/msgid/platform/37e56445-5430-43f1-935b-926ccb38cff9%40hspconsortium.org.

[Jim....@csiro.au]

This looks like the log of ontoserver working happily (there is not presently a log message generated after retrieving the file).

 

When you say the thread hangs, is Ontoserver unresponsive after that? Is the loinc code system available (e.g. in a CodeSystem search, or a $lookup operation)

 

 

Jim Steel PhD

Software Engineer

The Australian e-Health Research Centre

CSIRO

E jim....@csiro.au T +61 7 3253 3600 

Address: Level 5 - UQ Health Sciences Building, Royal Brisbane and Women's Hospital, Herston, QLD 4029 Australia

www.csiro.au | www.aehrc.com

 

PLEASE NOTE

The information contained in this email may be confidential or privileged. Any unauthorised use or disclosure is prohibited. If you have received this email in error, please delete it immediately and notify the sender by return email. Thank you. To the extent permitted by law, CSIRO does not represent, warrant and/or guarantee that the integrity of this communication has been maintained or that the communication is free of errors, virus, interception or interference. 

 

To view this discussion on the web visit https://groups.google.com/a/hspconsortium.org/d/msgid/platform/70CB8988-1021-4BF8-81E4-54678BAD0093%40asu.edu.

[Preston Lee]

Ok I’m seeing that now as well. I couldn’t get the server request handler to return a response to the client; I must not have been patient enough. :)  I see it showing up in the CodeSystem index now as well as some basic $lookup’s working as expected. Couple additional technical things before getting into authn/authz details…

 - A few of us have noticed that the JVM seems to like more max heap than whatever the default is. Could this be changed in a future patch release?

 - For SSL/TLS, the best practice we’ve established for HSPC core services is to let crypto exist at the proxy/balancer level and have service containers be blissfully unaware of layers 5/6 stuff. In other words, turn off SSL support at the application level completely and never give a container access to private .crt/.pem files. This is what we’re currently doing, and OntoServer is *correctly* generating URLs with the "https://“ prefix. Are URLs always generated from the “ontoserver.fhir.base” option, or are there situations where things like the HTTP Referer header is used? …or maybe any edge cases where something like X-Forwarded-For etc is needed? Is there anything we can do in configuration options to disable HTTPS support completely other than setting ONTOSERVER_INSECURE=true?

Preston

[Jim....@csiro.au]

Hi Preston

 

TLS:

• Yes, we’ve found that handling TLS in the cache/proxy is preferable.
• URLS for the FHIR API are generated using the ontoserver.fhir.base variable. URLS for the syndication api (e.g. links that appear in the syndication.xml file) are generated using the ontoserver.synd.base variable. If you don’t use these variables, then the request URLs are used.
• Disabling ONTOSERVER_SECURE should be sufficient to disable all TLS handling inside ontoserver

 

Memory:

• That memory limit is usually fine for anything except indexing – especially RF2 addition and indexing. Was that the activity that caused the problems?
• If you need to change it, its just a matter of putting a line in the docker-compose like ` - JAVA_OPTS=-Xmx56G` (in the environment section for the ontoserver container). 56G might be more than you need, of course ;)
• We tend to operate separate servers for indexing and “normal” operations. We only stand our indexing server up occasionally, for short periods, when a new code system needs to be indexed, or there’s a new version of ontoserver that necessitates or warrants a rebuild. The other server can then syndicate from the indexing server.

To view this discussion on the web visit https://groups.google.com/a/hspconsortium.org/d/msgid/platform/A230AAF5-6E44-492F-AB40-9C7E1EE13EDF%40asu.edu.

[prest...@prestonlee.com]

Re: Memory.. Ok, I'll try revert the heap back down to the default the next time it bounces and see what happens. It was probably just the /api calls that were causing problems. For potentially recursive operations such as $subsumes, is there any way of knowing (or establishing) an upper bound on heap allocation?

I'm hopeful this group doesn't have to establish a separate VM for indexing operations... assuming folks can provide them pre-built. :)

I still can't get reverse proxy path rewrites to work as I'd expect, so I'm just going to leave the defaults in place. It smells like the app server isn't detecting the route/path has been futzed with when generating URLs, but I didn't mess with it a ton.

Preston

 

Preston

 

 

 

-P

To unsubscribe from this group and stop receiving emails from it, send an email to platform+unsubscribe@hspconsortium.org.

To post to this group, send email to plat...@hspconsortium.org.
To view this discussion on the web visit https://groups.google.com/a/hspconsortium.org/d/msgid/platform/37e56445-5430-43f1-935b-926ccb38cff9%40hspconsortium.org.

-- 
You received this message because you are subscribed to the Google Groups "HSPC Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform+unsubscribe@hspconsortium.org.

To post to this group, send email to plat...@hspconsortium.org.
To view this discussion on the web visit https://groups.google.com/a/hspconsortium.org/d/msgid/platform/70CB8988-1021-4BF8-81E4-54678BAD0093%40asu.edu.

--
You received this message because you are subscribed to the Google Groups "HSPC Platform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform+unsubscribe@hspconsortium.org.

[prest...@prestonlee.com]

Hi Jim,

I was just updating us to 5.0.9 and was hoping to get a few additional indexes loaded. Could we get new binaries syndicated for LOINC 2.62 and SCT US edition? Appreciate your help!

We're still waiting to resolve our licensing issues holding up the service release, but I'm still trying to stay current in the interim.

Preston

[Preston Lee]

With CSIRO’s help, the September 2017 SNOMED CT US module and LOINC 2.63 terminology services are now live on the HSPC server!

Preston

> --
> You received this message because you are subscribed to the Google Groups "HSPC Platform" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to platform+u...@hspconsortium.org.

> To post to this group, send email to plat...@hspconsortium.org.

> To view this discussion on the web visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_a_hspconsortium.org_d_msgid_platform_6dc30a1a-2D6e64-2D43ce-2Da532-2D0a9f96a69898-2540hspconsortium.org&d=DwIBaQ&c=l45AxH-kUV29SRQusp9vYR0n1GycN4_2jInuKy6zbqQ&r=2tWwk6gOBLmYIR3lBn72E6jAn0Jrb63diTFAf1WGxIU&m=u9IYLzcjUd7znvjbm4bedNcdE3U5Q1MKhB1QIrF2-Xo&s=jtupGWfsgCO_QxHuLq6cepuhPlJiBBZZFsAgKqw9t-I&e=.