Subscriptions: tag DATABASE, IN-MEMORY, off-line management and mail configuration for TLS

[Marcelo Cabello]

Hi,
I'm working on Subscription resources. My proof of concept consists in 2 Fhir Servers. First server acts as publisher and second as suscriber respectively.
I would like to ask about.

1.     What means the tag's inyection after create a Subscription resource with values IN-MEMORY and DATABASE. The tag refers to a matching-strategy?, therefore I guess there are relevance in the server's behavior.
2.     If server 2 is off-line for a while:
1. How can server 2 asks for new changes to server 1 to trigger the update via subscription?
2. If both servers have different history ID for the same resource (example: server1/Patient/1/history/5, server2/Patient/history/1 ¿how the second server update the information?
3. For Email channelType how can I configure the property in hapi.propertiy to set up mail though TLS? Should I add a property such as props.put("mail.smtp.starttls.enable", "true"); or am I doing a mistake?
   

Any advice, documentation is welcome!
Thanks!

{
    "resourceType": "Subscription",
    "id": "example-subscription",
    "meta": {
        "versionId": "1",
        "lastUpdated": "2019-12-06T10:24:37.747-03:00",
        "tag": [
            {
                "system": "http://hapifhir.io/fhir/StructureDefinition/subscription-matching-strategy",
                "code": "IN_MEMORY",
                "display": "In-memory"
            }
        ]
    },
    "status": "requested",
    "end": "2021-01-01T00:00:00Z",
    "reason": "New resource detection",
    "criteria": "Patient??active=true",
    "channel": {
        "type": "rest-hook",
        "endpoint": "http://localhost:8200/hapi-fhir-jpaserver/fhir",
        "payload": "application/fhir+json",
        "header": [
            "Content-Type: application/json"
        ]
    }
}

That's the error I got after configured email-subscription in hapi-properties using smtp and TLS connection:

2019-12-06 12:09:30.806 [subscription-delivery-email-mail-subscription-1] ERROR c.u.f.j.s.m.s.BaseSubscriptionDeliverySubscriber [BaseSubscriptionDeliverySubscriber.java:84] Failure handling subscription payload for subscription: Subscription/mail-subscription
org.springframework.mail.MailSendException: Failed messages: com.sun.mail.smtp.SMTPSendFailedException: 530 5.7.0 Must issue a STARTTLS command first. w125sm6772404vkh.50 - gsmtp

        at org.springframework.mail.javamail.JavaMailSenderImpl.doSend(JavaMailSenderImpl.java:490)
        at org.springframework.mail.javamail.JavaMailSenderImpl.send(JavaMailSenderImpl.java:360)
        at org.springframework.mail.javamail.JavaMailSenderImpl.send(JavaMailSenderImpl.java:355)
        at ca.uhn.fhir.jpa.subscription.module.subscriber.email.JavaMailEmailSender.send(JavaMailEmailSender.java:132)
        at ca.uhn.fhir.jpa.subscription.module.subscriber.email.SubscriptionDeliveringEmailSubscriber.handleMessage(SubscriptionDeliveringEmailSubscriber.java:79)
        at ca.uhn.fhir.jpa.subscription.module.subscriber.BaseSubscriptionDeliverySubscriber.handleMessage(BaseSubscriptionDeliverySubscriber.java:76)
        at org.springframework.messaging.support.ExecutorSubscribableChannel$SendTask.run(ExecutorSubscribableChannel.java:144)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

[Marcelo Cabello]

That's happened in FHIR JPA STARTER 4.1

[nmmou...@gmail.com]

Hi Marcelo,

I am trying to implement a similar project. I have completed implementing CRUD operations on a Subscription resource. what i dont understand is how do we continuously poll the database to know when your subscription criteria is met..? Like in your example what do you use to search your database continuously to understand a new patient is created or a new patient with this observation is created..?

Do we need to use interceptors for this..? if so do we have any reference implementation..?

[Kevin Mayfield]

I working on a similar issue. My big picture diagram is below.

I'm thinking I need to seperate out the subscription component and use a transaction feed (MQ) from HAPI JPA. 

This will go to a Pub/Sub component which processes subscriptions and sends messages to delivery channels (probably outside the subscription service). 
The RHS of the diagram will predominantly be Apache Camel plus a MQ component. 

I believe I need to write add interceptor in HAPI FHIR JPA and hook into one of the hooks https://hapifhir.io/hapi-fhir/docs/interceptors/server_pointcuts.html
I think that would be SERVER_PROCESSING_COMPLETED_NORMALLY 

For the Subscription component, I believe this is contained within this repo https://github.com/jamesagnew/hapi-fhir/tree/master/hapi-fhir-jpaserver-subscription but that may be a SmileCDR focused component. (It does seem to be an external component for handling subscriptions) 

[Ken Stevens]

Marcelo,

Here are the answers to your three questions:

1. The IN-MEMORY vs DATABASE tag is there as a convenience to tell you which matching strategy the server expects to use when it matches incoming resources against the criteria for that subscription.  IN-MEMORY subscriptions can be matched against inbound resources "in-memory" whereas the DATABASE ones need to make a call out to the database.  For example, criteria with chained search parameters or _has search parameters need to go out to the database to determine a match since the inbound resource doesn't contain all the information to determine a match.  For now, this distinction is mostly useful for performance tuning (since in-memory matching is much faster than calls out to the database.) . In the future, we plan to support stand-alone subscription matching servers that don't connect to the database and simply match subscriptions.  (You can see some tests where I roughed this out in ca.uhn.fhir.jpa.subscription.module.standalone.)

2. Subscription matching currently happens inside the jpa-server via interceptors and queues.  Normally, the matching resource sent to a rest endpoint is the one that was matched in the interceptor.  However, if you always need to send the most recent version, you can set the "http://hapifhir.io/fhir/StructureDefinition/subscription-resthook-deliver-latest-version" extension on the subscription.

3. TLS is not currently supported in the email endpoint.  However, this morning, I created a couple of pull-requests to add support for it.  Let me know if you're available to help test it.  Here are the Pull-requests that implement TLS for email subscription:

https://github.com/jamesagnew/hapi-fhir/pull/1666

https://github.com/hapifhir/hapi-fhir-jpaserver-starter/pull/75

Ken