Proposed Development Sprint: Sharing the MVP/CIEL dictionary

[Darius Jazayeri]

Hi All,

I'd like the development work that we do to be proposed more directly by the community. In that vein, wearing an I-TECH & PIH hat, I'd like to propose a development sprint. And hopefully others copy this template going forwards...

-Darius

Goals and Concrete Deliverables

To make it possible to subscribe to the MVP/CIEL concept dictionary, using a module that will automatically pull down new concepts whenever you request them, or via a scheduled task.

Prior Work

Currently the MVP/CIEL dictionary is a great resource, but it gets less use than I think it should since using it requires installing sql dumps to get the updates. Using a combination of the MCL and the Metadata Sharing module, it's possible to download individual concepts from the MVP/CIEL dictionary, and convert them between different OpenMRS versions. With a bit more work, we could automate this process.

Primary Benefits

If people can get updates from the MVP/CIEL dictionary automatically, without needing to get sql dumps from a dropbox, and manually install them, more implementations would be able to leverage that dictionary, and more implementations would be working with a well-curated dictionary.

Secondary Benefits

The work we'd do on this sprint would also allow other groups besides MVP/CIEL to publish their concept dictionaries and have clients subscribe to them.

Proposed Work

• In the Metadata Sharing module: META-224
• Create a "Metadata Subscription" module that depends on Metadata Sharing.
• You configure this module with a URL to subscribe to, a type of metadata, and a frequency to check.
• With a scheduled process, this module will contact an OpenMRS server running Metadata Sharing at the given URL, and download an on-the-fly package of all metadata (of the given type) with a date-modified after a given timestamp
• The module will save a timestamp of the last time it checked, so it only needs to download newly-modified items
• After downloading the package, it will be automatically installed, updating your metadata tables.
• TODO: decide if automatically installing the package is good enough (e.g. we can assume people aren't editing concepts locally), or if we should also support interactive installation of downloaded packages.
• For purposes of this sprint we only care about fetching concepts from MCL, but the module should support any kind of metadata from any url.
• (Optional, hopefully done before this sprint) Allow Metadata Sharing to convert concepts between the 1.7 and 1.9 data models. 

What Am I Asking For?

• One design call to finish defining tickets.
• Minimum of one week with two devs.
• (Optionally) several days of Rafal's time to get MDS converting 1.7 concepts <-> 1.9 concepts. (I think he needs to be the one to do this since it requires non-trivial MDS refactoring.)

What Can I Provide?

• I can lead the design call discussion, create a design page, and create tickets.
• If we schedule it after the end of June, I should be able to work on the sprint as a developer.
• Hopefully I can convince another dev or two to join in. :-)

[Michael Seaton]

Hi Darius,

I really like this format and the idea that implementations who are able to specify generally useful, and reasonably well-specified projects like this can provide the basis for many of our development sprints.  For this project in particular, I'd propose that we might want to think about whether we would want a mechanism for marking particular metadata as published, and thus a candidate for automatic download to a subscriber.  I'm thinking in particular of metadata that might take a fairly long period to define - like reports and htmlforms.  Those who subscribe to these would likely only want published versions, and not works-in-progress which are most likely broken and/or incomplete as they are being authored.  I'm happy to be a part of any ongoing design calls.

Cheers,
Mike

---

Click here to unsubscribe from OpenMRS Developers' mailing list

[Ben Wolfe]

This looks like a great template for implementers/devs to use.  Do we have this on the wiki already?

How will this be integrated with the summer of code project for the MDS Server? https://wiki.openmrs.org/display/projects/Metadata+Sharing+Server+Project

That seems like the module to add in the publishing functionality.  Subscribing could be a separate module or combined with MDS.

Can you pick a Wednesday that you are available for the design call?  Add your topic to a notes page and announce the date here for all interested parties to mark their calendars.
https://wiki.openmrs.org/display/RES/Design+Forum

Ben

[Joaquín Blaya]

Darius,

I would suggest a slightly different view point (at least it's what I'm learning now).

In talking with Ellen, I think she has a very good point that you wouldn't want your concept dictionary to be updated without you knowing what was going on, rather you should oversee the process on a test or concept server and if it works then move it over to your production server. What would be great to automate would be that step from having tested it on your test/concept server sending that concept dictionary to other servers that depend on you.

This comes from my current situation (which is similar to that of Ellen's) which is that we will have multiple OpenMRS servers that we are trying to maintain with the same concept dictionary.

Specifically I would love to be able to

a. Update a test server from MVP or other dictionaries by manually or automatically downloading an MDS import file

b. Being able to select which concepts I want to transfer from my test server to a concept server

c. Automatically have the concept server update all of the production instances (could be dozens or hundreds)

Do you requirements include this last step?

Joaquín
___________________________________________________________________
Gerente de Desarrollo, eHealth Systems
Research Fellow, Escuela de Medicina de Harvard
Moderador, GHDOnline.org

-- OpenMRS Developers: http://go.openmrs.org/dev
Post: d...@openmrs.org
Unsubscribe: dev+uns...@openmrs.org

[Andrew Kanter]

Can someone confirm how MDS updates existing concepts that have changed, either by changing synonyms or index terms or by changing maps to references?

Thanks

Andy

 

--------------------
Andrew S. Kanter, MD MPH

Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology

Columbia University
Email: andrew...@dbmi.columbia.edu
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter

---

From: Joaquín Blaya <joaqui...@hms.harvard.edu>
To: "d...@openmrs.org" <d...@openmrs.org>
Sent: Thursday, May 31, 2012 10:36 AM
Subject: Re: Proposed Development Sprint: Sharing the MVP/CIEL dictionary

[Darius Jazayeri]

Joaquin,

My proposal is only about that last step.

I've written it up as "download updates from MVP/CIEL" but really the implementation would allow you to automate getting updates from whatever your master concept server is.

(Metadata Sharing already lets you do (a) and (b), i.e. it lets you move *some* concepts from multiple source dictionaries to your working dictionary. MCL provides a great example of this, i.e. it lets you search, find concepts, and do an MDS export of those. In this proposed sprint we would not touch that functionality, though. For this scope of work, I'm only interested in the "complete sync from master to children" use case.)

-Darius

[Joaquín Blaya]

Sorry, one thing I forgot to add, is that I think we could do what I requested via the sync module, don't know what the benefits of one versus the other, I was thinking of doing it to have a single module instead of two.

Joaquín
___________________________________________________________________
Gerente de Desarrollo, eHealth Systems
Research Fellow, Escuela de Medicina de Harvard
Moderador, GHDOnline.org

[Joaquín Blaya]

Ok, great, well, we would definitely use that functionality.

Joaquín
___________________________________________________________________
Gerente de Desarrollo, eHealth Systems
Research Fellow, Escuela de Medicina de Harvard
Moderador, GHDOnline.org

[Darius Jazayeri]

Andy,

Are you asking whether MDS will work correctly if:

1. you start from the MVP/CIEL dictionary
2. you import an update to MVP/CIEL which:
1. adds a synonym
2. changes a mapping from SNOMED:1234 to SNOMED:5678

Good question. I think part of this sprint should involve writing integration tests of real-world scenarios like that.

(I originally wrote this answer, but think I misunderstood the point of your question:)

It merges what you have locally with what is incoming.

When you're importing a metadata package you choose between "peer-to-peer" and "parent-child" modes. IIRC in peer-to-per (e.g. someone from another implementation sent you a form) it merges the incoming changes into yours. In parent-child (e.g. a new update from MVP/CIEL) it merges your local version into the incoming one.

-Darius

[Andrew Kanter]

I was asking specifically about taking something from CIEL, then sending an update from CIEL... Unlike the overwrite method of taking all from CIEL, using MDS sounds like it might not be sufficient. Merging is not what is needed if the edits done on the CIEL side involved deleted information... I thought MDS allows for a complete replace option which if originally came from CIEL, this shouldn't be a problem, right?

Andy

 

--------------------
Andrew S. Kanter, MD MPH

Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology

Columbia University
Email: andrew...@dbmi.columbia.edu
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter

---

From: Darius Jazayeri <dar...@openmrs.org>
To: d...@openmrs.org
Sent: Thursday, May 31, 2012 5:33 PM

[Darius Jazayeri]

I don't think MDS supports a complete overwrite option, because IIRC we were worried about the case where you've actually used a name/answer/mapping/etc, so we only merged additively rather than deleting.

Rafal, is this right?

-Darius

[Mark Goodrich]

As far as I understand, Darius is correct, MDS always does a merge when dealing with properties of bbjects which are collections. 

 

Re: the example Darius lays out... if on the MVP/CIEL dictionary you delete concept mapping SNOMED:1234  and add mapping SNOMED:6789, when you update this update elsewhere you will end up with both concept mappings.

 

However, if you *changed* the mapping from SNOMED:1234 to SNOMED:6789 (ie, if you literally changed the concept_map object and maintained the uuid... this probably isn't possible via the UI) then I would think that when updating elsewhere the mapping will be changed.

 

Rafal can comment if I am incorrect... :)  

 

Mark

 

 

---

From: djaz...@gmail.com [djaz...@gmail.com] On Behalf Of Darius Jazayeri [dar...@openmrs.org]
Sent: Thursday, May 31, 2012 9:57 PM
To: d...@openmrs.org

[Mark Goodrich]

 

And the reasoning behind this logic is as follows...

 

Say you are using the MVP/CIEL dictionary and you decide to start using a module like the MDR-TB module that comes with it's own concept set.  Say when importing the MDR-TB concept set there is a "SMEAR RESULT" concept with a mapping "MDR-TB:SMEAR_RESULT", and instead of importing it you want to use an existing "SMEAR RESULT" concept in the MVP/CIEL dictionary.  You'd tell MDS to use your existing concept, and MDS would add the MDRTB mapping to your existing SMEAR RESULT concept.  The next time you import updates from the MVP/CIEL dictionary, you don't want that mapping to be blown away. 

 

(Note that in reality the MVP/CIEL already has all the MDR-TB mapping, so you should never need to do this... this is just an example)

 

Mark

 

---

From: d...@openmrs.org [d...@openmrs.org] On Behalf Of Mark Goodrich [mgoo...@pih.org]
Sent: Thursday, May 31, 2012 10:30 PM
To: d...@openmrs.org
Subject: RE: Proposed Development Sprint: Sharing the MVP/CIEL dictionary

[Rafal Korytkowski]

Mark and Darius you're correct. I'd add that the only way to "remove"
is to set voided or retired flag on an object. If UUIDs match then MDS
will also void/retire an object while importing.

If you change a name of a concept then an old name is automatically
voided and a new one is created (it is a core feature). It means that
it will behave correctly when you import an updated version of a
concept. MDS will also void an old name and create a new one.

The problem is with objects that don't have the void/retire flag such
as concept mappings. They're completely deleted from the db and MDS
will never do that while importing.

-Rafał

[Darius Jazayeri]

@Andy,

Does this happen in the MVP/CIEL dictionary? I.e. is an existing mapping ever removed or modified? Or do you ever remove a possible answer from a question?

-Darius

[Andrew Kanter]

Mappings definitely change. The map source can change (SNOMED CT to SNOMED NP, for example), or the reference code can change. A SNOMED CT code could become a SNOMED NP code and a new SNOMED CT code be added.

We rarely remove answers from questions, but certainly add them. There could be examples where an answer was incorrectly added and has to be removed...

Andy

 

--------------------
Andrew S. Kanter, MD MPH

Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology

Columbia University
Email: andrew...@dbmi.columbia.edu
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter

---

From: Darius Jazayeri <dar...@openmrs.org>
To: d...@openmrs.org

Sent: Friday, June 1, 2012 1:06 PM
Subject: Re: FW: Proposed Development Sprint: Sharing the MVP/CIEL dictionary

[Darius Jazayeri]

So it seems like to let MDS support this, we need to add "voided" columns to concept_set, concept_answer, and concept_map (or whatever it's called now).

As a workaround for the changing mappings, I guess we could make sure you see a UI that lets you *modify* a mapping rather than remove and create a new one.

I can't think of an easy solution for removing answers.

-Darius

[Mark Goodrich]

This sounds correct to me.