Flexible File loader

187 views
Skip to first unread message

odys...@sysnetint.com

unread,
Mar 23, 2017, 10:12:22 AM3/23/17
to openempi-su...@sysnetint.com
Posted: Aug 28, 2015 13:36 by moforiappiah
------------------------------------------------------------------------------
I ma trying to import data into openempi using the file-loader-map-connectathon-v3-preconectathon.xml schema but I get the following error:

2015-08-28 15:21:03,079]ERROR766098[pool-1-thread-2] - org.openhie.openempi.loader.FlexibleFileLoader.loadMappingConfiguration(FlexibleFileLoader.java:183) - Unable to initialize the flexible file loader; the mapping file is not valid: java.lang.NullPointerException
java.lang.NullPointerException
at java.util.regex.Pattern.<init>(Pattern.java:1336)
at java.util.regex.Pattern.compile(Pattern.java:1022)
at org.openhie.openempi.loader.FlexibleFileLoader.loadMappingConfiguration(FlexibleFileLoader.java:166)
at org.openhie.openempi.loader.FlexibleFileLoader.init(FlexibleFileLoader.java:137)
at org.openhie.openempi.loader.FileLoaderManager.loadFile(FileLoaderManager.java:65)
at org.openhie.openempi.jobqueue.impl.FileImportJobTypeHandler.run(FileImportJobTypeHandler.java:51)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2015-08-28 15:21:03,080] WARN766099[pool-1-thread-2] - org.openhie.openempi.jobqueue.impl.FileImportJobTypeHandler.run(FileImportJobTypeHandler.java:55) - Failed while trying to load a file: java.lang.RuntimeException: Unable to initialize the flexible file loader; the mapping file is not valid.
java.lang.RuntimeException: Unable to initialize the flexible file loader; the mapping file is not valid.
at org.openhie.openempi.loader.FlexibleFileLoader.loadMappingConfiguration(FlexibleFileLoader.java:184)
at org.openhie.openempi.loader.FlexibleFileLoader.init(FlexibleFileLoader.java:137)
at org.openhie.openempi.loader.FileLoaderManager.loadFile(FileLoaderManager.java:65)
at org.openhie.openempi.jobqueue.impl.FileImportJobTypeHandler.run(FileImportJobTypeHandler.java:51)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Can someone help. I have tried several times using the test data on this link but with no success

www.openempi.org/confluence/display/openempi30/Flexible+Data+Loader


Posted: Aug 31, 2015 19:27 by chrishi
------------------------------------------------------------------------------

It looks like there is something wrong with the delimiter that is used in your mapping file. Can you post the section of the mapping file you are using?

Chris

Posted: Sep 03, 2015 09:05 by moforiappiah
------------------------------------------------------------------------------
This is the mapping file details: I

<file-loader-map
xsi:schemaLocation="http://configuration.openempi.openhie.org/fileloadermap fileloadermap.xsd"
xmlns="http://configuration.openempi.openhie.org/fileloadermap"
xmlns:fl="http://configuration.openempi.openhie.org/fileloadermap"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
delimeter=","
header-first-line="true">
<fields>
<field
datatype="String"
one-to-many="true"
is-identifier="true"
delimeter=":">
<subfields>
<field
datatype="String">
<column-index>1</column-index>
<field-name>identifier</field-name>
</field>
<field
datatype="String">
<column-index>2</column-index>
<field-name>namespaceIdentifier</field-name>
</field>
<field
datatype="String">
<column-index>3</column-index>
<field-name>universalIdentifier</field-name>
</field>
<field
datatype="String">
<column-index>4</column-index>
<field-name>universalIdentifierTypeCode</field-name>
</field>
</subfields>
</field>
<field
datatype="String">
<column-index>2</column-index>
<field-name>familyName</field-name>
</field>
<field
datatype="String">
<column-index>3</column-index>
<field-name>givenName</field-name>
</field>
<field
datatype="String">
<column-index>4</column-index>
<field-name>mothersMaidenName</field-name>
</field>
<field
datatype="Date"
date-format-string="yyyyMMdd">
<column-index>5</column-index>
<field-name>dateOfBirth</field-name>
</field>
<field
datatype="String">
<column-index>6</column-index>
<field-name>gender</field-name>
</field>
<field
datatype="String">
<column-index>7</column-index>
<field-name>address1</field-name>
</field>
<field
datatype="String">
<column-index>8</column-index>
<field-name>city</field-name>
</field>
<field
datatype="String">
<column-index>9</column-index>
<field-name>state</field-name>
</field>
<field
datatype="String">
<column-index>10</column-index>
<field-name>postalCode</field-name>
</field>
<field
datatype="String">
<column-index>11</column-index>
<field-name>phoneAreaCode</field-name>
</field>
<field
datatype="String">
<column-index>12</column-index>
<field-name>phoneNumber</field-name>
</field>
<field
datatype="String">
<column-index>13</column-index>
<field-name>ssn</field-name>
</field>
</fields>
</file-loader-map>


The data is in this format:

29050909GrinHA:2.16.840.1.113883.3.72.5.9.1:ISO,Paul,Solomons,Rebecca,19761215,M,378 Camila Road,Lynwood,GP,40202,502,566-5945,401-25-4355

It works when I remove the identity subfields and hard code both the domain name and delimiter in the source code.


Posted: Sep 04, 2015 15:57 by mdaly
------------------------------------------------------------------------------
Yes I agree, I am currently running into this same issue. I need the mapping file to describe where to find the values in the imported data for the identifier domain attributes and dynamically assign them.

I have discovered that using the <subfields> element and using these child fields:

<field-name>universalIdentifier</field-name>
<field-name>namespaceIdentifier</field-name>
<field-name>universalIdentifierTypeCode</field-name>
<field-name>identifierDomainName</field-name>

does not work, and attempts to import a new identifier domain with null attributes which fails on SQL NULL constraints error. However, changing the mapping file to remove the <subfields> element and contents and instead assign the following attributes to the field element that represents the singular data import field:

is-identifier="true"
identifier-domain-name="SSN"
namespace-identifier="2.16.840.1.113883.4.1"
universal-identifier="2.16.840.1.113883.4.1"
universal-identifier-type-code="SSN"

The data import was changed to not have ':' delimited subfields for the identifier fields, and just have the IDs. This works, and imports the records. But the values must be hardcoded into the XML map on the server. How can we define a map to dynamically assign these values using imported data?


UPDATE:

I was able to get this working using the first method. In the import mapping XML file, I had to define the fields that contain the identifiers like this:

<field
datatype="String"
one-to-many="true"
is-identifier="true"
enclosing-character="&quot;"
delimiter=":">
<field-name>identifier</field-name>
<column-index>37</column-index>
<subfields>
<field datatype="String">
<column-index>1</column-index>
<field-name>identifier</field-name>
</field>
<field datatype="String">
<column-index>2</column-index>
<field-name>namespaceIdentifier</field-name>
</field>
</subfields>
</field>

This will add the data in the format "ID:OID" within the import record to the list of identifiers for the person record. The namespaceID values must correspond to an existing record in the identifier domains list.

rrli...@gmail.com

unread,
Jan 21, 2018, 6:54:41 PM1/21/18
to OpenEMPI Support Forum, odys...@sysnetint.com
Hi.

My file xml don´t work.

<?xml version="1.0" encoding="UTF-8"?>
delimiter=","
header-first-line="false">
    <fields>
        <field
            datatype="String"
            one-to-many="true"
            is-identifier="true"
            delimeter=":">
    is-ignored="true">
            <column-index>4</column-index>
            <field-name>nametypecode</field-name>
        </field>
        <field
            datatype="String">
            <column-index>5</column-index>
            <field-name>mothersMaidenName</field-name>
        </field>
<field
            datatype="String"
    is-ignored="true">
            <column-index>6</column-index>
            <field-name>nametypecode</field-name>
        </field>
        <field
            datatype="Date"
            date-format-string="yyyyMMdd">
            <column-index>7</column-index>
            <field-name>dateOfBirth</field-name>
        </field>        
        <field
            datatype="String">
            <column-index>8</column-index>
            <field-name>gender</field-name>
        </field>        
        <field
            datatype="String">
            <column-index>9</column-index>
            <field-name>address1</field-name>
        </field>
        <field
            datatype="String">
            <column-index>10</column-index>
            <field-name>city</field-name>
        </field>        
        <field
            datatype="String">
            <column-index>11</column-index>
            <field-name>state</field-name>
        </field>
        <field
            datatype="String">
            <column-index>12</column-index>
            <field-name>postalCode</field-name>
        </field>
        <field
            datatype="String">
            <column-index>13</column-index>
            <field-name>phoneNumber</field-name>
        </field>
        <field
            datatype="String"
    is-ignored="true">
            <column-index>14</column-index>
            <field-name/>
        </field>
        <field
            datatype="String">
            <column-index>15</column-index>
            <field-name>ssn</field-name>
        </field>
    </fields>
</file-loader-map>


Can someone help me?
Thank´s

rrli...@gmail.com

unread,
Jan 21, 2018, 6:54:46 PM1/21/18
to OpenEMPI Support Forum, odys...@sysnetint.com
Hi.

I've changed my xml file according to the post but it's don't work.

<?xml version="1.0" encoding="UTF-8"?>

delimiter=","

header-first-line="false">

    <fields>

        <field

            datatype="String"

            one-to-many="true"

            is-identifier="true"

            delimeter=":">

            <column-index>1</column-index>

            <field-name>identifier</field-name>



            <subfields>

                <field

                    datatype="String">

                    <column-index>1</column-index>

                    <field-name>identifierDomainName</field-name>
    is-ignored="true">

            <column-index>4</column-index>

            <field-name>nametypecode</field-name>

        </field>

        <field

            datatype="String">

            <column-index>5</column-index>

            <field-name>mothersMaidenName</field-name>

        </field>

<field

            datatype="String"

    is-ignored="true">

            <column-index>6</column-index>

            <field-name>nametypecode</field-name>

        </field>

        <field

            datatype="Date"

            date-format-string="yyyyMMdd">

            <column-index>7</column-index>

            <field-name>dateOfBirth</field-name>

        </field>        

        <field

            datatype="String">

            <column-index>8</column-index>

            <field-name>gender</field-name>

        </field>        

        <field

            datatype="String">

            <column-index>9</column-index>

            <field-name>address1</field-name>

        </field>

        <field

            datatype="String">

            <column-index>10</column-index>

            <field-name>city</field-name>

        </field>        

        <field

            datatype="String">

            <column-index>11</column-index>

            <field-name>state</field-name>

        </field>

        <field

            datatype="String">

            <column-index>12</column-index>

            <field-name>postalCode</field-name>

        </field>

        <field

            datatype="String">

            <column-index>13</column-index>

            <field-name>phoneNumber</field-name>

        </field>

        <field

            datatype="String"

    is-ignored="true">

            <column-index>14</column-index>

            <field-name/>

        </field>

        <field

            datatype="String">

            <column-index>15</column-index>

            <field-name>ssn</field-name>

        </field>

    </fields>

</file-loader-map>

Can someone help me?

Thank's.

chr...@sysnetint.com

unread,
Jan 30, 2018, 10:30:55 AM1/30/18
to OpenEMPI Support Forum
If you can send us a couple of lines from your data file, we can help you fix the mapping file. Make sure you modify the data in the file to not include PHI in the records.

rrli...@gmail.com

unread,
Feb 3, 2018, 12:01:48 PM2/3/18
to OpenEMPI Support Forum
Hi.

This is my datafile csv.

Patient ID,Family Name,Given Name,Name Type Code,Mother Maiden Name,Name Type Code,DOB,Admin Sex,Street,City,State,Zip,Home Phone,Account Number,SSN  

JN-018:NIST2010:2.16.840.1.113883.3.72.5.9.1:ISO,NXEAL,JERMAINO,L,HAWKINSX,L,19781013,M,3715 Red Hawk Road,LONGWOOD,FL,32750,PRN PH 321 2169372,2590014,210-78-0118

BL-180:NIST2010:2.16.840.1.113883.3.72.5.9.1:ISO,LXEIGHTON,BARBARA,L,BRYANTX,L,19460717,F,2674 Farland Avenue,SAN ANTONIO,TX,78205,PRN PH 830 4876570,2180484,321-48-9511

SN-032:NIST2010:2.16.840.1.113883.3.72.5.9.1:ISO,NXEAL,SHAQUILLO,L,HAWKINSX,L,19720306,M,2169 Warner Street,BOSTON,MA,2101,PRN PH 401 1549812,2184999,598-99-5899

RN-480:NIST2010:2.16.840.1.113883.3.72.5.9.1:ISO,NXEAL,RUTH,L,HAWKINSX,L,19361120,F,2222 Raver Croft Drive,HAMPTON,TN,37658,PRN PH 435 9485847,1990844,588-97-4411

DV-301:NIST2010:2.16.840.1.113883.3.72.5.9.1:ISO,VASXQUEZ,DAVID,L,WILSONX,L,19670305,M,1589 Brentwood Drive,HUTTO,TX,78634,PRN PH 512 7592000,1007240,148-81-8414

KM-375:NIST2010:2.16.840.1.113883.3.72.5.9.1:ISO,MARXCHANT,KATHRINE,L,MCDOWELLX,L,19421105,F,4670 Raver Croft Drive,HAMPTON,TN,37658,PRN PH 423 7259587,1840084,633-00-0104

GB-481:NIST2010:2.16.840.1.113883.3.72.5.9.1:ISO,BXRACK,GERALD,L,MCGEEX,L,19800130,M,3005 Tori Lane, SALT LAKE CITY,UT,84104,PRN PH 801 6054813,5819980,647-80-8974

DM-026:NIST2010:2.16.840.1.113883.3.72.5.9.1:ISO,MOORXE,DEMI,L,GUYNESX,L,19621111,F,3046 Pooh Bear Lane,GREENVILLE,SC,29607,PRN PH 864 6209874,94819416,655-03-2186

RM-489:NIST2010:2.16.840.1.113883.3.72.5.9.1:ISO,MOORXE,ROGER,L,PARKSX,L,19271014,M,4510 Park Street,PITTSBURGH,CA,94565,PRN PH 925 1459742,1588908,604-41-5879

MM-971:NIST2010:2.16.840.1.113883.3.72.5.9.1:ISO,MOORXE,MICHAEL,L,NEWMANX,L,19540423,M,4756 Stoney Lane,IRVING,TX,75063,PRN PH 972 9154876,1.85E+08,644-03-0711

OpenEMPI Support Forum

unread,
Feb 3, 2018, 12:54:36 PM2/3/18
to OpenEMPI Support Forum
Try the following map file that is attached to this post. This worked fine for me. There was a typo and a mapping for the identifier missing.

file-loader-map-site.xml
Reply all
Reply to author
Forward
0 new messages