dozer file mapper reads the 1st record only and cannot find it by patient search

[Mostafa Elhefnawy]

Hi all,

I'm trying to upload and import the sample patient ID datasets however I can see it reads the 1st record only. Is it a line ending problem LF vs. CRLF? also this 1st record I cannot find it by searching for it on the openempi admin console.

Please find the "catalina.out" below.

Thanks and best regards,

[2020-07-07 18:00:56,044] INFO725851[pool-2-thread-3] - org.openhie.openempi.loader.SpringEntityLoaderManager.setupConnection(SpringEntityLoaderManager.java:75) - Will be doing an import instead of an add

[2020-07-07 18:00:56,045] INFO725852[pool-2-thread-3] - org.openhie.openempi.loader.SpringEntityLoaderManager.setupConnection(SpringEntityLoaderManager.java:83) - WIll skip the header line during import

[2020-07-07 18:00:56,045] INFO725852[pool-2-thread-3] - org.openhie.openempi.loader.SpringEntityLoaderManager.setupConnection(SpringEntityLoaderManager.java:91) - Will be doing an import for entity person

[2020-07-07 18:00:56,068] INFO725875[http-nio-8080-exec-9] - org.dozer.util.InitLogger.log(InitLogger.java:45) - Initializing a new instance of the dozer bean mapper.

[2020-07-07 18:00:56,074] INFO725881[pool-2-thread-3] - org.openhie.openempi.loader.ConcurrentFileLoader.init(ConcurrentFileLoader.java:66) - Initializing the Concurrent File Loader.

[2020-07-07 18:00:56,074] INFO725881[pool-2-thread-3] - org.openhie.openempi.loader.AbstractFileLoader.parseFile(AbstractFileLoader.java:152) - Parsing the file using character set encoding UTF-8

[2020-07-07 18:00:56,075] WARN725882[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[0]=1

[2020-07-07 18:00:56,075] WARN725882[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[1]=CHIP

[2020-07-07 18:00:56,075] WARN725882[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[2]=MOORE

[2020-07-07 18:00:56,075] WARN725882[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[3]=

[2020-07-07 18:00:56,075] WARN725882[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[4]=10 PINETREE

[2020-07-07 18:00:56,075] WARN725882[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[5]=

[2020-07-07 18:00:56,075] WARN725882[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[6]=WEBSTER

[2020-07-07 18:00:56,075] WARN725882[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[7]=63119

[2020-07-07 18:00:56,075] WARN725882[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[8]=MO

[2020-07-07 18:00:56,075] WARN725882[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[9]=19380224

[2020-07-07 18:00:56,075] WARN725882[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[10]=0

[2020-07-07 18:00:56,075] WARN725882[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[11]=555-555-5555

[2020-07-07 18:00:56,075] WARN725882[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[12]=

[2020-07-07 18:00:56,075] WARN725882[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[13]=0

[2020-07-07 18:00:56,075] WARN725882[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[14]=M

[2020-07-07 18:00:56,075] WARN725882[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[15]=White

[2020-07-07 18:00:56,075] WARN725882[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[16]=0000001&2.16.840.1.113883.4.357&2.16.840.1.113883.4.357&hl7

[2020-07-07 18:00:56,075] WARN725882[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[17]=PDQ113XX01&IHENA&1.3.6.1.4.1.21367.2010.1.2.300&ISO^LPDQ113XX01&IHELOCAL&1.3.6.1.4.1.21367.2010.1.2.310&ISO

[2020-07-07 18:00:56,075] WARN725882[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[18]=null

[2020-07-07 18:00:56,075] WARN725882[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[19]=null

[2020-07-07 18:00:56,076] WARN725883[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[20]=null

[2020-07-07 18:00:56,076] WARN725883[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[21]=null

[2020-07-07 18:00:56,076] WARN725883[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[22]=null

[2020-07-07 18:00:56,076] WARN725883[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[23]=null

[2020-07-07 18:00:56,076] WARN725883[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[24]=null

[2020-07-07 18:00:56,076] WARN725883[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[25]=null

[2020-07-07 18:00:56,076] WARN725883[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[26]=null

[2020-07-07 18:00:56,076] WARN725883[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[27]=null

[2020-07-07 18:00:56,076] WARN725883[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[28]=null

[2020-07-07 18:00:56,076] WARN725883[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[29]=null

[2020-07-07 18:00:56,076] WARN725883[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[30]=null

[2020-07-07 18:00:56,076] WARN725883[pool-2-thread-5] - org.openhie.openempi.loader.ConcurrentFileLoader$RecordParseTask.fixAttributeFields(ConcurrentFileLoader.java:158) - Fields[31]=null

[2020-07-07 18:00:57,065] INFO726872[pool-2-thread-3] - org.openhie.openempi.loader.ConcurrentFileLoader.shutdown(ConcurrentFileLoader.java:104) - Shutting down the concurrent file loader.

 

[Mostafa Elhefnawy]

Hi all,

I'm trying to upload and import the 2 patient identity datasets (test-data-6k.csv  and pixpdq_connectathon_patients.csv ) but I found the below logs in "catalina.out". I can see it reads the 1st record only. is it a line ending problem LF vs. CRLF? or something else. Also this 1st record I cannot find it by searching for it on the UI. Any response is appreciated.

Thanks.

[OpenEMPI Support Forum]

Mostafa,

You need to use the flexible file loader to load this data since the concurrent file loader uses a fixed file format. There is a blog entry that describes how to do this here:

https://www.openempi.org/2014/10/03/a-more-flexible-way-to-load-data-into-openempi/

Best,

sup...@sysnetint.com