Decrypted ndjson file is not valid JSON

[Billy Brown]

Using both swagger and curl to download a sample export via the user guides, I see the file beginning with: b' and ending with a '.  This is after I run decryption.   I tried removing these, but when I run a json check on the file I get errors for single quotes throughout the file, but they appear to be escaped \'

Does anyone have any suggestions?  Ultimately, I'm trying to load this data into a database.  I used emacs, textmate, and atom to view/edit the file.

[anthony smith]

I found the same thing and reported it on the workgroup a few weeks ago.

I ran through the file and removed the escaped quotes.   I needed to get the file so we could review the data content.

There should be two posts from me on the problems I found if you want to look.

Thanks

On Sat, 18 May 2019, 17:45 Billy Brown, <bbro...@gmail.com> wrote:

Using both swagger and curl to download a sample export via the user guides, I see the file beginning with: b' and ending with a '.  This is after I run decryption.   I tried removing these, but when I run a json check on the file I get errors for single quotes throughout the file, but they appear to be escaped \'

Does anyone have any suggestions?  Ultimately, I'm trying to load this data into a database.  I used emacs, textmate, and atom to view/edit the file.

--
You received this message because you are subscribed to the Google Groups "Developer Group - Beneficiary Claims Data API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bc-api+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bc-api/0657a919-e7bc-47e3-a559-2526fe6d6e7a%40googlegroups.com.

[Alex Hovancik]

Billy and Anthony,

Thanks for bringing this to our attention!  I'll be looking into this today and create a ticket in our internal tracking system.  I'd like to get a little more information from you first though.

How did you decrypt the file?  Did you use one of the scripts that we provided?  Which version of the programming language did you use to run the decryption script?  Any additional information you can think to include about the process that you went through would be helpful here as well.

Looking forward to hearing back from you!

Cheers,

Alex

To view this discussion on the web visit https://groups.google.com/d/msgid/bc-api/CAPqgrwx6Dd0zexr5AOOVH4_VXzC7ohUG4ZBhNvAQ2wkDUrmarw%40mail.gmail.com.

[Billy Brown]

Python 3.7.3 to decrypt.  Yes, I used the decrypt.py file provided on github.  Here is the full command I ran:

python decrypt.py \

--file aaad7bb8-45ec-4375-9b91-7d96a479df15.ndjson \

--pk ATO_private.pem \

--key  0159a646cce1d5d5ccefac42eb88e2c8a067db8dfe9020684abd43fce8f33016c9e4f189fbf21404b3e66fb5ff4ddf7a3683470b286fdd8738e22b8ff13d3aa0316702d23de5b1d3414b35668e97d7585bf53c962ed5e700f8e7b7556bbf0ea2eaaf8a91ce71100969adc3b3a73cc02f621c442f1c20ba57486aaf651a3c04e92af189f888b56114655a1f6559cb9f4cf3f4e2af51def3ea250a487a6193d0a05b64d127f32d44977db339ee12b8f8a9843603cdd0c30c7c492262455cb04e131edf1e3f45aa3ec88fb24bcde7dd82fe991ebe64f62eef6032d9d5edee881056e86efa14438ece6b6144f7af2fe47e08524ee06480e6271f899b58693c9c88e6 \

> decrypted_output.ndjson

It appears the decrypted file is a bytes object and the escape characters within it are hardcoded, eg \n and \' are just literal strings.  

On Monday, May 20, 2019 at 1:36:58 PM UTC-4, Alex Hovancik wrote:

Billy and Anthony,

Thanks for bringing this to our attention!  I'll be looking into this today and create a ticket in our internal tracking system.  I'd like to get a little more information from you first though.

How did you decrypt the file?  Did you use one of the scripts that we provided?  Which version of the programming language did you use to run the decryption script?  Any additional information you can think to include about the process that you went through would be helpful here as well.

Looking forward to hearing back from you!

Cheers,

Alex

On May 18, 2019, at 5:01 PM, anthony smith <aws...@gmail.com> wrote:

I found the same thing and reported it on the workgroup a few weeks ago.

I ran through the file and removed the escaped quotes.   I needed to get the file so we could review the data content.

There should be two posts from me on the problems I found if you want to look.

Thanks

On Sat, 18 May 2019, 17:45 Billy Brown, <bbro...@gmail.com> wrote:

Using both swagger and curl to download a sample export via the user guides, I see the file beginning with: b' and ending with a '.  This is after I run decryption.   I tried removing these, but when I run a json check on the file I get errors for single quotes throughout the file, but they appear to be escaped \'

Does anyone have any suggestions?  Ultimately, I'm trying to load this data into a database.  I used emacs, textmate, and atom to view/edit the file.

--
You received this message because you are subscribed to the Google Groups "Developer Group - Beneficiary Claims Data API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bc-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bc-api/0657a919-e7bc-47e3-a559-2526fe6d6e7a%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Developer Group - Beneficiary Claims Data API" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bc-...@googlegroups.com.

[Liani Lye]

Billy and Anthony — We are looking into this.  We would love to give everyone more information, but the BCDA Development Team is currently investigating.  We will report back once our information is complete.

Anthony — Thank you for your original post!  It was your post that sent us down this line of questioning initially.  Apologies for missing that, and thank you again.

All my best,
Liani

-- 

Liani Lye
Community Manager

Beneficiary Claims Data API

To unsubscribe from this group and stop receiving emails from it, send an email to bc-api+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bc-api/c3d02417-9555-433e-ac38-1f7acb17043d%40googlegroups.com.

[anthony smith]

I used the scripts you provided.   I'm out of the office this week.  I can't get the version until next Tuesday.   Thanks.

[Alex Hovancik]

Billy and Anthony,

Wanted to provide an update and a workaround for the ndjson issue you mentioned.  A ticket has been created in our internal tracking system and the work has been assigned to an engineer.  We expect to have a solution to the issue in about a week's time.

In the meantime, if you are willing and able, please try out one of the other decryption scripts we have provided (in C# and Go found here: https://github.com/CMSgov/bcda-app/tree/master/decryption_utils).  We would love to hear any feedback you have on using those.

Thanks again to you both for bringing this issue up to the google group.  Hope to hear from you again soon!

Cheers,

Alex

BCDA Engineer

[Liani Lye]

Hello Billy and Anthony,

A quick update:  We have identified the root cause regarding the decryption issue -- strange ‘a’ and ‘b’ prefixes and errors thrown during JSON check.  These issues occur when Python (3.7) is used; however, Python 2.7 works just fine.

As Alex mentioned, the other decryption scripts we have provided (in C# and Go found here) are verified to decrypt as expected.
Thank you for your patience!  In the meantime, our development team is working on a more solid Python solution to this issue.

All my best,
Liani

-- 

Liani Lye
Community Manager

Beneficiary Claims Data API

[Liani Lye]

Hello Billy and Anthony,

The BCDA engineering team has a more solid Python solution to the decryption issue, found at the BCDA File Encryption page.  The Python solution here should work for both Versions 2.7 and 3.4.  We would be incredibly grateful if either or both of you could try decrypting again.  If the issue persists, please let us know!