Base64 encoding to get around 'expired' statuses

[Ryan Link (VA Lighthouse)]

Some API consumers uploading documents to the Benefits Intake API have reported unusually high rates of expired statuses, meaning the PUT within 15 minutes to the location provided by an initial POST request never made it through and thus closed out the GUID.

We have determined the cause of this issue for at least one consumer to be that the VA firewall is falsely flagging PDF content as a threat when it is encoded as binary.

We've updated our code and documentation to allow the encoding of documents for upload using base64 instead appears to have fixed the issue for some. You can find the instructions for doing the base64 approach in our official documentation under Optional Base64 Encoding at https://developer.va.gov/explore/benefits/docs/benefits?version=current Please comment here if you have encountered this issue.

Additionally, the expired status can result from a TLS connection closing due to a failed handshake as well. If you are seeing unusual numbers of TLS handshake failures in your logs, please comment here as well.

[Ann Graham]

We did experience this issue prior to implementing base64 encoding.

We still see an occasional "expired" message, but the rate has dropped to less than 1 percent of all submissions.

I have not noticed any TLS errors in our logs.

[Kostya Tkachuk]

Hi. Not sure if that's a good place to ask this kind of questions. If not. please redirect me to correct location.

I'm trying to implement base64 encoding. Basically what I do is taking MultipartFormDataContent object (which works splendidly and submission gets to received state in sandbox) read bytes from it and then convert to base64 string.

After that I preface the result string with "data:application/pdf;base64," and close with "==".

Request itself gets submitted successfully, but then I get a error status with message "Unexpected end of payload".

What am I doing wrong? Should only the attachment part be encoded as base64, not the whole FormData? Any suggestions what to try to resolve it?

Any help would be appreciated.

For what it's worth, API is written in .Net Core 3.1. I'm using default HttpClient and send base64 string as StringContent (which I'm sure worked for me before while sending requests with base64 encoded data).

Thanks.

[Ryan Link (VA Lighthouse)]

Hi Kostya,

You'll want to encode the whole payload in base64 rather than individual attachments.  There are some sample apps here that might help.

Note that the base64 payload must start with data:multipart/form-data;base64 and the multipart boundaries i.e. ------WebKitFormBoundaryVfOwzCyvug0JmWYo and ending ------WebKitFormBoundaryVfOwzCyvug0JmWYo-- must also be included.

Hope this helps - please let us know how you make out.

Ryan Link

Customer Success Manager & Developer Support Lead

VA Lighthouse API Program

a...@va.gov

Visit developer.va.gov for more on the VA API Program

Visit valighthouse.statuspage.io to subscribe to API status alerts

Visit GovDelivery to subscribe to the Lighthouse API newsletter

[Kostya Tkachuk]

Hi Ryan,

First of all, thank you a lot! With your explanation and samples I tried changing different parts of FormData string and finally figured that I've missed WebKitFormBoundary part from boundary string (though it did work for FormData upload).

And also, I prefaced my payload with "data:application/pdf;base64," instead of "data:multipart/form-data;base64,".

Either I'm stupid or documentation should be changed, because it's kinda misleading.

But in any case, thank you so much again for helping me to get this work!

[Ryan Link (VA Lighthouse)]

Hi Kostya,

Glad to hear this is working for you.  Please let us know if anything else comes up.

Cheers,

Ryan

[Ted Nguyen]

Kostya Tkachuk can you share your working C# or VB.NET code? I've tried different combinations and keep getting  response like missing type or unexpected end of payload.

[lope...@yahoo.com]

[Ryan Link (VA Lighthouse)]

Thanks for posting Dan!  Ted, was this helpful?