Job expired after 46 hours

[Edisnel C. C.]

Hello BCDA team, 

I have been trying to download the files for this job: https://api.bcda.cms.gov/api/v2/jobs/66372

After 46 hours of execution, 59 files were downloaded out of a total of 125, but the job expired. When I check its status I got (with Python and Postman):

{

    "issue": [

        {

            "code": "exception",

            "details": {

                "coding": [

                    {

                        "code": "Not Found Error",

                        "display": "",

                        "system": "http://hl7.org/fhir/ValueSet/operation-outcome"

                    }

                ],

                "text": ""

            },

            "severity": "error"

        }

    ],

    "resourceType": "OperationOutcome"

}

- Postman is showing this status for the job: Status: 410 Gone 

When I try to download a file like this: https://api.bcda.cms.gov/data/66372/067368e7-c011-4e1e-b954-2f9c1214cb51.ndjson, this is the response: 404 page not found

Could you specify how long a job and the associated files are available for download?

Thank you very much for your help

[Beneficiary Claims Data API (BCDA) Community]

Hello!

Sorry for the trouble downloading the results from your recent job. Files expire after 24 hours. You will have 24 hours after the job completes to download the data file(s). Otherwise, the files will expire and you will need to restart the job.

Thanks,
The BCDA Team

[Edisnel C. C.]

Thank for you support

At the moment the job expired, the total size for downloaded files was 31.2 GB, each file with more than 500 MB.

Could you specify what is the maximum speed/rate at which it can be downloaded from the BCDA API? Maybe it is the machine from which I download the files, and we can improve the download speed.

Thank you very much

[Beneficiary Claims Data API (BCDA) Community]

Hello,

We recommend using the Accept-Encoding: gzip header to download compressed NDJSON files when using BCDA, especially if your downloads are taking more than 24 hours to download after the job completes. Additionally, you may download files concurrently rather than waiting for one download to complete before submitting your next request.

These tips should help to greatly improve download speeds from BCDA.

Thanks,
The BCDA Team

[Edisnel C. C.]

Hello,

I tried downloading the files concurrently and it's working fine.

Thank you very much