After restore, pipeline fails

[Michael Maley]

I am having a weird problem after restoring.

Pipeline A has a build stage and a deployment stage.

1. Take backup of Go Server via api call.

2. Change pipeline A (add another environment variable).

3. Run pipeline A (label 56) = success.

4. Run pipeline A again (label 57) = success.

5. Restore from the above backup following the documented steps to the letter.

6. Run pipeline A (label 56 again) = fails with a fetch verification error in the deployment stage.

[ERROR] Verification of the integrity of the artifact [primary/ExConsoleApp001.exe] failed. The artifact file on the server may have changed since its original upload.

7. Delete pipeline A's 57 folder in the artifactory.

8. Run pipeline A (label 57 again) = fails with the same error.

9. Run pipeline A (label 58, virgin label number) = success.

So the restore appears to work, but that is weird problem. I hate to run pipelines that will fail. just to burn label numbers. Not sure I would know what the next virgin label number is.

Any suggestions or thoughts?

Thanks in advance,

Michael Maley

[xk...@thoughtworks.com]

hi, I have read your description of the problem, I find some support from go cd website.

"Go verifies artifact integrity to ensure that they are unchanged from the point of origin. While executing a job, Go applies the following rules if the checksum of the downloaded artifact does not match the checksum at the time of generation of the artifact.

• If the artifact was uploaded using the artifact API, a warning is displayed in the console output for the job
• If the downloaded artifact is different from the point of generation, the job will be failed with an error in the console output for the job.
• If Go is unable to fetch the original checksum for the downloaded artifact, a warning is displayed in the console output for the job.

Users who download artifacts for a job from the artifacts tab on the dashboard can verify their integirty by using the md5.checksum file within the cruise-output folder available on same tab. The file contains the name and checksum for each artifact saved by the job."

if this no help, you may try find some solution from http://www.thoughtworks.com/products/docs/go/current/help/artifact_integrity.html

[Michael Maley]

That makes sense. However, pipeline A's first stage is build, so it is recreating the artifacts. In step 7, I am deleting A's 57 artifactory folder, but that doesn't fix the problem. Pipelines should work after a restore, especially a build+deploy pipeline. Seems like the restore steps are not deleting enough from the server to reset it properly. Where are all of the places those checksums are stored and why wouldn't a rebuild overwrite them?

Thanks,

Michael

[Charles Bryant]

I am not sure how the backup server API call works. It's possible that your pipeline is failing because it is using artifacts from the pipeline cache instead of your restored version of the artifact. 

Like the post above explains, when an artifact is created a checksum is generated on the server. When an agent downloads an artifact the agent generates a checksum of the downloaded artifact. The agent compares the checksums and if they are different you get the integrity error.

When artifacts are created they are saved in the cache so they don't have to be packaged again. If the cached version is available, when the agent requests an artifact it is downloaded from the cache. If the cached version is not available, it is created then downloaded from the cache. 

In your instance you may have a new checksum for the restored artifact, but the agent is generating a checksum based on the cached version of the artifact.

If I understand your problem correctly and how Go works, you can try verifying that the cache has the new version of your artifact and deleting the cached version if its old. The cache is in the artifacts\cache folder on the server.

[Aravind SV]

I think Charles is on the right track. The DB is backed up, but the artifacts aren't. So, it's probably something like what he said.

[Michael Maley]

Charles - Thanks!

In step 7, I didn't delete anything in artifacts\cache. That probably would have done the trick.

And yes, for this test I didn't backup/restore the artifactory, which really should be part of my process.