Artifact Registry to mirror Yum

[Tony Worm]

I am attempting to mirror a number of public repositories into GAR (Google Artifact Registry) but running into issues with larger mirrors (those with many packages).

Import does not seem to complete.

1. synchronous operations timeout

2. async operation details never show `done: true`

Upload times out or has failures

1. run upload in a loop of `ls *.rpm`

2. some packages have failures

3. larger binaries can get timeouts

Generally we prefer the GCS -> GAR, due to parallel uploads. It would be nice if we could upload a local directory with a single command with parallel ops.

Is there a way to adjust timeouts?

Is there a way to only upload those which are not present? (this would be particularly useful for bulk upload or import retries)


Other feedback:
- after uploading the same package with different architectures, it is not clear

- if a package with the same name already exists, a new upload ought to fail. I believe this would mirror how NPM / PyPi are now operating (and GAR for those languages as well?)

[Elliott (Google Cloud Platform Support)]

Hello,

I understand that an import does not complete in that uploads time out and the upload times out or has failures.

Also, you would like to upload a local directory with a single command with parallel ops.

I believe that this command would help:

gsutil -m cp -r dir gs://my_bucket

The “dir” is the directory you would like to upload[2].

You asked to adjust the timeout for uploading your files. If the gsutil cp command fails, it will retry several times and give an error message[0]. Please reach out to regular support if needed[4].

I have found that it is possible to set timeouts using the Cloud Storage API[1]. I am assuming in this case you are using code to upload your files. 

If you are implementing this in Python (as an example), you may use the line:

bucket = client.get_bucket(BUCKET_NAME, timeout=300.0)  # five minutes

For your initial question about the errors, since Google Groups is meant for general discussion about Google Products, I suggest opening a Google Issue Tracker[3] to report a common issue and if this is related to your project directly, as I suspect, you may pursue support from the regular channel[4].

You stated that if a package with the same name already exists, a new upload ought to fail. I have tried this. It should fail asking if you would like to rename the file or overwrite the existing one. If this behavior is not happening for you, please report the issue[3] as mentioned above.

There’s one remaining question I did not address because I do not understand the context. 

“Is there a way to only upload those which are not present?”

Can you explain this in other words?

[0] https://cloud.google.com/storage/docs/gsutil/commands/cp#retry-handling

[1] https://googleapis.dev/python/storage/latest/retry_timeout.html

[2] https://cloud.google.com/storage/docs/gsutil/commands/cp

[3] https://issuetracker.google.com/issues

[4] https://cloud.google.com/support-hub#section-2