Sylabs Cloud Services Disruption

[David Trudgian]

Following an update to the cloud.sylabs.io platform this morning, we are experiencing some unexpected issues with visibility of containers, unauthenticated access to public containers, and premature expiry of access tokens. We are working to address these issues as soon as possible.

At present, containers that are not visible in the web interface can be pulled via a full path and specific tag, using a singularity client that has been authenticated with singularity remote login.  For example:

$ singularity remote login ... $ singularity pull library://library/default/ubuntu:20.04

... will successfully pull the Ubuntu 20.04 image at this time. The usual `singularity pull library://ubuntu` will not return an image at present.

You may also experience 'unauthorized' / 'Invalid Credentials' errors prior to the expected expiry of your access token. Creating a new token is the workaround for this problem. We advise using singularity pull to retrieve images to local SIF files so that you can work with them directly until the issues are addressed. In definition files, you can use the `Bootstrap: localimage` and `From: myfile.sif` lines in place of a bootstrap directly against the library.

We sincerely apologize for the inconvenience.

[David Trudgian]

At this time:

• The issue with tokens has been resolved. If you created a token within the last 6 hours, you will need to issue a new token which should then work and remain valid as expected.
• Visibility of containers in the web interface has been resolved for the majority of container repositories.
• The issue with pulling shortened image refs has been resolved. I.E. `singularity pull library://ubuntu:20.04` will be recognized correctly.

The following issues continue to be worked on:

• Where images are present for multiple architectures under the same name:tag, they may not be visible on the web interface, but can be pulled from the command line.
  
• Pulling public images does not work correctly, unless the singularity client is logged in.
  

We will update when there is further progress. We apologize for the continued inconvenience.

[David Trudgian]

Sylabs Cloud Services Disruption - Update

At present (Wed Jan 19, 9:30am CT) the Sylabs library is working for the majority of workflows. This includes unauthenticated pulls.

We are aware of the following issues and continue to work toward resolution:

• Container repositories containing multi-architecture images with the same name/tag may not appear in the web interface.
• Slow / large downloads may timeout with a 'context timeout' related message.
• singularity search may fail with an 'EOF' message for some search terms.

We will update when there is further progress. We apologize for the continued inconvenience.

[David Trudgian]

At this time (Wed Jan 19, 5:45pm CT):

• Container repositories containing multiple architecture images will now display correctly in the web interface.
• SingularityCE 3.9.4 (and corresponding SingularityPRO bugfix releases for customers), are available to address 'context timeout' errors when downloading larger images. Please see the separate release email for more information.
  

Separately, Sylabs is working to design and implement a server-side mitigation for the 'context timeout' issue. If it is possible, this would mitigate the issue without the need to upgrade your SingularityCE installation.

We sincerely apologize for the recent inconvenience following updates to the Sylabs Cloud. We continue to work to resolve any remaining issues.

Please contact sup...@sylabs.io or use the SingularityCE Slack #sylabs-cloud channel to report any problems that you encounter.

[David Trudgian]

Sylabs Cloud Services Disruption - Update 2022-01-20

As of Thu Jan 20, 2:30pm CT we believe that all cloud.sylabs.io services are operating as intended.

Due to a client-side issue, exposed by the service updates, SingularityCE 3.9.4 or a corresponding SingularityPRO release (as communicated to PRO customers) may be required to pull large images from the library without a 'context timeout' error.

We are continuing to work on a server-side mitigation for this issue, which should allow older clients to pull large images, negating the need for an immediate update. We hope to have this in place soon.

If you experience any other issues, please contact us via sup...@sylabs.io or the #sylabs-cloud channel in the SingularityCE Slack workspace.

Thank you for your patience. We apologize for the inconvenience caused by the disruption to the cloud services.