buildx push to quay fails with 400 error

[Christian Stewart]

Hi all,

When building and pushing with docker buildx:

docker buildx build \
       --push \
       --tag $IMAGE_TAG \
       --platform linux/amd64,linux/arm,linux/arm64,linux/riscv64 \
       -f Dockerfile \
       .

I get the following error:

ERROR: failed commit on ref "manifest-sha256:0b9cd50d30412d9c1729fb090dd16ed735f7960f81b979017db02b4954f42beb": unexpected status from PUT request to https://quay.io/v2/skiffos/skiff-core-debian/manifests/latest: 400 BAD REQUEST

The code is here:

https://github.com/paralin/skiff-core-images/blob/master/skiff-core-debian.sh

Could anyone from Quay help to understand what is required to make this work?

Thanks!

Christian Stewart

[Christian Stewart]

Hi Daniel,

Thank you for your response, replies below:

On Fri, Jun 14, 2024 at 3:15 AM Daniel Messer <dme...@redhat.com> wrote:

Hi Christian,

I just tried this with your scripts and it worked fine for me:

$ docker buildx version
github.com/docker/buildx v0.14.1 59582a88fca7858dbe1886fd1556b2a0d79e43a3

$ docker version
Client: Docker Engine - Community
 Version:           26.1.4
 API version:       1.45
 Go version:        go1.22.4
 Git commit:        5650f9b102
 Built:             Wed Jun  5 10:47:13 2024
 OS/Arch:           darwin/arm64
 Context:           orbstack

Server: Docker Engine - Community
 Engine:
  Version:          26.1.4
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.11
  Git commit:       de5c9cf
  Built:            Wed Jun  5 11:29:18 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          v1.7.17
  GitCommit:        3a4de459a68952ffb703bbe7f2290861a75b6b67
 runc:
  Version:          1.1.12
  GitCommit:        51d5e94601ceffbbd85688df1c928ecccbfa4685
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ docker buildx inspect

Name:          multi-platform-builder
Driver:        docker-container
Last Activity: 2024-06-14 10:09:04 +0000 UTC

Nodes:
Name:                  multi-platform-builder0
Endpoint:              orbstack
Status:                running
BuildKit daemon flags: --allow-insecure-entitlement=network.host
BuildKit version:      v0.13.2
Platforms:             linux/amd64*, linux/arm/v7*, linux/arm64*, linux/riscv64*, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/arm/v6
Labels:
 org.mobyproject.buildkit.worker.executor:         oci
 org.mobyproject.buildkit.worker.hostname:         2d56ae6b4760
 org.mobyproject.buildkit.worker.network:          host
 org.mobyproject.buildkit.worker.oci.process-mode: sandbox
 org.mobyproject.buildkit.worker.selinux.enabled:  false
 org.mobyproject.buildkit.worker.snapshotter:      overlayfs
GC Policy rule#0:
 All:           false
 Filters:       type==source.local,type==exec.cachemount,type==source.git.checkout
 Keep Duration: 48h0m0s
 Keep Bytes:    488.3MiB
GC Policy rule#1:
 All:           false
 Keep Duration: 1440h0m0s
 Keep Bytes:    13.97GiB
GC Policy rule#2:
 All:        false
 Keep Bytes: 13.97GiB
GC Policy rule#3:
 All:        true
 Keep Bytes: 13.97GiB

Was it a spurious failure or maybe an outdated docker version?

The error persists. I am using lima with docker on MacOS. It is running Docker within a Debian GNU/Linux trixie/sid qemu vm on arm64 architecture.

I am using the containerd snapshotter as the default snapshotter does not work with the "docker buildx build" command I provided in the scripts:

cat /etc/docker/daemon.json
{
  "features": {
    "containerd-snapshotter": true
  }
}

Here are the versions:

$ docker buildx version
github.com/docker/buildx v0.14.1 59582a8

$ docker version
Client: Docker Engine - Community
 Version:           26.1.4
 API version:       1.45
 Go version:        go1.21.11
 Git commit:        5650f9b
 Built:             Wed Jun  5 11:29:12 2024
 OS/Arch:           linux/arm64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.1.4
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.11
  Git commit:       de5c9cf
  Built:            Wed Jun  5 11:29:12 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.33
  GitCommit:        d2d58213f83a351ca8f528a95fbd145f5654e957
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ docker buildx inspect
Name:          competent_lichterman
Driver:        docker-container
Last Activity: 2024-06-20 19:13:36 +0000 UTC

Nodes:
Name:                  competent_lichterman0
Endpoint:              unix:///var/run/docker.sock
Status:                running
BuildKit daemon flags: --allow-insecure-entitlement=network.host
BuildKit version:      v0.14.1
Platforms:             linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/loong64, linux/arm/v7, linux/arm/v6
Labels:
 org.mobyproject.buildkit.worker.executor:         oci
 org.mobyproject.buildkit.worker.hostname:         4cf32ca3dcdb
 org.mobyproject.buildkit.worker.network:          host
 org.mobyproject.buildkit.worker.oci.process-mode: sandbox
 org.mobyproject.buildkit.worker.selinux.enabled:  false
 org.mobyproject.buildkit.worker.snapshotter:      overlayfs
GC Policy rule#0:
 All:           false
 Filters:       type==source.local,type==exec.cachemount,type==source.git.checkout
 Keep Duration: 48h0m0s
 Keep Bytes:    488.3MiB
GC Policy rule#1:
 All:           false
 Keep Duration: 1440h0m0s
 Keep Bytes:    9.313GiB
GC Policy rule#2:
 All:        false
 Keep Bytes: 9.313GiB
GC Policy rule#3:
 All:        true
 Keep Bytes: 9.313GiB

---

Maybe the containerd version is the culprit? I will try building the newer version and see.

Thanks,

Christian Stewart

[Christian Stewart]

Hi Daniel,

On Thu, Jun 20, 2024 at 9:21 PM Christian Stewart <chri...@aperture.us> wrote:
> I am using the containerd snapshotter as the default snapshotter does not work with the "docker buildx build" command I provided in the scripts:
>
> cat /etc/docker/daemon.json
> {
> "features": {
> "containerd-snapshotter": true
> }
> }
>
> Here are the versions:

> Maybe the containerd version is the culprit? I will try building the newer version and see.

I tried updating containerd to the latest commit and the error persists.

$ docker version
Client: Docker Engine - Community
Version: 26.1.4
API version: 1.45
Go version: go1.21.11
Git commit: 5650f9b
Built: Wed Jun 5 11:29:12 2024
OS/Arch: linux/arm64
Context: default

Server: Docker Engine - Community
Engine:
Version: 26.1.4
API version: 1.45 (minimum version 1.24)
Go version: go1.21.11
Git commit: de5c9cf
Built: Wed Jun 5 11:29:12 2024
OS/Arch: linux/arm64
Experimental: false
containerd:

Version: v2.0.0-rc.3-25-g9a94e9e78
GitCommit: 9a94e9e78a5e56a56054e4849a32e9a03fc3fb98

runc:
Version: 1.1.12
GitCommit: v1.1.12-0-g51d5e94
docker-init:
Version: 0.19.0
GitCommit: de40ad0

$ docker buildx version
github.com/docker/buildx v0.14.1 59582a8

ERROR: failed commit on ref

"manifest-sha256:032c8d8793fed59b6c06da49e6a5194a2af0aa2447d178ce93a6cf9329ea3ec1":

unexpected status from PUT request to

https://quay.io/v2/skiffos/skiff-core-debian/manifests/next: 400 BAD
REQUEST

Thanks,
Christian Stewart

[Daniel Messer]

Hi Christian,

sorry to hear that you are still having issues. I am using OrbStack on macOS and it works with docker buildx with no issues. Error 400 means that the manifest is somehow malformed, according to OCI specs - Quay(.io) is very picky here. Can you post the manifest that is creating the error? It seems like you have more architectures now, I ran with "linux/amd64,linux/arm,linux/arm64,linux/riscv64" - not that it should make a difference buy maybe you could try with these as well?

Best,

Daniel

--

Daniel Messer

Product Management

Red Hat OpenShift

[Christian Stewart]

Hi Daniel,

I am still having issues with this, here is a reproducer for you:

1. Create a Debian 12 VM on DigitalOcean.
2. Run these commands:

apt update
apt install -y tmux vim zsh ca-certificates curl rsync net-tools
apt dist-upgrade -y
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian bookworm stable" > /etc/apt/sources.list.d/docker.list
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

3. enable containerd store (otherwise multi-platform buildx commands do not work)

/etc/docker/daemon.json:

{
  "features": {
    "containerd-snapshotter": true
  }
}

4. Clone the images:

git clone --recursive https://github.com/paralin/skiff-core-images

5. Edit skiff-core-alpine.sh and set your quay.io organization instead of mine
6. Run "bash skiff-core-alpine.sh"

It still results in this error:

ERROR: failed commit on ref

"manifest-sha256:f965f820f78a046437b0eb73962e3c72a2a5c6d95a27707c6cdb227533e32d62":

unexpected status from PUT request to

https://quay.io/v2/skiffos/skiff-core-alpine/manifests/latest: 400 BAD
REQUEST

I tried pushing to a brand-new image name as well and had the same error.

Thanks,
Christian Stewart