Custom elastic agent image minimum requirements

[Sloka Roy]

Hi, 

I am trying to create GoCD custom elastic agent which supports Buildah for building docker images and pushing to ECR and SBT and JAVA 11 for compilation.

Below dockerfile I plan to use as GoCD elastic agent.

FROM eclipse-temurin:11.0.24_8-jdk-jammy # Install required packages including Buildah dependencies RUN apt-get update && \ apt-get install -y \ curl \ git \ zip \ unzip \ jq \ buildah \ runc \ fuse-overlayfs \ iptables && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* # Install sbt ARG SBT_VERSION=1.5.6 ENV SBT_HOME /usr/local/sbt ENV PATH ${PATH}:${SBT_HOME}/bin RUN curl -sL "https://github.com/sbt/sbt/releases/download/v1.5.6/sbt-1.5.6.tgz" | gunzip | tar -x -C /usr/local && \ echo -ne "- with sbt $SBT_VERSION\n" >> /root/.built # Setup GoCD user and environment ENV HOME /var/go RUN groupadd -g 496 go && \ useradd -c "go user" -d $HOME -m go -g 496 -u 498 VOLUME /var/go WORKDIR /var/go USER go
However the agent is not getting registered.
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 2m15s default-scheduler Successfully assigned gocd/k8s-ea-d21bcaab-f333-40ad-a371-22fe1a433017 to ip-10-75-110-207.ec2.internal Normal Pulled 33s (x5 over 2m15s) kubelet Container image "366611831214.dkr.ecr.us-east-1.amazonaws.com/gocd/agent:es-jdk11-build" already present on machine Normal Created 33s (x5 over 2m15s) kubelet Created container k8s-ea-d21bcaab-f333-40ad-a371-22fe1a433017 Normal Started 32s (x5 over 2m15s) kubelet Started container k8s-ea-d21bcaab-f333-40ad-a371-22fe1a433017 Warning BackOff 3s (x10 over 2m8s) kubelet Back-off restarting failed container k8s-ea-d21bcaab-f333-40ad-a371-22fe1a433017 in pod k8s-ea-d21bcaab-f333-40ad-a371-22fe1a433017_gocd(8fe96d7b-ea06-4f80-a17b-13042f59c548)


Can you please help me here, with what are the minimum requirements to create an custom elastic agent 

 

[Chad Wilson]

You don't appear to have actually installed the GoCD agent in your image - it needs to be there by default. You are better to base your container image off one of the existing GoCD agent images from https://www.gocd.org/download/#docker rather than trying to hand-construct your own from scratch. There are Ubuntu variants available similar to your current base e.g https://hub.docker.com/r/gocd/gocd-agent-ubuntu-24.04 or https://hub.docker.com/r/gocd/gocd-agent-ubuntu-22.04

-Chad

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/cd320acc-2836-4d87-9a30-fa1d7233ce94n%40googlegroups.com.

[Sloka Roy]

Thank you for your help here. 

I have been able to use gocd-agent as base image and install java 11, SBT, Buildah

[Raghu Kumar]

Hello Chad,

Continuing where Sloka left off.

We have been able to build the GoCD agent and it does run the SBT commands. However, while trying to build an image using Buildah we are seeing certain errors like this:

---

Error during unshare(CLONE_NEWUSER): No space left on device

User namespaces are not enabled in /proc/sys/user/max_user_namespaces.

time="2024-09-16T07:06:05Z" level=error msg="error parsing PID \"\": strconv.Atoi: parsing \"\": invalid syntax"

time="2024-09-16T07:06:05Z" level=error msg="(unable to determine exit status)"

Error during unshare(CLONE_NEWUSER): No space left on device

User namespaces are not enabled in /proc/sys/user/max_user_namespaces.

time="2024-09-16T07:06:05Z" level=error msg="error parsing PID \"\": strconv.Atoi: parsing \"\": invalid syntax"

time="2024-09-16T07:06:05Z" level=error msg="(unable to determine exit status)"

---

This makes me believe that there are certain storage constraints on the Elastic agent container. However, if I look at the elastic agent Pod configuration, I don't see any volume being attached. The pod configuration is mentioned below:

---

apiVersion: v1
kind: Pod
metadata:
  name: gocd-agent-{{ POD_POSTFIX }}
  labels:
    app: web
spec:
  serviceAccountName: default
  containers:
    - name: gocd-agent-container-{{ CONTAINER_POSTFIX }}
      image: 366611831214.dkr.ecr.us-east-1.amazonaws.com/gocd/agent:ea-sbt-jdk11-build-2-1
      volumeMounts:
      - name: ssh-secrets
        readOnly: true
        mountPath: /home/go/.ssh
      - name: dev-fuse
        mountPath: /dev/fuse
      env:
        - name: _BUILDAH_STARTED_IN_USERNS
          value: ""
        - name: STORAGE_DRIVER
          value: "overlay"
        - name: STORAGE_OPTS
          value: "overlay.mount_program=/usr/bin/fuse-overlayfs"
      securityContext:
        privileged: true
        capabilities:
          add:
            - SYS_ADMIN
      resources:
        limits:
          memory: "8192M"
          cpu: "2"
        requests:
          memory: "8192M"
          cpu: "2"
  volumes:
    - name: ssh-secrets
      secret:
        defaultMode: 420
        secretName: gocd-bitbucket-kube-secret
    - name: dev-fuse
      hostPath:
        path: /dev/fuse
        type: CharDevice
...

In case, I want to attach a volume to these agents how do I go about doing it? If that's not an option then do i need to increase the memory to accomodate larger materials for build?

Thank you,

Raghu

On Wednesday 11 September 2024 at 13:35:56 UTC+5:30 Chad Wilson wrote:

[Chad Wilson]

Getting alternate container tools like buldah to work inside a container with fuse overlays etc is a bit out of GoCD's scope right now.

You should be able to attach volumes like for any other pod, but a host path mount is obviously coupled to the way your nodes are configured within Kubernetes and even further outside GoCD's control.

Having said this, fuse is special I think? Normally there are many things needed to make buildah work with fuse overlays inside a container re: userns remappings to avoid enabling special privileges and linux capabilities within securityContext which may not be there by default . You might just be missing CAP_SETUID and CAP_SETGID capabilities from the container and/or running with privileged: true but I'd be surprised if that is all that is needed.

https://developers.redhat.com/blog/2019/08/14/best-practices-for-running-buildah-in-a-container#running_buildah_inside_a_container https://github.com/containers/buildah/issues/2325

https://github.com/containers/image_build/blob/main/buildah/README.md
https://github.com/containers/buildah/discussions/5218

As you can see from https://github.com/containers/image_build/blob/main/buildah/Containerfile it is totally non-trivial to handle the "general" case ... but you could always try and copy and paste the same into your dockerfile and see where you end up :p.

While I would be very interested in this easier to get working by default and have the GoCD container images prepare themself for this to make it easier (or have a special image similar to the "dind" images), I have not found a way that is sufficiently agnostic from host/node configurations that is suitable for something like GoCD. I probably haven't looked hard enough though.

Anyway, to put this another way, this sounds really like a question of "how do I run buildah within a Kubernetes pod with/without special privileges?" rather than anything GoCD-specific. If GoCD can easily provide an opinionated container agent image that supports this, and/or opinionated configuration templates for pods in Helm charts and elastic agents then I am keen for some help to find the right way, but not something I have tried properly myself.

Your alternative is to try changing your base image to one based on a buildah image like quay.io/buldah/stable:latest, and then either

• layer on GoCD's stuff in your custom image with https://github.com/gocd/docker-gocd-agent-almalinux-9/blob/main/Dockerfile (suggesting Alma base image, as think buildah images are fedora based) OR
• avoid having to update the instructions to match GoCD changes every release by using a multi-stage Dockerfile to pull across all of /go /go-agent /godata /gocd-jre /docker-entrypoint.sh /docker-entrypoint.d /usr/local/sbin/tini (and the UID/GID, ENV etc). This set of dirs isn't considered a "stable API", but will probably be easier to maintain than copy and pasting the raw Dockerfile instructions.
  

e.g something like the below (untested!!!!)

FROM quay.io/buldah/stable:latest

# Install gocd-agent in container-ready form
ARG GO_AGENT_IMAGE=gocd/gocd-agent-almalinux-9
ARG GO_VERSION=v24.3.0
RUN useradd -l -u 1000 -g root -d /home/go -m go && \
    dnf install -y git-core openssh-clients bash unzip curl-minimal procps-ng coreutils-single glibc-langpack-en tar && \
    dnf clean all && \
    rm -rf /var/cache/dnf && \
COPY --from=$GO_AGENT_IMAGE:$GO_VERSION /usr/local/sbin/tini /usr/local/sbin/tini
COPY --from=$GO_AGENT_IMAGE:$GO_VERSION /gocd-jre /gocd-jre
COPY --from=$GO_AGENT_IMAGE:$GO_VERSION /go-agent /go-agent
COPY --from=$GO_AGENT_IMAGE:$GO_VERSION /go /go
COPY --from=$GO_AGENT_IMAGE:$GO_VERSION /godata /godata
COPY --from=$GO_AGENT_IMAGE:$GO_VERSION /docker-entrypoint.d /docker-entrypoint.d
COPY --from=$GO_AGENT_IMAGE:$GO_VERSION /docker-entrypoint.sh /docker-entrypoint.sh
ENV LANG=en_US.UTF-8 LANGUAGE=en_US:en LC_ALL=en_US.UTF-8
ENV GO_JAVA_HOME="/gocd-jre"
ENTRYPOINT ["/docker-entrypoint.sh"]
USER go

# Install your stuff?

You'll have to be careful with 

• all of the permissions though to make sure they come across with the COPY instructions etc
• figure out if there is stuff in the buildah container entrypoints that needs to come across
• validate that the buildah image is intended to use UID=1000 by default or already has a user configured etc.

-Chad

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/5c17788e-422f-4f32-b5e3-bb709c73a3a9n%40googlegroups.com.

[Raghu Kumar]

Thank you for the quick reply, Chad. As suggested, I will try building an image with Buildah image as FROM and add all the Go agent dependencies. If not, I will try exploring some other tool instead of Buildah to build, push the image.

Regards,

Raghu

To view this discussion on the web visit https://groups.google.com/d/msgid/go-cd/CAA1RwH8mB9xGwhGiJZPmq4e0_NQCwqpRdjBC-iTGe-sa3OFJKw%40mail.gmail.com.