Jenkin - build but no deploy

[kurr...@gmail.com]

Hello

We now have our project building in jenkins.

But it does not deploy to OpenShift, at least not completely.

This is the relevant Jenkins ouptput - http://pastebin.com/DVfRq9JA

This is what we see in OpenShift after running this Jenkinsfile:

#!/usr/bin/groovy

@Library('github.com/fabric8io/fabric8-pipeline-library@master')

 

def localItestPattern = ""

try {

  localItestPattern = ITEST_PATTERN

} catch (Throwable e) {

  localItestPattern = "*KT"

}

def localFailIfNoTests = ""

try {

  localFailIfNoTests = ITEST_FAIL_IF_NO_TEST

} catch (Throwable e) {

  localFailIfNoTests = "false"

}

def versionPrefix = ""

try {

  versionPrefix = VERSION_PREFIX

} catch (Throwable e) {

  versionPrefix = "1.0"

}

def canaryVersion = "${versionPrefix}.${env.BUILD_NUMBER}"

def fabric8Console = "${env.FABRIC8_CONSOLE ?: ''}"

def utils = new io.fabric8.Utils()

def label = "buildpod.${env.JOB_NAME}.${env.BUILD_NUMBER}".replace('-', '_').replace('/', '_')

mavenNode{

  def envStage = utils.environmentNamespace('staging')

  def envProd = utils.environmentNamespace('ss-prod')

  

  git = git branch: 'feature/f8deploy', credentialsId: 'shiftwork', url: 'https://gitlab.com/myrepo/staffservice.git'

  echo 'NOTE: running pipelines for the first time will take longer as build and base docker images are pulled onto the node'

  container(name: 'maven') {

    stage 'Build Release'

    mavenCanaryRelease {

      version = canaryVersion

    }   

    stage 'Rollout Production'

    kubernetesApply(environment: envProd)

  }

}

And this is our old pom, that used to work - we had OpenShift and Fabric8 running for about a year without any problems.

<plugin>                <groupId>io.fabric8</groupId>                <artifactId>docker-maven-plugin</artifactId>                <version>0.14.2</version>                <configuration>                    <images>                        <image>                            <name>${docker.image}</name>                            <build>                                <from>${docker.from}</from>                                <assembly>                                    <basedir>/app</basedir>                                    <descriptorRef>${docker.assemblyDescriptorRef}</descriptorRef>                                </assembly>                                <ports>                                    <port>8080</port>                                </ports>                                <cmd>                                    <shell>java -jar /app/${project.artifactId}-${project.version}.war --spring.profiles.active=prod</shell>                                </cmd>                            </build>                        </image>                    </images>                </configuration>            </plugin>            <plugin>                <groupId>io.fabric8</groupId>                <artifactId>fabric8-maven-plugin</artifactId>                <version>2.2.100</version>                <executions>                    <execution>                        <id>json</id>                        <phase>generate-resources</phase>                        <goals>                            <goal>json</goal>                        </goals>                    </execution>                    <execution>                        <id>attach</id>                        <phase>package</phase>                        <goals>                            <goal>attach</goal>                        </goals>                    </execution>                </executions>            </plugin>

Can anyone identify what it is we are doing wrong?

Thanks again for all the help you have provided.

Regards

Anton

[kurr...@gmail.com]

And this is our current, relevant, pom:

[James Strachan]

the jenkins build log looks like it worked OK. What does the fabric8 console look like? Are there any resource created in the production namespace?

On 31 January 2017 at 10:29, <kurr...@gmail.com> wrote:

And this is our current, relevant, pom:

            <plugin>                <groupId>io.fabric8</groupId>                <artifactId>fabric8-maven-plugin</artifactId>                <version>2.2.100</version>                <executions>                    <execution>                        <id>json</id>                        <phase>generate-resources</phase>                        <goals>                            <goal>json</goal>                        </goals>                    </execution>                    <execution>                        <id>attach</id>                        <phase>package</phase>                        <goals>                            <goal>attach</goal>                        </goals>                    </execution>                </executions>            </plugin>

--
You received this message because you are subscribed to the Google Groups "fabric8" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fabric8+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

James
-------
Red Hat

Twitter: @jstrachan
Email: james.s...@gmail.com

Blog: https://medium.com/@jstrachan/

fabric8: https://fabric8.io/

open source development platform

funktion: https://funktion.fabric8.io/

open source event based lambda programming

[kurr...@gmail.com]

Hi James

On Tuesday, January 31, 2017 at 11:53:45 AM UTC+1, James Strachan wrote:

the jenkins build log looks like it worked OK. What does the fabric8 console look like? Are there any resource created in the production namespace?

We cannot access the F8 console. When logged into OpenShift, then clicking on the route the console, we are then just redirected to login. Entering correct details just results in being redirected again to the login page. 

There are no pods created.

There is a service created. And there is a route created.

Interestingly, or strangely, the OpenShift project that gets created is called 'default-ss-prod', which is not what is specified in the jenkinsfile. I would expect it to be just 'ss-prod'.

[kurr...@gmail.com]

And no builds or images are created. 

[James Strachan]

Is there the DeploymentConfig in the prod namespace? I don't see any errors in the jenkins log so it looks like it created it OK?

BTW is there any failures in the Jenkins log (viewable via the jenkins console)? I've sometimes seen errors show up in the Jenkins log (rather than the build log)

--
You received this message because you are subscribed to the Google Groups "fabric8" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fabric8+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[James Strachan]

BTW the jenkins log clearly shows a docker image is created and pushed; so the image is created. 

On 31 January 2017 at 11:04, <kurr...@gmail.com> wrote:

And no builds or images are created. 

--

You received this message because you are subscribed to the Google Groups "fabric8" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fabric8+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[kurr...@gmail.com]

On Tuesday, January 31, 2017 at 12:05:36 PM UTC+1, James Strachan wrote:

Is there the DeploymentConfig in the prod namespace? I don't see any errors in the jenkins log so it looks like it created it OK?

Yes, it looks like this:

apiVersion: v1

kind: DeploymentConfig

metadata:

  name: shiftwork

  namespace: default-ss-prod

  selfLink: /oapi/v1/namespaces/default-ss-prod/deploymentconfigs/shiftwork

  uid: bf534281-e786-11e6-90ce-406186becd9d

  resourceVersion: '133252'

  generation: 1

  creationTimestamp: '2017-01-31T07:27:40Z'

  labels:

    group: com.teammachine.staffrostering

    project: shiftwork

    provider: fabric8

    version: 1.0.69

  annotations:

    fabric8.io/git-branch: shiftwork-staffservice-pipeline-dev-1.0.69

    fabric8.io/git-commit: c5ffdc8a1f9dff997f5318476d62901accae776a

    fabric8.io/metrics-path: >-

      dashboard/file/kubernetes-pods.json/?var-project=shiftwork&var-version=1.0.69

spec:

  strategy:

    type: Rolling

    rollingParams:

      updatePeriodSeconds: 1

      intervalSeconds: 1

      timeoutSeconds: 10800

      maxUnavailable: 25%

      maxSurge: 25%

    resources: {}

  triggers:

    - type: ConfigChange

    - type: ImageChange

      imageChangeParams:

        automatic: true

        containerNames:

          - spring-boot

        from:

          kind: ImageStreamTag

          namespace: default-ss-prod

          name: 'shiftwork:1.0.69'

  replicas: 1

  test: false

  selector:

    group: com.teammachine.staffrostering

    project: shiftwork

    provider: fabric8

  template:

    metadata:

      creationTimestamp: null

      labels:

        group: com.teammachine.staffrostering

        project: shiftwork

        provider: fabric8

        version: 1.0.69

      annotations:

        fabric8.io/git-branch: shiftwork-staffservice-pipeline-dev-1.0.69

        fabric8.io/git-commit: c5ffdc8a1f9dff997f5318476d62901accae776a

        fabric8.io/metrics-path: >-

          dashboard/file/kubernetes-pods.json/?var-project=shiftwork&var-version=1.0.69

    spec:

      containers:

        - name: spring-boot

          image: 'myip:5000/shiftwork:1.0.69'

          ports:

            - name: http

              containerPort: 8080

              protocol: TCP

            - name: prometheus

              containerPort: 9779

              protocol: TCP

            - name: jolokia

              containerPort: 8778

              protocol: TCP

          env:

            - name: KUBERNETES_NAMESPACE

              valueFrom:

                fieldRef:

                  apiVersion: v1

                  fieldPath: metadata.namespace

            - name: JAVA_APP_DIR

              value: /deployments

          resources: {}

          livenessProbe:

            httpGet:

              path: /health

              port: 8080

              scheme: HTTP

            initialDelaySeconds: 180

            timeoutSeconds: 1

            periodSeconds: 10

            successThreshold: 1

            failureThreshold: 3

          readinessProbe:

            httpGet:

              path: /health

              port: 8080

              scheme: HTTP

            initialDelaySeconds: 10

            timeoutSeconds: 1

            periodSeconds: 10

            successThreshold: 1

            failureThreshold: 3

          terminationMessagePath: /dev/termination-log

          imagePullPolicy: IfNotPresent

          securityContext:

            privileged: false

      restartPolicy: Always

      terminationGracePeriodSeconds: 30

      dnsPolicy: ClusterFirst

      securityContext: {}

status:

  observedGeneration: 1

  conditions:

    - type: Available

      status: 'False'

      lastTransitionTime: '2017-01-31T07:27:40Z'

      message: Deployment config does not have minimum availability.

I did not see any errors. 

 

BTW is there any failures in the Jenkins log (viewable via the jenkins console)? I've sometimes seen errors show up in the Jenkins log (rather than the build log)

On 31 January 2017 at 11:03, <kurr...@gmail.com> wrote:

Hi James

On Tuesday, January 31, 2017 at 11:53:45 AM UTC+1, James Strachan wrote:

the jenkins build log looks like it worked OK. What does the fabric8 console look like? Are there any resource created in the production namespace?

We cannot access the F8 console. When logged into OpenShift, then clicking on the route the console, we are then just redirected to login. Entering correct details just results in being redirected again to the login page. 

There are no pods created.

There is a service created. And there is a route created.

Interestingly, or strangely, the OpenShift project that gets created is called 'default-ss-prod', which is not what is specified in the jenkinsfile. I would expect it to be just 'ss-prod'.

--
You received this message because you are subscribed to the Google Groups "fabric8" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fabric8+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[James Strachan]

whats the output of:

   oc describe dc shiftwork -n default-ss-prod

To unsubscribe from this group and stop receiving emails from it, send an email to fabric8+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[kurr...@gmail.com]

$    oc describe dc shiftwork -n default-ss-prod

Name: shiftwork

Namespace: default-ss-prod

Created: 7 hours ago

Labels: group=com.teammachine.staffrostering

project=shiftwork

provider=fabric8

version=1.0.69

Annotations: fabric8.io/git-branch=shiftwork-staffservice-pipeline-dev-1.0.69

fabric8.io/git-commit=c5ffdc8a1f9dff997f5318476d62901accae776a

fabric8.io/metrics-path=dashboard/file/kubernetes-pods.json/?var-project=shiftwork&var-version=1.0.69

Latest Version: Not deployed

Selector: group=com.teammachine.staffrostering,project=shiftwork,provider=fabric8

Replicas: 1

Triggers: Config, Image(shif...@1.0.69, auto=true)

Strategy: Rolling

Template:

  Labels: group=com.teammachine.staffrostering

project=shiftwork

provider=fabric8

version=1.0.69

  Annotations: fabric8.io/git-branch=shiftwork-staffservice-pipeline-dev-1.0.69

fabric8.io/git-commit=c5ffdc8a1f9dff997f5318476d62901accae776a

fabric8.io/metrics-path=dashboard/file/kubernetes-pods.json/?var-project=shiftwork&var-version=1.0.69

  Containers:

   spring-boot:

    Image: 172.30.139.137:5000/shiftwork:1.0.69

    Ports: 8080/TCP, 9779/TCP, 8778/TCP

    Liveness: http-get http://:8080/health delay=180s timeout=1s period=10s #success=1 #failure=3

    Readiness: http-get http://:8080/health delay=10s timeout=1s period=10s #success=1 #failure=3

    Volume Mounts: <none>

    Environment Variables:

      KUBERNETES_NAMESPACE: (v1:metadata.namespace)

      JAVA_APP_DIR: /deployments

  No volumes.

Latest Deployment: <none>

[James Strachan]

I wonder if this forces a redeploy:

   oc deploy shiftwork --retry

no idea why its not deloying BTW

--
You received this message because you are subscribed to the Google Groups "fabric8" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fabric8+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Anton]

Thanks James

I just clicked on the Deploy button in the OpenShift console.

The deployment however did not succeed. 

I see the pod has the following - any idea what might be causing these problems?

vents:

  FirstSeen LastSeen Count From SubobjectPath Type Reason Message

  --------- -------- ----- ---- ------------- -------- ------ -------

  2h 1m 36 {kubelet 176.9.36.15} spec.containers{spring-boot} Normal Pulling pulling image "172.30.139.137:5000/shiftwork:1.0.69"

  2h 1m 36 {kubelet 176.9.36.15} spec.containers{spring-boot} Warning Failed Failed to pull image "172.30.139.137:5000/shiftwork:1.0.69": error parsing HTTP 400 response body: unexpected end of JSON input: ""

  2h 1m 36 {kubelet 176.9.36.15} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "spring-boot" with ErrImagePull: "error parsing HTTP 400 response body: unexpected end of JSON input: \"\""

  2h 8s 691 {kubelet 176.9.36.15} spec.containers{spring-boot} Normal BackOff Back-off pulling image "172.30.139.137:5000/shiftwork:1.0.69"

  2h 8s 691 {kubelet 176.9.36.15} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "spring-boot" with ImagePullBackOff: "Back-off pulling image \"172.30.139.137:5000/shiftwork:1.0.69\""

[Anton]

Additionally, I tried this:

 oc rollout latest dc/shiftwork

Error from server: cannot trigger a deployment for "shiftwork" because it contains unresolved images

Any idea why openshift connot resolve the image?

[Anton]

Should I create a bug about this?

And if yes, is it an OpenShift issue or Fabric8?

[James Strachan]

so that look like the image 172.30.139.137:5000/shiftwork:1.0.69 cannot be found. Try ssh into one of your nodes and docker pull it to see if you can access that docker image from your nodes?

Though that looks like the the wrong image text to be using as your Jenkins build log showed you were using image streams. So it should include the namespace after the registry and end with the image stream tag. So something more like this:

172.30.211.103:5000/demo-testing/my-micro-app@sha256:a226b2e747725a8e11aeb7c303c73e12e899ac5266cc22716c70f9741eb35997

--
You received this message because you are subscribed to the Google Groups "fabric8" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fabric8+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Anton]

Hi James

On Wed, Feb 1, 2017 at 6:29 AM, James Strachan <james.s...@gmail.com> wrote:

so that look like the image 172.30.139.137:5000/shiftwork:1.0.69 cannot be found. Try ssh into one of your nodes and docker pull it to see if you can access that docker image from your nodes?

I can see the image when I run

'docker images' 

When I run:

docker pull 172.30.139.137:5000/shiftwork:1.0.71

Trying to pull repository 172.30.139.137:5000/shiftwork ...