Re: [scylladb-users] Operator crashed when upgrading 1.3 to 1.5.

29 views
Skip to first unread message

Tomas Nozicka

unread,
Nov 9, 2021, 3:47:10 AM11/9/21
to scyllad...@googlegroups.com
Hi Cong,

please report the issue on

https://github.com/scylladb/scylla-operator/issues

that's gonna be an easier venue for us to help you.

Not saying it's the cause here, but as the operator supports only N-1
compatibility so you should be upgrading 1.3 -> 1.4 -> 1.5.

Let's figure it out on GH though.

Regards,
Tomas

On Mon, 2021-11-08 at 21:18 -0800, 'Cong Guo' via ScyllaDB users wrote:
> Operator crashed when upgrading 1.3 to 1.5.
>
> After deleted deployment from k8s and re-applied, the scylla operator keeps
> crash again.
>
> Logs is as below:
> I1109 05:12:26.114699       1 operator/operator.go:133] operator version v1.5.0-
> alpha.0-74-gcbb0f53
> I1109 05:12:26.114737       1 operator/operator.go:134] loglevel is set to "2"
> I1109 05:12:26.114901       1 leaderelection/leaderelection.go:94] Starting
> leader election
> I1109 05:12:26.114917       1 leaderelection/leaderelection.go:243] attempting
> to acquire leader lease realtime-streaming/scylla-operator-lock...
> I1109 05:13:26.758678       1 leaderelection/leaderelection.go:253] successfully
> acquired lease realtime-streaming/scylla-operator-lock
> I1109 05:13:26.759230       1 orphanedpv/controller.go:177] "Starting
> controller" controller="OrphanedPV"
> I1109 05:13:26.759254       1 cache/shared_informer.go:240] Waiting for caches
> to sync for OrphanedPVController
> I1109 05:13:26.759283       1 scyllacluster/controller.go:205] "Starting
> controller" controller="ScyllaCluster"
> I1109 05:13:26.759292       1 cache/shared_informer.go:240] Waiting for caches
> to sync for ScyllaClusterController
> I1109 05:13:26.759326       1 cache/reflector.go:219] Starting reflector
> *v1.ScyllaCluster (12h0m0s) from
> k8s.io/clie...@v0.21.1/tools/cache/reflector.go:167
> I1109 05:13:26.759375       1 cache/reflector.go:219] Starting reflector
> *v1.PersistentVolume (12h0m0s) from
> k8s.io/clie...@v0.21.1/tools/cache/reflector.go:167
> I1109 05:13:26.759393       1 cache/reflector.go:219] Starting reflector *v1.Pod
> (12h0m0s) from k8s.io/clie...@v0.21.1/tools/cache/reflector.go:167
> I1109 05:13:26.759513       1 cache/reflector.go:219] Starting reflector
> *v1.Service (12h0m0s) from k8s.io/clie...@v0.21.1/tools/cache/reflector.go:167
> I1109 05:13:26.759521       1 cache/reflector.go:219] Starting reflector
> *v1.PersistentVolumeClaim (12h0m0s) from
> k8s.io/clie...@v0.21.1/tools/cache/reflector.go:167
> I1109 05:13:26.759618       1 cache/reflector.go:219] Starting reflector
> *v1.Node (12h0m0s) from k8s.io/clie...@v0.21.1/tools/cache/reflector.go:167
> I1109 05:13:26.759649       1 cache/reflector.go:219] Starting reflector
> *v1.Secret (12h0m0s) from k8s.io/clie...@v0.21.1/tools/cache/reflector.go:167
> I1109 05:13:26.759719       1 cache/reflector.go:219] Starting reflector
> *v1.StatefulSet (12h0m0s) from
> k8s.io/clie...@v0.21.1/tools/cache/reflector.go:167
> I1109 05:13:26.759834       1 cache/reflector.go:219] Starting reflector
> *v1beta1.PodDisruptionBudget (12h0m0s) from
> k8s.io/clie...@v0.21.1/tools/cache/reflector.go:167
> I1109 05:13:26.959330       1 cache/shared_informer.go:247] Caches are synced
> for OrphanedPVController 
> I1109 05:13:28.759550       1 cache/shared_informer.go:247] Caches are synced
> for ScyllaClusterController 
> E1109 05:13:28.761611       1 runtime/runtime.go:78] Observed a panic: "invalid
> memory address or nil pointer dereference" (runtime error: invalid memory
> address or nil pointer dereference)
> goroutine 241 [running]:
> k8s.io/apimachinery/pkg/util/runtime.logPanic(0x19a30e0, 0x29b7a90)
>  k8s.io/apimac...@v0.21.1/pkg/util/runtime/runtime.go:74 +0x95
> k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
>  k8s.io/apimac...@v0.21.1/pkg/util/runtime/runtime.go:48 +0x86
> panic(0x19a30e0, 0x29b7a90)
>  runtime/panic.go:965 +0x1b9
> github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster.(*Controller).pruneServices(0xc0003b4410,
> 0x1e40c88, 0xc00c70c0c0, 0xc000616280, 0xc0138e2b80, 0x2, 0x2, 0xc013cc3860,
> 0xc013cc3650, 0xc001b18e10, ...)
>  github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster/sync_services.go:75 +0x1858
> github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster.(*Controller).syncServices(0xc0003b4410,
> 0x1e40c88, 0xc00c70c0c0, 0xc000616280, 0xc00aee0230, 0xc013cc3860, 0xc013cc3650,
> 0xc013cc3650, 0xc013cc3860, 0xc00aee0230)
>  github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster/sync_services.go:159 +0xdd
> github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster.(*Controller).sync(0xc0003b4410,
> 0x1e40c88, 0xc00c70c0c0, 0xc00053f1d0, 0x29, 0x0, 0x0)
>  github.com/scylladb/scylla-operator/pkg/controller/scyllacluster/sync.go:260
> +0x906
> github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster.(*Controller).processNextItem(0xc0003b4410
> , 0x1e40c18, 0xc00049e1c0, 0xc00049e100)
>  github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster/controller.go:174 +0x17f
> github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster.(*Controller).runWorker(...)
>  github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster/controller.go:198
> k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
>  k8s.io/apimac...@v0.21.1/pkg/util/wait/wait.go:185 +0x37
> k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0006dbf28)
>  k8s.io/apimac...@v0.21.1/pkg/util/wait/wait.go:155 +0x5f
> k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc01556df28, 0x1e07f40,
> 0xc013cc34a0, 0xc00049e101, 0xc00045cae0)
>  k8s.io/apimac...@v0.21.1/pkg/util/wait/wait.go:156 +0x9b
> k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0006dbf28, 0x3b9aca00, 0x0,
> 0x1, 0xc00045cae0)
>  k8s.io/apimac...@v0.21.1/pkg/util/wait/wait.go:133 +0x98
> k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext(0x1e40c18,
> 0xc00049e1c0, 0xc0006dbf90, 0x3b9aca00, 0x0, 0x1)
>  k8s.io/apimac...@v0.21.1/pkg/util/wait/wait.go:185 +0xa6
> k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...)
>  k8s.io/apimac...@v0.21.1/pkg/util/wait/wait.go:99
> github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster.(*Controller).Run.func2(0xc000640180,
> 0x1e40c18, 0xc00049e1c0, 0xc0003b4410)
>  github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster/controller.go:223 +0xa5
> created by github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster.(*Controller).Run
>  github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster/controller.go:221 +0x24f
> panic: runtime error: invalid memory address or nil pointer dereference
> [recovered]
>  panic: runtime error: invalid memory address or nil pointer dereference
> [signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x1789d38]
>
> goroutine 241 [running]:
> k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
>  k8s.io/apimac...@v0.21.1/pkg/util/runtime/runtime.go:55 +0x109
> panic(0x19a30e0, 0x29b7a90)
>  runtime/panic.go:965 +0x1b9
> github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster.(*Controller).pruneServices(0xc0003b4410,
> 0x1e40c88, 0xc00c70c0c0, 0xc000616280, 0xc0138e2b80, 0x2, 0x2, 0xc013cc3860,
> 0xc013cc3650, 0xc001b18e10, ...)
>  github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster/sync_services.go:75 +0x1858
> github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster.(*Controller).syncServices(0xc0003b4410,
> 0x1e40c88, 0xc00c70c0c0, 0xc000616280, 0xc00aee0230, 0xc013cc3860, 0xc013cc3650,
> 0xc013cc3650, 0xc013cc3860, 0xc00aee0230)
>  github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster/sync_services.go:159 +0xdd
> github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster.(*Controller).sync(0xc0003b4410,
> 0x1e40c88, 0xc00c70c0c0, 0xc00053f1d0, 0x29, 0x0, 0x0)
>  github.com/scylladb/scylla-operator/pkg/controller/scyllacluster/sync.go:260
> +0x906
> github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster.(*Controller).processNextItem(0xc0003b4410
> , 0x1e40c18, 0xc00049e1c0, 0xc00049e100)
>  github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster/controller.go:174 +0x17f
> github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster.(*Controller).runWorker(...)
>  github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster/controller.go:198
> k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
>  k8s.io/apimac...@v0.21.1/pkg/util/wait/wait.go:185 +0x37
> k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc0006dbf28)
>  k8s.io/apimac...@v0.21.1/pkg/util/wait/wait.go:155 +0x5f
> k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc01556df28, 0x1e07f40,
> 0xc013cc34a0, 0xc00049e101, 0xc00045cae0)
>  k8s.io/apimac...@v0.21.1/pkg/util/wait/wait.go:156 +0x9b
> k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0006dbf28, 0x3b9aca00, 0x0,
> 0x1, 0xc00045cae0)
>  k8s.io/apimac...@v0.21.1/pkg/util/wait/wait.go:133 +0x98
> k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext(0x1e40c18,
> 0xc00049e1c0, 0xc0006dbf90, 0x3b9aca00, 0x0, 0x1)
>  k8s.io/apimac...@v0.21.1/pkg/util/wait/wait.go:185 +0xa6
> k8s.io/apimachinery/pkg/util/wait.UntilWithContext(...)
>  k8s.io/apimac...@v0.21.1/pkg/util/wait/wait.go:99
> github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster.(*Controller).Run.func2(0xc000640180,
> 0x1e40c18, 0xc00049e1c0, 0xc0003b4410)
>  github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster/controller.go:223 +0xa5
> created by github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster.(*Controller).Run
>  github.com/scylladb/scylla-
> operator/pkg/controller/scyllacluster/controller.go:221 +0x24f
> --
> You received this message because you are subscribed to the Google Groups
> "ScyllaDB users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to scylladb-user...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/scylladb-users/dba56387-ff37-4cb9-9d18-799d74e2646fn%40googlegroups.com
> .

Reply all
Reply to author
Forward
0 new messages