Citus upgrade from 10.2 to 11.3

[Adnan Mohammad]

I have upgraded Citus from 10.2 to 11.3 (postgres 14). This is running in kubernetes. However, after the upgrade I see the below error in logs.  The upgrade and  metadata sync  seems to be successful 
Can someone help me fix this. 

Errors:
ERROR: cache lookup failed for pg_dist_background_task, called too early?

CONTEXT: Citus maintenance daemon for database 16920 user 10
LOG: background worker "Citus Maintenance Daemon: 16920/10" (PID 562) exited with exit code 1
LOG: starting maintenance daemon on database 16920 user 10
CONTEXT: Citus maintenance daemon for database 16920 user 10 WARNING: cache lookup failed for pg_dist_cleanup, called too early?
CONTEXT: Citus maintenance daemon for database 16920 user 10

[Naisila Puka]

Hi Adnan,

I was not able to reproduce this with a simple upgrade from 10.2-5 to 11.3-1 in my local machine. Normally  pg_dist_background_task and pg_dist_cleanup are created in Citus 11.1 in the pg_catalog schema so they should be there 🤔
Are you using the latest patch versions?
Could you share more details?

--
You received this message because you are subscribed to the Google Groups "citus-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to citus-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/citus-users/d2c15d83-59d2-4ae3-8b9e-8f9fab87e0f6n%40googlegroups.com.

[Adnan Mohammad]

Hi Naisila,

Thanks for your response. I was using 10.2 docker image and I was trying to upgrade to 11.2 using the docker image citusdata/citus:11.3.0-pg14. As soon as the pods are up, I connected to the pod and executed ALTER EXTENSION citus UPDATE;  and I confirmed that the version has been upgraded and than executed the command CALL citus_finish_citus_upgrade(); I am not sure if this is some kind of timing issue. I did check the database  and see that  pg_dist_background_task and pg_dist_cleanup are created in the databse. I got the same error for 11.2 upgrade as well  citusdata/citus:11.2.0-pg14.   

[Adnan Mohammad]

sorry for the typo " I was using 10.2 docker image in my exisiting deployment and I was trying to upgrade to 11.3"

[Naisila Puka]

Hi Adnan,

Your steps seem safe.

Did you run the update on the worker nodes as well?

-- run on coordinator and worker nodes
ALTER EXTENSION citus UPDATE;
SELECT * FROM citus_version();

-- run ONLY on coordinator

CALL citus_finish_citus_upgrade();

To view this discussion on the web visit https://groups.google.com/d/msgid/citus-users/eca29ddf-512f-459c-a055-a50799446ea2n%40googlegroups.com.

[Adnan Mohammad]

Yes, I did run the update on worker nodes as well. However What I noticed is as soon as I update the image to 11.3.0 the command  SELECT * FROM citus_version(); shows the version as 11.3 even before executing  ALTER EXTENSION citus UPDATE;