Upgrading Citus 5.0 from 4.0 on AWS ec2

[Sal]

Since I am getting errors about missing libraries, I think I might be missing something with my AWS upgrade of citus db 5.0. Here are the steps that I did for installing postgresql 9.5.1 on Citus 4.0 master node:

- Download, compile and install PostgreSQL 9.5.1 from source - installed in /usr/local/pgsql

- Install Citusdb extensions as below:

 

- Download and compile pg_shard-develop (version 1.2.3 in META.json) - confirm pg_shard.so has been copied to /usr/local/pgsql/lib
 - Download and compile cstore_fdw-1.4 (version 1.4.0 in META.json) - confirm cstore_fdw.so has been copied to /usr/local/pgsql/lib

It seems I have missed something in the above steps I cobbled together myself (since online documentation is not clear on how to upgrade from 4.0 to 5.0 with citus extensions). Now, when I run pg_upgrade command as documented here, I get the following errors - will appreciate pointers on resolving these errors:

Could not load library "$libdir/shard_rebalancer"
ERROR:  could not access file "$libdir/shard_rebalancer": No such file or directory
 
Could not load library "$libdir/hll"
ERROR:  could not access file "$libdir/hll": No such file or directory

[Metin Doslu]

Hi Sal,

I think that shard_rebalancer and hll extensions are installed on Citus 4.0, but not on your new PostgreSQL 9.5.1. Could you update shared_preload_libraries to include the only citus in postgresql.conf at your newly created data directory, most likely it is $PGDATA5_0/postgresql.conf

shared_preload_libraries = 'citus'

And could you make sure that you started the correct the data directory?

Best,
Metin

--
You received this message because you are subscribed to the Google Groups "citus-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to citus-users...@googlegroups.com.
To post to this group, send email to citus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/citus-users/9e8e63b5-95ac-4e84-a493-9769be750fd2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sal]

Hi Metin, 

What is a good way to validate that citus shared library is present before adding to postgresql.conf? If I add it to $PGDATA5_0/postgresql.conf, during server start, I get:

FATAL:  could not access file "citus": No such file or directory

I know for sure that pg_shard.so and cstore_fdw.so are present under Postgresql 9.5 lib. I don't think I ever compiled citus.so for Postgresql 9.5. In the downloads section, I can see only pg_shard and cstore_fdw extensions. Please let me know if I missed anything. 

[Sal]

Also, one more thing - I am currently stuck at step #6 documented here. Step #11 is where the documentation mentions adding citus extension as you recommended. 

[Metin Doslu]

Hi Sal,

One step back, can I ask you one question? Why do you start with Citus 4.0 and want to upgrade to Citus 5.0. If this is a new cluster for you, from the Cloudformation, you can just start with a Citus 5.0 cluster. Here is the link to the documentation:

https://www.citusdata.com/docs/citus/5.0/installation/multi_node_aws.html#multi-node-aws

For your current problem, I think the problem is that you didn't install citus extension (Citus became an extension with the 5.0 release).

Citus packages for RHEL are available through official PostgreSQL repositories (PGDG), and packages for Ubuntu will be made available in the upcoming days.

First, you need to install pgdg package depending on your OS version, such as;

sudo yum install -y https://download.postgresql.org/pub/repos/yum/9.5/redhat/rhel-6-x86_64/pgdg-ami201503-95-9.5-2.noarch.rpm

Then, you can install Citus. This will install the PostgreSQL 9.5 dependency and then Citus 5.0.

sudo yum install -y citus_95

Best,

Metin

To view this discussion on the web visit https://groups.google.com/d/msgid/citus-users/1ff69e9d-d1e2-47e5-9294-4e8e6016bb3a%40googlegroups.com.

[Sal]

Hi Metin,

It is an existing cluster that needs to be upgraded since we have API servers connecting to the databases on that cluster. So, instead of setting up load balancers and everything else for new cluster, it is just easier to upgrade it as long as it is quite painless. Otherwise I will stick with Citus 4.0 for now.

Followed the yum steps below, and got citus.so but under /usr/pgsql-9.5. Would it be ok to move them all to /usr/local/pgsql which I compiled with pg_shard and cstore_fdw? Or I can recompile postgresql against /usr/pgsql-9.5 if that is easier.

Thanks.

[Sal]

Ok, so now I downloaded and compiled citus v5.0.0 against my postgresql 9.5 install, and added "shared_preload_libraries = 'citus'" to 

$PGDATA5_0/postgresql.conf. Now, it is back to original problem of not able to locate "shard_rebalancer" and "hll" libraries in step #6. Is there a way to upgrade Citus 4.0 to 5.0? I can download and compile those if someone can point me to their source code.

[Andres Freund]

Hello Sal,

What exactly are you doing when you get those error messages? I guess
you are able to actually start the server, and get them when executing
certain queries? Are you actively using the hll datatype and/or the
shard rebalancer?

hll is available at https://github.com/aggregateknowledge/postgresql-hll

shard_rebalancer is not open source, it's part of citus' enterprise
version, and as such not part of citus 5.0.

Regards,

Andres Freund

> >>>>> <https://www.citusdata.com/docs/citus/5.0/installation/multi_node_aws.html#multi-node-aws>,

> >>>>> I get the following errors - will appreciate pointers on resolving these
> >>>>> errors:
> >>>>>
> >>>>> Could not load library "$libdir/shard_rebalancer"
> >>>>> ERROR: could not access file "$libdir/shard_rebalancer": No such
> >>>>> file or directory
> >>>>>
> >>>>> Could not load library "$libdir/hll"
> >>>>> ERROR: could not access file "$libdir/hll": No such file or directory
> >>>>>
> >>>>>
> >>>>> --
> >>>>> You received this message because you are subscribed to the Google
> >>>>> Groups "citus-users" group.
> >>>>> To unsubscribe from this group and stop receiving emails from it, send
> >>>>> an email to citus-users...@googlegroups.com.
> >>>>> To post to this group, send email to citus...@googlegroups.com.
> >>>>> To view this discussion on the web visit
> >>>>> https://groups.google.com/d/msgid/citus-users/9e8e63b5-95ac-4e84-a493-9769be750fd2%40googlegroups.com

> >>>>> <https://groups.google.com/d/msgid/citus-users/9e8e63b5-95ac-4e84-a493-9769be750fd2%40googlegroups.com?utm_medium=email&utm_source=footer>
> >>>>> .

> >>>>> For more options, visit https://groups.google.com/d/optout.
> >>>>>
> >>>>
> >>>> --
> >>> You received this message because you are subscribed to the Google
> >>> Groups "citus-users" group.
> >>> To unsubscribe from this group and stop receiving emails from it, send
> >>> an email to citus-users...@googlegroups.com.
> >>> To post to this group, send email to citus...@googlegroups.com.
> >>> To view this discussion on the web visit
> >>> https://groups.google.com/d/msgid/citus-users/1ff69e9d-d1e2-47e5-9294-4e8e6016bb3a%40googlegroups.com

> >>> <https://groups.google.com/d/msgid/citus-users/1ff69e9d-d1e2-47e5-9294-4e8e6016bb3a%40googlegroups.com?utm_medium=email&utm_source=footer>
> >>> .

> >>>
> >>> For more options, visit https://groups.google.com/d/optout.
> >>>
> >>
> >>
>
> --
> You received this message because you are subscribed to the Google Groups "citus-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to citus-users...@googlegroups.com.
> To post to this group, send email to citus...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/citus-users/2bc801ff-d262-452d-8e18-7f1a09fc0245%40googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

Greetings,

Andres Freund

[Sal]

I am just following the upgrade steps listed here. The errors that I mentioned happen in step 6.

Here is what I did:

- Download and compile postgresql 9.5

- Download and compile pg_shard, cstore_fdw, citus - install under postgresql 9.5

- Do steps 2-6 of documented upgrade steps. Ran into errors in step 6 (no data loading, just upgrade compatibility check)

I am puzzled why I have shard_rebalancer in my existing citus 4.0 installation which is just AWS cloud formation. I am happy to remove it or completely disable it in the upgrade if someone tells me how to.

I am not using hll datatype either - happy to remove it too if possible. It seems it too came from citus 4.0 install.

[Andres Freund]

Hi,

First off, there's no need to install pg_shard with citus 5.0 - its functionality is now included in citus itself.

With regard to the problem you're observing, you might be able to work around the problem by dropping the hll and shard_rebalancer extensions from all databases before upgrading (i.e. execute DROP EXTENSION ...;).

Regards,

Andres

To view this discussion on the web visit https://groups.google.com/d/msgid/citus-users/3a6888e6-ffb2-452f-a733-dd323484ac32%40googlegroups.com.

[Sal]

Perfect, thank you so much for clarification. Dropping both extensions did the trick of making pg_upgrade happy. I also dropped pg_shard extension, and removed it from postgresql.conf shared setting after successful pg_upgrade check.

Now, it is up and running on 5.0. I need to repeat this on slave server now.

Do we need cstore_fdw extension? Or is that too redundant by citus extension?

[Andres Freund]

Hi Sal,

Good to hear!

No cstore_fdw is not part of citus.

Regards,

Andres

On 2016-04-13 12:23:29 -0700, Sal wrote:
> Perfect, thank you so much for clarification. Dropping both extensions did
> the trick of making pg_upgrade happy. I also dropped pg_shard extension,
> and removed it from postgresql.conf shared setting after successful
> pg_upgrade check.
>
> Now, it is up and running on 5.0. I need to repeat this on slave server now.
>
> Do we need cstore_fdw extension? Or is that too redundant by citus
> extension?
>
> On Wednesday, April 13, 2016 at 2:51:54 PM UTC-4, Andres Freund wrote:
> >
> > Hi,
> >
> > First off, there's no need to install pg_shard with citus 5.0 - its
> > functionality is now included in citus itself.
> >
> > With regard to the problem you're observing, you might be able to work
> > around the problem by dropping the hll and shard_rebalancer extensions from
> > all databases before upgrading (i.e. execute DROP EXTENSION ...;).
> >
> > Regards,
> >
> > Andres
> >

> > On Wed, Apr 13, 2016 at 11:49 AM, Sal <sanket....@gmail.com <javascript:>>

> > wrote:
> >
> >> I am just following the upgrade steps listed here

> >> <https://www.citusdata.com/docs/citus/5.0/admin_guide/upgrading_citus.html>.

> >> The errors that I mentioned happen in step 6.
> >>
> >> Here is what I did:
> >> - Download and compile postgresql 9.5
> >> - Download and compile pg_shard, cstore_fdw, citus - install under
> >> postgresql 9.5
> >> - Do steps 2-6 of documented upgrade steps. Ran into errors in step 6 (no
> >> data loading, just upgrade compatibility check)
> >>
> >> I am puzzled why I have shard_rebalancer in my existing citus 4.0
> >> installation which is just AWS cloud formation. I am happy to remove it or
> >> completely disable it in the upgrade if someone tells me how to.
> >>
> >> I am not using hll datatype either - happy to remove it too if possible.
> >> It seems it too came from citus 4.0 install.
> >>
> >>
> >> On Wednesday, April 13, 2016 at 2:18:09 PM UTC-4, Andres Freund wrote:
> >>
> >>> Hello Sal,
> >>>
> >>> What exactly are you doing when you get those error messages? I guess
> >>> you are able to actually start the server, and get them when executing
> >>> certain queries? Are you actively using the hll datatype and/or the
> >>> shard rebalancer?
> >>>
> >>> hll is available at https://github.com/aggregateknowledge/postgresql-hll

> >>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Faggregateknowledge%2Fpostgresql-hll&sa=D&sntz=1&usg=AFQjCNEG_C_GTXGMwNGm5iRpIw-ENw_cVA>

> >> email to citus-users...@googlegroups.com <javascript:>.

> >> To post to this group, send email to citus...@googlegroups.com

> >> <javascript:>.

> >> To view this discussion on the web visit

> >> https://groups.google.com/d/msgid/citus-users/3a6888e6-ffb2-452f-a733-dd323484ac32%40googlegroups.com
> >> <https://groups.google.com/d/msgid/citus-users/3a6888e6-ffb2-452f-a733-dd323484ac32%40googlegroups.com?utm_medium=email&utm_source=footer>

> >> .
> >>
> >> For more options, visit https://groups.google.com/d/optout.
> >>
> >
> >
>
> --
> You received this message because you are subscribed to the Google Groups "citus-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to citus-users...@googlegroups.com.
> To post to this group, send email to citus...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/citus-users/f619e8f2-e882-4f31-ad5c-f4ec9fb48bd8%40googlegroups.com.