GPDB on Native Docker for Mac / CentOS

[Amil Khanzada]

Hi all,

In our team, we have been setting up mounted/native Docker on Mac OSX with CentOS 7 for development because we rely on resource groups which only work on Linux.

Good news is that it is at a point where our code can install and run on a gpdemo cluster in Docker!

As mentioned in the below thread ("Development and initial-trial platforms: governance over adding / removing "support""), it seems we have some out-of-date instructions already on GPDB for the older Docker machine. Unfortunately, they did not work on my Mac a couple of weeks back:

https://github.com/greenplum-db/gpdb/blob/master/README.md#development-with-docker

https://hub.docker.com/r/pivotaldata/gpdb-devel/

https://github.com/greenplum-db/gpdb/tree/master/src/tools/docker

Because we've got this working, for next steps I was thinking were:

1. Create a README.docker.md with what we've done

2. Possibly removing the current instructions and the src/tools/docker folder.

Any thoughts or ideas before we PR?

Amil

---------- Forwarded message ----------
From: Heikki Linnakangas <hlinna...@pivotal.io>
Date: Thu, Oct 12, 2017 at 11:25 PM
Subject: Re: Development and initial-trial platforms: governance over adding / removing "support"
To: Robert Eckhardt <reck...@pivotal.io>, Daniel Gustafsson <dgust...@pivotal.io>
Cc: Greenplum Developers <gpdb...@greenplum.org>, "C.J. Jameson" <cjam...@pivotal.io>


On 10/13/2017 06:10 AM, Robert Eckhardt wrote:

On Oct 12, 2017 6:07 PM, "Daniel Gustafsson" <dgust...@pivotal.io> wrote:

On 12 Oct 2017, at 17:56, C.J. Jameson <cjam...@pivotal.io> wrote:

3. On this particular question, should we veer towards more platforms /

vehicles, or fewer but higher quality?

Regardless of fewer/more (I have less of a strong opinion there), the ones
we
do adopt into the tree should be considered first class citizens that we as
a
community collectively have a responsibility to keep as green as the build
pipeline.

+1

We also need enough ways to have a GPDB cluster so that developers can
build things for gpdb.  Having a strong community means not only people
being able to contribute code to gpdb but also allowing them to build the
ecosystem of tools surrounding gpdb.

Agreed.

To make that possible, let's make GPDB easy to package. That means:

* Minimize dependencies. Reuse is good, but often re-implementing something is better, after all, if it avoids having to link with an extra library.

* Use programming languages, compilers, etc. that we already depend on.

* Use autoconf checks for any libraries or versions that you do depend on. That way you at least get a nice "libfoo version 1.23 or later is needed". Use #ifdefs and configure options to make it possible to build without extra dependencies.

* Write regression tests in a way that doesn't unnecessarily depend on optional features or tools.

Easy packaging will make the "what are the supported platforms?" question less relevant, because you can easily build and run it on any platform, with few changes. And it will make our life easier, by reducing the amount of platform-specific scripts and code that we need to maintain.

- Heikki

On Thu, Oct 12, 2017 at 5:56 PM, C.J. Jameson <cjam...@pivotal.io> wrote:

Hi all,

I wanted to follow-up on a discussion many of the committers held recently. On the one hand, we want many avenues for people to hack on our code and give Greenplum a spin. On the other hand, we're concerned about the quality and maintenance of these delivery mechanisms.

Especially to less-involved members of the Greenplum community, we're interested in your thoughts on this tradeoff, and your experiences getting started with GPDB!

A prime example is Docker. There is a Dockerfile at the root of the main repo (https://github.com/greenplum-db/gpdb/blob/master/Dockerfile) and lots of people are familiar with Docker and would love to use it. However, we don't publish an image from that dockerfile, it hasn't been updated in 4 months, and confusingly, we do use different Docker-compatible images for the majority of our Concourse testing pipelines.

Another example is the brew tap (https://github.com/greenplum-db/homebrew-tap) -- it's for MacOS convenience and currently works. But it's maintained by best-effort only, still requires a number of manual steps (not ideal customer experience), and MacOS isn't an OS that users would use in production.

In similar veins, we could talk about the Vagrantfile, the OS-specific READMEs, plans for rpm / deb packaging ( --> package managers? ).

1. What's our objective in having platforms beyond `./compile; make; make install`?

2. How do we make a decision on new platforms / vehicles? (how does governance work?)

3. On this particular question, should we veer towards more platforms / vehicles, or fewer but higher quality?

Thanks,
C.J.

[C.J. Jameson]

Hi Amil,

Sounds like a good plan, I'm excited for the enthusiasm from multiple corners to get our "Development in Docker" story sorted out, and have folks using similar things.

See the following pull request: https://github.com/greenplum-db/gpdb/pull/3549

I prefer your approach, using the same docker images that the tests running in Concourse use.

Regardless:

 - Yes, cleaning up old cruft, adding more documentation and tips, all this is good, so go for it

 - Per Daniel and others' urging, we want something that stays up-to-date in CI. I think that a bar we should aim for is: we should open source the Dockerfiles etc. that are used to create the images.

Thanks,
C.J.

[Amil Khanzada]

Hi C.J.,

Thanks for the quick reply and linking us to that PR!

Ben and I are planning to do this:

1. Create a README.docker.md and link to it in README.md as a new section ("Development with Native Docker Client"). Open a PR for this.

2. Create a new PR to remove the old section and src/tools/docker

-Ben and Amil

[Amil Khanzada]

Hi C.J.,

We have created the two PRs as promised.

https://github.com/greenplum-db/gpdb/pull/3588

https://github.com/greenplum-db/gpdb/pull/3589

Perhaps we can continue the discussions there.

Best,

Amil and Ben