Hello and multi-stage Docker build

[James Hillyerd]

Hi! Long time gopher first time keeper.

I wanted to run Perkeep on my NAS (intel based QNAP), which supports Linux containers.  Unfortunately, I found that the official Perkeep Docker builds are focused on integration tests and would be hard to setup as an automated build on hub.docker.com.

Thus, I created a new multi-stage Docker build, plus a bit of bash and Go to create the initial server-config.json and PGP key -- making it easy for people to get started.  For those that aren't aware, multi-stage means that perkeep is built inside one container with all the compiler tools, then installed into a much smaller one.  The final image is about 44MB.

I'd appreciate it if folks could sanity check both the Dockerfile and the template I'm using for the server-config.json.  I'm not clear on if Perkeep benefits from dependencies beyond libjpeg-turbo-utils, please let me know if there is any others I should add.

If you'd like to try the image out for yourself without building it, it's available at https://hub.docker.com/r/jhillyerd/perkeep/  Currently both :stable and :latest tags fetch Perkeep 2017-05-05, but I plan to move :latest to a newer version in the future.  Github repo: https://github.com/jhillyerd/perkeep-docker

Happy to contribute any/all back to the official project if people find it useful.

-james

[Brad Fitzpatrick]

Nice!

I can look more tomorrow or Friday, but sounds great.

We've wanted this for awhile now (e.g. https://github.com/perkeep/perkeep/issues/986)

Thanks!

--
You received this message because you are subscribed to the Google Groups "Camlistore" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camlistore+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mathieu Lonjaret]

On 22 February 2018 at 04:40, James Hillyerd <ja...@hillyerd.com> wrote:

Hi! Long time gopher first time keeper.

Hi,

 

I wanted to run Perkeep on my NAS (intel based QNAP), which supports Linux containers.  Unfortunately, I found that the official Perkeep Docker builds are focused on integration tests and would be hard to setup as an automated build on hub.docker.com.

I take it you have not seen what we have in ./misc/docker ?

Thus, I created a new multi-stage Docker build, plus a bit of bash and Go to create the initial server-config.json and PGP key -- making it easy for people to get started.  For those that aren't aware, multi-stage means that perkeep is built inside one container with all the compiler tools, then installed into a much smaller one.  The final image is about 44MB.

Yep, I believe that's what we do too for ./misc/docker/server

perkeep/server              latest              8e0ffbe8c1b2        5 weeks ago         31.5MB

:-)

 

I'd appreciate it if folks could sanity check both the Dockerfile and the template I'm using for the server-config.json.  I'm not clear on if Perkeep benefits from dependencies beyond libjpeg-turbo-utils, please let me know if there is any others I should add.

If you'd like to try the image out for yourself without building it, it's available at https://hub.docker.com/r/jhillyerd/perkeep/  Currently both :stable and :latest tags fetch Perkeep 2017-05-05, but I plan to move :latest to a newer version in the future.  Github repo: https://github.com/jhillyerd/perkeep-docker

Happy to contribute any/all back to the official project if people find it useful.

-james

--

[James Hillyerd]

Hi Mathieu, I did play around with misc/docker/server a bit, I remember having some trouble with the build process until I figured out I needed to move perkeep into my GOPATH for dock.go to work.

IMHO, some of the benefits of my Docker build:

• Builds entirely within a Docker container, no local Go or perkeep build environment required
• This makes the build more deterministic and documented
• The build compatible with hub.docker.com automated builds
  
• Docker hub means anybody can 'docker pull jhillyerd/perkeep', they don't need to build it themselves
• New Perkeep users setting up a NAS may not have access to a docker environment
• Build logs are available along with the Dockerfile that produced them, so you know what you are getting.
• The resulting image is a bit more user friendly, exporting distinct /config and /storage volumes that work out of the box
  

The downside right now is that the :stable and :latest tags are triggered by pushes to my github, and not the project itself.  It's also only an amd64 built right now, I've never tried to do docker with arm - I haven't surveyed the NAS landscape to know what is worth supporting via Docker.

[Mathieu Lonjaret]

On 22 February 2018 at 17:47, James Hillyerd <ja...@hillyerd.com> wrote:

Hi Mathieu, I did play around with misc/docker/server a bit, I remember having some trouble with the build process until I figured out I needed to move perkeep into my GOPATH for dock.go to work.

yeah probably, we're still "old school" about that.

IMHO, some of the benefits of my Docker build:

• Builds entirely within a Docker container, no local Go or perkeep build environment required

right. because it's simpler for us to write Go than shell, we use ./misc/docker/dock.go to bind all the pieces together. But that does come with the price of requiring Go and (a small bit, that could be removed) of Perkeep.

• This makes the build more deterministic and documented
• The build compatible with hub.docker.com automated builds
  
• Docker hub means anybody can 'docker pull jhillyerd/perkeep', they don't need to build it themselves

Yes. We've been wanting to push more images at  https://hub.docker.com/r/camlistore but we've been busy with other things. We do have https://storage.googleapis.com/camlistore-release/docker/camlistored.tar.gz that we keep relatively up to date though.

• New Perkeep users setting up a NAS may not have access to a docker environment
• Build logs are available along with the Dockerfile that produced them, so you know what you are getting.

Neat, I did not know about that.

 

• The resulting image is a bit more user friendly, exporting distinct /config and /storage volumes that work out of the box
  

Yes, that's nice. We haven't worked on such subtleties yet because our main use for the docker image is to deploy on GCE, where camlistored already knows by itself to work with buckets, and where we don't really expect the user to tweak things.