zotonic CI deployment best practice for 2019

[laye...@gmail.com]

Hi,

I have followed zotonic for the last couple of years and am now interested in setting up a production server for zotonic.

I have just spent some time searching zotonic users and zotonic developers for an up to date guide on how you would go about developing and deploying zotonic to a production environment.

most of the links I found range from between 2010 - 2012.

I have looked through the zotonic documentation as well.

I have also installed locally using the zotonic full docker container and the manual install.

I am looking for a guide for Git / continuous integration / continuous deployment guide and any best practice tips.

Is something like this available?

Many thanks

Eddie L

(Uk based)

[ll...@writersglen.com]

Hi Eddie,

 

I'm in a similar situation--- planning soon to release a Nitrogen web application but feeling profound apprehension since I don't have a clear understanding of risks and how to minimize them.

 

No doubt the flip answer is, "It depends..." Are we talking cloud hosting or on-premises? What are projected traffic patterns? Etc. Etc.

 

But "It depends..." doesn't help folks like us who lack experience or organizational support.

 

One can find considerable information on the web covering firewalls, proxy servers, load balancers, site hardening, etc. But, like you, I've found little in the Erlang corpus that provides sufficiently clear patterns and guidelines to assuage my Erlang release/production apprehensions.

 

It may well be a book-length topic or more, but even a thoughtful, thorough tutorial, or even a checklist, would be helpful.

 

Please do let me know what you come up with.

 

All the best,

 

Lloyd

--

---
You received this message because you are subscribed to the Google Groups "Zotonic developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zotonic-develop...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marc Worrell]

Hi Eddie and Lloyd,

“It depends” is indeed the correct answer...

Most Zotonic installations fall in two categories:

1. Deployment via a Docker container, which is updated before deployment

2. Deployment via git (manual or automatic)

Some follow (1),  I follow (2).

Nice thing of (2) is that it allows for hot code upgrades and quick turnaround of small patches.

When we are i a development cycle for new features we might have 10 to 20 deployments per day.

This goes very smooth, we even deploy minor updates to Zotonic and dependencies using hot code upgrades.

Zotonic itself can watch changes in the file system and dynamically load new files.

Most of the updates we deploy have only minor changes.

Think of changes to templates, css, or translations.

And minor Erlang changes (ie. not completely new apps or other dependencies).

That is why we can have updates-via-git without service interruption.

The orgs that deploy via Docker (option 1) - have typically less frequent updates and perform periodic major updates.

As the whole container is updated this also restarts the Zotonic server.

Of course it is also possible to deploy using the OTP release mechanism.

This is something that will be possible with the 1.0 (really soon now) of Zotonic, as that version is OTP compliant.

I personally never had the need to use OTP releases as the version control via git is good enough for us.

(Especially with rebar3 managing versions of dependencies.)

Cheers, Marc

[ll...@writersglen.com]

Hi Marc,

 

Many thanks for your prompt response.

 

This addresses one part of the puzzle. I can't speak for Eddie, but as a total DevOps noob, a big-picture perspective would give me greater comfort as I move into my own deployments.

 

I'm guessing that there are different issues if we're considering on-premises vs. cloud deployment, so considering the two hosting scenarios...

 

What are the specific security, scaling, or other issues we need to attend to? And how do we implement them? For instance, if on-premises: Do we need a dmz to protect our LAN or is port-forwarding sufficient? Should we put a proxy server such as nginx or HAProxy in front of our app? How can we most effectively harden our servers and applications against exploits?

 

I understand much of this goes beyond the realm of Zotonic, Erlang, and Nitrogen. But when I step out into the web for answers I'm totally overwhelmed with confusing and oft contradictory info.

 

I'd love to create a living-document tutorial and checklist for, at least, the simplest deployments, but wouldn't know where to start.

 

Thanks again,

 

Lloyd

[Marc Worrell]

Hi Lloyd (and rest of list),

For Zotonic I have seen two setups:

1. Only Zotonic (Erlang)  (single machine)

With this setup external port 80 is redirected (using iptables prerouting rules) to 127.0.0.1:8000.

And port 443 to 8433. Erlang is directly handling all (SSL) traffic.

Same is done with the SMTP port 25, which is usually mapped to 2525.

2. Using a proxy (one or more machines)

In this haproxy or nginx are used to terminate the SSL connections and proxy requests to Zotonic/Erlang.

The Zotonic node is either running locally or on some other host in local network.

In this setup it is also common to have a mail server running to forward incoming email to the Zotonic/Erlang node.

Setup 1 (direct) is the easiest for small single-server operations.

Setup 2 (proxy) is when you want to have multiple machines and a local network.

We have setup 1 running on a €50/month VPS, handling monthly traffic of 1.5+ TB (~2M monthly visitors).

Another company is running setup 1 on dedicated hardware with more than 1M hourly requests.

(That server is mostly idle…)

So for most (98%?) sites the simple setup is actually very realistic.

You just need to be sure that you have a good backup scheme, as it is a single machine.

I know of some people that are working on a “lukewarm” failover setup.

With the single VPS server solution we didn’t have any significant down time in the last years and

usually have 100% monthly uptime (according to pingdom).

So, for me, the single server solution works best.

Especially with a VPS where the hosting company moves the VPS in case of any hardware problems.

Cheers,

Marc

[laye...@gmail.com]

Hello Marc and Lloyd,

Lloyd thanks for Joining in the conversation. Hope we can both get something running on Zotonic, or in your case Nitrogen.

Marc thanks for the additional update. You seem to have answered some of my questions which were around the server spec and the Zotonic configuration

My interest in Zotonic (Erlang) runs alongside my existing interest in Elixir and Phoenix.

Certainly like to go with Zotonic 1.0 when its available.

Both types of deployment interest me - container v manual installation. They both have advantages / disadvantages. It would be good to see both types documented too. 

But to start off I would like to use your proven Git deployment process if thats possible. 

The least path of resistance for now and build up experience from there.

That also goes with your recommendation that a single server will suffice for most cases.

Do you have a server setup you could share?

I am looking at deploying zotonic to a server with the capacity to support a number of individual sites using digital Oceon or https://gigalixir.com/

See https://gigalixir.readthedocs.io/en/latest/main.html#tiers for info

Gigalixir is an Erlang / elixir friendly provider but not sure how economical it will be for hosting Zotonic or elixir apps for that matter.

 

As for the Zotonic configuration. Does it make sense to use  setup 2 on one machine initially? Do you still recommend varnish too for the proxy or just go with Haproxy / Nginx?

Happy to document Gigalixir setup and share on Github with Lloyd (with everybody).

At the end of the day I would like to have a setup doc for Zotonic using one server that could be used to also to build a secure production container.

Using this as an example of a manual server build - https://www.digitalocean.com/community/tutorials/how-to-automate-elixir-phoenix-deployment-with-distillery-and-edeliver-on-ubuntu-16-04

Although Giglixir has a lot of that built in.

So in summary target is to install proxy + Zotonic + N sites on single Gigalixir VPS or container. 

Eddie L

Hi Eddie and Lloyd,

To unsubscribe from this group and stop receiving emails from it, send an email to zotonic-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Zotonic developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zotonic-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Zotonic developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zotonic-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Zotonic developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zotonic-developers+unsub...@googlegroups.com.

[laye...@gmail.com]

Hi Marc,

Just talking to some colleagues who suggest that gigalixir might be over the top. They are suggesting that I do something like Git push > circleCI -> deploy to heroku.(or other provider)

something along these lines.

http://www.theproduct.works/blog-of-works/continuous-integration-deployment-with-erlang-18

Eddie 

On Tuesday, January 15, 2019 at 8:40:05 AM UTC, Marc Worrell wrote:

Hi Eddie and Lloyd,

To unsubscribe from this group and stop receiving emails from it, send an email to zotonic-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Zotonic developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zotonic-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Zotonic developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zotonic-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Zotonic developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zotonic-developers+unsub...@googlegroups.com.

[ll...@writersglen.com]

Hi Marc,

 

Really helpful!

 

You're the best.

[ll...@writersglen.com]

Hi Eddie,

 

By all means lets keep in touch on this. We're running on parallel tracks.

 

Thanks for opening up the thread.

 

Best wishes,

Hi Eddie and Lloyd,

To unsubscribe from this group and stop receiving emails from it, send an email to zotonic-develop...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Zotonic developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zotonic-develop...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Zotonic developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zotonic-develop...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Zotonic developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zotonic-develop...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Zotonic developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zotonic-develop...@googlegroups.com.

[heiheshang]

I use docker and hosting for $ 5 and everything works

понедельник, 14 января 2019 г., 23:28:01 UTC+9 пользователь laye...@gmail.com написал:

[Lloyd R. Prentice]

Thanks. Good to know.

Lloyd

Sent from my iPad

--

[laye...@gmail.com]

Hello Marc,

1. Do you have a date yet for 1.0?

I am trying to build a zotonic image to run on digital oceon. Once I can see it running in a manually created droplet I want to build and save an image on digital oceon using hashicorp Packer

then I can build a webserver using hashicorp terraform. Trying to get a repeatable build setup using Hashicorps popular tooling.

I have installed the basics using

sudo apt-get update

sudo apt-get install build-essential git erlang imagemagick postgresql

I have not carried out any extra configuration on this digitaloceon server at all - its just the basic docker + ubuntu 18.04 and I am running as root. 

(I will delete this droplet as soon as I have zotonic working and build a more secure basic server)

2. I just tried getting zotonic running in a basic digitaloceon droplet running docker on ubuntu 18.04 using the cmd below from the zotonic docs.

it fails like this... 

root@docker-s-1vcpu-1gb-lon1-01:~/zotonic# docker run -v `pwd`:/opt/zotonic/user/sites -p 8443:8443 zotonic/zotonic-heavy
+ HOME=/tmp
+ ZOTONIC_PIDFILE=/tmp/zotonic.pid
+ ZOTONIC_CONFIG_DIR=/etc/zotonic
+ SHELL=/bin/sh
+ export HOME ZOTONIC_PIDFILE ZOTONIC_CONFIG_DIR SHELL
+ touch /run/zotonic.pid
+ chown zotonic /run/zotonic.pid
+ mkdir /opt/zotonic/priv
+ chown -R zotonic /opt/zotonic/priv
+ chown -R zotonic /opt/zotonic/_build/default/lib/mimetypes/ebin
chown: /opt/zotonic/_build/default/lib/mimetypes/ebin: No such file or directory
+ chown -R zotonic /opt/zotonic/_build/default/lib/zotonic_site_status/priv/
chown: /opt/zotonic/_build/default/lib/zotonic_site_status/priv/: No such file or directory
+ sed -i -e s/{password, ""}/{password, ""}/ /etc/zotonic/zotonic.config
+ exec /usr/bin/gosu zotonic /opt/zotonic/bin/zotonic start-nodaemon
2019-01-28 17:09:07.467 UTC [9] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2019-01-28 17:09:07.468 UTC [9] LOG:  could not bind IPv6 address "::1": Address not available
2019-01-28 17:09:07.468 UTC [9] HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
2019-01-28 17:09:07.471 UTC [9] LOG:  listening on Unix socket "/run/postgresql/.s.PGSQL.5432"
2019-01-28 17:09:07.476 UTC [9] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2019-01-28 17:09:07.530 UTC [55] LOG:  database system was shut down at 2019-01-23 17:31:35 UTC
2019-01-28 17:09:07.534 UTC [9] LOG:  database system is ready to accept connections
Zotonic has not been compiled and cannot run. Exiting.
2019-01-28 17:09:12.870 UTC [9] LOG:  received smart shutdown request

3. Installing from git it fails with this after running bin/zotonic debug (it compiles OK with erlang 20.0)

16:56:36.867 [info] Application zotonic_notifier started on node 'zotonic001@docker-s-1vcpu-1gb-lon1-01'
16:56:36.867 [info] Application zotonic_core started on node 'zotonic001@docker-s-1vcpu-1gb-lon1-01'
16:56:36.906 [error] CRASH REPORT Process exec with 0 neighbours exited with reason: bad return value: "Port program /root/zotonic/_build/default/lib/erlexec/priv/x86_64-pc-linux-gnu/exec-port with SUID bit set is not allowed to run without setting effective user!" in gen_server:init_it/6 line 357
16:56:36.906 [error] Supervisor exec_app had child exec started with exec:start_link([]) at undefined exit with reason bad return value: "Port program /root/zotonic/_build/default/lib/erlexec/priv/x86_64-pc-linux-gnu/exec-port with SUID bit set is not allowed to run without setting effective user!" in context start_error
16:56:36.906 [error] CRASH REPORT Process <0.310.0> with 0 neighbours exited with reason: {{shutdown,{failed_to_start_child,exec,{bad_return_value,"Port program /root/zotonic/_build/default/lib/erlexec/priv/x86_64-pc-linux-gnu/exec-port with SUID bit set is not allowed to run without setting effective user!"}}},{exec_app,start,[normal,[]]}} in application_master:init/4 line 134
16:56:36.907 [error] zotonic:49 Zotonic start error: {{shutdown,{failed_to_start_child,exec,{bad_return_value,"Port program /root/zotonic/_build/default/lib/erlexec/priv/x86_64-pc-linux-gnu/exec-port with SUID bit set is not allowed to run without setting effective user!"}}},{exec_app,start,[normal,[]]}}
16:56:36.907 [info] Application erlexec exited with reason: {{shutdown,{failed_to_start_child,exec,{bad_return_value,"Port program /root/zotonic/_build/default/lib/erlexec/priv/x86_64-pc-linux-gnu/exec-port with SUID bit set is not allowed to run without setting effective user!"}}},{exec_app,start,[normal,[]]}}

Many thanks

Edlay

Hi Eddie and Lloyd,

To unsubscribe from this group and stop receiving emails from it, send an email to zotonic-developers+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Zotonic developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zotonic-developers+unsub...@googlegroups.com.

[Marc Worrell]

Hi Eddie,

Hope David (or someone else) can chime in on the docker question, as that is quite unknown territory for me.

For the 1.0, we are currently working on getting the sessions re-worked.

This would make the admin working again, and gets us a lot closer to merging cotonic into zotonic.

Then we would need to:

 - Merge in some changes from 0.x

 - Fix uploads of files

 - Tests

 - Update the docs (so people don’t get too confused)

At this point we are postponing major UI changes.

Only the logon/logout forms will see quite some changes, as they interact with the new transports and session management.

After the first release we will add deeper integration of the new UI (Cotonic) code and security system.

I still hope to get quite far by February.

However, got some high priority projects during this period which might push it to March.

Cheers, Marc

To unsubscribe from this group and stop receiving emails from it, send an email to zotonic-develop...@googlegroups.com.