Docker KoBoToolbox behind reverse proxy server with SSL support

[jpstaub]

WARNING! - Use this information at your own risk. I am not a web developer. I am not a data security specialist. I am a guy making open source software work through determination rather than fundamental understanding. 

This an informational post that describes a method to run a Docker KoBoToolbox instance behind a reverse proxy server with SSL support. The method set up in the included files supports either http or http access to KoBoToolbox using one domain name. Currently it does not support alias domains since the domain name and its implementation is bound up in the mechanization of kobotoolbox_nginx. From a technical standpoint, there is no reason why alias domains could not be incorporated. It hasn't been done here more for time considerations than anything else. 

Q1: Why bother with the additional complexity of a reverse proxy?
A1: Click here to read all about it.

Q2: If SSL is so good why is it such a pain in the ass to implement?
A2: I have no idea. Fortunately, there are people in this world that believe in helping a brother/sister out.

Background:
Docker-KoBoToolbox is a great way to get up and running on a Docker host that isn't being used for much other than KoboToolbox. My problem was that I am using a Docker host for a bunch of services including other web servers. Also, I prefer to give the services I run domain names so other people can interact with them. And when a service is meant for the web, I prefer to secure it with SSL. That's why I use jwilder/nginx-proxy (reverse proxy) combined with docker-letsencrypt-nginx-proxy-companion (letsencrypt).

Unfortunately, Docker-KoBoToolbox, neither local or server versions, is set up to run behind a reverse proxy . Additionally, the way to use domain names is by using the server version.  But the server version doesn't appear to work without an SSL certificate. For people that work with SSL certificates regularly that's probably not a big deal. For me it's a pain in the ass especially if all I want to do is run the service on the LAN. Plus, self signed certificates these days give browsers massive chest pains even if you manage to create them by painstakingly following all the poorly contrived instructions on the web.

Solution:
In general, the solution amounted to hijacking "docker-compose.server.yml" and "nginx_site_https.conf.tmpl" for my own purposes. "docker-compose.server.yml" was modified to make kobotoolbox_nginx compatible with the reverse proxy & letsencrypt combo. "nginx_site_https.conf.tmpl" was stripped down so kobotoolbox_nginx acts as an http server only. This way, KoBoToolbox can be run locally as an http service using domain names. KoBoToolbox can also be run over the web as an https service with a trusted certificate provided and updated automagically by letsencrypt. The modified "docker-compose.server.yml" allows for the quick & easy configuration of KoBoToolbox to run either locally (http) or over the web (https).  

To implement the solution:

• Prerequisites - 1. jwilder/nginx-proxy 2. docker-letsencrypt-nginx-proxy-companion. Both containers should be up prior to starting KoBoToolbox.
• Configure "envfile.server.txt" with the domain and subdomain names of choice.
• Swap out "docker-compose.server.yml" and "nginx_site_https.conf.tmpl" with the attached files.
• Configure "docker-compose.server.yml" for http or https and the domain and subdomain names of choice.
• Give it a try and troubleshoot as required.
• Do others a solid and post additional information/solutions.

Todo:

• Implement domain alias
• Implement domain/subdomain variables within "docker-compose.server.yml" to take advantage of settings in "envfile.server.txt"
  

Regards,

Jake

[Kaleem Qureshi]

Please help me I am stuck in deployment. All the services running perfectly. but no application is accessible thriough browser. My setup is given below:


1. Virtual Box 4.3

2. Linux ubuntu 14.04 LTS

3. Docker version

 Client:
 Version:      17.03.1-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:10:36 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.1-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:10:36 2017
 OS/Arch:      linux/amd64
 Experimental: false
When I use docker-compose up , a lot of error comes.


Regards,
Kaleem

[jpstaub]

Hello Kaleem,

I have limited experience with Virtual Box. I gave Virtual Box up early on in my virtualization efforts. I use VMware ESXi as a hypervisor.

That said, you did not list a version of Docker Compose. If by chance you have not installed Docker Compose click here for a nice tutorial put together by DigitalOcean. Also, you did not mention a version for jwilder/nginx-proxy or its companion letsencrypt-nginx-proxy-companion.

With an application as complex as KoBoToolbox I would recommend verifying your basic setup by getting a simple web server up and running via Docker Compose to the point where you can observe a demonstration web page on your browser at an address of your choice. Once that's working reliably deploying KoBoToolbox should go rather smoothly.

Regards,
Jake

[Kaleem Qureshi]

Thanks JP. I have test it locally  on the network . I want to work with jwilder/nginx-proxy and letsencrypt-proxy-comparison

Let me tell only two things(jwilder/nginx-proxy and letsencrypt-proxy-comparison) will be added in existing server installation.Currently I have setup with vmware server with ubuntu 16.04 woth docker version 17.03.1-ce and docker compose version 1.12.0.

--
You received this message because you are subscribed to a topic in the Google Groups "Kobo Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kobo-users/kh8QEkLzVGI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kobo-users+unsubscribe@googlegroups.com.
To post to this group, send email to kobo-...@googlegroups.com.
Visit this group at https://groups.google.com/group/kobo-users.
For more options, visit https://groups.google.com/d/optout.

--

With Best Regards,

Kaleem Ahmad Qureshi

Regional ICT Infrastructure Engineer South Asia Region

Country Office Pakistan, 

Islamic Relief World Wide UK ,

Cell: 92 300 8521825 , 92 333 5135815.

[jpstaub]

Good afternoon Kaleem,

That KoBoToolbox is working locally is good news.

Just in case you're trying to use a LAN - SSL Certificate I'll mention this is not supported by letsencrypt.

Once you get jwilder and letsencrypt containers set up add a web server container and access it via public DNS. At that point you will be ready to implement KoBoToolbox.

Regards,
Jake

[Kaleem Qureshi]

Thanks Jake, Then If we buy a certificates from third party, then We have to follow the method which mentioned on https://github.com/kobotoolbox/kobo-docker.

if we dont have a certificate then we have to use the  jwilder and letsencrypt  by using the self certificates for test SSL environment. 

Is that correct, What I am saying  

--
You received this message because you are subscribed to a topic in the Google Groups "Kobo Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kobo-users/kh8QEkLzVGI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kobo-users+unsubscribe@googlegroups.com.
To post to this group, send email to kobo-...@googlegroups.com.
Visit this group at https://groups.google.com/group/kobo-users.
For more options, visit https://groups.google.com/d/optout.

[jpstaub]

Good morning Kaleem,

Yes. If you get certificates from a third party then follow the instructions on GitHub which you referenced below.

If you don't have a certificate and don't want to buy a certificate you can generate one yourself. Then you can follow the instructions on GitHub to set up a test SSL environment. One of the primary drawbacks of using a self generated SSL certificate is that web browsers complain about security because they cannot confirm that the SSL certificate was generated by a Certificate Authority. If you're using Firefox and want to fix browser complaints rather than add an exception to the browser you can add a root certificate to the computer you're using. Other browsers should follow a similar process.

If you need a reverse proxy then it's best to use jwilder. If you do not need a reverse proxy then using jwilder is not necessary.

If you do need a reverse proxy and are using jwilder then letsencrypt can provide an SSL certificate. However, if you do not need an SSL certificate you do not need to use letsencrypt. letsencrypt is only necessary if you would like your deployment to have SSL protection.

Regards,
Jake

On Friday, May 19, 2017 at 3:03:49 PM UTC-4, Kaleem Qureshi wrote:

Thanks Jake, Then If we buy a certificates from third party, then We have to follow the method which mentioned on https://github.com/kobotoolbox/kobo-docker.

if we dont have a certificate then we have to use the  jwilder and letsencrypt  by using the self certificates for test SSL environment. 

Is that correct, What I am saying  

On Fri, May 19, 2017 at 8:59 PM, jpstaub <jpst...@gmail.com> wrote:

Good afternoon Kaleem,

That KoBoToolbox is working locally is good news.

Just in case you're trying to use a LAN - SSL Certificate I'll mention this is not supported by letsencrypt.

Once you get jwilder and letsencrypt containers set up add a web server container and access it via public DNS. At that point you will be ready to implement KoBoToolbox.

Regards,
Jake

On Friday, May 19, 2017 at 8:22:27 AM UTC-4, Kaleem Qureshi wrote:

Thanks JP. I have test it locally  on the network . I want to work with jwilder/nginx-proxy and letsencrypt-proxy-comparison

Let me tell only two things(jwilder/nginx-proxy and letsencrypt-proxy-comparison) will be added in existing server installation.Currently I have setup with vmware server with ubuntu 16.04 woth docker version 17.03.1-ce and docker compose version 1.12.0.

--
You received this message because you are subscribed to a topic in the Google Groups "Kobo Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kobo-users/kh8QEkLzVGI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to kobo-users+...@googlegroups.com.

To post to this group, send email to kobo-...@googlegroups.com.
Visit this group at https://groups.google.com/group/kobo-users.
For more options, visit https://groups.google.com/d/optout.

[Kaleem Qureshi]

Dear Jake

Thanks , God bless you.



regards,
kaleem

[Kaleem Qureshi]

Dear Jake

Now I have test it on LAN and it is working fine. Now I am using this with server using http only, but again it generate a lot of errors with docker-compose up.

Please let me tell that is it possible with HTTP? or we have to use with HTTPS only.


Regards,
Kaleem

On Saturday, May 20, 2017 at 6:13:29 PM UTC+5, jpstaub wrote:

[jpstaub]

Hello Kaleem,

It is possible to use KoboToolbox with HTTP.

It is difficult to provide assistance without seeing the errors you are experiencing.

Regards,
Jake

[Kaleem Qureshi]

Thanks Jake , I want to deploy the docker with http server method. I perform the following steps

1. Change the envfile.server.txt

2. change the docker-compose.server.yml with own IP address

3. Put the 3 entries  in etc/hosts file as mention on web site .

Only the enketo_express_1 is running fine, but kpi and kobocat creating problem. screen shot attached.

Please oblige.

[Kaleem Qureshi]

[jpstaub]

Good evening Kaleem,

Thanks for the additional information regarding your end goal. Based on what you have written below it appears you are mixing Kobo methods with my method. As stated at the start:

• Prerequisites - 1. jwilder/nginx-proxy 2. docker-letsencrypt-nginx-proxy-companion. Both containers should be up prior to starting KoBoToolbox.
• Configure "envfile.server.txt" with the domain and subdomain names of choice.
• Swap out "docker-compose.server.yml" and "nginx_site_https.conf.tmpl" with the attached files.
• Configure "docker-compose.server.yml" for http or https and the domain and subdomain names of choice.

My method does not rely on IP addresses. Rather it relies on a domain and subdomains. Entries made in etc/hosts will not produce desirable results.

I apologize for not explicitly stating it, but to implement my method requires that you set up a DNS Server on your local network. I do not know your hardware configuration so I cannot comment on what the best method is to set up a DNS Server. Click here for a nice explanation on setting up a DNS Server if you are not familiar with the concept. 

Ultimately, you need a functioning DNS Server and the ability to make DNS entries for the domain and each subdomain you used when configuring envfile.server.txt. This same domain and subdomains are also used to configure "VIRTUAL_HOST" in docker-compose.server.yml. When making DNS entries the IP address the entries should point to is the IP address assigned to the machine that hosts Docker.

You also need to ensure that the machine you are using to browse to your local network http Docker KoboToolbox instance is configured such that the primary DNS Server is the local DNS Server that you set up. Otherwise, the local addresses (subdomain.domain) will not resolve properly.

What helped my understanding when I developed this method was to keep in mind how a request from the browser is routed to KoboToolbox. The routing goes like this:

1. Type address in browser (kobotoolbox.mylocalnetwork for example).
   
2. Browser contacts the local DNS Server which was set as the primary DNS Server on the browsing computer.
   
3. Local DNS Server directs the browsing computer to the IP address for kobotoolbox.mylocalnetwork which is the IP address assigned to the machine that hosts Docker.
   
4. jwilder/nginx-proxy container running on Docker resolves kobotoolbox.mylocalnetwork to the proper Docker container and passes on the request for information.
   
5. kobotoolbox responds to the browsing computer.
   

Regards,
Jake

[Kaleem Qureshi]

Dear Jake

Once again thanks for your support. I am using the internal DNS(windows 2008 server), I have pull the clone jwilder/nginx proxy, But I am facing the problem that how can I bind the kobo containers with jwilder/nginx proxy .

Secondly, Is it must to install the DNS on a sever where kobo toolbox is hosted.

Waiting for your kind reply.

Regards,
kaleem

On Wednesday, November 9, 2016 at 9:11:56 AM UTC+5, jpstaub wrote:

[Kaleem Qureshi]

Dear Jake

Once again thanks for your support. I am using the internal DNS(windows 2008 server), I have pull the clone jwilder/nginx proxy, But I am facing the problem that how can I bind the kobo containers with jwilder/nginx proxy .

Secondly, Is it must to install the DNS on a sever where kobo toolbox is hosted.

One more thing, I have login from super user , working fine . but when I create the user with djang adminsitration. User can not create the form and deploy the project?

Please advise.

Waiting for your kind reply.

Regards,
kaleem

[jpstaub]

Good morning Kaleem,

But I am facing the problem that how can I bind the kobo containers with jwilder/nginx proxy .

The only kobo container that is bound to jwilder/nginx proxy is the kobo nginx container. The kobo nginx container acts as a reverse proxy for the containers of the KoboToolbox installation. Ensure that you have set the following lines under docker-compose.server.yml << nginx:

# manually replace host variables below with envfile.server.txt subdomains and domain    
 - VIRTUAL_HOST=KOBOFORM_PUBLIC_SUBDOMAIN.PUBLIC_DOMAIN_NAME,
   KOBOCAT_PUBLIC_SUBDOMAIN.PUBLIC_DOMAIN_NAME,ENKETO_EXPRESS_PUBLIC_SUBDOMAIN.PUBLIC_DOMAIN_NAME

 - HTTPS_METHOD=noredirect

# - LETSENCRYPT_HOST=KOBOFORM_PUBLIC_SUBDOMAIN.PUBLIC_DOMAIN_NAME,
   KOBOCAT_PUBLIC_SUBDOMAIN.PUBLIC_DOMAIN_NAME,ENKETO_EXPRESS_PUBLIC_SUBDOMAIN.PUBLIC_DOMAIN_NAME

# - LETSENCRYPT_EMAIL=admin@PUBLIC_DOMAIN_NAME

If you have made the appropriate edits above and are still having trouble getting jwilder/nginx proxy to bind to the KoboToolbox nginx container try a restart of the jwilder/nginx proxy container. If that doesn't work, regenerate the jwilder/nginx proxy container. If that doesn't work, remove KoboToolbox containers and the jwilder/nginx proxy container. Generate a jwilder/nginx proxy container then start a KoboToolbox installation.

Secondly, Is it must to install the DNS on a sever where kobo toolbox is hosted.

You can install the DNS server anywhere that is accessible to the computer that you intend to use to access KoboToolbox.

Regards,
Jake

[jpstaub]

Good morning Kaleem,

One more thing, I have login from super user , working fine . but when I create the user with djang adminsitration. User can not create the form and deploy the project?

It sounds like you have not set User permissions correctly for the user in question. On Django site administration see:
Home >> Authentication and Authorization >> Users >> User of Interest >> Permissions (Check Active) >> User permissions (select required permission for User of Interest).

Regards,
Jake

[Kaleem Qureshi]

Thanks Jake, When we run the jwilder/nginx and kobo toolbox, then error comes "driver failed programming external connectivity kobo docker nginx jwilder* . I think port conflict  problem. please let me that is there any change needed in docker-compose.yml file in nginx/jwilder .

Regards,
kaleem

[jpstaub]

Hello Kaleem,

No changes should be required in jwilder/nginx-proxy. See attached docker-compose file for what I use to start a jwilder/nginx-proxy container.

To eliminate potential port conflict problems stop all containers that are not part of the KoBoToolbox deployment.

Also, to ensure the Ubuntu server firewall is not creating problems disable the firewall during deployment troubleshooting efforts.

Regards,
Jake 

[Kaleem Qureshi]

[Kaleem Qureshi]

Dear Jake

Thanks for great support to enable me the kobo in test environment. Now I am facing the problem SSL certificate integration with kobo toolbox in production environment( kobo server on Esx(i)  in DMZ )

I have configured the Global DNS with the following sub domains:

1.   kobo.myorg.org (do'nt mention the actual DNS)

2.    kc.myorg.org

3.    ee. myorg.org

SSL certificate from Godadday

My  question are :

1. For each sub-domain we need the chained/bundle certificate, if yes, so I have concatenate the certificate in one file(ssl.crt) and create one private key file (ssl.key) and put in secrets folder of kobo docker as mentioned on github.

2. Will i put the configurations for sub-domains and location of certificates in nginx_site_https_conf.tmpl . Well I am attaching this file with this. Please correct if there is any mistake.

I also attaching other configuration files with screen shots. Please help me , because I have deadline till Monday.

Regards,

Kaleem

--

You received this message because you are subscribed to a topic in the Google Groups "Kobo Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kobo-users/kh8QEkLzVGI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to kobo-users+unsubscribe@googlegroups.com.

To post to this group, send email to kobo-...@googlegroups.com.
Visit this group at https://groups.google.com/group/kobo-users.
For more options, visit https://groups.google.com/d/optout.

[jpstaub]

Good morning Kaleem,

The method that I developed to implement SSL makes use of Let's Encrypt certificates that are automatically generated and maintained by the LetsEncrypt companion container for nginx-proxy. By trying to use GoDaddy certificates you are attempting to mix my method with the method implemented by the KoboToolbox developers. You cannot use both at the same time. You either need to implement my method from start to finish or abandon it entirely in order to set up a working KoboToolbox deployment. If you want to use the method implemented by the KoboToolbox developers you must seek their support. I have no experience with their method and as a result cannot offer support.
 
Three files are attached that should help you implement my method if you would like to continue down that path.

1. docker-compose.yml -- This file is what I use to start jwilder/nginx-proxy and jrcs/letsencrypt-nginx-proxy-companion. This combination of containers is what provides reverse proxy and SSL support to a KoboToolbox instance.
   
2. docker-compose.server.yml -- This file is what I use to start a KoboToolbox instance. I have modified the file to reflect the domain names that you listed as examples. The only settings you should ever change are under: nginx >> environment >> VIRTUAL_HOST / LETSENCRYPT_HOST / LETSENCRYPT_EMAIL. Do not change any other settings.
3. envfile.server.txt -- This file is what holds domain and security settings. I have modified the file to reflect the domain names that you listed as examples. You need to provide your own security settings according to the directions given in the file. When you change domain names in this file you must also change the domain names in docker-compose.server.yml.    
   

With the three attached files you have everything you need to deploy a KoboToolbox instance with self generated and maintained SSL certificates presuming your server is publicly accessible at the domain name you are using. You can see a working example that was produced using the three files at kobo.ripcordengineering.com. Note: you cannot register a new account because I have disabled account registration.

If you cannot successfully deploy KoboToolbox but still need your own instance I am available on a professional basis.

Regards,
Jake

[Kaleem Qureshi]

Thanks Jake. but no error comes after docker-compose up. But one last thing I want to confirm that for auto letsencrypt certificate we have to configure the certbot..


Regards,

Kaleem

[Kaleem Qureshi]

It works perfectly with HTTP only on LAN and no error generate after docker-compose with nginx-proxy and letsencrypt container . But main theme is that it work on https. Is there any certificates needed from letsencrpt for kobo.myorg.org ee.myorg.org kc.myorg.org or we have to configure the certbot

[Kaleem Qureshi]

Dear Jake 

Please share you skype id or whats app id to discuss it professionally.

My skype id is kaleemqureshi1 and whatsapp +923008521825

--
You received this message because you are subscribed to a topic in the Google Groups "Kobo Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kobo-users/kh8QEkLzVGI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kobo-users+unsubscribe@googlegroups.com.
To post to this group, send email to kobo-...@googlegroups.com.
Visit this group at https://groups.google.com/group/kobo-users.
For more options, visit https://groups.google.com/d/optout.

[gonu...@gmail.com]

Hi Jake,

I'm a new member in this forum as well as on this topic and will really need your help to install and configure kobotoolbox on my local computer as proof of concept. Here is my email address gee...@hotmail.com. Please let me hear from you so we can discuss further privately on this on how to help us achieve our goal by using your process. We have attempted twice trying to install this application but now. Urgent please.  

Thanks,

Joe

On Tuesday, November 8, 2016 at 11:11:56 PM UTC-5, jpstaub wrote:

[Kaleem Qureshi]

Are you available now

--

[Kaleem Qureshi]

Dear Jake


I have test your method completely in my office and it is accessible from Internet and also generate the certificates from Letsencrypt. Those who are struggling with kobo Toolbox deployment on private server, Please follow the Jake's Method.

Regards,

Kaleem

[Kaleem Qureshi]

Sorry When I create the form and deploy it , It generate Server Error 500. I have check on both version http and https. 

 

It works perfactly when we test on local machine . Each and every thing working properly, I face the problem on deployment