Unable to deploy form on local Docker Kobo Toolbox instance

[jpstaub]

Hopefully these symptoms look familiar to someone in the group. I'm at a loss as to how to correct the problem that is preventing the deployment of forms on a local Docker Kobo Toolbox instance. The synopsis follows below:
 
When I try to deploy a simple one question test form created by the local Form Builder the result is:

===
unable to deploy

your form cannot be deployed because it contains errors:

HTTPConnectionPool(host='192.168.1.104', port=8001): Max retries exceeded with url: /api/v1/forms (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out',))
===

In an effort to troubleshoot, a simple form was created on the official Kobo Toolbox site and was uploaded on the docker implementation for deployment. When I try to deploy that form I get:

===
unable to deploy, server error 500, if this problem persists, contact sup...@kobotoolbox.org
===

Additionally, I cannot preview forms. The end result of attempts is:

===
Loading Error
Gateway Timeout
===

However, when I go to KoBoCAT (port=8001) I can upload a draft test form downloaded from KPI but cannot "Enter Web Form." After a lengthy delay a "502 Bad Gateway," response is given. The logs show trouble for kobocat:

===
kobocat_1         | Sun Oct 23 20:42:58 2016 - *** HARAKIRI ON WORKER 1 (pid: 146, try: 1) ***
kobocat_1         | Sun Oct 23 20:42:58 2016 - HARAKIRI !!! worker 1 status !!!
kobocat_1         | Sun Oct 23 20:42:58 2016 - HARAKIRI [core 0] 192.168.1.154 - GET /kobo/forms/a8b79dWjg5suLK3d5WW6rJ/enter-data since 1477269657
kobocat_1         | Sun Oct 23 20:42:58 2016 - HARAKIRI !!! end of worker 1 status !!!
postgres_1        | 2016-10-24 00:42:58 GMT LOG:  unexpected EOF on client connection with an open transaction
kobocat_1         | DAMN ! worker 1 (pid: 146) died, killed by signal 9 :( trying respawn ...
kobocat_1         | Respawned uWSGI worker 1 (new pid: 148)
kobocat_1         | Sun Oct 23 20:44:59 2016 - *** HARAKIRI ON WORKER 2 (pid: 147, try: 1) ***
kobocat_1         | Sun Oct 23 20:44:59 2016 - HARAKIRI !!! worker 2 status !!!
kobocat_1         | Sun Oct 23 20:44:59 2016 - HARAKIRI [core 0] 192.168.1.154 - GET /kobo/forms/a8b79dWjg5suLK3d5WW6rJ/enter-data since 1477269778
kobocat_1         | Sun Oct 23 20:44:59 2016 - HARAKIRI !!! end of worker 2 status !!!
postgres_1        | 2016-10-24 00:44:59 GMT LOG:  unexpected EOF on client connection with an open transaction
kobocat_1         | [pid: 148|app: 0|req: 7/10] 192.168.1.154 () {38 vars in 774 bytes} [Sun Oct 23 20:45:00 2016] GET /favicon.ico => generated 0 bytes in 644 msecs (HTTP/1.1 301) 4 headers in 190 bytes (1 switches on core 0)
kobocat_1         | DAMN ! worker 2 (pid: 147) died, killed by signal 9 :( trying respawn ...
kobocat_1         | Respawned uWSGI worker 2 (new pid: 149)
===

The startup logs had a few squawks but I don't know whether or not the discrepancies are relevant.

===
#1 Preview form at koboCAT=8000
Loading Error
Gateway Time-out

#2 redis_cache_1 redis_main_1
 WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
 
 #3
 postgres_1        | WARNING: enabling "trust" authentication for local connections
 postgres_1        | 2016-10-23 20:57:06 GMT LOG:  unexpected EOF on client connection with an open transaction
 postgres_1        | 2016-10-23 18:25:43 GMT LOG:  incomplete startup packet
 
 #4
 enketo_express_1  | >> Local Npm module "grunt-concurrent" not found. Is it installed?
 enketo_express_1  | >> Local Npm module "grunt-contrib-jshint" not found. Is it installed?
 enketo_express_1  | >> Local Npm module "grunt-contrib-watch" not found. Is it installed?
 enketo_express_1  | >> Local Npm module "grunt-jsbeautifier" not found. Is it installed?
 enketo_express_1  | >> Local Npm module "grunt-karma" not found. Is it installed?
 enketo_express_1  | >> Local Npm module "grunt-mocha-test" not found. Is it installed?
 enketo_express_1  | >> Local Npm module "grunt-nodemon" not found. Is it installed?
 enketo_express_1  | >> Local Npm module "grunt-shell" not found. Is it installed?
 
 #5
 kobocat_1         |  *** WARNING: you have enabled harakiri without post buffering. Slow upload could be rejected on post-unbuffered webservers ***
===

I have a feeling I have a wrong setting somewhere but don't have enough experience to know what it might be based on the above symptoms. Diagnostic help would be appreciated.

Regards,
Jake

[jpstaub]

I could still use some help getting Kobo to function fully as a local Docker deployment. How about hint?

Regards,

Jake

[Donald Lobo]

hey jake

can you give more details on what platform you are running on? We've been successfully running the docker instances on various versions of ubuntu

lobo

[jpstaub]

Hi lobo,

Platform details:

Host = Ubuntu 16.04.1 LTS

Docker version = 1.12.2

Browser 1 = Firefox 49.0.1

Browser 2 = Safari 9.1.2

It shouldn't matter, but just in case, the Host is a VM running on top of vSphere 6. That portion of the configuration is a known good as there are several services running normally on the Host. Kobo is nearly there. All the basic parts appear functional. 

Regards,

Jake  

[jpstaub]

Hi lobo,

Can you give more details on the configuration(s) you're using to successfully run docker instances?

Regards,

Jake

On Friday, October 28, 2016 at 10:58:14 PM UTC-4, Donald Lobo wrote:

[jpstaub]

Still chipping away on this problem. I started playing around with the Enketo API via the terminal to see if I could see something useful. A form "test" was uploaded to Kobocat in order to troubleshoot.

Using the "ENKETO_API_TOKEN" set in envfile.local.txt produces the following:

$ curl --user abc: "http://192.168.1.104:8005/api/v1/survey/preview?server_url=http://192.168.1.104:8001&form_id=test"

{

    "preview_url": "http://192.168.1.104:8005/preview/::YYY8",

    "code": 200

Using the "ENKETO_API_TOKEN" and previewing for superuser kobo produces the following:

$ curl --user  abc: "http://192.168.1.104:8005/api/v1/survey/preview?server_url=http://192.168.1.104:8001/kobo&form_id=test"

{

   "message": "Survey not found",

    "code": 404

Using kobo's api-token (found by: http://192.168.1.104:8001/kobo/api-token) and previewing produces the following:

$  curl --user ceb39a81a93b6d4d7a2f558e89ae7d4f98f9e5e: "http://192.168.1.104:8005/api/v1/survey/preview?server_url=http://192.168.1.104:8001/kobo&form_id=test"

{

    "code": 401,

    "message": "Not Allowed. Invalid API key."

To my untrained eye, it looks like there's a permission problem that's preventing KPI, Kobocat, and Enketo from interacting correctly when a user is involved. This renders the local Docker Kobo Toolbox instance nearly useless. 

What am I missing here? It can't be this hard to get a local Docker instance working. The whole point of Docker is to mitigate configuration/dependency problems.

Jake

[Donald Lobo]

hey jake:

sorry for the delayed reply. was a bit tied up :(

a couple of things:

1. i'm running on a ubuntu box: 16.04

2. How did u figure out the IP address: 192.168.1.104? I'm assuming you followed the instructions in envfile.local.txt

3. did you update to the latest code both via a git pull and a docker-compose pull

lobo

[jpstaub]

Hello lobo,

No worries.

192.168.1.104 is the IP of my Ubuntu box that's running Docker. That IP works for all the other services I'm running. So, it'd be pretty dicked up indeed if Kobo refused to work under that IP.

Out of desperation I figured I'd do some work and inspect Kobo containers to see what the network situation looks like. Before I got that far I noticed the kobocat container has port 8000 exposed. The kobocat Dockerfile backs this up. Shouldn't kobocat have 8001 exposed in order for the system to work properly?

kpi is sitting on port 8000 as well. That nginx is configured to resolve ports 8000 and 8001 suggests the kobocat Dockerfile is exposing the wrong port. 

Since you've got a working system, what are you seeing as far as ports are concerned?  

Jake

[Donald Lobo]

You've got the ports right. But i think u've got the IP wrong. check the comments at top of  envfile.local.txt, there are instructions there which tell u to set the IP to the output of

$ docker-compose run --rm kpi /sbin/ip route|awk '/default/ { print $3 }'

I think this is inherited from docker, i dont know what/why/how, but i think i was stuck on this also during my first install

lobo

# The address where your KoBo Toolbox instance can be reached.

# NOTE: If you are not connected to a network or don't intend to permit access

#   from other devices on your network, use the IP address of Docker's network

#   interface. This can be obtained e.g with the following command:

#   docker-compose run --rm kpi /sbin/ip route|awk '/default/ { print $3 }'

HOST_ADDRESS=172.17.0.1

[jpstaub]

Hi lobo,

My envfile.local.txt says the following:

# The address where your KoBo Toolbox instance can be reached; generally the IP address of your computer.

# DO NOT use "localhost" or "127.0.0.1" or containers will not be able to communicate with each other.

# NOTE: If you are not connected to a network or don't intend to permit access 

#   from other devices on your network, use the IP address of Docker's network 

#   interface. This can usually be obtained e.g with the following command:

#   docker-compose run --rm kpi /sbin/ip route|awk '/default/ { print $3 }'

...generally the IP address of your computer.

is the bit I was relying on to set 192.168.1.104.

I tried:

$ docker-compose run --rm kpi /sbin/ip route|awk '/default/ { print $3 }'

which yielded:

172.17.0.1

Setting HOST_ADDRESS=172.17.0.1 resulted in the inaccessibility of kpi at 192.168.1.104:8000 although enketo_express was reachable at 192.168.1.104:8005. For reference, 172.17.0.1 is the default Docker network which I would not expect to be accessible from another machine on the LAN. 

For configuration reference:

docker-compose version 1.9.0-rc1, build 28788bd

After noticing that "docker-compose.local.yml" is Version 1 which is scheduled for deprecation I made an attempt to upgrade to Version 2. All that got me was bitching about "-wait" for rabbit and kpi as well as kobocat entering a perpetual restarting state.

Honestly lobo, I appreciate the effort. And I've seen a few of your other posts which were informative. It sure would be nice if the developers gave some direction. They must have seen something similar at some point.

Regards,

Jake

[jpstaub]

This is my final post regarding Kobo Toolbox. I have decided to throw Kobo onto the trash heap for the following reasons:

1. Try as I might I cannot coax Kobo into either previewing or publishing a form. As a result, Kobo rendered itself completely useless.
2. For practical purposes, Kobo offers no technical support. The documentation is scattered and inefficient.
3. The complexity of Kobo is unjustifiable for what is a very simple concept: the gathering of form responses.

I suspect more than a few people are drawn to Kobo because of its offline capability as I was. Frankly, there doesn't appear to be much available in terms of open source offline functionality at first blush. However, upon closer examination and a better focus in internet searches I arrived at the following proven, relatively straight forward, and open source tool chain:

1. Form Tools for form generation as well as administration. Form Tools also allows just about any type of external form to be incorporated.
2. HTML5 Web Storage to provide offline form storage capability within the browser.
3. Jotform to build HTML forms apart from Form Tools.
  a. Jotform source code is generated by following these simple instructions.
  b. Jotform includes HTML5 Web Storage capability by following these simple instructions.
4. To incorporate Web Storage on any form follow these simple instructions.

Good bye Kobo. You were a prolific time waster.
Jake

[Esmail Fadae]

Hi, Jake.

Sincerest apologies for the frustrations you had with your `kobo-docker` setup and the lack of response to your requests for support. I'm currently digging my way through a backlog of `kobo-docker` support requests and got bogged down on a data retention/backup issue relevant to another user.

I concur with Lobo's suspicions that there was a network problem of some sort. You're clearly able to access all three services through your browser at the `HOST_ADDRESS` you entered, and the containers can communicate with each other over the Docker network bridge (e.g. when KoBoCAT executes `scripts/wait_for_kpi.bash` for initialization), but for some reason the apps (KPI, KoBoCAT, and Enketo) don't seem to be able to reach the `HOST_ADDRESS`.

Assuming you haven't made any changes outside your `envfile.local.txt` everything should be working fine (I use the same `kobo-docker` "local" setup as my sole development environment), so my only guess is there's something relating to your vSphere environment that's causing your `HOST_ADDRESS` to be unresolvable within your VM (or at least within the Docker containers). I know in consumer-grade VM software like VirtualBox, this can happen if the VM's network adapter is set to NAT mode instead of bridged mode.

If you're willing to give it another shot, your issue might be resolved by mimicking the workaround for DNS resolution issues in `kobo-docker` "server" setups (commented-out KoBoCAT example here). You'd need to first find your Docker network bridge IP (assumed to be 172.17.0.1 in the following) and add `hosts` file overrides to your `docker-compose.local.yml` using the `extra_hosts` directive as follows:

kobocat:
...
  extra_hosts:
    - '192.168.1.104:172.17.0.1'

kpi:
...
  extra_hosts:
    - '192.168.1.104:172.17.0.1'

...

enketo_express:
...
  extra_hosts:
    - '192.168.1.104:172.17.0.1'

...

If you feel you've already sunk enough time into setting up KoBo and don't want to attempt the workaround, I understand and apologize again for your frustrations and for letting you feel like giving KoBo a shot was a waste of your time.

Regards,
Esmail


P.S. Thanks for all the help, Lobo.

[jpstaub]

Hello Esmail,

Thanks for taking the time to respond despite a critique biased by frustration.

I tried what you suggested. No dice. However, you got my attention with the following:

You're clearly able to access all three services through your browser at the `HOST_ADDRESS` you entered, and the containers can communicate with each other over the Docker network bridge (e.g. when KoBoCAT executes `scripts/wait_for_kpi.bash` for initialization), but for some reason the apps (KPI, KoBoCAT, and Enketo) don't seem to be able to reach the `HOST_ADDRESS`.

The hairpin of the apps to the 'HOST_ADDRESS' was the tipper I needed to disable the Docker host's firewall (Ubuntu's Uncomplicated Firewall). Disabling the firewall resulted in the proper functioning of Kobo. Why I could communicate with the applications but the applications couldn't communicate with each other is beyond me. Given the links between containers, the need to loop back for the apps to communicate with one another is not what I would expect. Suffice it to say, it would be worth including a check of the Docker host firewall in troubleshooting documentation if for no other reason than to remind those like me who aren't familiar with networking that firewalls create problems.

I'll also put another plug in for a version 2 of the docker-compose files. For those that already have services running on the default Docker network or that add services in the future, it would be worthwhile to isolate Kobo on its own network. I've included my stab at a solution. It doesn't work but it's a useful way to bypass formatting grunt work.

Finally, how do I go about turning the nginx server into an http server only? Using jwilder/nginx-proxy and letsencrypt-nginx-proxy-companion is a very nice way to reverse proxy and add trusted ssl to web services via letsencrypt.

Regards,
Jake

[Esmail Fadae]

Hi, Jake. Thanks for giving it another try and following up with us with your results. Glad to hear you got things working. Like you, I really wasn't expecting this to be related to a firewall issue. I'll work a warning about potential firewall issues (as well as the NAT mode VM network adapter one) into the README.

Thanks for the Compose v2 file. We'll likely be going that direction in the coming weeks, so it'll be a nice reference.

Re. HTTPS, our current "server" setup path makes a HTTPS-only server, but requires a wildcard or SAN certificate+key pair usable on the three subdomains for the three apps. There's already an effort underway to consolidate to a single subdomain (or port, in the "local" setup), at which point, I agree, it would be nice to start using letsencrypt, but it's difficult to say when I'll be getting around to either at the moment.

Regards,
Esmail

[jpstaub]

Hello Esmail,

I developed a solution to implement reverse-proxy/lentsencrypt and made a new post about it. You may want to try it out, identify weaknesses, and comment about them for others that may be following along. As the disclaimer says, I've made something that works but not because I understand the full extent of what's going on within KoBoToolbox.

Regards,
Jake

[Martin Andrews]

Hi Jake,

Where is the post for your solution? I am trying to get kobotoolbox up with a letsencrypt certificate.

Martin

[jpstaub]

Hi Martin,

Click here for the post.

You can also search for: Docker KoBoToolbox behind reverse proxy server with SSL support.

Regards,
Jake