Question re: Docker Setup

200 views
Skip to first unread message

JamiroquaiMX

unread,
Aug 15, 2018, 3:04:31 AM8/15/18
to Bots Open Source EDI Translator
Hi everyone,

I've been testing BOTS for quite some time and I'm very pleased so far with what it can do. I just have some infrastructure-related question, hopefully you can provide some guidance on it.

Can I have the bots-jobqueueserver.py created as a separate Docker container, split from bots-engine.py? I have my dev environment set up as a Docker container, but I'm looking into setting up new containers for each major task. I plan to only load the mappings and grammars on the bots-engine container. This instance then contacts the jobqueueserver when a new job is run. I then plan the scheduler/cron to be a third container.

Any suggestions too re: running two BOTS containers simultaneously? I'm trying to see if I can cluster two together for failover and reduncancy.

Thanks in advance for your feedback.

Eppye Bots

unread,
Aug 15, 2018, 3:20:17 AM8/15/18
to 'Chuck Turco' via Bots Open Source EDI Translator
I do not have a lot of docker experience.
want to help sure.

note that before you reach the point where bots can not handle stuff you do a lot of edi.
I do have cloud-things, where more install run, but these are different accounts.

don't make it too complicated.
I have 'tuned back' complicated setup - which where just plain overblown and over-engineerd. 
start simple.
keep it simple.
if you have not done edi before: keep it very very simple.



kind regards, Henk-Jan Ebbers

--
You received this message because you are subscribed to the Google Groups "Bots Open Source EDI Translator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to botsmail+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/botsmail.
For more options, visit https://groups.google.com/d/optout.

JamiroquaiMX

unread,
Aug 16, 2018, 3:01:41 AM8/16/18
to Bots Open Source EDI Translator
Thanks Henk-Jan. Understood on your point in keeping things simple.

I guess I can share my initial Dockerfile for reference. This is based off the instructions I saw on https://bots.readthedocs.io/en/latest/installation/.

As part of my housekeeping, I created the following folders: configs, grammars/csv, mappings/csv. I work mostly with CSV and XLS files on my job, but anyone can add more folders here :) The configs folder contain my edited bots.ini (debug plus a few more tweaks) and requirements.txt (contains the list of pip modules required for BOTS).

Hope this helps. Appreciate the group's input on this.


kind regards, Henk-Jan Ebbers

To unsubscribe from this group and stop receiving emails from it, send an email to botsmail+u...@googlegroups.com.
Dockerfile
requirements.txt

Wassily

unread,
Aug 16, 2018, 6:31:31 AM8/16/18
to Bots Open Source EDI Translator
Hi JamiroquaiMX 

Thanks for your question, this motivated me to fix my Dockerfile that I used for a long time and finally make it smaller by switching to Alpine Linux instead of Debian. 
You can check my latest Dockerfile here: https://github.com/chadgates/bots_build or pull the docker image from dockerhub: https://hub.docker.com/r/chadgates/bots-build/
I do start the job-queue server within the same Docker Container and also a "pseudo" cron service. 

JamiroquaiMX

unread,
Aug 17, 2018, 2:39:01 AM8/17/18
to Bots Open Source EDI Translator
No worries, Wassily. Thanks too for sharing your Docker setup, these are great references so far. Biggest takeaway from me is the usage of crond and supervisord.

Robert Rauer

unread,
Jun 14, 2021, 9:54:21 AMJun 14
to Bots Open Source EDI Translator
Hi all,

I am using Wassily's docker compose and docker file. It runs fine. I wanted to share the local directories as described

volumes: 
 - ./config:/usr/local/lib/python2.7/dist-packages/bots/config 
 - ./botssys:/usr/local/lib/python2.7/dist-packages/bots/botssys 
 - ./usersys:/usr/local/lib/python2.7/dist-packages/bots/usersys

but they are empty as bots holds the data in "site-packages". Changing the path in the docker compose file prevents bots from starting up correctly:

Recreating bots_build-master_bots-server_1 ... done
Attaching to bots_build-master_bots-server_1
bots-server_1  | Traceback (most recent call last):
bots-server_1  |   File "/usr/local/bin/bots-webserver.py", line 5, in <module>
bots-server_1  |     webserver.start()
bots-server_1  |   File "/usr/local/lib/python2.7/site-packages/bots/webserver.py", line 37, in start
bots-server_1  |     botsinit.generalinit(configdir)     #find locating of bots, configfiles, init paths etc.
bots-server_1  |   File "/usr/local/lib/python2.7/site-packages/bots/botsinit.py", line 47, in generalinit
bots-server_1  |     raise botslib.PanicError(u'In initilisation: path to configuration does not exists: "%(configdir)s".',{'configdir':configdir})
bots-server_1  | bots.botslib.PanicError: In initilisation: path to configuration does not exists: "config".
bots_build-master_bots-server_1 exited with code 1

My target is to share these 3 folders with the host to change data from there while the container is running. Am I missing something?

Thanks for help in advance.

Best regards,
Robert
Message has been deleted

Wassily

unread,
Jun 15, 2021, 4:32:35 AMJun 15
to Bots Open Source EDI Translator
Hi Robert

If I recall correctly, you should NOT change the dist-packages folder to site-packages.
Did you find that it did not work with the dist-packages folder?
It has been 3 years since I had done that, so I don't recall why I had to use dist-packages as also I had the issue. Nevertheless, I think it is time I review that, Docker has evolved and there are better ways to build this now then back then and I would probably also switch away from Alpine Linux in this case.
May I suggest you try with the dist-packages folder. Yes, it does seem wrong and counter intuitive, but I do think this is the way it works in that container. I'll review the Dockerfile in the next couple of days and try to improve it.

Robert Rauer

unread,
Jun 15, 2021, 10:47:32 AMJun 15
to Bots Open Source EDI Translator
Hi Wassily,

thanks a lot for checking the docker configuration after all that years. I used the dist-packages in my docker-compose.yaml as follows:

version: '3'

# Running BOTS with JobQueue Server enabled.
# The Supervisor uses environment variables SUPERVUSER for HTTP user login and
# SUPERVPASS for password.

services:
  bots-server:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "0.0.0.0:8080:8080"
      - "0.0.0.0:9001:9001"
    environment:
      - SUPERVUSER=bots
      - SUPERVPASS=botsbots
    volumes:
      - ./config:/usr/local/lib/python2.7/dist-packages/bots/config
      - ./botssys:/usr/local/lib/python2.7/dist-packages/bots/botssys
      - ./usersys:/usr/local/lib/python2.7/dist-packages/bots/usersys

This configuration works but bots is installed in /usr/local/lib/python2.7/site-packages/bots
Which means the directories stay empty. Adding and removing files to the mapped directories works so the docker part is fine.

It seems that changing volumes to 

    volumes:
      - ./config:/usr/local/lib/python2.7/site-packages/bots/config
      - ./botssys:/usr/local/lib/python2.7/site-packages/bots/botssys
      - ./usersys:/usr/local/lib/python2.7/site-packages/bots/usersys
 
make them unaccessible for /usr/local/bin/bots-webserver.py on startup.
This is hard for me to understand. 
Is there a way to link the /usr/local/lib/python2.7/site-packages/bots/* folders to /usr/local/lib/python2.7/dist-packages/bots/  ?
I tried to symlink them but that doesn't work in docker.

Thanks for your help!
Regards,
Robert

Wassily

unread,
Jun 17, 2021, 12:55:47 PMJun 17
to Bots Open Source EDI Translator
Hi Robert

Have updated the repo and Dockerfile. Switched from the alpine to the -slim python container. Everything should be proper now in site-packages. Also upgraded some of the packages to newer versions to remove the number of vulnerabilities.

You can pull updated image as before with:
docker pull chadgates/bots-build:latest 

Hope that fixes is for you.

Cheers
Wassily

Robert Rauer

unread,
Jun 18, 2021, 2:10:24 PMJun 18
to Bots Open Source EDI Translator
That's great Wassily!

Thank you very much for upgrading. That solves the volume problem. Everything is in its place now :-)

Thanks.
Robert

duf...@gmail.com

unread,
Jul 2, 2021, 6:06:53 PMJul 2
to Bots Open Source EDI Translator
Hi Wassily,

   I'm attempting to use the docker and docker-compose files as specified at https://github.com/chadgates/bots_build, but am not able to replicate Robert's success. It runs okay out of the box, but attempting to utilize the volumes in docker-compose.yaml prevents the apps from starting correctly; supervisord complains on every entry. Investigating, I found that the volumes are not wired together properly, and if I go look in  /usr/local/lib/python2.7/site-packages/bots/config, there is no crontab. The crontab is present and everything fires off fine if I don't include the volumes entry in docker-compose.yaml. Can you verify this docker-compose.yaml is correct if using volumes as you intended?

Thanks,
Jason

duffy@joyce ~/PycharmProjects/bots_build $ cat docker-compose.yaml
version: '2'


# Running BOTS with JobQueue Server enabled.
# The Supervisor uses environment variables SUPERVUSER for HTTP user login and
# SUPERVPASS for password.

services:
  bots-server:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "0.0.0.0:8080:8080"
      - "0.0.0.0:9001:9001"
    environment:
      - SUPERVUSER=bots
      - SUPERVPASS=botsbots
    command: /usr/local/bin/supervisord -c /etc/supervisor/supervisord.conf

    volumes:
      - ./config:/usr/local/lib/python2.7/site-packages/bots/config
      - ./botssys:/usr/local/lib/python2.7/site-packages/bots/botssys
      - ./usersys:/usr/local/lib/python2.7/site-packages/bots/usersys

Robert Rauer

unread,
Jul 3, 2021, 8:13:53 AMJul 3
to Bots Open Source EDI Translator
Hi Jason,

not sure if this helps:
as I didn't use Wassily's docker file as it is I can't say if this worked out of the box with mounted volumes.
I had to do some manual changes:
1. copied my botssys, usersys and config to the mounted directories from the host system (including crontab to config) before starting the container
2. changed the path in crontab to 05,15,25,35,45,55      *     * * *   python /usr/local/bin/bots-job2queue.py /usr/local/bin/bots-engine.py -cconfig --new
as per default there were errors in stderr (no such file)
3. for directory monitor to work I put pyinotify to requirements/base.txt

What error messages do you get?

Regards,
Robert

Wassily

unread,
Jul 5, 2021, 2:44:45 AMJul 5
to Bots Open Source EDI Translator
Hi Jason

Can you share the errors you are getting as when running locally, supervisor start the required service without any problem. Did you get the image from dockerhub or did you check out the git repo and build it from there? I would assume that the later would lead to success, but using the dockerhub image probably requires an adjusted docker-compose file.

Please note that the dockerfile as is can get you started but needs adjustments to your local setup, it is somewhat of a compromise. Initially, everything lives within the docker container. So I'd suggest you do two stage adjustement:
1.) only mount the usersys and botssys from the host, and leave the config path in the container.
2.) when you find things working, copy the config path details from the container to your host, so you can make adjustments to them to your liking.

I do NOT recommend to use directory monitoring this way, as Docker adds a file system layer and will not get notified about host file system changes reliably. If you want to have a file system monitoring active within a container, I suggest you use other means. I have used the watchdog package to do this here (but also this documentation requires updating...): https://django-pyas2.readthedocs.io/en/latest/detailed-guide/extending.html

Hope above helps.

Greets
Wassily

Jason Duffy

unread,
Jul 13, 2021, 5:57:30 PMJul 13
to bots...@googlegroups.com
Hi Robert. Thank you for the reply! The error messages I got led me to notice the empty botsys, usersys, and config directories. Your #1 "copied my botssys, usersys, and config..." step was the advise I needed. Thank you kindly!

Thanks,
Jason

You received this message because you are subscribed to a topic in the Google Groups "Bots Open Source EDI Translator" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/botsmail/K2oFSsVw_Eo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to botsmail+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/botsmail/9e359ea7-7bf3-499a-9bdb-975ba5deea16n%40googlegroups.com.

Jason Duffy

unread,
Jul 13, 2021, 6:03:32 PMJul 13
to bots...@googlegroups.com
Hi Wassily. Thank you for the response! The error messages I got were about basically about empty botssys and config directories. Copying the botssys, config, and usersys directories from the python zip up to the directory containing docker-compose.yaml enabled the setup to work out of the box. I was pulling from the git repo btw.

Yeah I'm not planning to use this in a production setup; I'll dig in to those details if we find a need for bots. I needed to do a quick demo for a new client and found your github repository from Google. I've now found it does work on both my Macbook and a Linux box if I add that copy step to your instructions. Might save someone else the trouble if you add the step to your readme before the volumes section.

Thanks again for putting it together and publishing!

Thanks,
Jason


You received this message because you are subscribed to a topic in the Google Groups "Bots Open Source EDI Translator" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/botsmail/K2oFSsVw_Eo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to botsmail+u...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages