Question re: Docker Setup

[JamiroquaiMX]

Hi everyone,

I've been testing BOTS for quite some time and I'm very pleased so far with what it can do. I just have some infrastructure-related question, hopefully you can provide some guidance on it.

Can I have the bots-jobqueueserver.py created as a separate Docker container, split from bots-engine.py? I have my dev environment set up as a Docker container, but I'm looking into setting up new containers for each major task. I plan to only load the mappings and grammars on the bots-engine container. This instance then contacts the jobqueueserver when a new job is run. I then plan the scheduler/cron to be a third container.

Any suggestions too re: running two BOTS containers simultaneously? I'm trying to see if I can cluster two together for failover and reduncancy.

Thanks in advance for your feedback.

[Eppye Bots]

I do not have a lot of docker experience.

want to help sure.

note that before you reach the point where bots can not handle stuff you do a lot of edi.

I do have cloud-things, where more install run, but these are different accounts.

don't make it too complicated.

I have 'tuned back' complicated setup - which where just plain overblown and over-engineerd. 

start simple.

keep it simple.

if you have not done edi before: keep it very very simple.

kind regards, Henk-Jan Ebbers

--
You received this message because you are subscribed to the Google Groups "Bots Open Source EDI Translator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to botsmail+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/botsmail.
For more options, visit https://groups.google.com/d/optout.

[JamiroquaiMX]

Thanks Henk-Jan. Understood on your point in keeping things simple.

I guess I can share my initial Dockerfile for reference. This is based off the instructions I saw on https://bots.readthedocs.io/en/latest/installation/.

As part of my housekeeping, I created the following folders: configs, grammars/csv, mappings/csv. I work mostly with CSV and XLS files on my job, but anyone can add more folders here :) The configs folder contain my edited bots.ini (debug plus a few more tweaks) and requirements.txt (contains the list of pip modules required for BOTS).

Hope this helps. Appreciate the group's input on this.

kind regards, Henk-Jan Ebbers

To unsubscribe from this group and stop receiving emails from it, send an email to botsmail+u...@googlegroups.com.

[Wassily]

Hi JamiroquaiMX 

Thanks for your question, this motivated me to fix my Dockerfile that I used for a long time and finally make it smaller by switching to Alpine Linux instead of Debian. 

You can check my latest Dockerfile here: https://github.com/chadgates/bots_build or pull the docker image from dockerhub: https://hub.docker.com/r/chadgates/bots-build/

I do start the job-queue server within the same Docker Container and also a "pseudo" cron service. 

[JamiroquaiMX]

No worries, Wassily. Thanks too for sharing your Docker setup, these are great references so far. Biggest takeaway from me is the usage of crond and supervisord.

[Robert Rauer]

Hi all,

I am using Wassily's docker compose and docker file. It runs fine. I wanted to share the local directories as described

volumes: 

 - ./config:/usr/local/lib/python2.7/dist-packages/bots/config 

 - ./botssys:/usr/local/lib/python2.7/dist-packages/bots/botssys 

 - ./usersys:/usr/local/lib/python2.7/dist-packages/bots/usersys

but they are empty as bots holds the data in "site-packages". Changing the path in the docker compose file prevents bots from starting up correctly:

Recreating bots_build-master_bots-server_1 ... done

Attaching to bots_build-master_bots-server_1

bots-server_1  | Traceback (most recent call last):

bots-server_1  |   File "/usr/local/bin/bots-webserver.py", line 5, in <module>

bots-server_1  |     webserver.start()

bots-server_1  |   File "/usr/local/lib/python2.7/site-packages/bots/webserver.py", line 37, in start

bots-server_1  |     botsinit.generalinit(configdir)     #find locating of bots, configfiles, init paths etc.

bots-server_1  |   File "/usr/local/lib/python2.7/site-packages/bots/botsinit.py", line 47, in generalinit

bots-server_1  |     raise botslib.PanicError(u'In initilisation: path to configuration does not exists: "%(configdir)s".',{'configdir':configdir})

bots-server_1  | bots.botslib.PanicError: In initilisation: path to configuration does not exists: "config".

bots_build-master_bots-server_1 exited with code 1

My target is to share these 3 folders with the host to change data from there while the container is running. Am I missing something?

Thanks for help in advance.

Best regards,

Robert

[Wassily]

Hi Robert

If I recall correctly, you should NOT change the dist-packages folder to site-packages.

Did you find that it did not work with the dist-packages folder?

It has been 3 years since I had done that, so I don't recall why I had to use dist-packages as also I had the issue. Nevertheless, I think it is time I review that, Docker has evolved and there are better ways to build this now then back then and I would probably also switch away from Alpine Linux in this case.
May I suggest you try with the dist-packages folder. Yes, it does seem wrong and counter intuitive, but I do think this is the way it works in that container. I'll review the Dockerfile in the next couple of days and try to improve it.

[Robert Rauer]

Hi Wassily,

thanks a lot for checking the docker configuration after all that years. I used the dist-packages in my docker-compose.yaml as follows:

version: '3'

# Running BOTS with JobQueue Server enabled.

# The Supervisor uses environment variables SUPERVUSER for HTTP user login and

# SUPERVPASS for password.

services:

  bots-server:

    build:

      context: .

      dockerfile: Dockerfile

    ports:

      - "0.0.0.0:8080:8080"

      - "0.0.0.0:9001:9001"

    environment:

      - SUPERVUSER=bots

      - SUPERVPASS=botsbots

    volumes:

      - ./config:/usr/local/lib/python2.7/dist-packages/bots/config

      - ./botssys:/usr/local/lib/python2.7/dist-packages/bots/botssys

      - ./usersys:/usr/local/lib/python2.7/dist-packages/bots/usersys

This configuration works but bots is installed in /usr/local/lib/python2.7/site-packages/bots

Which means the directories stay empty. Adding and removing files to the mapped directories works so the docker part is fine.

It seems that changing volumes to 

    volumes:

      - ./config:/usr/local/lib/python2.7/site-packages/bots/config

      - ./botssys:/usr/local/lib/python2.7/site-packages/bots/botssys

      - ./usersys:/usr/local/lib/python2.7/site-packages/bots/usersys

 

make them unaccessible for /usr/local/bin/bots-webserver.py on startup.

This is hard for me to understand. 

Is there a way to link the /usr/local/lib/python2.7/site-packages/bots/* folders to /usr/local/lib/python2.7/dist-packages/bots/  ?

I tried to symlink them but that doesn't work in docker.

Thanks for your help!

Regards,

Robert

[Wassily]

Hi Robert

Have updated the repo and Dockerfile. Switched from the alpine to the -slim python container. Everything should be proper now in site-packages. Also upgraded some of the packages to newer versions to remove the number of vulnerabilities.

You can pull updated image as before with:
docker pull chadgates/bots-build:latest 

Hope that fixes is for you.

Cheers

Wassily

[Robert Rauer]

That's great Wassily!

Thank you very much for upgrading. That solves the volume problem. Everything is in its place now :-)

Thanks.

Robert

[duf...@gmail.com]

Hi Wassily,

   I'm attempting to use the docker and docker-compose files as specified at https://github.com/chadgates/bots_build, but am not able to replicate Robert's success. It runs okay out of the box, but attempting to utilize the volumes in docker-compose.yaml prevents the apps from starting correctly; supervisord complains on every entry. Investigating, I found that the volumes are not wired together properly, and if I go look in  /usr/local/lib/python2.7/site-packages/bots/config, there is no crontab. The crontab is present and everything fires off fine if I don't include the volumes entry in docker-compose.yaml. Can you verify this docker-compose.yaml is correct if using volumes as you intended?

Thanks,

Jason

duffy@joyce ~/PycharmProjects/bots_build $ cat docker-compose.yaml
version: '2'

# Running BOTS with JobQueue Server enabled.
# The Supervisor uses environment variables SUPERVUSER for HTTP user login and
# SUPERVPASS for password.

services:
  bots-server:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "0.0.0.0:8080:8080"
      - "0.0.0.0:9001:9001"
    environment:
      - SUPERVUSER=bots
      - SUPERVPASS=botsbots

    command: /usr/local/bin/supervisord -c /etc/supervisor/supervisord.conf

    volumes:
      - ./config:/usr/local/lib/python2.7/site-packages/bots/config
      - ./botssys:/usr/local/lib/python2.7/site-packages/bots/botssys
      - ./usersys:/usr/local/lib/python2.7/site-packages/bots/usersys

[Robert Rauer]

Hi Jason,

not sure if this helps:

as I didn't use Wassily's docker file as it is I can't say if this worked out of the box with mounted volumes.
I had to do some manual changes:

1. copied my botssys, usersys and config to the mounted directories from the host system (including crontab to config) before starting the container

2. changed the path in crontab to 05,15,25,35,45,55      *     * * *   python /usr/local/bin/bots-job2queue.py /usr/local/bin/bots-engine.py -cconfig --new

as per default there were errors in stderr (no such file)

3. for directory monitor to work I put pyinotify to requirements/base.txt

What error messages do you get?

Regards,

Robert

[Wassily]

Hi Jason

Can you share the errors you are getting as when running locally, supervisor start the required service without any problem. Did you get the image from dockerhub or did you check out the git repo and build it from there? I would assume that the later would lead to success, but using the dockerhub image probably requires an adjusted docker-compose file.

Please note that the dockerfile as is can get you started but needs adjustments to your local setup, it is somewhat of a compromise. Initially, everything lives within the docker container. So I'd suggest you do two stage adjustement:

1.) only mount the usersys and botssys from the host, and leave the config path in the container.

2.) when you find things working, copy the config path details from the container to your host, so you can make adjustments to them to your liking.

I do NOT recommend to use directory monitoring this way, as Docker adds a file system layer and will not get notified about host file system changes reliably. If you want to have a file system monitoring active within a container, I suggest you use other means. I have used the watchdog package to do this here (but also this documentation requires updating...): https://django-pyas2.readthedocs.io/en/latest/detailed-guide/extending.html

Hope above helps.

Greets

Wassily

[Jason Duffy]

Hi Robert. Thank you for the reply! The error messages I got led me to notice the empty botsys, usersys, and config directories. Your #1 "copied my botssys, usersys, and config..." step was the advise I needed. Thank you kindly!

Thanks,

Jason

You received this message because you are subscribed to a topic in the Google Groups "Bots Open Source EDI Translator" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/botsmail/K2oFSsVw_Eo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to botsmail+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/botsmail/9e359ea7-7bf3-499a-9bdb-975ba5deea16n%40googlegroups.com.

[Jason Duffy]

Hi Wassily. Thank you for the response! The error messages I got were about basically about empty botssys and config directories. Copying the botssys, config, and usersys directories from the python zip up to the directory containing docker-compose.yaml enabled the setup to work out of the box. I was pulling from the git repo btw.

Yeah I'm not planning to use this in a production setup; I'll dig in to those details if we find a need for bots. I needed to do a quick demo for a new client and found your github repository from Google. I've now found it does work on both my Macbook and a Linux box if I add that copy step to your instructions. Might save someone else the trouble if you add the step to your readme before the volumes section.

Thanks again for putting it together and publishing!

Thanks,

Jason

You received this message because you are subscribed to a topic in the Google Groups "Bots Open Source EDI Translator" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/botsmail/K2oFSsVw_Eo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to botsmail+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/botsmail/01f0ed4e-b472-4d38-b990-34f327f1e9e6n%40googlegroups.com.

[Antonio Pizarro]

Hi Vassily

Try to make the edi bots application as a Docker

I follow up your link and this is my result

sudo docker pull chadgates/bots-build:latest

Status: Downloaded newer image for chadgates/bots-build:latest
docker.io/chadgates/bots-build:latest

This created the below image O.K

REPOSITORY TAG IMAGE ID CREATED SIZE chadgates/bots-build latest 1379dd195352 9 months ago 596MB

I am stuck at this level I did a lot of research but I don't know what to do next.

Could you please share some point in order to make this image into a container and run it properly

Regards

Antonio

[Andres Oliva]

Hi Antonio

I am new to bots edi but I can share what I did and is working fine for me in my Linux environment.

First get the zip files from this site https://github.com/chadgates/bots_build

1.- Click on Code and Download the zip file into your Download folder
2.- unzip the file bots_build-master.zip
3.- Create a directory in your home directory that you will use for this bots edi docker-compose
4.- copy all the files and folder into your new directory in this example I call it "bots-images"

See my directory  bots-images

[andres@bots-images]$ ls -ltr
total 460
-rw-r--r-- 1 andres andres    857 Jun 17  2021 supervisord.conf
-rw-r--r-- 1 andres andres   3259 Jun 17  2021 README.md
-rw-r--r-- 1 andres andres    477 Jun 17  2021 postinstall.py
-rw-r--r-- 1 andres andres   1329 Jun 17  2021 Dockerfile
-rw-r--r-- 1 andres andres    459 Jun 17  2021 docker-compose.yaml
-rw-r--r-- 1 andres andres   1120 Jun 17  2021 crontab
-rw-r--r-- 1 andres andres    817 Jun 17  2021 bots_build_hdi_settings.json
-rw-r--r-- 1 andres andres 435467 Jun 17  2021 bots-3.2.0.tar.gz
drwxr-xr-x 2 andres andres   4096 Apr  2 17:29 requirements

After you make sure that your docker.service is running

only after that enter the below command to build the images and the container for bots edi

sudo docker-compose up -d

Hope that will help you.

Regards

Andres