Docker

[David]

To aid in testing, security, and extremely easy deployments; I added a Dockerfile to the dev branch (dev on feature happens in feature\docker). There's still more that can be added, but as is, you only need to add your instance.py file to the image and Pytomation runs. The default docker config.py will work without edit; it just won't have an admin user predefined.

It seems to work well... Except python-openzwave no longer successfully builds (not related to this change).

I'm not fully sure what happened to python-openzwave, as I can't seem to get it to successfully build on the same computer that I originally used to test on, and is still working on (was already built). It must be something that was updated in both my system and the standard Python docker image.

This is significant because we can't merge dev to master until this is resolved. Only the latest version of python-openzwave works in Python 3.

[George Farris]

I'm just about to get a Pi3 up and running and am going to document what I had to do to get it going on debian stretch. I'm going to have a go at getting the Python3 branch running, I know there are changes that have to be applied to the insteon, weeder and w800rf32 devices.

I'll upload diffs to git when I'm done. I'll also have a go at python-openzwave and see if I can get it to compile.

George

--
You received this message because you are subscribed to the Google Groups "pytomation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pytomation+...@googlegroups.com.
To post to this group, send email to pytom...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[George Farris]

Oh yes and nice work on the docker image!!

On Thu, 2018-01-11 at 20:36 -0800, David wrote:

--

[David]

I ended up getting python-openzwave to build. It wasn't building in Docker because of the image was missing libudev-dev. I'm not sure why it wasn't building on my test computer; it is now. It was probably just a disk space issue or something, since I was building and testing a lot of Docker images without pruning.

The Python3 branch might work better on the Pi3, since it seems to use less resources.

The Docker image uses the Python image, which is based on the Debian Jessie image, just one version behind. I'm sure stretch isn't far away.

[David]

I also uploaded the image to docker hub. You can install it with the command "docker pull dheaps/pytomation". You'll just need to override the instance directory and, if you use them, the ssl and openzwave config directories.

One cool thing about running it in a docker container is multiple web servers can be running on the same machine, with different IPs, all pointing to port 80 and 443.

[David]

Well, it turns out the image I released didn't fully work, just in the interactive mode I had it in. I fixed that and pushed the new image.

To get that image to run you'd run something like this (after pulling it). Here, I'm using my Insteon test instance:
sudo docker run -t -v /home/heaps/pytotest/insteon:/home/pytomation/instances pytomation:latest

While testing I did find a bug in the web UI, where the slider control isn't updating properly. I'll take a look at that. I have a feeling it's related to the changes I was making to get the Android app to compile again, with the new versions of Apache Cordova.

There are 3 issues that I see keeping me from releasing dev to master:

1. Dev branch has not been tested on Python2 and will likely need modifications to be backwards compatible
2. Dev branch does not have the Android app working
1. I'm trying to not auto-accept invalid SSL certs, and only accept the one the user created... but for some reason the new Android security, starting in version 7.0 makes this extremely difficult!
1. network-security-config has been ignored in the Cordova App (to allow for user imported certs)
2. Installing the certs directly to the system cert store, using root, has also been ignored too
   
1. This is using the standard instructions that I link to in Readme.md, seems to work on other Android versions...
   
3. I may end up having to work around or disable security, but there has to be a way for this to work... So many companies have complained I'm sure Google has done something about this (just Google it)... or they should at least be working on it
3. Users security still needs to be able to limit commands on the devices
   

I haven't made many changes to Insteon.py and no changes to the other interfaces. If you're still on Python2, you may need to run the 2to3.py script on them first, but they should otherwise merge cleanly.

[David]

I fixed the web UI bug, it actually wasn't related to the UI, Android app, or the Dockerfile. I had missed a UTF encoding on the standard HTTP Server, which is what that particular instance was testing.

I committed the change and Docker should be auto-building the new image shortly.

[David]

One last note on the Docker images. The automated builds use the branch name to tag the image, with Git 'master' mapped to Docker 'latest'. Since master does not have a Dockerfile, pytomation:lastest was removed as soon as I initiated the auto-builds.

Currently the images are:
dheaps/pytomation:dev *will be built after feature_user
dheaps/pytomation:feature_python3 *built now
dheaps/pytomation:feature_user *being built right now

You can pull using a command like:
sudo docker pull dheaps/pytomation:dev

And run it using a command similar to:
sudo docker run -t -v /home/heaps/pytotest/insteon:/home/pytomation/instances pytomation:dev

[George Farris]

Hey have you run any performance tests onm using the docker image as opposed to running Pytomation natively? I'm thinking it's probably better not to run the docker image on the RPi.

George

--

[David]

Two questions here, let me answer the one you're getting at first:
No, I would not run the Docker image on the Pi. Docker is currently best suited for a 64-bit server. The standard images being referenced are for the amd64 (Intel/AMD 64-bit) architecture, so it would force docker to run a VM... and no... not on the Pi. I could build an arm docker image, which would have near identical performance as hosting natively, but I'm not sure it make sense on the Pi.

Docker performance:
No performance tests yet. But the standard docker performance benchmarks applies. When running a docker image in a virtual machine it will, of course, run slower. You only do this if you're running a different host though, like a Windows machine. If you run docker on Linux, it basically runs in a chroot environment. The only performance degradation is with the file systems being stacked (minimal) and with the NAT network. You can actually disable NAT and go full bridge, if you want though.

Docker is used all over, in every single hosting environment worth mentioning. It's used so widely largely because of the wonderful performance, while still keeping everything in a container, for security and easy implementation. Google, Amazon, and even Netflix all use it. I think Netflix is probably the most extreme: every time you open Netflix, you're presented with your own personal docker container (which physically runs geographically close to you). Netflix still has the extreme performance to pump out 4K video to more users than I can count.

[David]

Ok, one more docker update. I ran into an issue with trying to maintain good security with the config files, since UIDs of files don't necessarily match from host to container.

To combat this (and make the run command much simpler) I gave Docker a config directory structure to ensure that only the pyto user has read access to it.

On the host, you'd generally have these files owned by root and not accessible by anyone else, especially things that hold passwords, keys, and encryption certificates. Now if you follow this structure, you can still do that and the security will be maintained in the container, with the exception that the containers pyto user will be able to read them.

This allows you to configure by placing files the following structure :
/
   Intances\ <--Your inteface files here
   Config\config.py <--Pytomation Config
   Config\openzwave.xml <-- OpenZwave options.xml (to store network key)
   ssl\ <-- Your ssl encryption certificates

Then you could run using a command like this:
sudo docker run -p 192.168.249.59:443:8443 -v /etc/pytomation:/secured dheaps/pytomation:dev

Note that a cool trick, if you're hosting several things on one large server is to add multiple IP address to the same computer (this is a *nix thing).
sudo ip address add 192.168.249.59/24 dev eth0

And for persistence, in /etc/network/interfaces:
iface eth0 inet dhcp

iface eth0 inet static
    address 192.168.249.59/24

[David]

Well, I couldn't get the timezone sync part of the Docker entry script to consistently update the container's timezone. Although the timezone always get set on my test laptop, running Ubuntu 17.10, it's always changes to UTC on my server, running 16.04.3... making testing a bit challenging. I found several suggestions on the causes and fixes, but so fare none work on that server (even though I wanted one of the fixes set the time, on my laptop).

I hope to spend some more time debugging this weekend.

[Jason Sharpee]

It's odd that I have the same timezone issue with the misterhouse docker image I built from phusion .  Please share if you ever get that fixed.

--

You received this message because you are subscribed to the Google Groups "pytomation" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pytomation+unsubscribe@googlegroups.com.

[David]

I'm not sure if I consider this fixed, but last night I ran the docker image in interactive mode (using -i -t) on the server, just as I did on my laptop. I figured that maybe it was giving me an error... no, it set the timezone... and the timezone stuck that time. So then I thought, maybe it's an error while in daemon mode; so I purged the system (deleted the interactive container) and ran it in daemon mode... but it set the timezone that time.

At least that's consistent with what happened on the laptop... but I still have no explanation as to why I had to run the docker image in interactive mode first, for the timezone to get set in daemon mode. The system was not rebooted in-between tests; but it does auto-update nightly, so it is possible an Ubuntu update occurred on the host, which allowed it to work.

I'm going to try and reproduce this a few more times. It'd be interesting if doing that on your image fixes it for you too. The code I run in the pytomation entry script is below (I run docker with the "-e TZ="America/Los_Angeles"" parameter):

if [ "x" != "x$TZ" ]

then

echo "Setting Timezone $TZ"

echo $TZ > /etc/timezone && dpkg-reconfigure -f noninteractive tzdata && ln -snf /usr/share/zoneinfo/$TZ /etc/localtime

fi

To unsubscribe from this group and stop receiving emails from it, send an email to pytomation+...@googlegroups.com.

[David]

So, the problem with the timezone not getting set is you need -t to provide the tty, to allow the environment variables. You can use -t with -d.

Also, a recent update made devices that are given access to the image, using the --device flag, can only be accessed by root. Since Pytomation runs as the pyto user, I added a DEVICES environment variable, to change the ownership of the devices it contains to be owned by pyto. This only affects the docker image, not the host device. It's a semi-colon separated list

Example docker command:
sudo docker run -td -p yourip:443:8080 --device=/dev/zwave --device=/dev/insteon --hostname=pytomation.yourdomain.net --name=pytomation.yourdomain.net -v /etc/pytomation:/secured -e "TZ=America/Los_Angeles" --restart unless-stopped -e "DEVICES=/dev/zwave;dev/insteon" dheaps/pytomation:dev

or for interactive, replace d with i:
sudo docker run -ti -p yourip:443:8080 --device=/dev/zwave --device=/dev/insteon --hostname=pytomation.yourdomain.net --name=pytomation.yourdomain.net -v /etc/pytomation:/secured -e "TZ=America/Los_Angeles" --restart unless-stopped -e "DEVICES=/dev/zwave;dev/insteon" dheaps/pytomation:dev

Remember, you can create additional IP addresses on your host machine, for the docker image to consume, like this (be sure the IP is on the same subnet):

sudo ip address add 192.168.249.59/24 dev eth0

And for persistence, in /etc/network/interfaces:
iface eth0 inet dhcp

iface eth0 inet static
    address 192.168.249.59/24