Major GrowSense updates/fixes/improvements completed

[John JR]

Short version:

I just completed major update to the GrowSense system including improving, fixing and/or adding functionality to the web based UI, the plug and play system, the distributed system functionality (ie. working with multiple garden computers linked together), and the automatic upgrade system.

It seems like everything is working and stable.

Cory just installed the latest version and connected up an arduino (after uploading the irrigator sketch to it) and reported (after 1 small bug fix) that it was detected and shows up on the web based UI as expected.

The latest updates have now been rolled out to the "lts" (long term support, aka stable) branch and is ready for people to use.

Long version:

Here's a more detailed description of what I did...

1) The web based UI now has:

a) Authentication (you log in using the same username and password that is set for the MQTT broker during install) so no-one can use it without the username and password

b) The ability to rename devices (even if it's on a different garden computer)

c) The ability to delete a device (even if it's on a different garden computer). So if a ghost entry (a device still registered but not connected) ever occurs it can easily be deleted.

d) More functionality for setting irrigator and soil moisture monitor calibration settings (including a "default settings") option. I need to double check the default settings are accurate but they should be pretty close.

e) Improved ability to detect added/removed devices and subscribe to/unsubscribe from their MQTT topics.

f) The ability to reconnect to the MQTT broker if the connection ever drops out (eg. the MQTT broker on another server is rebooted, or there's a network connection glitch).

g) The ability to link another garden computer to the system using a simple form.

2) The device names used to be automatically generated every time they were detected and configured by the plug and play system.

This generally worked fine but occasionally if you unplugged and plugged back in a device (or the USB connection just glitched temporarily) at exactly the wrong moment (after the plug and play service has checked for removed devices and before it checks for new devices) it would result in a duplicate device entry; irrigator1 would become irrigator2 and irrigator1 would still be registered but no longer connected.

Another downside to this approach was that if you had 2 irrigators (irrigator1 and irrigator2) connected then the USB port numbers got reassigned in a different order (which can happen after a reboot or if the USB connections glitch temporarily and then get restored) the two would potentially swap names and irrigator1 would become irrigator2, and irrigator2 would become irrigator1.

While this didn't alter the settings because they're stored on the device this would potentially cause confusion for users, when they have in their head that irrigator1 is connected to one plant, irrigator2 is connected to another one, and after a reboot or a glitch the names get swapped.

The fix was to make a pretty large update to the plug and play service, the plug and play scripts it launches, as well as every device in the GrowSense collection.

Now the first time you plug in a device the name is automatically generated, then the name is sent as a command to the device and stored in EEPROM (the memory of the arduino).

Once stored in EEPROM even if you unplug the device and plug it back in, or if the USB connections glitch, or if the USB port numbers are reassigned, whatever happens it will reuse the same name.

Even if you unplug the device from one garden computer and plug it back into a different computer it will still keep the same name.

The only time it will regenerate a new name is if another device with the same name is detected on a different garden computer on the same network. This functionality is to prevent conflicts between devices on different garden computers. We don't want 2 devices on the same network with the same name.

3) The automatic upgrade system (which automatically upgrades the garden computer and any devices attached when I release a new version) was flawed.

It's launched at a specific interval by a "supervisor" script/service which runs in the background.

The problem was that the upgrade process requires stopping all services including the supervisor service, upgrading the system, then reinstalling all services (in case they've been updated), then restarting all services.

The reason this was flawed is that the supervisor service would launch the upgrade script, the upgrade script would stop the supervisor service, and in doing so it would stop the upgrade script itself (because it was being run by the supervisor service).

The fix (after messing around with one potential solution I ended up abandoning) was to separate the two services. Now the upgrade script is still launched by the supervisor script but as a separate systemctl service. Then when the upgrade script stops the supervisor service the upgrade service still continues until it's complete, and restarts the supervisor service once it's done. And the upgrade service just sits there doing nothing until the supervisor triggers it again.

Along with that I reorganized the upgrade script itself and fixed a few limitations it had, as well as adding reporting (to email and to the message list which shows up on the UI) when an upgrade starts and when it completes.

One limitation that still exists with the upgrade system is that when it stops all services it also stops the web based UI, and stops any publishing of data to MQTT.

So during an upgrade the web based UI goes offline. What should really happen is for it to swap over to a friendly "this system is upgrading" web page. I'll eventually figure out how to implement that but it's not critical for the system to operate.

I'll need to create a separate web page/site on the GrowSense system which gets launched after the web UI is taken offline, then shut down before the web UI is brought back online.

For now if the system is automatically upgrading the web UI on the device will simply be offline and you'll get a "page not found" error until the upgrade completes. This is a little bit annoying but you just have to wait a minute or two and hit refresh and the web UI will be back online.

If the web UI never comes back online let me know and I'll figure out why and fix it.

4) The supervisor service used to pull device and message info from other connected garden computers each time it looped. This worked but sometimes the supervisor service is busy checking the status of the system or the devices which takes some time to complete.

If you plugged in a device to a remote garden computer during this time it wouldn't show up on the web based UI until all the status checks completed, potentially minutes depending on how many devices the supervisor needs to check the status of.

The fix was to separate the "pull devices/messages from remotes" functionality into a separate systemctl service/script called the "mesh manager" (the term mesh referring to a network of interlinked computers), so now it runs every 5 seconds even if the supervisor script is busy doing something else.

Now if you plug in a device on any remote garden computer it should only take 5 to 10 seconds to be not only detected and configured on that computer, but also detected by the web based UI on another garden computer.

5) Cory found a bug when he installed the GrowSense system which I fixed by changing from python 2 to python 3. This was a necessary fix because python 2 is being deprecated and potentially why it prevented platform.io from installing even though the previous version of the install script worked fine on my systems.

[BL]

Hello John and fellow Sensoricans :)

I tinkered around to make a logo. It's a first draft, still needs work but I think it reflects the brand we are trying to build. 

I will work on it some more, to fix the colours and work on the font as well. I will send you something by the end of the week and integrate any comments you may have (don't be shy!).

Tibi published it here:

https://sites.google.com/view/growsense/home

Waiting to hear from you!

Bev

--
Go to SENSORICA home
https://sites.google.com/site/sensoricahome/home
Go to our Working Space
https://sites.google.com/site/sensoricahome/home/working-space
--
You received this message because you are subscribed to SENSORICA group.
To post to this group, send email to Sens...@googlegroups.com
To unsubscribe from this group, send email to
Sensorica+...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/Sensorica?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups "SENSORICA" group.
To unsubscribe from this group and stop receiving emails from it, send an email to Sensorica+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/Sensorica/CAK73SL29DW20yO%2B5RiLh6MJDU8nWXSiW2JLN0bgVcZQZMfE4gw%40mail.gmail.com.