Updating Ubuntu 16.04 LTS to 20.04 LTS

[Yotta]

Wondering if anyone still monitors these conversations.

So upgraded my Linux server and it naturally broke my WFrog. So far its looking a lot like it might be fatal, but opening this thread in case I manage to get it back up and running.

Starting point:

- It looks like the upgrade left the /etc/wfrog directories intact

- The startup scripts in /etc/init.d/ were removed so I have recreated them, and have run

$ sudo update-rc.d wfrender defaults 99

$ sudo update-rc.d wflogger defaults 99

- have also reinstalled most of the required packages though it looks like some have changed enough they are likely not directly compatible.

$ sudo apt install python-cheetah

$ sudo apt install python-lxml

$ sudo apt install python3-serial

$ sudo apt install python-yaml

$ pip pygooglecharts

$ pip pyusb

However on startup, nothing is loading

$ ps -ef | grep "wfrog"
$ yotta    18381 16987  0 22:19 pts/2    00:00:00 grep --color=auto wfrog

And that's about where I am stuck. For my last upgrade wfrog was still loading as a single process, and I was able to track down the issues through the logs, but its not getting far enough to populate any of the wfrog / wfrender / wflogger logs currently

[Yotta]

Looks like this will be a project beyond my capabilities given I don't know python at all and this is a quite complex application. What I have learnt so far:

• "print" became a function in python3 and now requires brackets around the content
• "string" has become "str" and apparently is now a core module which does not need to be imported
• Linux has a tool called "2to3" which can update the most basic changes (like adding brackets to print) but is not sufficient to get the code working with python3
  
• Next issue seems to center around the "import" function. It does not throw an error itself, but does the first time an imported function is called. Have not determined what in particular is failing, whether import works differently, the modules have moved, modules have been renamed or how the module functions are called has changed.

 

[Yotta]

After having some success with fixing a different app, it seems that "pip" installs packages for Python2.7 and I needed to use "pip3" to install them for Python3.8

$ pip3 install cheetah3

$ pip3 install lxml

$ pip3 install pyserial

$ pip3 install pyyaml

$ pip3 install pygooglechart

$ pip3 install pyusb

Note: All following changes are after using the "2to3" utility.

In /opt/wfrog/wfrender/renderer/chart.py

-- NameError on line 829 for "chart". Looking at this file it seemed like the "chart" was defined within the GoogleChartWindRadarRenderer class, so raised all lines from 787 onwards to be included in this class which got me past the error (though whether that has later implications is to be seen)

In /opt/wfrog/wfrender/renderer/datatable.py

-- TabError: inconsistent use of tabs and spaces on line 45. Opening this file there was a tab then several spaces to indent this line. Replaced the tab with spaces, and made indent depth consistent with line 47 which got me past this error.

In /opt/wfrog/wfcommon/config.py

-- NameError on line 106 for "file". In this case, based on the syntax which seems to expect a file pointer, changed "file" to "open". Additionally yaml.load needs an explicit loader so added the parameter ", Loader=yaml.Loader" after the file argument which got me past this error.

-- NameError on line 110 for "Template". Similar to previous, replaced "file" with "open" and added the loader parameter for "yaml.load" then discovered that the "from Cheetah.Template import Template" had been commented out which got me past this error

-- SyntaxWarning on line 79. "is not" with a literal. Did you mean "!=". The offending line is "if element[0] is not '!'" and doing some research it does seem like this should be a '!='. != is a value comparison which is what these seems to be, whereas "is not" checks if the same memory address is used and it seems unlikely that a "!" and an array returned by a function would ever be the same memory space so I think this would currently be a constant true statement.

In opt/wfrog/wfcommon/generic/include.py

-- NameError on line 62 for "Template". Similar to previous, added the loader parameter for "yaml.load" then discovered that the "from Cheetah.Template import Template" had been commented out which got me past this error

-- NameError on line 61 for "file". changed "file" to "open" which got me past this error.

At this point it looked like wfrog actually ran, but had permission errors trying to access the USB (was running as user)

So ran:

$ sudo wfrog

And got a different set of errors which seems to stem from the superuser having a different environment path than standard user. After some digging, the modules above needed to be installed using "sudo pip3 install xxxx"

At this point it looks like wfrog is completing as both user and sudo. However its still throwing continuous errors (ie application continues to run).

In /opt/wfrog/wflogger/collector/xmlfile.py

-- NameError on line 112 for "E". Looks like "from lxml import etree" and "from lxml.builder import E" were commented out.

-- TypeError on line 108 write() argument must be str not bytes. Changed the file open arguments to binary "file = open(self.path, 'wb')"

In /opt/wfrog/wfrender/renderer/http.py"

-- AttributeError on line 116 for module cgi has no attribute 'parse_qsl". Parse_qsl was depreciated in Python3.8. Replaced "cgi.parse_qsl" with "urllib.parse.parse_qsl"

In /opt/wfrog/wfcommon/database.py

-- AttributeError on line 118'NoneType' object has no attribute "connect". The offending line was "self.dbObject = kinterbasdb.connect(dsn=self.db,". KinterbasDB has been depreciated, installed "fdb" and replaced all instances of "kinterbasdb" with "fdb"

-- DatabaseError - Could not connect. The Firebird2.5 install was broken by the upgrade. Installed Firebird3.0 as 2.5 is not available in the standard repositories for Ubuntu 20.04. Firebird 3.0 is not backwards compatible with 2.5 so now need to convert the original 2.5 database into a 3.0 database. 

Looks like this is as far as I'll get tonight, but progress has been made.

On Monday, December 20, 2021 at 10:29:18 PM UTC-6 Yotta wrote:

[Yotta]

So working through the firebird issues.

-- Firebird3.0 is not backwards compatible with Firebird2.5, recommended path is to backup 2.5 databases using the gbak tool then restore them as 3.0 databases. gbak is part of the firebird3.0-utils package:

$ sudo apt-get install firebird3.0-utils

-- Unfortunately discovered this tool is also not backwards compatible with 2.5 so:

-- Need to get the 2.5 version, but this is not an available package in 20.04

--> downloaded the classic server tarball from https://www.firebirdsql.org/en/firebird-2-5/

--> Unpacked it using

$ tar -xvzf FirebirdCS-2.5.9.27129-0.amd64.tar.gz

--> had to kill the 3.0 server for the 2.5 install to work

$ ps -ef | grep -i fire[b]ird

$ sudo kill [ProcessID]

--> Entered the created folder and ran the install

$ cd FirebirdCS-2.5.9.27129-0.amd64

$ sudo ./install.sh

--> Went into the firebird2.5 install directory

$ cd /opt/firebird/bin

--> and ran the 2.5 version of gbak to backup my database

$ sudo ./gbak -backup /var/lib/firebird/2.5/data/wfrog.db /home/user/Downloads/FBBackup/wfrog.bak

--> then using the 3.0 version of gbak restored the database to the new location

$ sudo gbak -rep /home/user/Downloads/FBBackup/wfrog.bak /var/lib/firebird/3.0/data/wfrog.db

--> discovered the permissions weren't correct and the data directories are tricky to get to so

$ sudo -i

$ cd /var/lib/firebird/3.0/data

$ chown firebird:firebird wfrog.db

$ exit

--> You then need to restart the firebird3.0 server (I simply rebooted as restarting the fguard daemon as a user is apparently not liked)

At this point I was able to log in and connect to my new 3.0 database. While I did not check the original while I had 2.5 available, the new database has 268298 records going back to when I first set up this server so seems like it is complete

$ isql-fb -u sysdba -p [password]

> connect 'localhost:/var/lib/firebird/3.0/data/wfrog.db';

> select count(*) from METEO;

Back to running:

$ sudo wfrog

In /opt/wfrog/wfcommon/database.py

-- DatabaseError on line 118: username and password not defined. Seems the core issue is the new db is not being passed to the function correctly. Adding a print command confirmed the old db was still being requested.

In /opt/wfrog/wfcommon/storage/firebird.py

-- Updated the default database to the 3.0 database /var/lib/firebird/3.0/data/wfrog.db  However this is not the DB passed to the function so needed to find the config file

In /opt/wfrog/wfcommon/config/storage.yaml

-- Updated to point to the 3.0 database, however this is not the DB passed to function so looking for other config files

In /etc/wfrog/wfcommon/config/storage.yaml

-- Finally found the source, updated to point to the 3.0 database

Now running $ sudo wfrog appears to be working, have a periodic usb timeout error but that was known to exist prior to this upgrade.

Starting the services instead

$ sudo /etc/init.d/wflogger start

$ sudo /etc/init.d/wfrender start

Confirmed they started successfully

$ ps -ef | grep wfrog

So new issue is no logs are being created that I can see. The renderer does appear to be up as it successfully throws a 500 error but without logging I can't determine that issue yet.

[Yotta]

wflogger is now working as far as I can tell, over 100 entries since last night, interval looks consistent with previous records and the data looks correct.

-- It is not logging errors

wfrender is not working, it throws a 500 error and also does not log errors.

Stopped the services and restarted as a direct application call in verbose mode:

$ sudo /etc/init.d/wflogger stop

$ sudo /etc/init.d/wfrender stop

$ sudo wfrog -v

In /opt/wfrog/wfrender/renderer/chart.py line 637

-- NameError '_valid_color'. Looks like the previous fix for "chart" was a bad idea and came back to bite me. Everything 787 onwards needs to be outside the GoogleChartWindRadarRenderer class. Reversing this change did not recreate the original error.

-- TypeError %x format is integer not float. Inserted a print command and indeed a float was being fed to this function. Added an Int() to convert.

In /opt/wfrog/wfrender/renderer/chart.py line 692

-- TypeError set_axis_style accepts 3 to 5 arguments but 7 were given. Commented out the last three values

In /opt/wfrog/wfrender/renderer/chart.py line 708

-- NameError "Chart" not defined. Suspect this is a stray capitalization

In /opt/wfrog/wfrender/renderer/chart.py line 259

-- TypeError 'cmp' is an invalid keyword for sort(). This one is going to be difficult as it appears cmp was depreciated with python3. Need to understand the sort function better to continue.