Importing data for custom observations

[Zach]

I'm trying to track the water level of a lake so I added a new column to the database (lakeSurfaceLevel), created a service to populate the data, and modified my skin to display it. All of that works great, however when I try to import data for that new observation using weectl, I get a error "KeyError: 'lakeSurfaceLevel'

Any help would be greatly appreciated!

Attached are the csv.conf and a data file

[gjr80]

Thanks, can you post the complete and exact output to the console when weectl import is run. Also, what does the weectl log show?

Gary

[Zach]

Sorry, I should have clicked reply all. ;)

Here is the output

Using configuration file /Users/zach/weewx-data/weewx.conf

This is a dry run. Nothing will actually be done.

Starting weectl import...

A CSV import from source file '/Users/zach/Downloads/Untitled.csv' has been requested.

The following options will be used:

     config=/Users/zach/weewx-data/weewx.conf, import-config=/Users/zach/weewx-data/csv.conf

     source=/Users/zach/Downloads/Untitled.csv, from=None, to=None

     dry-run=True, calc_missing=False, ignore_invalid_data=True

     tranche=250, interval=derive, date/time_string_format=%Y-%m-%d %H:%M:%S

     delimiter=',', wind_direction=[-360.0, 360.0]

     UV=False, radiation=False

Using database binding 'wx_binding', which is bound to database 'weewx.sdb'

Destination table 'archive' unit system is '0x01' (US).

The following imported field-to-WeeWX field map will be used:

     source field 'datetime' in units 'unix_epoch' --> WeeWX field 'dateTime'

     source field 'lakeSurfaceLevel' in units 'foot' --> WeeWX field 'lakeSurfaceLevel'

Imported records will not overwrite existing database records.

All WeeWX UV fields will be set to None.

All WeeWX radiation fields will be set to None.

This is a dry run, imported data will not be saved to archive.

Starting dry run import ...

Obtaining raw import data for period 1 ...

Raw import data read successfully for period 1.

Mapping raw import data for period 1 ...

Traceback (most recent call last):

  File "/Users/zach/weewx-venv/bin/weectl", line 8, in <module>

    sys.exit(main())

  File "/Users/zach/weewx-venv/lib/python3.9/site-packages/weectl.py", line 67, in main

    namespace.func(namespace)

  File "/Users/zach/weewx-venv/lib/python3.9/site-packages/weectllib/__init__.py", line 90, in dispatch

    namespace.action_func(config_dict, namespace)

  File "/Users/zach/weewx-venv/lib/python3.9/site-packages/weectllib/import_cmd.py", line 85, in import_func

    weectllib.import_actions.obs_import(config_dict,

  File "/Users/zach/weewx-venv/lib/python3.9/site-packages/weectllib/import_actions.py", line 58, in obs_import

    source_obj.run()

  File "/Users/zach/weewx-venv/lib/python3.9/site-packages/weeimport/weeimport.py", line 406, in run

    _mapped_data = self.map_raw_data(_raw_data, self.archive_unit_sys)

  File "/Users/zach/weewx-venv/lib/python3.9/site-packages/weeimport/weeimport.py", line 976, in map_raw_data

    weewx.units.obs_group_dict[_field])

  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/collections/__init__.py", line 941, in __getitem__

    return self.__missing__(key)            # support subclasses that define __missing__

  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/collections/__init__.py", line 933, in __missing__

    raise KeyError(key)

KeyError: 'lakeSurfaceLevel'

[gjr80]

OK, I suspect the problem is when you added field lakeSurfaceLevel to WeeWX you didn't tell WeeWX what unit group the field belongs to. Consequently, when you attempt to import data into lakeSurfaceLevel weectl import conducts some checks to see if any source field unit conversion is required. That is what is causing the KeyError error.

The simple fix is to tell WeeWX what unit group is used by the WeeWX field lakeSurfaceLevel. There are a number of ways to do this, the most common is to add a few lines to ~/weewx-data/bin/user/extensions.py. You could try adding the following to ~/weewx-data/bin/user/extensions.py:

import weewx.units
weewx.units.obs_group_dict['lakeSurfacelevel'] = 'group_altitude'

save extensions.py and try the import agin.

Gary

[Zach]

I have that line in my custom service for pulling the data from an API. Should I also put it in the extension.py?

[gjr80]

The problem is weectl import is a standalone utility run independent of WeeWX and as such does not load the WeeWX services specified in the WeeWX config file (well it actually does cause some of the services to be loaded, but later on after the import has occurred when missing obs are/may be calculated). I've had some second thoughts since my previous post and I am not even sure that adding those lines to extensions.py will work. If it is easy enough to try please try it, it will either work or not, nothing will be broken/corrupted.

Failing that there will be a small change required to weectl import that I should be able to implement later today.

Let me know how you get on.

Gary

[Zach]

Unfortunately, adding it to extensions.py did not work. I'll sit tight until a fix is completed. But no hurry!

Thanks Gary!

[gjr80]

OK. Thanks.

Gary

[gjr80]

Zach,

Did you by chance copy and paste those two lines from my earlier post into extensions.py? If so if you look closely I had a typo, lakeSurfaceLevel was incorrectly capitalised. I've just run weectl import with your data and import config file with the correct two lines in extensions.py and the import worked just fine.

Gary

On Wednesday 24 July 2024 at 08:53:39 UTC+10 Zach wrote:

[Zach]

That was it! Thank you so much for your help!!!

[Graham Eddy]

i.e. add new field to the schema, as well as just add column in database

⊣GE⊢

[Zach]

I thought the schema is only used during database creation. Did I min-read that?

[Graham Eddy]

i meant it in broader context of the defining the types

re the schema columns table in extensions.py, i have seen tom several times say it is used only on creation but i have encountered it being used at different times while fiddling

⊣GE⊢

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/cc874a65-e25b-42af-ae4e-24a637ce4d74n%40googlegroups.com.

[Zach]

Ok, thank you for the clarification!