Import from Weathercat

[Ville Helander]

Hello all!

I have searched alot from the net and Weewx github to understund importing from Weathercat.

I can look the wee_import code and there is mention of weathercat and some info. 

What should I put to import.conf file? 

I see the examples of CSV, WU and Cumulus import conf files, but not example of importing from Weathercat monthly files. 

Does anyone done importing from Weathercat?

-Ville-

[gjr80]

Hi,

I recall doing some work with WeatherCat log files and wee_import at home, but I don’t recall ever pushing anything to the WeeWX repo. I had a quick look just now and I can find nothing. Can you point me to the information that makes you believe wee_import can import WeatherCat data.

Gary

[Ville Helander]

Hi,

Thank’s for answering.

When you watch the code of wee_import utility you can see the weathercat mention there.

Here is link: https://github.com/weewx/weewx/blob/master/bin/wee_import

gjr80 <gjrod...@gmail.com> kirjoitti 12.10.2019 kello 14.58:

Hi,

I recall doing some work with WeatherCat log files and wee_import at home, but I don’t recall ever pushing anything to the WeeWX repo. I had a quick look just now and I can find nothing. Can you point me to the information that makes you believe wee_import can import WeatherCat data.

Gary

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/zc4atfmQIco/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/df89b214-b2ee-45af-bab3-4497a7132a71%40googlegroups.com.

[gjr80]

Thank you, seems I did commit WeatherCat support back in April 2017 but it appears (for reasons unknown) to have been largely removed. I will have a look at resurrecting it (it shouldn’t be too hard) though it was a first cut so may still have some rough edges plus documentation is still to be updated. I will try to include it in the 4.0 release though I have a few other things to finish first plus free time is somewhat scarce at the moment. Fortunately your WeatherCat data is not going anywhere :)

Gary

[Ville Helander]

Thank you very much!
I wait to get it into the development version of Weewx and then can test it. :)

-Ville-

> On 13 Oct 2019, at 0.06, gjr80 <gjrod...@gmail.com> wrote:
>
> Thank you, seems I did commit WeatherCat support back in April 2017 but it appears (for reasons unknown) to have been largely removed. I will have a look at resurrecting it (it shouldn’t be too hard) though it was a first cut so may still have some rough edges plus documentation is still to be updated. I will try to include it in the 4.0 release though I have a few other things to finish first plus free time is somewhat scarce at the moment. Fortunately your WeatherCat data is not going anywhere :)
>
> Gary
>

> --
> You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/zc4atfmQIco/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/a9342c34-fcb5-4b6d-9aa1-faa43253f160%40googlegroups.com.

[Ville Helander]

Hello Gary,

WeeWX beta1 is just released, is this WeatherCat import available with that beta1 release?

-Ville-

> On 13. Oct 2019, at 0.06, gjr80 <gjrod...@gmail.com> wrote:
>
> Thank you, seems I did commit WeatherCat support back in April 2017 but it appears (for reasons unknown) to have been largely removed. I will have a look at resurrecting it (it shouldn’t be too hard) though it was a first cut so may still have some rough edges plus documentation is still to be updated. I will try to include it in the 4.0 release though I have a few other things to finish first plus free time is somewhat scarce at the moment. Fortunately your WeatherCat data is not going anywhere :)
>
> Gary
>

> --
> You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/zc4atfmQIco/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/a9342c34-fcb5-4b6d-9aa1-faa43253f160%40googlegroups.com.

[gjr80]

Hi,

No unfortunately it has not been included in b1. I will get to it, unfortunately other things have taken priority.

Gary

On Saturday, 16 November 2019 21:10:32 UTC+10, Ville Helander wrote:

Hello Gary,

WeeWX beta1 is just released, is this WeatherCat import available with that beta1 release?

-Ville-


> On 13. Oct 2019, at 0.06, gjr80 <gjrod...@gmail.com> wrote:
>
> Thank you, seems I did commit WeatherCat support back in April 2017 but it appears (for reasons unknown) to have been largely removed. I will have a look at resurrecting it (it shouldn’t be too hard) though it was a first cut so may still have some rough edges plus documentation is still to be updated. I will try to include it in the 4.0 release though I have a few other things to finish first plus free time is somewhat scarce at the moment. Fortunately your WeatherCat data is not going anywhere :)
>
> Gary
>
> --
> You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/zc4atfmQIco/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to weewx...@googlegroups.com.

[Ville Helander]

Hello Gary and all,

WeeWX 4 is released, thank you for all the hard work everyone has made!!

I see that import from WeatherCat is not included there and would like to know is there happening any progress for making import working?

I can test the importing when there is somethink to test. :)

Again, thanks for everyone who work with weewx!! 

-Ville-

To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/f6f9db7a-d353-4343-a3c3-cbf5b8752f33%40googlegroups.com.

[gjr80]

Hi Ville,

My apologies that WeatherCat support for wee_import did not make it into 4.0.0. However, I have started adding back the the WeatherCat code that removed back in 2017. I see no reason why it will not appear in the next release. I will certainly get you to do some testing, my sample WeatherCat data is of somewhat dubious heritage.

Gary

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/f6f9db7a-d353-4343-a3c3-cbf5b8752f33%40googlegroups.com.

[Ville Helander]

Ok,

No problem with waiting, just wanted to hear somethink of WeatherCat import state. 

But it sound’s promising! 

-Ville-

To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/ed4ade9e-e49e-4825-87df-98f61684917f%40googlegroups.com.

[Ryan Stasel]

Hi Ville,

Just a heads up. I literally JUST imported 11 years worth of data out of WC into Weewx using a csv export from WC (5min intervals) and then using wee_import to import into weewx. Certainly not as clean as a native import, but it seems to have worked. Initially tested on my Pi4 and it took about 4 hours to do the import of 1.2M rows. Did it a second time on a ubuntu VM and took about 1.5 hours. The import is definitely single threaded (or close). So I'd run on the fastest single core you have. =)

Ok,

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/ed4ade9e-e49e-4825-87df-98f61684917f%40googlegroups.com.

[Ville Helander]

Hi Ryan,

Sounds great! I tryet importing from csv file some time ago, but newer get it to work. I guess it was user error..

Can you share more your configs how you did the import?

-Ville-

To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/91a8a6bb-f324-43c4-a297-eb61e3085e84%40googlegroups.com.

[gjr80]

Ville,

I have the WeatherCat import working under WeeWX4/python3 (yet to confirm python2) the only outstanding issue I have is units. I do not have access to WeatherCat so I am relying on the User Manual and whatever I can glean from Google. I was hoping you might be able to confirm one assumption and answer another couple of question relating to units:

1. The manual says the unit settings under Preferences, Units/Misc1 are applied throughout WeatherCat, I assume this means the .cat files as well?

2. What units are available under each of temperature (and dewpoint), pressure, windspeed, wind direction and precipitation?

3. I see that wind direction shows the setting 'compass', does that mean degrees or cardinal/intercardinal compass points (eg N, S, NE, NNW etc)?

If you cannot help with 2 and 3 that is fine, it won't greatly affect the wee_import code but I would like to include some appropriate guidance on import settings in the example WeatherCat import config file.

If you do intend to use the WeatherCat wee_import module there is no need to upgrade to WeeWX 4 if you have not already done so, I will be able to provide you with fairly straightforward instructions to run the 4.0 code side by side with WeeWX 3.x.

Gary

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/91a8a6bb-f324-43c4-a297-eb61e3085e84%40googlegroups.com.

[Ville Helander]

I try to answer.

1. 

Yes. 

Here is screenshot from my WeatherCat Preferences, Units/Misc1 and copy of this (May) months .cat file.

[Ryan Stasel]

Yup. Only concern (though not huge) is pressure vs barometer. According to WC author, what's in the csv is "pressure". But that doesn't make sense to me since WC displays "Pressure (rel)" which to me means "Barometer" unless I'm completely misunderstanding. 

My csv.conf is attached. I did end up moving the recorded pressures to barometer in my final import. Things look correct, but I haven't done (and likely won't do) a thorough check. 

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/91a8a6bb-f324-43c4-a297-eb61e3085e84%40googlegroups.com.

[gjr80]

I think we need to take the authors advice at face value given the following is at the top of each .cat file:

WeatherCat data file ***If you modify this file, be sure to delete the relevent .hrs file***.
t and V are not optional, all other fields are.
t is the day, hour and minute (2 digits each), T is outside temperature, Ti is internal temperature, T1 to T8 is auxiliary temperatures, D is dew point, Pr is barometric pressure, W is wind speed,
Wd is wind direction, Wc is wind chill, Wg is wind gust, Ph is hourly precipitation, P is total precipitation,
H is outside humidity, Hi is internal humidity, H1 to H8 are auxiliary humidity sensors, S is solar,
Sm1 to Sm4 is soil moisture, Lw1 to Lw4 is leaf wetness, St1 to St4 is soil temperature,
Lt1 to Lt4 is leaf temperature, U is UV, Pm is monthly precipitation, Py is annual precipitation, Ed is daily ET, Em is monthly ET, Ey is yearly ET, C is current conditions (delimited by double quotes), V is validation.

Gary

[gjr80]

Thanks Ville, that all falls pretty much into line with what I have done. Not exactly sure what is meant by/the differences between degrees, compass and cardinal. At present all that wee_import will support is degrees, though I have some cardinal/intercardinal code elsewhere in wee_import that I can use to support the cardinal/intercardinal directions.

If you have WeeWX 4 running that makes it easier, there will just be a few files to download and update. Will do a quick check on WeeWX4/python2 this afternoon and should have this posted tonight.

Gary

[Ville Helander]

For some reason, I see only part of my yesterdays message.

I send it again, let’s hope it works ok now.

[Ville Helander]

Thanks for this config file. I check it also.

Maybe the config can be uploaded to weewx github?

-Ville-

To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/85440857-6bb5-4e2e-8763-6af44fd6f19e%40googlegroups.com.
<csv.conf>

[Ville Helander]

2. 

Temperature: °C and °F

Dewpoint: °C and °F

Pressure: mBar, in Hg, hPa

Windspeed: km/h, MPH, Knots, m/s

Wind direction: Degrees, Compass, Cardinal

Preciputation: mm, inches

Height: Metres, Feet

Time: 12 Hr, 24 Hr

3.

I think compass settings is like N, S, NE, etc..

Here is link for my stations tag file where is all kind of data. I don’t know does it help anythink, but just in case.

https://kauttua.net/allindex.html

Here is also link to the Trixology site’s tag file, and there can see the compass points. 

https://live.trixology.com/custom/

I hope these help you!

I can try import any time, I have WeeWX 4 running right now via virtualvp to test things.

-Ville-

[gjr80]

Ville,

I received all of your post in the email summary, think the embedded image may have caused some hiccups for google groups.

I have yet to write the weathercat instructions for the Utilities Guide but they will be very similar to the Cumulus and Weather Display sections, the process will be very similar so you may get some good background by reading those sections.

The easiest was to use the weathercat import and not interfere with your current install is to clone the WeeWX Github development branch and run wee_import from there. Instructions as follows:

1. install git if not already installed:

$ sudo apt-get install git

2. create a directory in which the WeeWX repo will be cloned and change to that directory:

$ mkdir /var/tmp/src
$ cd /var/tmp/src

3. clone the development branch of the WeeWX GitHub repo:

$ git clone -b development https://github.com/weewx/weewx

4. check wee_import works:

$ sudo PYTHONPATH=/var/tmp/src/weewx/bin /var/tmp/src/weewx/bin/wee_import --help
$ sudo PYTHONPATH=/var/tmp/src/weewx/bin /var/tmp/src/weewx/bin/wee_import --version

you should see the wee_import help and the version should be 0.5

5. copy the example weathercat import config file to /var/tmp:

$ cp /var/tmp/src/weewx/util/import/weathercat-example.conf /var/tmp/wcat.conf

6. edit /var/tmp/wcat.conf and have a read through the settings and the comments. Change settings as required. Likely you will only need to change the directory setting, the interval setting and perhaps the [[Units]] settings. The directory setting should be the path to the directory containing the year directories containing your .cat files. The interval setting should be set to the interval used in the .cat files, in the .cat files I have seen this has been 1 minute so I have used interval = 1. Change the [[Units]] settings to suit the data in your .cat files, remember the setting must use WeeWX unit codes, the comments should help you.

7. stop WeeWX and backup your database

8. first try a dry run, the import command will look something like:

$ sudo PYTHONPATH=/var/tmp/src/weewx/bin /var/tmp/src/weewx/bin/wee_import --config=/home/weewx/weewx.conf --import-config=/var/tmp/wcat.conf --dry-run

You need to use your existing weewx.conf, I have assumed it is in /home/weewx, depending on your install it might be in /etc/weewx, change as required. You will likely see a lot of warnings indicating a field was mapped but no data was found, this is due to the fields that are in the wee_import .cat field map but for which there is no data in your .cat files, mostly it is the less common fields such as soil moisture, leaf temperature etc. Unless there are obvious discrepancies you can ignore these. You can suppress the warnings using the optional --suppress-warnings command line option.

I suggest that for your first dry run import just have a few months of .cat files in your import directory, or limit your import with --from and --to command line options, otherwise you might be waiting for some time.

9. if the dry run goes OK do a live import by removing the --dry-run command line option.

10. finally, you can remove the trial code code using:

$ rm -R /var/tmp/src

Any problems/issues/feedback please let me know.

Gary

[Ville Helander]

Thx for this! 

-Ville-

Ryan Stasel <rcst...@gmail.com> kirjoitti 12.5.2020 kello 23.30:



To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/85440857-6bb5-4e2e-8763-6af44fd6f19e%40googlegroups.com.

<csv.conf>

[Ville Helander]

Good to hear that email was send succesfully.

I try to get time on weekend to test it, I can make whole new virtual machine to try that import. 

Since yesterday I had to take my laptop (Macbook pro from 2016 with very shitty keyboard) to repair shop and have to work with space computer so it is not so easy, but I think I can locally attach keyboard and monitor to my server and try it locally.

I answer how thinks go.

-Ville-

 

gjr80 <gjrod...@gmail.com> kirjoitti 13.5.2020 kello 17.36:



--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/zc4atfmQIco/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/814c912a-06db-452f-ae7c-fe2ce7907aaa%40googlegroups.com.

[gjr80]

OK, should have the first cut of the WeatherCat section of wee_import portion of the Utilities Guides done today. Will post a link once done, check back before you do the import as the Utilities Guide will be an easier read than comments in a file.

Gary

To unsubscribe from this group and all its topics, send an email to weewx...@googlegroups.com.

[gjr80]

Ville,

I have pushed the first cut of the Utilities Guide that includes WeatherCat import details to GitHub. You can view the document via this link. Just a word of caution, the link provided will let you view and scroll through the Utilities Guide but you will likely find the navigation panes on the left hand side of the screen will not work, likewise for some of the embedded links - the vagaries of viewing version controlled HTML in GitHub.

Gary

[gjr80]

Ville,

Did you get around to trying the wee_import WeatherCat module?

Gary

[Ryan Stasel]

Hi Gary,

I'm just looking at this using weewx 4.1.1, and I'm seeing an odd result importing from WC using dryrun.

It's throwing lots of errors related to LeafTemp and SoilTemp not being mapped (several for each since it's 4 sensors per). Which, makes sense, as I don't have those sensors (in WC or Weewx).

Thoughts?

[Ryan Stasel]

Realized I missed including actual errors.

"Import field 'Lt2' is mapped to weewx field 'leafTemp2' but the import field 'Lt2' could not be found in one or more records.
WeeWx field 'leafTemp2' will be set to 'None' in these records"

[Ryan Stasel]

--suppress-warnings fixed it. =) And it seems to be importing fine.

Sorry for the noise!

[gjr80]

You worked it out, I don’t think of those messages as errors rather they are warnings to check that the import is setup correctly. Hard to do it any other way with fixed format sources unless you add a lot of complexity.

Gary

[Ryan Stasel]

Cool! It just was hard to track what was going on given the errors/warnings.

Shame we can't just say ignore those (fields, or lack there of) during import, unless I'm missing something obvious.

[gjr80]

On second thoughts there probably is some scope to have some sort of ignore list. I can see it relying on the user having a reasonable understanding of his/her data. I will make a note to look at this again.

Gary

[Ryan Stasel]

So, did get it working. Interesting note. When I was previously doing imports from a WC exported csv, my weewx.sdb, after import, was about 160MB (10 years of data). After importing from weewx, my weewx.sdb is over 600MB. 

Would there be any benefit to a VACUUM on sqlite post import? not entirely sure what would be taking up all the extra space. 

[gjr80]

Sorry, almost missed this one. I thought WeeWX used a vacuum somewhere but it seems not. (it was discussed when we were working on a database patch a couple of years back and I am aware of a couple of extensions that insert/delete data in some secondary databases that use it via  schedule). I had it in my mind the adding/deleting caused the bloat whereas adding only (as wee_import does) does not.

I guess you did in fact do a vacuum, did it bring the size down? Happy to look at implementing it somehow if there is a demonstrated benefit.

Gary