Importing data from a CSV file

[MRL]

I have about 4 years of Weather-Cat data that I can convert to a CSV file.

Here is the what the data looks like before conversion

day month year hour minute temperature   humidity     dewpoint   barometer    windspeed   gustspeed direction  rainlastmin    dailyrain  monthlyrain   yearlyrain  heatindex

 1  1 2016  0  0 44.1  45 24.1 29.954 0.1 2.0 246  0.000 0.000 0.000 0.000 44.1

 1  1 2017  0  1 44.1  46 24.6 29.953 0.2 0.0 245  0.000 0.000 0.000 0.000 44.1

 1  1 2017  0  2 44.1  46 24.6 29.953 0.0 0.0  60  0.000 0.000 0.000 0.000 44.1

 1  1 2017  0  3 44.1  46 24.6 29.953 0.3 3.0  12  0.000 0.000 0.000 0.000 44.1

 1  1 2017  0  4 44.1  46 24.6 29.954 0.8 2.0  12  0.000 0.000 0.000 0.000 44.1

...

The data is by hour/minute

I can convert the data/time data to the correct format as well as to save data as a CSV file.

I assume the first record stays and that I connect the CSV name with the weewx name in the CSV config file

Correct?

Thanks

[MRL]

Correction: The sample is from Weather-Display where I only have less than a year. Still working om my 3+ years of Weather-Cat.

[gjr80]

Don't worry about converting to csv, I would much rather have wee_import import from the raw files. Easiest thing is just to attach a few files, not sure if you will need to change the extension though, can't remember how fussy google groups is.

Gary

[MRL]

What about the date/time?

The date/time is in 5 fields? Hows do I set that up in the config file?

I can easily convert to a CSV file with my text editor.

I was planning to code a simple python program to transform the date fields into a single field - no big deal.

How could the import utility deal with the strange date/time format?

My Weather-Cat data is in PDF files. Fortunately I have a program that can convert PDF files to text files. Then a simple python program to turn the output into CSV files.

[gjr80]

On Friday, 21 April 2017 21:35:21 UTC+10, MRL wrote:

What about the date/time?

The date/time is in 5 fields? Hows do I set that up in the config file?

 

If you are going to go down the path of importing from a CSV file your date-time data must be in one field. The field can be an epoch timestamp or some fixed format combination of day, month, year, hour, minute (and second if necessary) as long as it uniquely defines a moment in time. The format must be consistent throughout all rows in a given import. The way you tell wee_import about the format is through Python strptime() format codes. This is covered with examples under the heading raw_datetime_format in the wee_import section of the Utilities Guide.

 

How could the import utility deal with the strange date/time format?

As explained above.

 
I would suggest you make use of the --dry-run command line option until you are happy with what you are importing. Either that or import into a test database/binding.

My Weather-Cat data is in PDF files. Fortunately I have a program that can convert PDF files to text files. Then a simple python program to turn the output into CSV files.

That's too bad, I was looking forward to making wee_import WeatherCat aware. Are the pdfs of the monthly .cat files? If so would it be possible to obtain an extract from some of the pdfs, I would prefer the raw data and am interested in the comments I believe are at the start of each file that detail the format.

Gary

[MRL]

I am ready to do a dry run.

It took a trivial python program to convert:

day month year hour minute temperature   humidity     dewpoint   barometer    windspeed   gustspeed direction  rainlastmin    dailyrain  monthlyrain   yearlyrain  heatindex

 1  1 2017  0  1 44.1  46 24.6 29.953 0.2 0.0 245  0.000 0.000 0.000 0.000 44.1

 1  1 2017  0  2 44.1  46 24.6 29.953 0.0 0.0  60  0.000 0.000 0.000 0.000 44.1

 1  1 2017  0  3 44.1  46 24.6 29.953 0.3 3.0  12  0.000 0.000 0.000 0.000 44.1

into:

datetime,temperature,humidity,dewpoint,barometer,windspeed,gustspeed,direction,rainlastmin,dailyrain,monthlyrain,yearlyrain,heatindex

2017-01-01 00:01:00,44.1,46,24.6,29.953,0.2,0.0,245,0.000,0.000,0.000,0.000,44.1

2017-01-01 00:02:00,44.1,46,24.6,29.953,0.0,0.0,60,0.000,0.000,0.000,0.000,44.1

2017-01-01 00:03:00,44.1,46,24.6,29.953,0.3,3.0,12,0.000,0.000,0.000,0.000,44.1

2017-01-01 00:04:00,44.1,46,24.6,29.954,0.8,2.0,12,0.000,0.000,0.000,0.000,44.1

2017-01-01 00:05:00,44.1,45,24.1,29.955,0.4,0.0,12,0.000,0.000,0.000,0.000,44.1

I will get back to you about the WeatherCat data as soon as the import of the weather-view data works.

[MRL]

I am about to import 400,000 records contained in 9 files. I did dry runs and no errors. This is 9 months of Weather-Display data in one-minute intervals.

I would like to make a copy of the database before I start. I have used Sqlite before. Not sure how to use your schema to do this. Can I just copy the database and rename it? Do I need to stop weewx before copying the database?  I might wait until the current weather stabilizes before stopping weewx.

Thanks.