modeling interval data in pvlib

[Will Holmgren]

I'd like to hear about workflows and challenges that pvlib python users have when modeling interval data in pvlib. We see this most frequently with hourly average data, but the same concepts apply to any interval and can be extended to other statistics.

Please respond with any comments, but here are some questions to spur discussion:

Do you treat the time labels as instantaneous without care for the averaging convention?

Do you treat the time labels as instantaneous but shift them by half an interval (e.g. subtract 30 minutes from hourly ending timestamps)?

Do you upsample your data before using pvlib functions and later resample to regain the original intervals?

Would you like to see functions or arguments to help with upsampling and resampling data in pvlib? What would these interfaces look like?

No wrong answers!

Thanks,

Will

[Silvana Ayala]

I often save data in TMY3 format and then read it with pvlib; one of the issues I've run into is that if the interval is not 1 hour it does not add the minutes; it does however return a column "Time (HH:MM) that can be used to fix the datetime indexes: 

        (tmydata, metadata) = pvlib.iotools.tmy.read_tmy3(filename=tmyfile) 

        tmydata['Datetime'] = pd.to_datetime(tmydata['Date (MM/DD/YYYY)'] + ' ' + tmydata['Time (HH:MM)'])

        tmydata = tmydata.set_index('Datetime').tz_localize(int(metadata['TZ'] * 3600))

Also, on running the modeling comparison between bifacial softwares we discovered that they all do something different. SAM and PVsyst are calculating the time 30 mins after the hour, even for TMY3 data which is supposed to be 30 minues before the hour sun position. If you pass the minute column then SAM calculates the sunposition at exactly that timestamp. We're getting around that by right-averaging for TMY3 data and left-averaging for SAM data and then shifting SAM's results 1 hour forward so that the sunposition and tracker position match.

-----

Another  consideration is handling of the sunrise and sunset hours. There are a lot of hours when there is DNI and DHI values for sunrise hour (let's say sun rises at 7:40 AM), so there are values at 8 AM for the 7 to 8 am period, but if the sun position is calculated at 7:30 like most hours then it's under the horizon and the tracker angle fails; I've noted PVSyst adds the irradiance to the following hour, but what we do is use PVLib to calculate sunrise, and then calculate the sunposition in the midpoint between sunrise and the hour ---> for this example that would be 7:50 AM      

And if it is left-averaged data like SAM style (11 AM --> 11:30 AM Sun position), for sunrise --- the values of irradiance are at 7 AM for the 7-8 AM period, so sun would be calculated at 7:50 AM. 

            if deltastyle == 'TMY3':

                print("Calculating Sun position with a delta of -30 mins. i.e. 12 is 11:30 sunpos")

                sunup= pvlib.irradiance.solarposition.sun_rise_set_transit_spa(datetimetz, lat, lng) 

    

                sunup['minutedelta']= int(interval.seconds/2/60) # default sun angle 30 minutes before timestamp

                # vector update of minutedelta at sunrise

                sunrisemask = sunup.index.hour-1==sunup['sunrise'].dt.hour

                sunup['minutedelta'].mask(sunrisemask,np.floor((60-(sunup['sunrise'].dt.minute))/2),inplace=True)

                # vector update of minutedelta at sunset

                sunsetmask = sunup.index.hour-1==sunup['sunset'].dt.hour

                sunup['minutedelta'].mask(sunsetmask,np.floor(((sunup['sunset'].dt.minute)/2-60)),inplace=True)

                # save corrected timestamp

                sunup['corrected_timestamp'] = sunup.index-pd.to_timedelta(sunup['minutedelta'], unit='m')

        

            elif deltastyle == 'PVSyst' or 'SAM':

                print("Calculating Sun position with a delta of +30 mins. i.e. 12 is 12:30 sunpos")

                sunup= pvlib.irradiance.solarposition.sun_rise_set_transit_spa(datetimetz, lat, lng) 

        

                sunup['minutedelta']= int(interval.seconds/2/60) # default sun angle 30 minutes after timestamp

                # vector update of minutedelta at sunrise

                sunrisemask = sunup.index.hour==sunup['sunrise'].dt.hour

                sunup['minutedelta'].mask(sunrisemask,np.floor((60-(sunup['sunrise'].dt.minute))/2+(sunup['sunrise'].dt.minute)),inplace=True)

                # vector update of minutedelta at sunset

                sunsetmask = sunup.index.hour==sunup['sunset'].dt.hour

                sunup['minutedelta'].mask(sunsetmask,np.floor((sunup['sunset'].dt.minute)/2),inplace=True)

                # save corrected timestamp

                sunup['corrected_timestamp'] = sunup.index+pd.to_timedelta(sunup['minutedelta'], unit='m')

        solpos = pvlib.irradiance.solarposition.get_solarposition(sunup['corrected_timestamp'],lat,lng,elev)   

        solpos['Sun position time'] = solpos.index

        solpos.index = datetimetz

[Anton Driesse]

I think pvlib/pandas/python already offer quite a few ways to deal with different conventions.  The hardest part for me is usually figuring out what other people's conventions and assumptions were/are. 

pvlib-python could perhaps a) propose a recommended convention b) provide some documented conversion functions for common alternatives.

--
You received this message because you are subscribed to the Google Groups "pvlib-python" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pvlib-python...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pvlib-python/18e7c7dd-cab4-4d0f-aa36-2ad08601189bn%40googlegroups.com.

-- 
Photovoltaic Performance Labs Germany
Emmy-Noether-Str. 2
79110 Freiburg
Germany

+49-761-8973-5603 (Office)
+49-174-532-7677 (Mobile)

www.pvperformancelabs.com 

[Anton Driesse]

Since I just came across this again, to the specific questions...

On 2020-09-18 23:23, Will Holmgren wrote:

I'd like to hear about workflows and challenges that pvlib python users have when modeling interval data in pvlib. We see this most frequently with hourly average data, but the same concepts apply to any interval and can be extended to other statistics.

Please respond with any comments, but here are some questions to spur discussion:

Do you treat the time labels as instantaneous without care for the averaging convention?

No, I often need to think about what averaging was done.

Do you treat the time labels as instantaneous but shift them by half an interval (e.g. subtract 30 minutes from hourly ending timestamps)?

Occasionally.  Mainly if I want mid-point sun positions.

Do you upsample your data before using pvlib functions and later resample to regain the original intervals?

No.

Would you like to see functions or arguments to help with upsampling and resampling data in pvlib? What would these interfaces look like?

I sense that something could be offered, but I don't know what it would be.

No wrong answers!

Thanks,

Will

--
You received this message because you are subscribed to the Google Groups "pvlib-python" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pvlib-python...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pvlib-python/94230269-3421-4d50-a49c-79e46c3c815fn%40googlegroups.com.

-- 
PV Performance Labs Germany
Emmy-Noether-Str. 2
79110 Freiburg
Germany

[Michael Deceglie]

I try to pay attention to instantaneous vs average values, but sometimes mistakes slip through the cracks. IMO given how common interval averaged data is in this field, adding some functionality to subsample would be great to avoid adding pandas work arounds external to pvlib. A nice interface might be optional kwags:

label= 'instantaneous', 'left', or 'right'
subsample_interval = timedelta

Default behavior could be preserved with default label='instantaneous'

On Friday, September 18, 2020 at 3:23:05 PM UTC-6 Will Holmgren wrote:

[Eliot Bryant]

Hello,

I concur with Michael, it would be very helpful to have the functionality for integral average calculations to align with SCADA data better.

Averaging convention cannot be discounted for anything complex due to some sensitive calculations that take measured data and compare to clearsky models. I am having that issue right now indeed.

As a result I would like the option to calculate the the clearsky calculations with respect to instantaneous, left or right. Instantaneous could remain the default, but the ability to display modelled data in the same way that SCADA data presents data is potentially helpful.

Potentially useful, I do not know if this is in PVlib already is something similar to the "timeshift" feature in PVSyst, where the average timeshift is estimated from a dataset (vs clearsky model) and one can use this for automatic adjustments. Though this timeshift estimation in PVSyst is not always perfect and I've occasionally seen it make errors. But such a check being part of a workflow, without need for prior knowledge of the datasource calculation method could be useful.

Eliot

--

You received this message because you are subscribed to the Google Groups "pvlib-python" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pvlib-python...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/pvlib-python/d8ce8206-c680-4fa1-82f4-3ddbf0a248b1n%40googlegroups.com.

[Kevin Anderson]

I found myself wanting a visualization of transposition error when failing to account for interval average vs instantaneous data and came up with the below plots comparing naive transposition with one that incorporates a half-interval shift.  Would something like this make a good gallery example?  It's more of a "cautionary tale" than something a user might want to copy/paste as a starting point for their own code, so I'm not sure it fits in the gallery.   Curious what other people think.

  

Kevin

To view this discussion on the web visit https://groups.google.com/d/msgid/pvlib-python/CAK9hGhOn7pPSxG9Bd48Q_TGv2XCXSuUKUK-dt7kyE6rFfx9r2w%40mail.gmail.com.

[cwh...@sandia.gov]

I would like to see this in the gallery for now, and when we get around to re-organizing documentation, perhaps it could be grouped with materials that are more educational rather than code examples.