Questions on HXG data

[jproc...@gmail.com]

Hello ISCCP community!

I've begun using the HXG product and have run across a few questions:

1) Both vrefta and vtauic measure cloud optical depth. If I want total column cloud optical depth regardless of the cloud type (i.e. for ice and water clouds together), which should I use?

2) There are places where there is no cloud cover (i.e. cloud mask indicates clear -- set equal to 0) but optical depth --vtauic-- has a value (i.e. is > 0). How should I interpret these points?

3) How do I see whether it's "night" or not? I see that there is a scene variable but i'm having trouble finding out how to access the is_night binary. There are both masks and flags with many different values and I'm not sure how to make sense of the output. I'm reading the nc file in using R. I want just the day_night flag and none of the other information in the scene variable. 

Thanks so much!

Jon

[Ken Knapp - NOAA Federal]

Hi Jon

Thanks for your questions. I hope this helps you (and others) understand the data better. The answers are provided below.

On Tue, Aug 8, 2017 at 6:45 PM, <jproc...@gmail.com> wrote:

1) Both vrefta and vtauic measure cloud optical depth. If I want total column cloud optical depth regardless of the cloud type (i.e. for ice and water clouds together), which should I use?

VREFTA: long_name = "VIS-retrieved liquid cloud TAU or surface reflectance"

VTAUIC: long_name = "VIS retreived ice cloud tau"

 

Please note that the VREFTA variable actually stores two variables: VIS-retrieved liquid cloud TAU (when cloudy) and surface reflectance (when clear)

So the optical depth is calculated:

if (CLOUD == 1) then

  TAU = TAUTAB(VREFTA) + TAUTAB(VTAUIC)

endif

where TAUTAB is the table that converts the unsigned byte (0-255) from VREFTA to an actual optical depth value.

Also, please note that you convert to optical depth units _then_ add the values. The conversion table TAUTAB is not linear, so you cannot add the index values prior to converting to optical depth units.

2) There are places where there is no cloud cover (i.e. cloud mask indicates clear -- set equal to 0) but optical depth --vtauic-- has a value (i.e. is > 0). How should I interpret these points?

This is an interesting question. Please note that the missing value for VTAUIC = 255b. The file I checked had missing values everywhere that CLOUD=0. If you are seeing a different behavior, please send us the filename of the ISCCP HXG file and we can look into it.

 

3) How do I see whether it's "night" or not? I see that there is a scene variable but i'm having trouble finding out how to access the is_night binary. There are both masks and flags with many different values and I'm not sure how to make sense of the output. I'm reading the nc file in using R. I want just the day_night flag and none of the other information in the scene variable. 

The SCENE variable provides information by encoding bytes. These can be separated by performing integer division and modulo. 

For instance, for snow/ice:

SNOICE_flag=SCENE mod 4 

This produces values of 

0 - no_snowice

1 - partial_seaice_or_landice

2 - full_seaice_or_land_snow

3 - ice_or_snow_margin

GLINT=(SCENE/4) mod 2

0 = no_glint

1 = is_glint

ROUGH_TOPO = (SCENE / 8) mod 2

0 = smooth topography

1 = rough topography

SHORE = (SCENE / 16) mod 2

0 = not shore

1 = is shore

WATER = (SCENE / 32) mod 2

0 = land

1 = water

NIGHT = (SCENE / 64) mod 2

0 = day

1 = night

 VEGCODE = SCENE / 128

This should be a value from 0-18. However, the file I tested didn't have these values.

I hope this helps.

Best regards.

-Ken

--

Ken Knapp, Ph.D.

Meteorologist, Center for Weather and Climate

NOAA National Centers for Environmental Information

formerly NOAA’s National Climatic Data Center

E/NE31

151 Patton Ave
Asheville, NC 28801
828-271-4339 (voice) 828-271-4328 (fax)
http://www.cyclonecenter.org/ where your clicks count!

"Leave this world a little better than you found it."
                             ---Baden-Powell's Last Message (1945)

Disclaimer:
The opinions expressed in this email are those of the author. They do not necessarily reflect the official views or policies of NOAA, Department of Commerce, or the US Government.

[Jonathan Proctor]

Dear Alisa and Ken,

Thank you so much for your response and sincere apologies for my delayed reply. Your answers all make sense. 

With respect to the observations with clear sky (i.e. no clouds) and readings for optical depth (vtauic) I've attached a script (in R) that replicates these findings. The HXG file is the example file that I downloaded from your website a ~2 months ago (ISCCP.HXG.v01r00.GLOBAL.1987.12.30.1500.GPC.10KM.CS0161501645.EQ0.10.nc).

 Essentially I read in vtauic, converted it to actual values using the tautab (I didn't find any raw vtauic values equal to 255 but I'm not sure what 255b is), then made both a histogram and plot of the values of tau where the cloud mask was equal to 0 (indicating clear conditions). There were many more tau observations during cloudy conditions but still some during clear conditions. This is likely an error on my end of how I'm interpreting the data but I thought I'd share in case this was helpful to you in any way. 

I've switched the majority of my analysis over to the ISCCP basic files to speed up processing (processing the HXG data was taking too much computer time) and they are a little more user friendly as well. Thanks so much for making those! The ISCCP products are fantastic. 

cheers,

Jon

On Mon, Aug 14, 2017 at 10:50 AM, Alisa Young - NOAA Federal <alisa...@noaa.gov> wrote:

Hi Ken,

I've already responded to Jon. Next time, I'll make sure to reply all. Hopefully, his issue will be resolved by the combination of responses. I think your answer for day/night is more direct than what I originally provided. 

Thanks,

Alisa

Alisa H. Young, Ph.D.
National Centers for Environmental Information (NCEI) - CO

Center for Weather and Climate

Mail Station:  E/NE3
325 S. Broadway St.
Boulder, CO 80305
(office) 303.497.4327  
(cell) 256.431.5353

The newly formed NCEI merges the National Oceanographic Data Center (NODC), the National Climatic Data Center (NCDC), and the National Geophysical Data Center (NGDC).

We will be updating our social media soon. Follow our current Facebook (NOAA National Climatic Data Center and NOAA National Oceanographic Data Center) and Twitter (@NOAANCDC and @NOAAOceanData) accounts for the latest information.

--
You received this message because you are subscribed to the Google Groups "ISCCP-QA" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isccp-qa+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isccp-qa/CAKxMmD6zxKtOVt2_zQ4cmAoZn0OHHF6bEXvw4wzfgH0Pmv4W8g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Jonathan Proctor]

Here's the file if that helps too ​

 ISCCP.HXG.v01r00.GLOBAL.1987.12.30.1500.GPC.10K...

​

[ken....@noaa.gov]

Hi Jon

I was going through messages and saw that this was never responded. Sorry about that. I hope you're finding the ISCCP Basic useful.

I have looked into the phenomena that you mentioned. It does appear that some of the clear sky pixels in the HXG file do have a cloud optical depth value. I looked at the file that you attached and it appears in a region near the Antarctic. It looks like it is limited to the AVHRR data. We will need to look into this. It does appear to not happen often, which is good.

Thanks for bringing this to our attention.

And please keep us informed how you use ISCCP. IT helps justify further improvements and continues support. [best done through the registration page]

https://docs.google.com/a/noaa.gov/forms/d/e/1FAIpQLSfuUKCGQjaDYfXPAccelzQnmPfyS7I1dbAE2TmYsfb1u0ePzA/viewform

Best regards-

-Ken

[spence...@gmail.com]

Ken,

I would like to confirm the formula's for converting the 'scene' variable within the HXG product is correct. When applying the formulas provided within this forum there seems something is off. The file I am using is 'ISCCP.HXG.v01r00.GLOBAL.2014.07.01.0000.GPC.10KM.CS1132411354.EQ0.10.nc'. The scene variable of interest are 'NIGHT' Which is defined as:

NIGHT = (SCENE / 64) mod 2

%         0 = day
%         1 = night

It seems though the day and night values are inverted depending whether the observation is over land or ocean and s screen shot of this is shown below. Can you clarify this please?

P.S. As you stated in the previous post there is no VEGCODE even for this file. Is this always the case and why?

[Ken Knapp - NOAA Federal]

Hi 

I have verified that the equation works for night for the file you specified.

It appears that the plot you made is somehow combining the is_land flag and the is_night flag. Can you recheck your code or post the code that made the plot?

As to VEGCODE .... yep. I just went through the code. For HXG, the VEGCODE will always be zero. That's a bug. The good news is that (As best as I can tell) the vegcode is more of a convenience value. It is never used in calculations (at least, not the value that was written to the HXG file, which is likely why this wasn't noticed). It should be present in other files that have different resolutions, but the HXG file is missing VEGCODE. Sorry about that.

Also, please register your usage of ISCCP by clicking here. The more information you provide on how you use ISCCP, the better we will be able to justify continued support, updates and improvements! Also, if you continue to use ISCCP in new ways, you can refill out the form.

Hope this helps.

-Ken

[spence...@gmail.com]

Ken,

Below is MATLAB code that created the image:

%Color    RGB Triplet
yellow    = [1,1,0];
black    = [0,0,0];
cmp = [yellow;black];

len = 2; %two descriptions: 0='day';1='night'};
SCENE = h5read([input_dir,files(file_inc).name],'/scene'); %Scene identification as type: int16

night_int16 = mod(SCENE/64,2);
figure;
    imagesc(night_int16);colorbar;
    handles.ColorbarTickLabels = {'day','night'}';
    caxis([-0.5 len-0.5]) %Use this to acknowledge bow tie deleted pixels
    colorbar('Ticks',0:len,'TickLabels',handles.ColorbarTickLabels,'fontsize',12,'fontweight','bold');
    colormap(gcf,cmp);

On Tuesday, August 8, 2017 at 6:45:15 PM UTC-4, jproc...@gmail.com wrote: