Creating Custom Color Tables in Satpy

[Kziolkowski]

Hello,

I have been attempting to create my own color tables in Satpy and I am running into some issues that I am not sure on what exactly is happening.

I have wrote a script to convert the RGB values of the color table to a .npy file. I use the color table editor in SIFT to get the colors for the custom color tables to look how I want them to, and then take those values and save them and run them through the conversion script. I then point to the color table in my enhancement generic.yaml file.

Now here is where I begin to run into problems. When I run the IR custom color table in Satpy it looks nothing at all as it does in SIFT. The colors are not matching and there are some colors showing up that I don't even have in my RGB palette (there is a green that is showing up). I have attached two images as an example of the IR color table in SIFT and the results when I run my Satpy script. I have also attached the CSV file that has my list of colors and associated temperatures (not, the temperatures are in degrees C in the csv file but I convert them to K in my conversion script).

So I am not sure where I am going wrong. I don't know if I need to have a higher resolution as far as colors go in my table that I feed Satpy? Also, would it be possible to have Satpy ingest the output color table .json file from SIFT?

Thanks,

Kyle Ziolkowski

[David Hoese]

Hi Kyle,

What does your YAML enhancement configuration look like?

If you turn on debug log messages are you sure the right enhancement is
being applied?

from satpy.utils import debug_on; debug_on()

Another theory, what if you flip the color table? Right now you have
cold to hot temperatures, what if you made it hot to cold temperatures.
Does Satpy show anything different?

Dave

> satpy_ir_custom_colortable.pngsift_ir_custom_colortable.png
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pytroll/017594bf-a778-4206-955e-a8a7b0390e04n%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/017594bf-a778-4206-955e-a8a7b0390e04n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Kziolkowski]

Hey Dave,

I do have my debug on. I have attached a screenshot of the output. Also here is my enhancements YAML:

ir_colortable1:

standard_name: ir_colortable1

operations:

- name: colorize

method: *colorizefun

kwargs:

palettes:

- {filename: /home/ziolkowskik/satdev/ConfigDir/colortables/ir_colortable1.npy}

here is my composites YAML:

  ir_colortable1:
    compositor: !!python/name:satpy.composites.GenericCompositor
    prerequisites:
      - name: C13
    standard_name: ir_colortable1

And when I flip the color table I just get a purple image, which is the color at -75.

Thanks,

Kyle

[David Hoese]

Did you flip only the colors? I was thinking flip all the rows, so first
row is last and last row is first, but the value -> color pairing is
still the same.

Looks like the colorize function is being used properly from your log.

Dave

On 2/23/21 7:08 AM, Kziolkowski wrote:
> Hey Dave,
>
> I do have my debug on. I have attached a screenshot of the output. Also
> here is my enhancements YAML:
>
> ir_colortable1:
> standard_name: ir_colortable1
> operations:
> - name: colorize
> method: *colorizefun
> kwargs:
> palettes:
> - {filename:
> /home/ziolkowskik/satdev/ConfigDir/colortables/ir_colortable1.npy}
>
> here is my composites YAML:
>   ir_colortable1:
>     compositor: !!python/name:satpy.composites.GenericCompositor
>     prerequisites:
>       - name: C13
>     standard_name: ir_colortable1
>
>
> And when I flip the color table I just get a purple image, which is the
> color at -75.
>
>
> Thanks,
>
> Kyle
>
>

> <https://groups.google.com/d/msgid/pytroll/017594bf-a778-4206-955e-a8a7b0390e04n%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/pytroll/017594bf-a778-4206-955e-a8a7b0390e04n%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pytroll/dd82ddbf-bc1a-4a19-a409-62159e322b75n%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/dd82ddbf-bc1a-4a19-a409-62159e322b75n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[David Hoese]

What is the shape of the array in your `.npy` file?

[Kziolkowski]

They shape of the array is (9, 5).

And yes I flipped them as so and just got a purple image:

40,0,0,0,0
-30.999,255,255,255,0
-31,81,183,232,255
-43,11,26,127,255
-50,251,248,83,255
-60,229,55,35,255
-62,7,1,1,255
-64,245,242,246,255
-75,174,54,183,255

Kyle

[David Hoese]

I have other ideas that you could try, but I'm not sure any of them make
sense based on your results. I'm very confused how you get an image that
is pretty similar to what you expect. Meaning, you don't get a solid
color image and you don't get an error. Being a core developer on SIFT
and Satpy, it makes me wonder which one is actually wrong here.

If you probe the data in SIFT, do you get the expected values for the
colors you are looking at?

My other guesses:

1. Is your .npy file actually in the range of 0 to 255? Or is it
floating point? What data type is it saved as?
2. What if your "Values" were specified as 0 to 1 and then you added a
"min_value" and "max_value" to your palettes definition in the YAML?
3. What if instead of an npy file you put the color table directly into
the YAML as a list.

I'm basically re-reading this and wondering if I'm missing something
about your setup:
https://github.com/pytroll/satpy/blob/21ebf26c2181c91a27e7b88d1287b71b5be3f0eb/satpy/enhancements/__init__.py#L226-L293

But everything you're doing seems like it should be correct.

Dave

> <https://groups.google.com/d/msgid/pytroll/dd82ddbf-bc1a-4a19-a409-62159e322b75n%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/pytroll/dd82ddbf-bc1a-4a19-a409-62159e322b75n%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
> >>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pytroll/2d17ced2-81ed-4bb7-a5b1-0406e7d906ben%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/2d17ced2-81ed-4bb7-a5b1-0406e7d906ben%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Kziolkowski]

I do get the expected values in SIFT when I restrict the color table range from -30 to -75 (so it matches my color table file).

Also my .npy file is in VRGBA format. So I set the temperatures in first column of the array followed by the RGBA values.

If I just used an RGBA file instead of specifying the temperature would that be the same as setting the values between 0-1?

I am also confused as to why there is green showing up in the image I create in Satpy when I am not even specifying that as a color. I find that behavior quite odd.

Kyle

[David Hoese]

> If I just used an RGBA file instead of specifying the temperature would
> that be the same as setting the values between 0-1?

It will be the same if your colors were equally spaces I think (it takes
row_idx/num_rows to produce the 0-1 values). Otherwise you'd have to do
V / (40 - -75), right?

For the green, I'm not sure which specific green you are talking about
but I definitely see green colors on the edge of the "blobs" in your
SIFT image.

I'm not sure I'll have time to look more deeply into this, but could you
send the `.npy` file you're using?

Dave

> <https://groups.google.com/d/msgid/pytroll/2d17ced2-81ed-4bb7-a5b1-0406e7d906ben%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/pytroll/2d17ced2-81ed-4bb7-a5b1-0406e7d906ben%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pytroll/cd713f25-379f-476a-860d-fb098a0f1659n%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/cd713f25-379f-476a-860d-fb098a0f1659n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Kziolkowski]

I keep getting an error message when I attach my .npy file and try to post the message. But this is what the array looks like when I load it.

array([[198.15 , 174.   ,  54.   , 183.   , 255.   ],
       [209.15 , 245.   , 242.   , 246.   , 255.   ],
       [211.15 ,   7.   ,   1.   ,   1.   , 255.   ],
       [213.15 , 229.   ,  55.   ,  35.   , 255.   ],
       [223.15 , 251.   , 248.   ,  83.   , 255.   ],
       [230.15 ,  11.   ,  26.   , 127.   , 255.   ],
       [242.15 ,  81.   , 183.   , 232.   , 255.   ],
       [242.151, 255.   , 255.   , 255.   ,   0.   ],
       [313.15 ,   0.   ,   0.   ,   0.   ,   0.   ]])

Kyle

[David Hoese]

What about:

from satpy.enhancements import create_colormap
cmap = create_colormap({'filename':
'/home/ziolkowskik/satdev/ConfigDir/colortables/ir_colortable1.npy'})

print(cmap.values)
print(cmap.colors)

Dave

> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pytroll/27c905ac-bbb0-4141-aa59-1c0348723f78n%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/27c905ac-bbb0-4141-aa59-1c0348723f78n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Yufei]

Hi Kyle, Dave,

I can reproduce Kyle's result in satpy, I think the colour map and satpy colorization are OK. The reason there is green in the image is because colourization function of satpy interpolation the colors between two adjacent custom define values/colors. I don't know how SIFT colorization works, however, the legend in Kyle's image is a color palette  with only few colors, not sure how SIFT interpolate between those colors.

Yufei

[David Hoese]

SIFT should also interpolate between the colors, but the colorbar shown
in Kyle's screenshot of SIFT is not from SIFT. Kyle how did you generate
that colorbar?

Dave

> <https://groups.google.com/d/msgid/pytroll/27c905ac-bbb0-4141-aa59-1c0348723f78n%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/pytroll/27c905ac-bbb0-4141-aa59-1c0348723f78n%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pytroll/cb868f8e-3898-47ec-9808-074a3bb09d25n%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/cb868f8e-3898-47ec-9808-074a3bb09d25n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Kyle Ziolkowski]

I cloned the PuBu (I think this was the one) and then I added control points and adjusted the colours on each of the control points and saved it. I only have 6-8 control points but still get the results I want in SIFT. So not sure if SatPy handles the interpolation differently.

Yufei, you can’t re-produce the SIFT colour palette either (the one I posted)? You get similar results?

Also are the colour palettes in SIFT produced by PyTroll? And if so, wouldn’t SatPy handle the colour palettes the same as SIFT?



Kyle


> On Feb 24, 2021, at 7:46 PM, David Hoese <dho...@gmail.com> wrote:
>
> SIFT should also interpolate between the colors, but the colorbar shown in Kyle's screenshot of SIFT is not from SIFT. Kyle how did you generate that colorbar?

> To unsubscribe from this group and stop receiving emails from it, send an email to pytroll+u...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/pytroll/39377078-f6e6-fbd8-95ed-430f22e4fed0%40gmail.com.

[Yufei]

Hi Kyle,

I only played the colortable you provided in satpy. I am not a SIFT use, so I can not help you on that one.

Yufei

[David Hoese]

SIFT's colormaps are either borrowed from VisPy or created custom from
the AWIPS visualization client that the USA National Weather Service
uses. Can you take a screenshot of your colorbar when you use it in SIFT
(in the Layer Details)?

I can't think of a way/reason why SIFT and Satpy would do the
interpolation any differently.

Dave

[Kyle Ziolkowski]

I have attached a few screenshots from SIFT and the output from satpy.enhancements create_colormap function. The sift windows show the colourbar I created in SIFT and the colour table editor window has the control points I used to create the colour table. The color table I cloned and edited is the PuGr colour table. So even with a few control points I achieve the results I want in SIFT but can’t carry this over to sappy for some reason. 

Kyle

You received this message because you are subscribed to a topic in the Google Groups "pytroll" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pytroll/nOkDdshHSqE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pytroll+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/pytroll/97b327b8-db44-5c7a-6eef-9ae5b72ba494%40gmail.com.

[Yufei]

Hi Kyle,

I think the problem is the when you try to reproduce the color table from SIFT to satpy, you did not take enough control points, so that satpy interpolate the colors between two control points. You may add more control points evenly between two key control points you have picked then applied in satpy.

Another way I can think of to verify is to export an unmodified color table from SIFT and the used it in satpy then compare the outputs. So that you can confirm if the 'transfer' of color table works in both system.

hope this helps

Yufei

[David Hoese]

Yufei,

Have you seen issues like this in Satpy before? That is, too few control
points? I don't see why that should matter as the interpolation should
still be linear.

Dave

On 2/25/21 11:44 AM, Yufei wrote:
> Hi Kyle,
>
> I think the problem is the when you try to reproduce the color table
> from SIFT to satpy, you did not take enough control points, so that
> satpy interpolate the colors between two control points. You may add
> more control points evenly between two key control points you have
> picked then applied in satpy.
> Another way I can think of to verify is to export an unmodified color
> table from SIFT and the used it in satpy then compare the outputs. So
> that you can confirm if the 'transfer' of color table works in both system.
>
> hope this helps
>
> Yufei
> On Thursday, February 25, 2021 at 9:49:42 AM UTC-5 Kziolkowski wrote:
>
> I have attached a few screenshots from SIFT and the output from
> satpy.enhancements create_colormap function. The sift windows show
> the colourbar I created in SIFT and the colour table editor window
> has the control points I used to create the colour table. The color
> table I cloned and edited is the PuGr colour table. So even with a
> few control points I achieve the results I want in SIFT but can’t
> carry this over to sappy for some reason.
>
>
> Kyle
>
>
>

>>>>> <http://googlegroups.com>
>>>>>     > <mailto:pytroll+u...@googlegroups.com
>>>>> <http://googlegroups.com>>.

>>>>>     > To view this discussion on the web, visit
>>>>>     >
>>>>> https://groups.google.com/d/msgid/pytroll/27c905ac-bbb0-4141-aa59-1c0348723f78n%40googlegroups.com
>>>>> <https://groups.google.com/d/msgid/pytroll/27c905ac-bbb0-4141-aa59-1c0348723f78n%40googlegroups.com>
>>>>>    <https://groups.google.com/d/msgid/pytroll/27c905ac-bbb0-4141-aa59-1c0348723f78n%40googlegroups.com
>>>>> <https://groups.google.com/d/msgid/pytroll/27c905ac-bbb0-4141-aa59-1c0348723f78n%40googlegroups.com>>
>>>>>     >
>>>>>    <https://groups.google.com/d/msgid/pytroll/27c905ac-bbb0-4141-aa59-1c0348723f78n%40googlegroups.com?utm_medium=email&utm_source=footer
>>>>> <https://groups.google.com/d/msgid/pytroll/27c905ac-bbb0-4141-aa59-1c0348723f78n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>    <https://groups.google.com/d/msgid/pytroll/27c905ac-bbb0-4141-aa59-1c0348723f78n%40googlegroups.com?utm_medium=email&utm_source=footer
>>>>> <https://groups.google.com/d/msgid/pytroll/27c905ac-bbb0-4141-aa59-1c0348723f78n%40googlegroups.com?utm_medium=email&utm_source=footer>>>.
>>>>> --
>>>>> You received this message because you are subscribed to the
>>>>> Google Groups "pytroll" group.
>>>>> To unsubscribe from this group and stop receiving emails from
>>>>> it, send an email to pytroll+u...@googlegroups.com
>>>>> <mailto:pytroll+u...@googlegroups.com>.
>>>>> To view this discussion on the web, visit
>>>>> https://groups.google.com/d/msgid/pytroll/cb868f8e-3898-47ec-9808-074a3bb09d25n%40googlegroups.com
>>>>> <https://groups.google.com/d/msgid/pytroll/cb868f8e-3898-47ec-9808-074a3bb09d25n%40googlegroups.com>

>>>>> <https://groups.google.com/d/msgid/pytroll/cb868f8e-3898-47ec-9808-074a3bb09d25n%40googlegroups.com?utm_medium=email&utm_source=footer

>>>>> <https://groups.google.com/d/msgid/pytroll/cb868f8e-3898-47ec-9808-074a3bb09d25n%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>>>>
>>>> --
>>>> You received this message because you are subscribed to the
>>>> Google Groups "pytroll" group.
>>>> To unsubscribe from this group and stop receiving emails from
>>>> it, send an email to pytroll+u...@googlegroups.com.
>>>> To view this discussion on the web, visit
>>>> https://groups.google.com/d/msgid/pytroll/39377078-f6e6-fbd8-95ed-430f22e4fed0%40gmail.com

>>>> <https://groups.google.com/d/msgid/pytroll/39377078-f6e6-fbd8-95ed-430f22e4fed0%40gmail.com>.
>>
>> --

>> You received this message because you are subscribed to a topic in
>> the Google Groups "pytroll" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/pytroll/nOkDdshHSqE/unsubscribe

>> <https://groups.google.com/d/topic/pytroll/nOkDdshHSqE/unsubscribe>.

>> To unsubscribe from this group and all its topics, send an email
>> to pytroll+u...@googlegroups.com.
>> To view this discussion on the web, visit
>> https://groups.google.com/d/msgid/pytroll/97b327b8-db44-5c7a-6eef-9ae5b72ba494%40gmail.com

>> <https://groups.google.com/d/msgid/pytroll/97b327b8-db44-5c7a-6eef-9ae5b72ba494%40gmail.com>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "pytroll" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pytroll+u...@googlegroups.com
> <mailto:pytroll+u...@googlegroups.com>.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pytroll/fc879e97-1372-47a9-b0ea-62793e1945f0n%40googlegroups.com
> <https://groups.google.com/d/msgid/pytroll/fc879e97-1372-47a9-b0ea-62793e1945f0n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Yufei]

Hi Dave,

I don't think it is an issue. I think Kyle's case are special, he is trying to reproduce the color map from SIFT. For example I think the color between yellow and blue in the SIFT color map is certain yellowish color; Kyle only picked yellow and blue as control points from the SIFT color table and passed to satpy, satpy then linearly interpolated the colors between yellow and blue which generate some greenish color which is different from the colors in the SIFT color table. If Kyle picked few more colors from the original SIFT color  table as control points, then satpy will linearly interpolate between those colors, so that it won't generate a color map as close as the SIFT one, which will not have the green colors in the middle.

Another example in satpy's colormaps, you may compare the yellow(0.5) to blue(1.0) part of 'rdylbu' vs 'spectral',  the yellow(0.5) and blue(1.0) are same, but the colors between defined by the control points between them are different.

https://github.com/pytroll/trollimage/blob/master/doc/_static/rdylbu.png

rdylbu = Colormap((0.0, (165 / 255.0, 0 / 255.0, 38 / 255.0)),
(0.1, (215 / 255.0, 48 / 255.0, 39 / 255.0)),
(0.2, (244 / 255.0, 109 / 255.0, 67 / 255.0)),
(0.3, (253 / 255.0, 174 / 255.0, 97 / 255.0)),
(0.4, (254 / 255.0, 224 / 255.0, 144 / 255.0)),
(0.5, (255 / 255.0, 255 / 255.0, 191 / 255.0)),
(0.6, (224 / 255.0, 243 / 255.0, 248 / 255.0)),
(0.7, (171 / 255.0, 217 / 255.0, 233 / 255.0)),
(0.8, (116 / 255.0, 173 / 255.0, 209 / 255.0)),
(0.9, (69 / 255.0, 117 / 255.0, 180 / 255.0)),

(1.0, (49 / 255.0, 54 / 255.0, 149 / 255.0)))

https://github.com/pytroll/trollimage/blob/master/doc/_static/spectral.png

spectral = Colormap((0.0, (158 / 255.0, 1 / 255.0, 66 / 255.0)),
(0.1, (213 / 255.0, 62 / 255.0, 79 / 255.0)),
(0.2, (244 / 255.0, 109 / 255.0, 67 / 255.0)),
(0.3, (253 / 255.0, 174 / 255.0, 97 / 255.0)),
(0.4, (254 / 255.0, 224 / 255.0, 139 / 255.0)),
(0.5, (255 / 255.0, 255 / 255.0, 191 / 255.0)),
(0.6, (230 / 255.0, 245 / 255.0, 152 / 255.0)),
(0.7, (171 / 255.0, 221 / 255.0, 164 / 255.0)),
(0.8, (102 / 255.0, 194 / 255.0, 165 / 255.0)),
(0.9, (50 / 255.0, 136 / 255.0, 189 / 255.0)),

(1.0, (94 / 255.0, 79 / 255.0, 162 / 255.0)))

Hope I make it clear.

Yufei