Cyclone tutorial - Issue on attributes

[Nicolas Steinmetz]

Hi all,

Starting with Warp10 for a few days now and quiet happy so far with what I achieved !

Follwoing the cyclone tutrial on Warp10 10.2.6 on my local machine I met an issue around attributes.

Dataset is there and we can see the "name" attribute : https://warp10.io/assets/files/ibtracs-2010-2015.gts

But for step 5 of the Category 1 tutorial "Names of Tropical Cyclones in 2015", the code did not work for me (https://warp10.io/content/04_Tutorials/05_Cyclones_on_Sandbox/10_Category_1)

If I do

[

  $read_token
  'gov.noaa.storm.wind'
  {}
]

FIND

The JSON output is always empty for the "a" field.

Like :

[

{"c":"gov.noaa.storm.wind","l":{"serial":"2012189N10262",".app":"io.warp10.bootstrap"},"a":{},"la":0,"v":[]},
...
]

Same if I do a FETCH.

[{"c":"gov.noaa.storm.wind","l":{"serial":"2015251N31298",".app":"io.warp10.bootstrap"},"a":{},"la":0,"v":[[1441951200000000,35.69999998435378,-60.30000007711351,40],[1441929600000000,34.299999992363155,-60.70000000298023,40],[1441908000000000,33.19999996572733,-60.90000000782311,40],[1441886400000000,32.19999998342246,-60.90000000782311,40],[1441864800000000,31.599999968893826,-60.90000000782311,45],[1441843200000000,31.19999995920807,-60.80000004731119,45],[1441821600000000,30.89999999385327,-60.80000004731119,45],[1441800000000000,30.799999991431832,-61.000000052154064,40],[1441778400000000,30.799999991431832,-61.20000005699694,35],[1441756800000000,30.799999991431832,-61.40000006183982,35],[1441735200000000,30.799999991431832,-61.600000066682696,30],[1441713600000000,30.89999999385327,-61.9000000320375,25],[1441692000000000,31.09999999869615,-62.20000008121133,20]]}

And if I do something like:

[ 

  $read_token

  'gov.noaa.storm.wind'

  {}

  '2016-01-01T00:00:00.000z'  // End date

  '2010-01-01T00:00:00.000z'  // Start date

]

FETCH // It is possible to avoid fetching data for the whole the year but we keep it simple for now

0 GET

ATTRIBUTES

It's still empty.

I deleted the gts and tried to reupload it from curl or from studio and it's always the same.

Any idea on how to fix this?

Thanks,

Nicolas

[Fabien Tencé]

Hi Nicolas,

What you experience is because your Warp 10 instance does not use the same configuration as the Sandbox one.

Namely, the ingress.parse.attributes = true, in 10-ingress.conf, is not set in your configuration. It enables the /update endpoint to parse attributes in the GTS input format. So currently, they are ignored on your instance.

By default, this option is disabled because it can degrade the ingestion performance at lot if attributes are often set, even if there are always the same, and GTSs are mixed. This is not the case in the .gts file for the tutorial but using this functionality should be done with care.

So set this configuration, restart your instance and reupload the data and you should be good.

For the reference, if you don't want to enable this option, you can set the attributes via the /meta endpoint, see https://www.warp10.io/content/03_Documentation/03_Interacting_with_Warp_10/08_Meta.

[Nicolas Steinmetz]

Hi Fabien,

Sorry for the late reply, I didn't see your answer in my emails :-/

On Friday, June 19, 2020 at 11:12:05 AM UTC+2, Fabien Tencé wrote:

What you experience is because your Warp 10 instance does not use the same configuration as the Sandbox one.

Namely, the ingress.parse.attributes = true, in 10-ingress.conf, is not set in your configuration. It enables the /update endpoint to parse attributes in the GTS input format. So currently, they are ignored on your instance.

By default, this option is disabled because it can degrade the ingestion performance at lot if attributes are often set, even if there are always the same, and GTSs are mixed. This is not the case in the .gts file for the tutorial but using this functionality should be done with care.

So set this configuration, restart your instance and reupload the data and you should be good.

For the reference, if you don't want to enable this option, you can set the attributes via the /meta endpoint, see https://www.warp10.io/content/03_Documentation/03_Interacting_with_Warp_10/08_Meta.

 
Thanks, it works as expected. I can go further now !
Have a nice day,
Nicolas