Question about quality index

[Lechémia Théo]

Hello,

I'm importing data in a procedure and I'm a bit confused about quality index.

I have set quality index for an observed property in a procedure: the data must be between 0 and 25.

After inserting, for data in the good range, the value 200 is set (QC1 - value aggregated with < 0.6), and for the others (which are negative of greater than 25), I have the the quality index "QC0 - raw (the format is correct)". 

Looking at the code, I saw that these values are configurable ( https://github.com/istSOS/istsos2/blob/master/services/default.cfg#L8-L12

And here, that there are two levels of verification (https://github.com/istSOS/istsos2/blob/master/istsoslib/responders/IOresponse.py#L504)

Tell me if I understand well :

The first check is for the "observed property level", which set the "correct_qi" value.

The second check is for the "procedure level and set the "stat_qi"

 value ?

So if the value never match with those two checks, the value don't change and it remain set at the value of "default_qi"

So if I want to my "wrong" values to be set as 0 (erroneous), I have to set the defaut "default_qi" to 0. And set the two other value to 909 (HQ - correct).

Is that the good way of doing things ?

many thanks,

Théo

[Lechémia Théo]

Ok it's better to read the doc before asking a question ... 

http://istsos.org/en/latest/doc/ws_datavalidation.html?highlight=quality

I just don't know why I have different value for the quality index description in my database .. 

I'll change my values according to the doc .  

Sorry for the flood ..

[Massimiliano Cannata]

Hi Theo,

1st check is for observed property (is the value meaningful, e.g.: rain is not negative)

2nd check that observed property in that location is meaningful? E.g. In desert area temperature are ok between 0 and 50, in the Alps at 2000 between -30 and 40

The default value is what is assigned initially, then if check 1 is passed it get new value,. Then if test 2 is passed get new qi value, and so on with other self made tests...

These two test are implemented because are the only test you can automatically perform on a single observation for real time consume of data (at least flag probably wrong data)...

Then you need more observation for other tests (last hours to see variability for example, or other stations to see space variability)

Hope this help

Maxi

--
You received this message because you are subscribed to the Google Groups "istSOS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to istsos+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/istsos/ad901385-c6b6-4f90-b57d-e9f5b8bb9493n%40googlegroups.com.

[Lechémia Théo]

Ok, thanks for your response.

So if the defaut value is "100" (as provided by default), my "wrong" values will never be tags as "erroneous" even if they don't match the two checks. And if I want to put "0" as de default value, all my procedure or observed properties without quality check will be tags as "erroneous"...

How do you usually proceed to tag "wrong data" ? 

Would it be more logical to proceed the two checks is a "falsy" way : I mean if the checks don't pass -> set the "erroneous" value (could be set in config).

Théo

[Massimiliano Cannata]

Hi Theo,

the standard way to proceed is:

- Any data I get are initially classified as raw data (100)

- A number of tests are run to verify increasingly strict checks, if the test is assed the data get the new qi code (so I know any data until when it goes in the quality check process)

- After automatic check there are manual check, they decide if a data is set to invalid, if it is exceptional value but valid, if you need to modify and mark it a s manually adjusted or what so ever...

So, in your process you can alway set an erroneous data and assign it to the observation.

For any process you should take only data that pass at least the first test.... (which means have a meaning, or if you take those that didn't pass the test they are most probably errors)

Maxi

To view this discussion on the web visit https://groups.google.com/d/msgid/istsos/d8ac3281-d684-41a2-8886-f75e2f34c6f5n%40googlegroups.com.

--

Massimiliano Cannata

Professore SUPSI in ingegneria Geomatica

Responsabile settore Geomatica

Istituto scienze della Terra

Dipartimento ambiente costruzione e design

Scuola universitaria professionale della Svizzera italiana

Campus Mendrisio, Via Flora Ruchat-Roncati 15

CH – 6850 Mendrisio

Tel. +41 (0)58 666 62 14

Fax +41 (0)58 666 62 09

massimilia...@supsi.ch

www.supsi.ch/ist

[Camille PNE]

Thanks for the replies.

If I understand it means that we can't distinguish : 

- Data that have QC to 100 because they didnt passed the tests (= error data)

- Data that have QC to 100 because there is no checks defined on its procedure nor observed property (good data without checks defined)
?

Thanks.

Camille Monchicourt
Ecrins national Park

-----------

[Massimiliano Cannata]

Dear Camille,

yes and no :-)

100 means it is raw data

200 means it passed the meaningfulness test

210 means it passed the station meaningful test

220 means it passed the time consistency test

and so on....

But yes, if it is 100 t means or didn't undergo any test or it is not meaningful...

;-)

To view this discussion on the web visit https://groups.google.com/d/msgid/istsos/0d799666-3d3b-4b1e-85c5-0ecfbe647c05n%40googlegroups.com.