evaluator - timing, classification errors bug

[David Garcia]

Jeremy Mange wrote:
> > Mr. Garcia,
> >
> > I am working on the ADAPT-Lite problem for the Diagnostic Competition,
> > and I had a quick question about timing that I was hoping you might be
> > able to answer.
> >
> > As a preliminary test, I was attempting to isolate and correctly
> > diagnose a simple error in a known scenario. I chose the
> > "Exp_1187_pb_ADAPT-Lite" scenario, since it involves a relatively simple
> > fault that can be identified with a boolean sensor (ESH244A).
> >
> > My question was about how the timing works -- in the scenario file, the
> > faultInjection line has the time 131600. In the output from the
> > Evaluator after a run, however, it is shown as having been injected at
> > time 131638 (this varied a bit across runs).

The number shown is the one found in the results file for the scenario
(adjusted relative to the scenario start). This is the real time
recorded by the ScenarioRecorder during a run, and so minor delays are
normal.

> > According to the scenario
> > file, the first sensor reading after that time is at 131720, which seems
> > like the very earliest the fault could conceivably be identified by a
> > DA. My simple test did identify it as such, but this was recorded as a
> > classification error*. It seems that the Evaluator only considers a
> > classification correct if identifies the exact correct FaultTime, but
> > assuming I understand the framework correctly, it doesn't seem like
> > there is any possibility of this happening, since the earliest a fault
> > could be recognized is the first sensor reading after it is injected.
> >
> > Am I misunderstanding something, or is there indeed little chance of
> > identifying the FaultTime exactly correctly? I would appreciate any
> > information you would be able and willing to provide.
> >

The time used by the evaluator for timing metrics isn't the FaultTime
parameter, it's the timestamp on the diagnosis message itself. The
FaultTime parameter is only used by the Oracle to determine if a fault
should cause an abort, in other words, it's the time you *think* the
fault was first injected, not the time you detected its presence.

> > Thanks,
> > Jeremy Mange
> >
> > * I suspect this is an Evaluator bug -- this was actually recorded as 2
> > classification errors. In testing some things, I found that if I edited
> > the results file to change either the component name or error name to
> > something not in the system catalog, the number of classification errors
> > dropped back to 1. But when it is the correct component and the correct
> > fault, but the wrong FaultTime parameter, it is counted as 2
> > classification errors.
> >

You're right, this is a bug. The evaluator should ignore parameters
altogether when calculating classification errors. Your example should
give 0 classification errors, the invalid fault example should give 1
(an invalid fault is treated as an empty fault), and a valid but
incorrect fault should give 2. This will be fixed in the next release,
probably at the end of this week. For now, you can omit fault
parameters
when sending diagnoses (just make sure to add them in when querying
the
Oracle).

David G