Validity of audio tests
Mark DeBellis
meaningful variable is a property of an extended passage, not a short
snippet. Then a subject's failure to accurately distinguish or
re-identify may be due to an inability to retain the property in
memory.
I am thinking by way of contrast to visual examples. I just made two
prints of a photograph using different settings on my printer. I am
looking at the face of the subject and I can see that the contrast is
higher in one than in the other. That is a Gestalt property of a
meaningful chunk of the picture, not a property of a few pixels (cf.
notes). The difference with the musical case is that I can compare
the contrast of the two pictures directly, whereas in music no
immediate comparison is possible. At best I have to keep the property
in memory, and maybe the relevant variable is something not easily
retained.
Is the existing empirical confirmation for tests recommended in audio
based largely on visual data? If so, perhaps they rely on factors
that apply to the visual domain (i.e., possibility of immediate
comparison) but do not transfer easily to audio.
Or are there cases in the scientific literature in which the relevant
kinds of tests have been found valid to measure the detection of
Gestalt properties of aural, temporally extended signals?
Gary Eickmeier
> I have the following worry about audio listening tests. Suppose the
> meaningful variable is a property of an extended passage, not a short
> snippet. Then a subject's failure to accurately distinguish or
> re-identify may be due to an inability to retain the property in
> memory.
Nobody uses short snippets to do audio listening tests or comparisons.
We usually use a complete song or passage of an extended work.
>
> I am thinking by way of contrast to visual examples. I just made two
> prints of a photograph using different settings on my printer. I am
> looking at the face of the subject and I can see that the contrast is
> higher in one than in the other. That is a Gestalt property of a
> meaningful chunk of the picture, not a property of a few pixels (cf.
> notes). The difference with the musical case is that I can compare
> the contrast of the two pictures directly, whereas in music no
> immediate comparison is possible. At best I have to keep the property
> in memory, and maybe the relevant variable is something not easily
> retained.
Not true. You can perform rapid switching as often and as many as you
want. You level match the two sources so that the only difference you
hear is the sound quality differences between the two. Rapid switching
is the audio equivalent of a direct comparison.
Your complaint would apply only to long term comparisons, where you
listen first to a complete song on one source, then switch to the other
source and listen all over again.
Gary Eickmeier
nab...@hotmail.com
> I have the following worry about audio listening tests. Suppose the
> meaningful variable is a property of an extended passage, not a short
> snippet. Then a subject's failure to accurately distinguish or
> re-identify may be due to an inability to retain the property in
> memory.
Then it's not audible. End of discussion.
bob
Ban
I second Garys comment. Some additions:
There are ABX switchboxes available, I made one myself with relays to switch
between two speaker cables simultaneously on both ends. You can switch any
time as much as you like. The music or test signal (I use mostly pink noise)
is completely at your disposition, use what you feel gives the best results.
The faster and cleaner the switching action, the more subtle differences can
be discovered. If you are not into electronics it is better to get a ready
made box, to avoid any difference between the channels, like the sound of
the relais being different making/breaking or so.
--
ciao Ban
Bordighera, Italy
Mark DeBellis
wrote:
>Mark DeBellis wrote:
>> I have the following worry about audio listening tests. Suppose the
>> meaningful variable is a property of an extended passage, not a short
>> snippet. Then a subject's failure to accurately distinguish or
>> re-identify may be due to an inability to retain the property in
>> memory.
>
>Nobody uses short snippets to do audio listening tests or comparisons.
>We usually use a complete song or passage of an extended work.
>>
>> I am thinking by way of contrast to visual examples. I just made two
>> prints of a photograph using different settings on my printer. I am
>> looking at the face of the subject and I can see that the contrast is
>> higher in one than in the other. That is a Gestalt property of a
>> meaningful chunk of the picture, not a property of a few pixels (cf.
>> notes). The difference with the musical case is that I can compare
>> the contrast of the two pictures directly, whereas in music no
>> immediate comparison is possible. At best I have to keep the property
>> in memory, and maybe the relevant variable is something not easily
>> retained.
>
>Not true. You can perform rapid switching as often and as many as you
>want.
OK but I am saying, suppose the meaningful variable is a property of
an extended passage. So you have to listen to an uninterrupted
passage in order to perceive the property. Switching back and forth
will defeat the purpose, yes?
Mark
Mark DeBellis
If you hear something but do not retain a memory of it (sufficient to
carry out a certain kind of test), you still heard it. No?
nab...@hotmail.com
In a sense, I suppose, but in that case you can't carry out any kind of
a comparison at all. So how could you be conscious of it under any
conditions? And if you can't be conscious of it under any conditions,
how can you say that you "heard" it?
Though I suspect you meant something slightly different, based on
another post, so I'll respond to that one.
bob
Gary Eickmeier
> OK but I am saying, suppose the meaningful variable is a property of
> an extended passage. So you have to listen to an uninterrupted
> passage in order to perceive the property. Switching back and forth
> will defeat the purpose, yes?
I would say this is a false premise. But perhaps you could give an
example of a meaningful variable that is a property of a longer passage.
Gary Eickmeier
nab...@hotmail.com
> OK but I am saying, suppose the meaningful variable is a property of
> an extended passage. So you have to listen to an uninterrupted
> passage in order to perceive the property. Switching back and forth
> will defeat the purpose, yes?
Yeah, but. First, the problem, if it were a problem, could easily be
solved by listening to longer passages. No one's ever heard differences
between competent amps/cables doing it that way, either.
Second, the research demonstrates pretty clearly that our memory for
subtle sonic differences is very limited. In other words, contrary to
your conjecture, switching back and forth quickly and frequently really
is more effective.
bob
Mark DeBellis
>Gary Eickmeier wrote:
>> Mark DeBellis wrote:
>>> I have the following worry about audio listening tests. Suppose the
>>> meaningful variable is a property of an extended passage, not a short
>>> snippet. Then a subject's failure to accurately distinguish or
>>> re-identify may be due to an inability to retain the property in
>>> memory.
>I second Garys comment. Some additions:
>There are ABX switchboxes available, I made one myself with relays to switch
>between two speaker cables simultaneously on both ends. You can switch any
>time as much as you like.
Sorry I guess I didn't explain my idea very well. Suppose that in
order to perceive the relevant property a listener has to hear an
uninterrupted stretch of music from the same source. That is, suppose
the relevant property is not a property that belongs to any short
snippet of the signal but is rather a property that belongs only to a
whole, longer passage, say 5 mins. in length or a whole movement.
What I am thinking of here is the SACD vs. CD issue discussed on
another thread. I am wondering if the unit over which perception can
differ meaningfully can be an extended passage not a brief interval;
if so, my switching back and forth between SACD and CD would not be a
relevant test, because I would hear neither SACD nor CD as an unbroken
extended passage. I guess I am asking basically whether the existing
protocols for audio tests make room for the possibility that there can
be auditory perception of properties of longer, extended passages, and
are sufficient to measure such perception.
Perhaps the answer would be that there could not be a difference in
perceptible properties of longer passages without a detectable
difference in frequency response, which could be heard in quick-switch
tests; but is that obvious?
Mark
Buster Mudd
I'm pretty sure I've mentioned this book here before: Daniel Dennett's
_Conciousness Explained_. Probably my favorite treatese on the physical
process of cognition & perception -- thought-provoking, challenging,
elucidating and funny to read! Highly recomended.
Anyway, I mention this book because, in a nutshell, according to
Dennett, the answer to your question is "No."
In a slightly bigger nutshell, Dennett goes on to explain (with far
more conviction & evidence than I could possibly muster in a newsgroup
posting) that there is often a significant & meaningful difference
between What We Perceived, and What We Think We Perceived.
Harry Lavo
news:d8vne...@news4.newsguy.com...
There is a very simple, very powerful way to determine this. But it is not
practical or possible for one individual. It is called monadic testing. It
requires listening to the segment of music, and rating that musical
reproduction *immediately afterwards* using a series of rating criteria.
Such criteria might include, for example, a five point scale ranging from:
"bass sounded extrememly punchy" to "bass sounded flabby and undynamic".
When hundreds of people do this, statistics can be applied to determine if
there are in fact perceivable differences, and if so, on what criteria.
If I were Harmon Industries, I might design and sponsor such a test on
occasion. Frankly, Sony blew an opportunity to do such a test (it would be
expensive) for their SACD launch. Imagine if the introductory campaign had
included "proof" that SACD sounded better. We'd now have a viable second
format.
If I were the AES, I might sponsor such a test as a "control test" for
single-person tests such as the much bally-hooed ABX test, to advance the
state of the art..
But for a given individual it is not a practical test.
nab...@hotmail.com
> Sorry I guess I didn't explain my idea very well. Suppose that in
> order to perceive the relevant property a listener has to hear an
> uninterrupted stretch of music from the same source. That is, suppose
> the relevant property is not a property that belongs to any short
> snippet of the signal but is rather a property that belongs only to a
> whole, longer passage, say 5 mins. in length or a whole movement.
> What I am thinking of here is the SACD vs. CD issue discussed on
> another thread. I am wondering if the unit over which perception can
> differ meaningfully can be an extended passage not a brief interval;
> if so, my switching back and forth between SACD and CD would not be a
> relevant test, because I would hear neither SACD nor CD as an unbroken
> extended passage. I guess I am asking basically whether the existing
> protocols for audio tests make room for the possibility that there can
> be auditory perception of properties of longer, extended passages, and
> are sufficient to measure such perception.
Yes. There's nothing that would make a DBT involving full 5-minute
samples invalid. However, there's also no reason to think they would
work better, as I noted yesterday.
(To be completely accurate, the protocols DO require that the subject
have the ability to switch any time he wants. But there is nothing that
requires him to switch more often than once every 5 minutes if he so
chooses.)
> Perhaps the answer would be that there could not be a difference in
> perceptible properties of longer passages without a detectable
> difference in frequency response, which could be heard in quick-switch
> tests; but is that obvious?
Yep. And you're more likely to notice it if you switch quickly and
frequently between choices.
bob
Mark DeBellis
>Mark DeBellis wrote:
>> OK but I am saying, suppose the meaningful variable is a property of
>> an extended passage. So you have to listen to an uninterrupted
>> passage in order to perceive the property. Switching back and forth
>> will defeat the purpose, yes?
>
>Yeah, but. First, the problem, if it were a problem, could easily be
>solved by listening to longer passages. No one's ever heard differences
>between competent amps/cables doing it that way, either.
It would not solve the problem because once you had listened to the
first passage you would have to remember the property for the duration
of the second passage, and I am hypothesizing that you don't have
reliable memory for that. That is the problem.
FWIW, here I am thinking of SACD vs. CD rather than amps or cables. I
don't know if it makes a difference, but the intuition is about music
not white noise (say).
>
>Second, the research demonstrates pretty clearly that our memory for
>subtle sonic differences is very limited. In other words, contrary to
>your conjecture, switching back and forth quickly and frequently really
>is more effective.
Is the research that demonstrates this based entirely on the tests
that I am saying would not be sensitive to such possibilities? Isn't
that a circular argument? If not, what is the relevant research?
Mark
Mark DeBellis
>Mark DeBellis wrote:
>> On 16 Jun 2005 03:14:23 GMT, nab...@hotmail.com wrote:
>>
>> >Mark DeBellis wrote:
>> >> I have the following worry about audio listening tests. Suppose the
>> >> meaningful variable is a property of an extended passage, not a short
>> >> snippet. Then a subject's failure to accurately distinguish or
>> >> re-identify may be due to an inability to retain the property in
>> >> memory.
>> >
>> >Then it's not audible. End of discussion.
>>
>> If you hear something but do not retain a memory of it (sufficient to
>> carry out a certain kind of test), you still heard it. No?
>
>In a sense, I suppose, but in that case you can't carry out any kind of
>a comparison at all. So how could you be conscious of it under any
>conditions? And if you can't be conscious of it under any conditions,
>how can you say that you "heard" it?
I don't see how it follows from the fact that you're unable to do a
comparison at a later point that you weren't conscious of said
property when you experienced it.
Mark DeBellis
wrote:
>Mark DeBellis wrote:
>> On 16 Jun 2005 03:14:23 GMT, nab...@hotmail.com wrote:
>
>> If you hear something but do not retain a memory of it (sufficient to
>> carry out a certain kind of test), you still heard it. No?
>
>
>
>I'm pretty sure I've mentioned this book here before: Daniel Dennett's
>_Conciousness Explained_. Probably my favorite treatese on the physical
>process of cognition & perception -- thought-provoking, challenging,
>elucidating and funny to read! Highly recomended.
>
>Anyway, I mention this book because, in a nutshell, according to
>Dennett, the answer to your question is "No."
That is an interesting book, and I do remember reading it a while
back, and I second your recommendation, but what exactly is the reason
to think "No"? There are cases all the time when people perceive
things and then forget them.
>
>In a slightly bigger nutshell, Dennett goes on to explain (with far
>more conviction & evidence than I could possibly muster in a newsgroup
>posting) that there is often a significant & meaningful difference
>between What We Perceived, and What We Think We Perceived.
True, but how does that difference play a role here?
Mark DeBellis
>Mark DeBellis wrote:
>
>> Sorry I guess I didn't explain my idea very well. Suppose that in
>> order to perceive the relevant property a listener has to hear an
>> uninterrupted stretch of music from the same source. That is, suppose
>> the relevant property is not a property that belongs to any short
>> snippet of the signal but is rather a property that belongs only to a
>> whole, longer passage, say 5 mins. in length or a whole movement.
>> What I am thinking of here is the SACD vs. CD issue discussed on
>> another thread. I am wondering if the unit over which perception can
>> differ meaningfully can be an extended passage not a brief interval;
>> if so, my switching back and forth between SACD and CD would not be a
>> relevant test, because I would hear neither SACD nor CD as an unbroken
>> extended passage. I guess I am asking basically whether the existing
>> protocols for audio tests make room for the possibility that there can
>> be auditory perception of properties of longer, extended passages, and
>> are sufficient to measure such perception.
>
>Yes. There's nothing that would make a DBT involving full 5-minute
>samples invalid.
OK, then please explain to me where I'm going wrong. I am
hypothesizing that there are properties (1) that can only be perceived
over long stretches and (2) are not retained in memory. If there are
such properties, the kind of test I'm thinking of won't be sufficient
to measure the perception of them, because at the end of the second
5-minute sample, the subject won't remember the first one well enough
to make an accurate comparison. A test of this sort will not be
sensitive to the phenomenon. Please tell me why the reasons I have
given for my conclusion are not good ones.
> However, there's also no reason to think they would
>work better, as I noted yesterday.
>
>(To be completely accurate, the protocols DO require that the subject
>have the ability to switch any time he wants. But there is nothing that
>requires him to switch more often than once every 5 minutes if he so
>chooses.)
>
>> Perhaps the answer would be that there could not be a difference in
>> perceptible properties of longer passages without a detectable
>> difference in frequency response, which could be heard in quick-switch
>> tests; but is that obvious?
>
>Yep.
Well, it's not obvious to me, so if you could give me some indication
why I should think it's true, that would be most appreciated!
Mark
nab...@hotmail.com
> On 18 Jun 2005 02:27:00 GMT, nab...@hotmail.com wrote:
> >
> >Yes. There's nothing that would make a DBT involving full 5-minute
> >samples invalid.
>
> OK, then please explain to me where I'm going wrong. I am
> hypothesizing
This is where you're going wrong. You are NOT hypothesizing. You are
engaging in idle speculation. "There might be something" is not a
hypothesis. If you can tell us what you think that something is, and
give us some reason to believe it might be a factor, then you have a
hypothesis.
> that there are properties (1) that can only be perceived
> over long stretches and (2) are not retained in memory.
Aren't these two things mutually contradictory? Certainly you are
relying on memory when you perceive something over long stretches. And,
to repeat myself, what is it? We're still waiting.
> If there are
> such properties, the kind of test I'm thinking of won't be sufficient
> to measure the perception of them, because at the end of the second
> 5-minute sample, the subject won't remember the first one well enough
> to make an accurate comparison. A test of this sort will not be
> sensitive to the phenomenon. Please tell me why the reasons I have
> given for my conclusion are not good ones.
>
>
> > However, there's also no reason to think they would
> >work better, as I noted yesterday.
> >
> >(To be completely accurate, the protocols DO require that the subject
> >have the ability to switch any time he wants. But there is nothing that
> >requires him to switch more often than once every 5 minutes if he so
> >chooses.)
> >
> >> Perhaps the answer would be that there could not be a difference in
> >> perceptible properties of longer passages without a detectable
> >> difference in frequency response, which could be heard in quick-switch
> >> tests; but is that obvious?
> >
> >Yep.
>
> Well, it's not obvious to me, so if you could give me some indication
> why I should think it's true, that would be most appreciated!
Can you name some sonic distinction that isn't a partial loudness
difference?
bob
Gary Eickmeier
> It would not solve the problem because once you had listened to the
> first passage you would have to remember the property for the duration
> of the second passage, and I am hypothesizing that you don't have
> reliable memory for that. That is the problem.
OK, so let me get this straight: You listen to one component, hear
certain properties, then listen to another component, and hear certain
other properties, but by the time it's all over with you can't remember
which was which? This pretty much dooms any listening test, doesn't it?
Not sure I see the point of your question.
Gary Eickmeier
nab...@hotmail.com
> On 17 Jun 2005 03:11:07 GMT, nab...@hotmail.com wrote:
>
> >Mark DeBellis wrote:
> >> OK but I am saying, suppose the meaningful variable is a property of
> >> an extended passage. So you have to listen to an uninterrupted
> >> passage in order to perceive the property. Switching back and forth
> >> will defeat the purpose, yes?
> >
> >Yeah, but. First, the problem, if it were a problem, could easily be
> >solved by listening to longer passages. No one's ever heard differences
> >between competent amps/cables doing it that way, either.
>
> It would not solve the problem because once you had listened to the
> first passage you would have to remember the property for the duration
> of the second passage, and I am hypothesizing that you don't have
> reliable memory for that. That is the problem.
Then how do you know it's a meaningful variable?
> FWIW, here I am thinking of SACD vs. CD rather than amps or cables. I
> don't know if it makes a difference, but the intuition is about music
> not white noise (say).
>
> >
> >Second, the research demonstrates pretty clearly that our memory for
> >subtle sonic differences is very limited. In other words, contrary to
> >your conjecture, switching back and forth quickly and frequently really
> >is more effective.
>
> Is the research that demonstrates this based entirely on the tests
> that I am saying would not be sensitive to such possibilities? Isn't
> that a circular argument? If not, what is the relevant research?
It's based on tests of human hearing. Your ability to remember partial
loudness differences lasts a couple of seconds, tops. You are
speculating that there exists something that violates this established
fact. What is it, and how do you know?
bob
Mark DeBellis
wrote:
>Mark DeBellis wrote:
It is that the test would be inadequate to measure the detection of
said properties.
The background to this is that I said I thought SACD sounded better
than CD, and Chung suggested that I try a simple blind test, to see if
I could reliably identify which was SACD and which was CD (after
matching levels; SACD and CD layers of same disc). Indeed I could
not, at least on one set of trials. Should the conclusion be that
SACD sounds the same as CD? Or is it possible that the test I applied
is inadequate in some way? How *could* it be possible that the test
is inadequate? My question is basically an attempt to explain how
that might be. Suppose when I am listening to recorded music (1) I
hear properties of temporally extended passages and (2) I can't retain
a memory of those properties long enough to make a comparison, if at
all. Then I could have perceived different things, although this
difference would not show up in the kind of test I performed.
Mark
Buster Mudd
> On 17 Jun 2005 23:48:24 GMT, "Buster Mudd" <mr_fu...@mail.com>
> wrote:
>
> >Mark DeBellis wrote:
> >> On 16 Jun 2005 03:14:23 GMT, nab...@hotmail.com wrote:
> >
> >> If you hear something but do not retain a memory of it (sufficient to
> >> carry out a certain kind of test), you still heard it. No?
> >
> >
> >
> >I'm pretty sure I've mentioned this book here before: Daniel Dennett's
> >_Conciousness Explained_. Probably my favorite treatese on the physical
> >process of cognition & perception -- thought-provoking, challenging,
> >elucidating and funny to read! Highly recomended.
> >
> >Anyway, I mention this book because, in a nutshell, according to
> >Dennett, the answer to your question is "No."
>
> That is an interesting book, and I do remember reading it a while
> back, and I second your recommendation, but what exactly is the reason
> to think "No"?
Because "hearing" is a cognitive process; it takes place in the brain,
not in the ear. So if your brain tells you you didn't hear it, even if
soundwaves did strike your eardrum...and even (!) if at an earlier time
your brain told you that you did hear it...for all intents & purposes,
you didn't hear it. Saying "I heard it" is only useful if you can
access the perception in order to make subsequent discriminations.
> There are cases all the time when people perceive
> things and then forget them.
At which point any information they may have gleaned from perceiving
that thing is lost to them. Hence, the distinction between whether they
actually perceived it & then forgot it, or never perceived it in the
first place, is moot.
> >In a slightly bigger nutshell, Dennett goes on to explain (with far
> >more conviction & evidence than I could possibly muster in a newsgroup
> >posting) that there is often a significant & meaningful difference
> >between What We Perceived, and What We Think We Perceived.
>
> True, but how does that difference play a role here?
It goes to the core of your initial question: How valid can an audio
test be if it's measuring the perception of phenomena which may not
actually exist?
nab...@hotmail.com
IOW, you did a test, you didn't like the result, so now you're
demanding that we give you some basis for rejecting the result of the
test.
The test is adequate, assuming you did it with reasonable care. That's
why scientists have used it to test just about everything we know about
hearing. Why should a test become inadequate just because some hobbyist
wants to believe something that isn't true?
bob
Harry Lavo
Sorry to sound like a broken record, but since you assert this constantly I
can only do the same in response....the testing you favor has never been
validated for the open-ended evaluation of reproduced music. Period. If it
had been and could be demonstrated to have been so, it would be widely used
and accepted by most every audiophile. The fact that it has not been, is
not accepted, and flies in the face of so much otherwise different consensus
means that to continue asserting it as you do, is an act of faith, nothing
else.
Mark DeBellis
>Mark DeBellis wrote:
>> OK but I am saying, suppose the meaningful variable is a property of
>> an extended passage. So you have to listen to an uninterrupted
>> passage in order to perceive the property. Switching back and forth
>> will defeat the purpose, yes?
>I would say this is a false premise. But perhaps you could give an
>example of a meaningful variable that is a property of a longer passage.
I missed your post until now because, for some reason, it didn't
download (I'm using Free Agent) from my news server.
An example of a property that belongs to a temporally extended passage
without belonging to short slices of it is being a descending C major
scale, one octave long. That is a property a listener can perceive
the passage as having, but it is a property is one that belongs to the
whole, not short parts.
Another perceivable property that belongs to a temporal whole is the
property, belonging to a spoken sentence, of being syntactically well
formed.
Mark
Mark DeBellis
>Mark DeBellis wrote:
>> On 17 Jun 2005 03:11:07 GMT, nabo...@hotmail.com wrote:
>
>> >Mark DeBellis wrote:
>> >> OK but I am saying, suppose the meaningful variable is a property of
>> >> an extended passage. So you have to listen to an uninterrupted
>> >> passage in order to perceive the property. Switching back and forth
>> >> will defeat the purpose, yes?
>
>> >Yeah, but. First, the problem, if it were a problem, could easily be
>> >solved by listening to longer passages. No one's ever heard differences
>> >between competent amps/cables doing it that way, either.
>
>> It would not solve the problem because once you had listened to the
>> first passage you would have to remember the property for the duration
>> of the second passage, and I am hypothesizing that you don't have
>> reliable memory for that. That is the problem.
>
>Then how do you know it's a meaningful variable?
>
>> FWIW, here I am thinking of SACD vs. CD rather than amps or cables. I
>> don't know if it makes a difference, but the intuition is about music
>> not white noise (say).
>
>> >Second, the research demonstrates pretty clearly that our memory for
>> >subtle sonic differences is very limited. In other words, contrary to
>> >your conjecture, switching back and forth quickly and frequently really
>> >is more effective.
>
>> Is the research that demonstrates this based entirely on the tests
>> that I am saying would not be sensitive to such possibilities? Isn't
>> that a circular argument? If not, what is the relevant research?
>
>It's based on tests of human hearing. Your ability to remember partial
>loudness differences lasts a couple of seconds, tops. You are
>speculating that there exists something that violates this established
>fact. What is it, and how do you know?
>
I'm not claiming to know that. I'm saying that *if* such properties
exist, certain tests would not be capable of demonstrating that people
perceive them. It is a point about whether the methodology is
sensitive to a certain class of properties.
It is not contrary to "established fact" to assert that properties of
temporally extended wholes exist and can be perceived. That is
demonstrated every time you recognize a melody.
Mark
nab...@hotmail.com
the scientific method, this is what I meant:
Harry Lavo wrote:
>
> Sorry to sound like a broken record, but since you assert this constantly I
> can only do the same in response....the testing you favor has never been
> validated for the open-ended evaluation of reproduced music. Period. If it
> had been and could be demonstrated to have been so, it would be widely used
> and accepted by most every audiophile. The fact that it has not been, is
> not accepted, and flies in the face of so much otherwise different consensus
> means that to continue asserting it as you do, is an act of faith, nothing
> else.
To believe this, one must ignore reams of scientific data that conflict
with what you believe. In particular, one must assert, against all
evidence, that human hearing operates differently when listening to
reproduced music than it does at all other times, and that generations
of scientists have just been deluding themselves.
This is how Creation Science operates. Let's hope the audio field never
sinks so low.
bob
Thepork...@aol.com
> Mark DeBellis wrote:
> > On 17 Jun 2005 03:11:07 GMT, nab...@hotmail.com wrote:
> >
> > >Mark DeBellis wrote:
> > >> OK but I am saying, suppose the meaningful variable is a property of
> > >> an extended passage. So you have to listen to an uninterrupted
> > >> passage in order to perceive the property. Switching back and forth
> > >> will defeat the purpose, yes?
> > >
> > >Yeah, but. First, the problem, if it were a problem, could easily be
> > >solved by listening to longer passages. No one's ever heard differences
> > >between competent amps/cables doing it that way, either.
> >
> > It would not solve the problem because once you had listened to the
> > first passage you would have to remember the property for the duration
> > of the second passage, and I am hypothesizing that you don't have
> > reliable memory for that. That is the problem.
>
> Then how do you know it's a meaningful variable?
It seems his point is that yo don't know it isn't either.
>
> > FWIW, here I am thinking of SACD vs. CD rather than amps or cables. I
> > don't know if it makes a difference, but the intuition is about music
> > not white noise (say).
> >
> > >
> > >Second, the research demonstrates pretty clearly that our memory for
> > >subtle sonic differences is very limited. In other words, contrary to
> > >your conjecture, switching back and forth quickly and frequently really
> > >is more effective.
> >
> > Is the research that demonstrates this based entirely on the tests
> > that I am saying would not be sensitive to such possibilities? Isn't
> > that a circular argument? If not, what is the relevant research?
>
> It's based on tests of human hearing. Your ability to remember partial
> loudness differences lasts a couple of seconds, tops.
Now that is interesting given that small barely audible level
differences can lead one to form a prefernce. How can that be?
You are
> speculating that there exists something that violates this established
> fact. What is it, and how do you know?
How do you explain the fact that small level diferences can lead to
prefeences if we can't remember them in our comparisons?
Scott Wheeler
NYO...@peoplepc.com
....the testing you favor has never been
>validated for the open-ended evaluation of reproduced music. Period.
Which evalutaion method that you approve of has been so validated?
What did the BBC use when they did their ABX comparisons for the
purpose of upgrading their speakers?
> If it
>had been and could be demonstrated to have been so, it would be widely used
>and accepted by most every audiophile.
The fact that double blind level matched comparisons are the standard
for those doing the serious research on all other forms of sound hasn't
seemed to convince audiophiles that it is valid, so, it would seem that
many of them simply refuse to accept what is known, probably because it
doesn't give the results that they believe they should get.
>The fact that it has not been, is
>not accepted, and flies in the face of so much otherwise different consensus
>means that to continue asserting it as you do, is an act of faith, nothing
>else.
And accepting less reliable, non-bias controlled tests, is not?
The problem is not consensus, since clearly audiophiles are in the
minority when it comes to ABX and it's reliability.
Mark DeBellis
>Because "hearing" is a cognitive process; it takes place in the brain,
>not in the ear. So if your brain tells you you didn't hear it, even if
>soundwaves did strike your eardrum...and even (!) if at an earlier time
>your brain told you that you did hear it...for all intents & purposes,
>you didn't hear it. Saying "I heard it" is only useful if you can
>access the perception in order to make subsequent discriminations.
I agree with the basic idea that perception is something that
influences behavior, but why does the behavior have to be restricted
to comparison and identification? Suppose a listener gives higher
approval ratings to one set of (blind) stimuli than another, without
ever trying to say which stimuli were the same and which were
different. This would be an influence on behavior, but of a weaker
sort than is required by the "can you reliably identify" type of test.
>
>> There are cases all the time when people perceive
>> things and then forget them.
>
>At which point any information they may have gleaned from perceiving
>that thing is lost to them. Hence, the distinction between whether they
>actually perceived it & then forgot it, or never perceived it in the
>first place, is moot.
Not if the information is still doing work somewhere in your cognitive
economy, even though it can't be brought to consciousness, or is not
specific enough to enable one to perform the identification task.
Mark
Greg Lee
...
> The background to this is that I said I thought SACD sounded better
> than CD, and Chung suggested that I try a simple blind test, to see if
> I could reliably identify which was SACD and which was CD (after
> matching levels; SACD and CD layers of same disc). Indeed I could
> not, at least on one set of trials. Should the conclusion be that
> SACD sounds the same as CD? Or is it possible that the test I applied
> is inadequate in some way? How *could* it be possible that the test
> is inadequate? My question is basically an attempt to explain how
Another possibility is suggested by an informal experiment on smells
that I did on myself. I took a selection of spices that were all
in the same type of bottle, closed my eyes, shuffled the bottles
around, and opened and sniffed them one by one, trying to identify
the spice by name. I was sure I could do it but was quite amazed to
discover that I couldn't do it at all. With my eyes closed, I could
tell the differences perfectly well, but I just couldn't connect the
names with the smells.
So I think it's *possible* that in unsighted comparisons, the part
of the brain that does symbolic analysis and associated judgment
is not in perfect communication with other parts.
--
Greg Lee <gr...@ling.lll.hawaii.edu>
Mark DeBellis
of course.
Quick-switch tests are said to be the most sensitive, therefore best,
yes? Because they permit finer discriminations. An observation: the
ultimate purpose of an audio test is not discrimination per se. That
is, it's not as if the job that has to be performed here is to
discriminate two sources if they can be, to do the best job of
discrimination we can, as if that were the real goal. The real purpose
of the test is rather to find out what information is available to the
listener in the context of use, or perhaps to estimate an upper bound
on that information.
It is possible that two sources in the ordinary context of use do not
present different information to the listener, even if there are ways
to set up testing situations (e.g., at higher volume) where a person
could discriminate the sources. All well and good if we do, but it's
not like having a test that permits such discrimination is a valuable
achievement in itself; what we are basically interested in is making
sure that if there are differences of information presented in the
ordinary context of use, then they will show up and get discriminated
in the test.
Question: for all the resolving power quick-switch tests have, for all
the power they have to put the stimulus under a microscope and discern
small differences of detail, are there certain sorts of properties they
are *not* so good at picking up? Is there perhaps a forest-for-trees
phenomenon lurking somewhere out there?
Here's an off the cuff example. Suppose I have two digital photographs
that are identical except that one is 1.01 the size of the other.
First I compare them (this is the analogue of the quick-switch test) by
comparing small portions of one with the other. The comparison is set
up in such a way that when I compare a square portion of one with a
square portion of the other, one of them is 1.01 as large as the other.
However, I cannot see the difference because these are small areas and
the difference in size is below my threshold of discrimination.
However, when I compare the wholes I can see the difference in size,
because the difference is now greater than the threshold, since the
whole is much larger than any of those parts.
An auditory example would be tempo. Suppose I am listening to two
sources, where the only difference is that one of them has a speed of
1.01 times the other. If I listen to short excerpts any difference is
below the just-noticeable-difference, but if the whole example is the
Ring cycle, I will notice that one finishes before dark and the other
doesn't, I get hungry during one but not the other, etc.
So even if quick-switch tests, on balance, are the most sensitive, that
doesn't mean there can't be things out there that don't get caught in
their net (though they may be detectable in other ways).
To come back to the SACD/CD example, my concern is whether, even if the
quick-switch test were a "null," there could be differences that the
test does not do a good job of proving the existence of. Rather than
feel assured that science tells us there could not be such differences,
it seems to me pretty apparent that every test has its limitations.
Sound plausible?
Mark
Mark DeBellis
one place; here is a corrected version.
Gary Eickmeier <geic...@tampabay.rr.com> wrote:
>Mark DeBellis wrote:
>> OK but I am saying, suppose the meaningful variable is a property of
>> an extended passage. So you have to listen to an uninterrupted
>> passage in order to perceive the property. Switching back and forth
>> will defeat the purpose, yes?
>I would say this is a false premise. But perhaps you could give an
>example of a meaningful variable that is a property of a longer passage.
I missed your post until now because, for some reason, it didn't
download (I'm using Free Agent) from my news server.
An example of a property that belongs to a temporally extended passage
without belonging to short slices of it is being a descending C major
scale, one octave long. That is a property a listener can perceive
the passage as having, but it is a property that belongs to the whole,
Harry Lavo
> "Buster Mudd" <mr_fu...@mail.com> wrote:
>
>>Because "hearing" is a cognitive process; it takes place in the brain,
>>not in the ear. So if your brain tells you you didn't hear it, even if
>>soundwaves did strike your eardrum...and even (!) if at an earlier time
>>your brain told you that you did hear it...for all intents & purposes,
>>you didn't hear it. Saying "I heard it" is only useful if you can
>>access the perception in order to make subsequent discriminations.
>
> I agree with the basic idea that perception is something that
> influences behavior, but why does the behavior have to be restricted
> to comparison and identification? Suppose a listener gives higher
> approval ratings to one set of (blind) stimuli than another, without
> ever trying to say which stimuli were the same and which were
> different. This would be an influence on behavior, but of a weaker
> sort than is required by the "can you reliably identify" type of test.
>
Yet if 200 people do the same thing, you can apply statistical measures of
difference and determine with high accuracy whether or not subjectively
there is a difference, and if so, in what characteristics (assuming the
scalar data is pertinent to the differences heard). That is exactly the
kind of validation that is missing that would prove (or prove in the
negative) whether quick-switch, "short form" tests such as ABX can measure
the same thing.
>>
>>> There are cases all the time when people perceive
>>> things and then forget them.
>>
>>At which point any information they may have gleaned from perceiving
>>that thing is lost to them. Hence, the distinction between whether they
>>actually perceived it & then forgot it, or never perceived it in the
>>first place, is moot.
>
> Not if the information is still doing work somewhere in your cognitive
> economy, even though it can't be brought to consciousness, or is not
> specific enough to enable one to perform the identification task.
Certainly. If you hear it again and respond the same way, it can influence
either favorably or unfavorably your reception to the music being played /
your evaluation of the system it is being played on. Thus most audiophiles
emphasis on long term listening.
Harry Lavo
I repeat, the test has never been directly validated for the purposes
espoused here.
And the vast majority of the scientific work has *NOT* been on music and
certainly virtually none has been dedicated to the worth of abx testing as a
means of open-ended evaluation of audio components.
nab...@hotmail.com
Or maybe it's possible that when you make the test tougher, you don't
do as well?
You're doing an open identification test. That would be the equivalent
of listening to one amp and deciding whether it was a Rotel, Adcom,
Krell, or Hafler. To succeed at such a test, even if it were possible,
would require intense practice. All evidence suggests, however, that it
would not be possible--that even if there were some slight sonic
differences between those amps, which you could tell apart in a
quick-switching comparison, you probably wouldn't be able to remember
what is distinctive about each of them long enough to identify a single
example.
Whereas I'll bet lots of professional chefs, with a fair bit more
training than you, would have no trouble putting the right labels on
your spice rack.
Olfactory analogies, like visual ones, don't work.
bob
nab...@hotmail.com
> A couple of further ideas to toss out, from an onlooker's perspective
> of course.
>
> Quick-switch tests are said to be the most sensitive, therefore best,
> yes? Because they permit finer discriminations. An observation: the
> ultimate purpose of an audio test is not discrimination per se.
No, but if you can't discriminate between two things, then the
differences between them are irrelevant.
> That
> is, it's not as if the job that has to be performed here is to
> discriminate two sources if they can be, to do the best job of
> discrimination we can, as if that were the real goal. The real purpose
> of the test is rather to find out what information is available to the
> listener in the context of use, or perhaps to estimate an upper bound
> on that information.
Well, we know that. That's what the science of psychoacoustics is all
about. Check it out sometime.
> It is possible that two sources in the ordinary context of use do not
> present different information to the listener, even if there are ways
> to set up testing situations (e.g., at higher volume) where a person
> could discriminate the sources. All well and good if we do, but it's
> not like having a test that permits such discrimination is a valuable
> achievement in itself; what we are basically interested in is making
> sure that if there are differences of information presented in the
> ordinary context of use, then they will show up and get discriminated
> in the test.
>
> Question: for all the resolving power quick-switch tests have, for all
> the power they have to put the stimulus under a microscope and discern
> small differences of detail, are there certain sorts of properties they
> are *not* so good at picking up? Is there perhaps a forest-for-trees
> phenomenon lurking somewhere out there?
No there is not, according to all experts in the field. Or do you think
you know more than the experts?
> Here's an off the cuff example. Suppose I have two digital photographs
> that are identical except that one is 1.01 the size of the other.
> First I compare them (this is the analogue of the quick-switch test) by
> comparing small portions of one with the other. The comparison is set
> up in such a way that when I compare a square portion of one with a
> square portion of the other, one of them is 1.01 as large as the other.
> However, I cannot see the difference because these are small areas and
> the difference in size is below my threshold of discrimination.
>
> However, when I compare the wholes I can see the difference in size,
> because the difference is now greater than the threshold, since the
> whole is much larger than any of those parts.
Irrelevant and off-point visual analogy. Visual analogies don't work.
Ever.
>
> An auditory example would be tempo. Suppose I am listening to two
> sources, where the only difference is that one of them has a speed of
> 1.01 times the other. If I listen to short excerpts any difference is
> below the just-noticeable-difference, but if the whole example is the
> Ring cycle, I will notice that one finishes before dark and the other
> doesn't, I get hungry during one but not the other, etc.
But you're not discriminating between the two by listening to them.
You're discriminating between them by looking at the clock. (And I
don't know what the threshold is for speed variation, but at some point
you really would be able to distinguish between them in a standard
DBT.)
> So even if quick-switch tests, on balance, are the most sensitive, that
> doesn't mean there can't be things out there that don't get caught in
> their net (though they may be detectable in other ways).
So far you haven't come up with a single one. That's because there
aren't any.
> To come back to the SACD/CD example, my concern is whether, even if the
> quick-switch test were a "null," there could be differences that the
> test does not do a good job of proving the existence of. Rather than
> feel assured that science tells us there could not be such differences,
> it seems to me pretty apparent that every test has its limitations.
>
> Sound plausible?
No. It sounds like you're grasping at straws because you don't like
what the science is telling you. If you live in Kansas, I suggest you
run for the state board of ed.
bob
nab...@hotmail.com
> "Buster Mudd" <mr_fu...@mail.com> wrote:
>
> >Because "hearing" is a cognitive process; it takes place in the brain,
> >not in the ear. So if your brain tells you you didn't hear it, even if
> >soundwaves did strike your eardrum...and even (!) if at an earlier time
> >your brain told you that you did hear it...for all intents & purposes,
> >you didn't hear it. Saying "I heard it" is only useful if you can
> >access the perception in order to make subsequent discriminations.
>
> I agree with the basic idea that perception is something that
> influences behavior, but why does the behavior have to be restricted
> to comparison and identification?
It doesn't. You can easily do a DBT as, say, a preference test. If the
subject reports the same preference at a statistically significant
rate, we can assume that the two are different.
> Suppose a listener gives higher
> approval ratings to one set of (blind) stimuli than another, without
> ever trying to say which stimuli were the same and which were
> different. This would be an influence on behavior, but of a weaker
> sort than is required by the "can you reliably identify" type of test.
>
> >
> >> There are cases all the time when people perceive
> >> things and then forget them.
> >
> >At which point any information they may have gleaned from perceiving
> >that thing is lost to them. Hence, the distinction between whether they
> >actually perceived it & then forgot it, or never perceived it in the
> >first place, is moot.
>
> Not if the information is still doing work somewhere in your cognitive
> economy, even though it can't be brought to consciousness, or is not
> specific enough to enable one to perform the identification task.
But it isn't, because of our short aural memory for partial loudness
differences.
bob
Harry Lavo
> Mark DeBellis wrote:
>> "Buster Mudd" <mr_fu...@mail.com> wrote:
>>
>> >Because "hearing" is a cognitive process; it takes place in the brain,
>> >not in the ear. So if your brain tells you you didn't hear it, even if
>> >soundwaves did strike your eardrum...and even (!) if at an earlier time
>> >your brain told you that you did hear it...for all intents & purposes,
>> >you didn't hear it. Saying "I heard it" is only useful if you can
>> >access the perception in order to make subsequent discriminations.
>>
>> I agree with the basic idea that perception is something that
>> influences behavior, but why does the behavior have to be restricted
>> to comparison and identification?
>
> It doesn't. You can easily do a DBT as, say, a preference test. If the
> subject reports the same preference at a statistically significant
> rate, we can assume that the two are different.
But at the current level of validation, you can't be sure the test allows
you to actually perceive some of the differences that might factor into a
longer-term preference.
>
>> Suppose a listener gives higher
>> approval ratings to one set of (blind) stimuli than another, without
>> ever trying to say which stimuli were the same and which were
>> different. This would be an influence on behavior, but of a weaker
>> sort than is required by the "can you reliably identify" type of test.
>>
>> >
>> >> There are cases all the time when people perceive
>> >> things and then forget them.
>> >
>> >At which point any information they may have gleaned from perceiving
>> >that thing is lost to them. Hence, the distinction between whether they
>> >actually perceived it & then forgot it, or never perceived it in the
>> >first place, is moot.
>>
>> Not if the information is still doing work somewhere in your cognitive
>> economy, even though it can't be brought to consciousness, or is not
>> specific enough to enable one to perform the identification task.
>
> But it isn't, because of our short aural memory for partial loudness
> differences.
This assumes all aural differences are a function of loudness in one form or
another. Some of us feel otherwise and would like to see this underlying
premises actually validated with regard to phase differences, or harmonic
structure differences, or impulse response differences, or frequency
coherence throughout a dynamic volume change. In other words, is it volume,
or is it a more complex brain processing within a high-complex of aural
stimulae.
Mark DeBellis
there will a perceptible difference between two passages only if there
are short corresponding portions of those passages which, when
juxtaposed, will exhibit a perceptible loudness difference. OK, yes,
that does seem plausible (*).
What I am getting from this then is that the relevant task is not
necessarily "Which one is SACD and which is CD?" but "Are A and B the
same or different?" (This is for me the most important point.)
And for the latter, yes, I can see, intuitively at least, why
quick-switch tests are the most sensitive.
The quick-switch test will catch a difference that exists between the
passages only if the samples include points of divergence; if too few
samples are taken, they might just miss them.
(*) About the principle stated above which, if true, is a fact of
psychology. It seems to be saying more or less that differences are
perceptible in context only if they are perceptible in isolation. OK,
suppose I am looking at a photograph with a continuous gradation from
light to dark. And I can see the difference between that and a patch
of constant tone. Then the principle would say that I can do this only
if I can see the difference between small patches where the sample
taken from the first photograph is more or less constant in tone.
But not too small, because once the areas get very small I can't
reliably compare them any more.
OK now the auditory case. There is a signal that gets louder, and I
can hear the difference between that and a signal of constant loudness.
The principle says that I can do this only if I can hear the
difference in loudness between short corresponding portions.
Seems plausible ... and if the portions get to be too short then would
reliability go down, just as in the visual case?
Does it matter how the short portions are "juxtaposed"? Separated by
silence or one followed continously by the other?
Mark
NYO...@peoplepc.com
>I took a selection of spices that were all
>in the same type of bottle, closed my eyes, shuffled the bottles
>around, and opened and sniffed them one by one, trying to identify
>the spice by name. I was sure I could do it but was quite amazed to
>discover that I couldn't do it at all.
IIRC the limit is 3 scents, after that the nose gets overloaded.
Ban
>
>
> Now that is interesting given that small barely audible level
> differences can lead one to form a prefernce. How can that be?
>
>
more small levels rise above the threshold of hearing. Since the change is
small, you cannot detect more significant tones, but just a general feeling
of higher resolution.
>
>
> You are
>> speculating that there exists something that violates this
>> established fact. What is it, and how do you know?
>
>
>
>
>
> How do you explain the fact that small level diferences can lead to
> prefeences if we can't remember them in our comparisons?
>
>
If there is a significant level difference, you could in fact detect more
acoustical events, like the noise when the musicians turn the page of their
partitures. In this case you can remember this additional sound, but it has
to be coined and recognized again in each trial. This would not be direct
comparison, because the recognition doesn't give a vague pointer, but a
destinctive indicator. It also needs to be learned by training and you will
need a focused attention.
So whatever people say about long-term and short-term testing, the opposite
seems to be the case, and in fact has been validated by research.
Long-term evaluation with long pauses between the trials will require an
evaluation that is tied to certain distinctive passages, where or where not
a certain sound/noise can be heard. The fast switching will give you more a
general impression of the music as a whole, without the need to concentrate
on separate noises. It will be more joyful, you listen equally to the
instruments, it is more what happens during a concert.
>
>
> Scott Wheeler
--
ciao Ban
Bordighera, Italy
Mark DeBellis
We want to prove that two sources provide the same information to the
listener in the context of ordinary use.
We appeal now to the following Principle:
If two sources provide the same information in ordinary use, then in a
test in which small, corresponding portions are compared, the listener
will not be able to discriminate them.
In particular, the type of discrimination task is the following: on any
trial, two short excerpts are played, either both from the same source
(a repetition) or from different sources, and the listener has to say
if they're from the same or different sources, and he passes the test
if he says same or different better than chance.
Two observations. First of all, in a test like this there is plenty of
information coming from either source that may not make it to the
listener, such as the "temporally extended" properties I have
mentioned. (Maybe that's obvious.) The thing is, the Principle says
that that's OK because those properties have a certain dependence
relation on the, as it were, "atomic" properties that get compared in
such a test. The relation is, no difference in information presented
unless there is a difference in atomic properties. In other words,
information "supervenes" on atomic properties. It does so even if it
does not *consist* exclusively in atomic properties (because, when I
hear the Brahmsian style of a recorded performance, that is not any
property of a short snippet). (To put the Principle better,
information supervenes on *discernible* atomic properties.)
Second, what is key here is the nature of the test. If I am being
asked whether to say A and B are the same or different, that is one
thing. But if I am trying to say which source a given sample is coming
from, where these have been labelled in advance (like SACD or CD), that
is a different task. And the results might not be the same.
Moreover, there might be information of the temporally extended sort
that is available to me when I listen to longer excerpts but which are
not directly picked up in any test. This would include quick-switch
testing as well as comparison of longer excerpts. It would not be
directly picked up in quick-switch testing because I wouldn't hear
long-enough excerpts to perceive the temporally extended property; it
would elude comparison of longer excerpts if the information fails to
be kept in memory in a way that permits long-term comparison. That was
my initial worry. That worry is allayed to some extent by the
existence of "same/different" testing (if that testing is sensitive to
things on which said information supervenes), but my point is that if
the question is "Was that SACD or CD?" then the result is not a valid
test. The failure to get a correct answer better than chance does not
prove anything. It makes a big difference what the task is.
Make sense?
Mark
Buster Mudd
> >actually perceived it & then forgot it, or never perceived it in the
> >first place, is moot.
>
> Not if the information is still doing work somewhere in your cognitive
> economy, even though it can't be brought to consciousness, or is not
> specific enough to enable one to perform the identification task.
>
How would you go about *proving* that something was in someone's
"cognitive economy" if that someone was not conscious of that
something? How would you go about *proving* that something was in
someone's "cognitive economy" if that something could not enable that
someone to perform a task, any task?
Mark DeBellis
>
>... analogies, like visual ones, don't work.
You keep insisting that, but there is a wonderful book, Art and
Illusion by Ernst Gombrich, from which the opposite conclusion may be
drawn. Gombrich writes about the development of "lifelike" art and
the very concept of "lifelike." Though he doesn't focus directly on
music, suggesting analogies only here and there, he discusses general
psychological principles that apply across various arts and makes
numerous comparisons between different domains. A classic, along with
Nelson Goodman's Languages of Art, which I've mentioned already.
Mark
Mark DeBellis
>Mark DeBellis wrote:
>>
>> ... if the information is still doing work somewhere in your cognitive
>> economy, even though it can't be brought to consciousness, or is not
>> specific enough to enable one to perform the identification task.
>
>But it isn't, because of our short aural memory for partial loudness
>differences.
>
Doesn't it follow from what you say that musical form can make no
difference? When I am listening to Theme B, it makes no difference
what the character of Theme A was, what motives made it up, and so on.
If it doesn't follow from what you say, why not?
If this is what psychoacoustics teaches us, I'm moving to Kansas!
Mark
Mark DeBellis
>Mark DeBellis wrote:
>> I agree with the basic idea that perception is something that
>> influences behavior, but why does the behavior have to be restricted
>> to comparison and identification?
>
>It doesn't. You can easily do a DBT as, say, a preference test. If the
>subject reports the same preference at a statistically significant
>rate, we can assume that the two are different.
>IOW, you did a test, you didn't like the result, so now you're
>demanding that we give you some basis for rejecting the result of the
>test.
I think it makes a difference whether the task is "Was that SACD or
CD" or "Were A and B the same or different?" It makes a difference
whether the task is comparison-and-identification or simple
discrimination. There is a reason to reject the result of the test I
did because it was the wrong kind of test.
Mark
Harry Lavo
Better order the moving van....Bob didn't warn me and I hit the student
rush! :-)
Seriously, this is what the proponents of quick-switch, comparative testing
*believe* psychoacoustics teaches us. However, as recent experience in this
newsgroup has shown, when they give us a specific reference, which is rare,
it turns out to have much more complex and nuanced information than is
proposed, some of which supports the more complex issues raised here. Then
we are told we can't possibly understand the concepts in these books unless
we devote a full course of study (and preferably a liftetime) to them. Is
this coming from neuropsychologists or neurophysisists or audiologists?
No, it is coming from folks no more or less "literate" in these areas than
ourselves.
Ain't newsgroups wonderful? :-)
nab...@hotmail.com
> OK now the auditory case. There is a signal that gets louder, and I
> can hear the difference between that and a signal of constant loudness.
> The principle says that I can do this only if I can hear the
> difference in loudness between short corresponding portions.
>
> Seems plausible ... and if the portions get to be too short then would
> reliability go down,
No. There is only one point at which the volume of the two passages is
identical, so unless you are listening at exactly that point, and only
that point, the two are going to sound different to you, no matter how
short the samples are. So unless you're extremely unlucky in your
choice of snippets to compare, you'll have no trouble discerning the
difference. And since any good listening test involving musical
passages allows the subject to control the switch, you'll have no
trouble finding other portions of the passage where the differences are
obvious.
Now, you're going to ask, Okay, but what if I'm only given the short
snippet where the levels are the same? In that case, you will hear no
difference. But if that's all you're given, then you can't extrapolate
from that to the remainder of the signal. A listening test is only
valid for what you're listening to.
> just as in the visual case?
>
> Does it matter how the short portions are "juxtaposed"? Separated by
> silence or one followed continously by the other?
A silence of more than a few seconds (maybe even less) will doom the
test.
bob
nab...@hotmail.com
> OK, is this the idea.
>
> We want to prove that two sources provide the same information to the
> listener in the context of ordinary use.
>
> We appeal now to the following Principle:
>
> If two sources provide the same information in ordinary use, then in a
> test in which small, corresponding portions are compared, the listener
> will not be able to discriminate them.
>
> In particular, the type of discrimination task is the following: on any
> trial, two short excerpts are played, either both from the same source
> (a repetition) or from different sources, and the listener has to say
> if they're from the same or different sources, and he passes the test
> if he says same or different better than chance.
>
> Two observations. First of all, in a test like this there is plenty of
> information coming from either source that may not make it to the
> listener,
No, there isn't. The only information coming to the listener is the
information that's actually coming to the listener. If you're comparing
two short snippets, that's all you're comparing, and you can't
extrapolate from that to anything that isn't part of the test.
Since the subject of this thread is "validity of audio tests," let me
be clear: Listening tests are valid for what you're listening to. They
can't tell you anything about what you're not listening to. So please
give up this sophistry.
bob
nab...@hotmail.com
> On 22 Jun 2005 03:05:31 GMT, nab...@hotmail.com wrote:
>
> >Mark DeBellis wrote:
> >>
> >> ... if the information is still doing work somewhere in your cognitive
> >> economy, even though it can't be brought to consciousness, or is not
> >> specific enough to enable one to perform the identification task.
> >
> >But it isn't, because of our short aural memory for partial loudness
> >differences.
> >
>
> Doesn't it follow from what you say that musical form can make no
> difference? When I am listening to Theme B, it makes no difference
> what the character of Theme A was, what motives made it up, and so on.
> If it doesn't follow from what you say, why not?
The difference between two themes is not a partial loudness difference!
Do you even know what a partial loudness difference is? I thought I was
arguing with someone who at least had a rudimentary understanding of
what he was talking about.
> If this is what psychoacoustics teaches us, I'm moving to Kansas!
I'd say you're already there in spirit.
bob
Mark DeBellis
wrote:
Excellent question. I would say a statement like that is confirmed if
it is part of the best explanation we have of the data, in other
words, part of the best psychological theory we have. (I use the term
"confirmed," not proved, because confirmation is never final or
certain.) For psychology, the data consists of behavior, as well as
other things that are observed.
Psychologists postulate unconscious representation all the time. You
don't have to be conscious of something in order to have perceived it;
so that is not a necessary criterion.
If you are saying, if a certain hypothesis about somebody's mental
states has no implications whatever for behavior then it has no real
content, I basically agree with that. But I think that to suppose
that someone can perceive something and then not have a memory of it
does not run afoul of that principle, so long as there are *some*
connections with behavior.
Here is an everyday example. Did I say "Unconscious representation is
postulated by psychologists," or "Psychologists postulate unconscious
representation," in the first sentence of the second paragraph above?
You remember that I said something like that but if you are like me
you have to go back and check the formulation to see which exactly it
was. But surely you parsed the sentence when you read it earlier and
had some mental representation of its syntax, though you did not
retain a memory of the exact formulation. (The reasons for thinking
you parsed the sentence are of a general nature: you regularly
comprehend what you read; how could that happen if you did not
regularly parse the sentences?) What I am supposing is going on with
music is something along these lines.
Mark
Mark DeBellis
my claim, as you seem to intend it, then much will follow about the
futility of musical form.
But really, it's OK. I think I answered my own question, along lines I
have reported in other posts. If anyone cares to make a substantive
response to them, to confirm or correct, that will be appreciated.
Gary Eickmeier
> An auditory example would be tempo. Suppose I am listening to two
> sources, where the only difference is that one of them has a speed of
> 1.01 times the other. If I listen to short excerpts any difference is
> below the just-noticeable-difference, but if the whole example is the
> Ring cycle, I will notice that one finishes before dark and the other
> doesn't, I get hungry during one but not the other, etc.
>
> So even if quick-switch tests, on balance, are the most sensitive, that
> doesn't mean there can't be things out there that don't get caught in
> their net (though they may be detectable in other ways).
>
> To come back to the SACD/CD example, my concern is whether, even if the
> quick-switch test were a "null," there could be differences that the
> test does not do a good job of proving the existence of. Rather than
> feel assured that science tells us there could not be such differences,
> it seems to me pretty apparent that every test has its limitations.
>
> Sound plausible?
You can't do a "quick switch" test with two sources that run at
different speeds because you can't synchronize them, which would be a
dead giveaway in itself, so that is a bad example.
If you want to use that example, you will have to listen first to one,
then the other, in its entirety, then decide if the speed difference is
audible. If so, then do a blind series, listening to a known version,
then to a randomly chosen one, and decide whether it is the same or
different. In this manner you will eventually arrive at a number for a
speed differential that is at the audible threshold. That is the basic
idea of how audio research is done. You may find that speed differences
of 1.01 will be inaudible to most, but audible to some with perfect
pitch. If this is interesting enough a question for you, then do the
research and report it.
Gary Eickmeier
Mark DeBellis
Exactamundo. The research may have not been intended to answer
certain questions, so it can't be looked to for definitive answers
about them. And it would be circular to argue that those questions
don't matter because everything that's relevant has already been
treated by science.
But look, whatever anyone thinks of my original question or subsequent
meanderings, I think it is apparent that there is a need to explain
better, to non-experts like me, how audio tests work, in the sense of
what they demonstrate and how they demonstrate it, what their logic
and rationale are, what the structure of the reasoning is that leads
from data to conclusions. I started with what seemed to me a prima
facie problem about the SACD/CD test I undertook. It seemed to me
there was a reason to think that the outcome of that particular test
fails to demonstrate that there is no difference between what I hear
in SACD and what I hear in CD (and hence that there is no sonic
advantage to SACD). If someone wants to tell me that the test does
demonstrate that, then I would be interested to know why what I think
is a reason, an obstacle, is not in fact a good reason. My own
assessment of this is that I performed the wrong kind of test, and
this is worth pointing out because it is very easy to assume that the
test does demonstrate said conclusion, since it is very easy not to
notice the difference between that test and other, better tests.
There is an initial plausibility to the idea that the failure to
identify SACD vs. CD means that one can't sound better than the other.
But I think that plausibility is illusory and betrays a lack of
clarity, from which I was certainly not myself immune, about what the
test really demonstrates and why.
Mark
Harry Lavo
> Mark DeBellis wrote:
>
>> OK now the auditory case. There is a signal that gets louder, and I
>> can hear the difference between that and a signal of constant loudness.
>> The principle says that I can do this only if I can hear the
>> difference in loudness between short corresponding portions.
>>
>> Seems plausible ... and if the portions get to be too short then would
>> reliability go down,
>
> No. There is only one point at which the volume of the two passages is
> identical, so unless you are listening at exactly that point, and only
> that point, the two are going to sound different to you, no matter how
> short the samples are. So unless you're extremely unlucky in your
> choice of snippets to compare, you'll have no trouble discerning the
> difference. And since any good listening test involving musical
> passages allows the subject to control the switch, you'll have no
> trouble finding other portions of the passage where the differences are
> obvious.
>
> Now, you're going to ask, Okay, but what if I'm only given the short
> snippet where the levels are the same? In that case, you will hear no
> difference. But if that's all you're given, then you can't extrapolate
> from that to the remainder of the signal. A listening test is only
> valid for what you're listening to.
>
Of course, this presumes that the brain understands and processes, in
conjunction with the ear, exactly the same thing in short-snippet,
qucik-switching, as it does in longer term contextual listening and
post-evaluation. That has never been done to verify that such is the case.
nab...@hotmail.com
It presumes no such thing. Indeed, any such presumption would be
absurd. It's actually more plausible to believe that the reason the
brain has a harder time distinguishing audible sounds over longer time
intervals is precisely because it is processing the sounds differently.
But it is demonstrably true that the brain has a harder time
distinguishing between audible sounds as the time interval between them
increases. There's not even a shred of evidence to the contrary.
bob
Buster Mudd
attempting to discuss auditory memory as it applies to musical themes,
motifs, and forms...whereas most of the other folks here have been
responding as if the discussion were about auditory memory as it
applies to identifying sonic differences between audio components.
Intuitively (for whatever *that's* worth!) it strikes me that there's a
fundamental difference between these two types of auditory memories
that may explain the obstacles to understanding one another:
There is a certain amount of "content" in a musical composition that is
objectively verifiable: e.g., Theme B is an inversion of Theme A, or,
this passage is in the relative minor key of the opening statement,
etc. These types of information can be easily confirmed by anyone with
a copy of the score. And these are examples of the sort of content that
can most definitely be percieved subconciously but have an affect on
one's subsequent perception of a musical passage. I don't think anyone
would disagree that this type of auditory memory exists; it's the
entire basis upon which Western Art Music was founded!
But the whole point of performing a valid audio test to determine
whether or not there exists a difference between two audio components
is to find out if there *is* any objectively verifiable "content".
We're not asking whether or not the subject has retained a memory
(conciously or otherwise) of This Thing; we're performing these tests
to first determine whether or not This Thing even exists.
Buster Mudd
> On 23 Jun 2005 00:09:36 GMT, "Buster Mudd" <mr_fu...@mail.com>
> wrote:
>
> >Mark DeBellis wrote:
> >> "Buster Mudd" <mr_fu...@mail.com> wrote:
> >>
> >> >Hence, the distinction between whether they
> >> >actually perceived it & then forgot it, or never perceived it in the
> >> >first place, is moot.
> >>
> >> Not if the information is still doing work somewhere in your cognitive
> >> economy, even though it can't be brought to consciousness, or is not
> >> specific enough to enable one to perform the identification task.
> >>
> >
> >
> >How would you go about *proving* that something was in someone's
> >"cognitive economy" if that someone was not conscious of that
> >something? How would you go about *proving* that something was in
> >someone's "cognitive economy" if that something could not enable that
> >someone to perform a task, any task?
>
> Excellent question. I would say a statement like that is confirmed if
> it is part of the best explanation we have of the data, in other
> words, part of the best psychological theory we have.
But that sounds suspiciously like "Philosopher's Syndrome": mistaking a
failure of imagination for an insight into necessity. I'm under the
impression that no self-respecting scientist would ever claim they'd
"confirmed" anything if it is simply the best explanation they have of
the data. All they've done is "postulated" a theory or "presumed" a
solution, which they would then seek to "confirm" via reproduceable
experiment.
> For psychology, the data consists of behavior, as well as
> other things that are observed.
>
> Psychologists postulate unconscious representation all the time. You
> don't have to be conscious of something in order to have perceived it;
> so that is not a necessary criterion.
>
But in order for a psychologist to postulate unconscious representation
they need to observe something in a subject's behavior that suggests
that Perceived-But-Not-Brought-To-Consciousness thing *was* affecting
the subject's cognitive economy. This gets right back to my previous
question: How would you go about *proving* (confirming? demonstrating?)
that something was in someone's "cognitive economy" if that something
could not enable that someone to perform a task? Relying on the "it
happens all the time so why shouldn't it be happening now" school of
philosophy is extremely suspect.
Harry Lavo
Sure it presumes exactly what I mentioned. You are presuming that if
something "disappears" when the time interval is lengthened, then it is not
important. You have no evidence for this whatsoever unless you can show
that the results of the short snippet, quick-switch test give the same
results as a longer-term, monadic, cross-population, post-listening-analysis
test. Such a test does not rely on one person's aural memory; instead it
measures response to the full longer term set of stimulae across the
population, and uses statistics to determine signifcant difference or lack
of same. Yet this test more closely duplicates the actual listening
conditions under which people use the equipment and listen to the music. If
the quick-switch, snippet test duplicates the results within reason, then
fine, you've got a winner. But if it doesn't, then the basic assumption
that the test is valid for the open-ended evaluation of musical reproduction
has been an error, and the testing is worthless for this purpose.
nab...@hotmail.com
> Exactamundo. The research may have not been intended to answer
> certain questions, so it can't be looked to for definitive answers
> about them.
But the research was intended to answer exactly the questions you are
asking. If you want to know how to determine whether two things sound
different, there are certain tests that are known to be reliable for
doing this. The only reason you are arguing this is that you don't like
what those tests tell you. I'm sorry we can't rearrange the laws of the
physical universe to your wishes, but we can't.
> And it would be circular to argue that those questions
> don't matter because everything that's relevant has already been
> treated by science.
>
> But look, whatever anyone thinks of my original question or subsequent
> meanderings, I think it is apparent that there is a need to explain
> better, to non-experts like me, how audio tests work, in the sense of
> what they demonstrate and how they demonstrate it, what their logic
> and rationale are, what the structure of the reasoning is that leads
> from data to conclusions.
You can't understand any of this unless you have a decent grasp of the
basics of psychoacoustics and the physics of sound, which your recent
posts indicate that you do not have. So the very first thing you should
do is pick up a textbook or two actually read the things. Then you'll
be able to pose informed questions, instead of throwing up uninformed
speculation.
> I started with what seemed to me a prima
> facie problem about the SACD/CD test I undertook.
Yeah, it didn't give you the result you wanted.
> It seemed to me
> there was a reason to think that the outcome of that particular test
> fails to demonstrate that there is no difference between what I hear
> in SACD and what I hear in CD (and hence that there is no sonic
> advantage to SACD).
NO test can demonstrate that there is no difference. Any such test has
one of two possible outcomes:
1) it can demonstrate that there IS a difference;
2) it can fail to determine whether there is or is not a difference.
Of course, if you keep getting result #2, that should tell you
something.
(This, by the way, is another very basic concept which you have not yet
grasped. I don't point this out to belittle you, but to indicate
further that your lack of background knowledge is hindering your
ability to understand what people are saying to you. So please take my
advice and read up a little.)
> If someone wants to tell me that the test does
> demonstrate that, then I would be interested to know why what I think
> is a reason, an obstacle, is not in fact a good reason.
I believe several people explained why your reasoning was faulty.
Rather than engage them, you've simply persisted in posting the same
question over and over again.
> My own
> assessment of this is that I performed the wrong kind of test, and
> this is worth pointing out because it is very easy to assume that the
> test does demonstrate said conclusion, since it is very easy not to
> notice the difference between that test and other, better tests.
> There is an initial plausibility to the idea that the failure to
> identify SACD vs. CD means that one can't sound better than the other.
> But I think that plausibility is illusory and betrays a lack of
> clarity, from which I was certainly not myself immune, about what the
> test really demonstrates and why.
The lack of clarity is entirely yours. I've suggested a remedy.
bob
Harry Lavo
> Mark DeBellis wrote:
>snip<
>
>> I started with what seemed to me a prima
>> facie problem about the SACD/CD test I undertook.
>
> Yeah, it didn't give you the result you wanted.
>
>> It seemed to me
>> there was a reason to think that the outcome of that particular test
>> fails to demonstrate that there is no difference between what I hear
>> in SACD and what I hear in CD (and hence that there is no sonic
>> advantage to SACD).
>
> NO test can demonstrate that there is no difference. Any such test has
> one of two possible outcomes:
> 1) it can demonstrate that there IS a difference;
> 2) it can fail to determine whether there is or is not a difference.
>
> Of course, if you keep getting result #2, that should tell you
> something.
>
Yes, either the test is wrong or you need more trials. :-)
>snip<
nab...@hotmail.com
> <nab...@hotmail.com> wrote in message news:d9fuo...@news2.newsguy.com...
> > Harry Lavo wrote:
> >>
> >> Of course, this presumes that the brain understands and processes, in
> >> conjunction with the ear, exactly the same thing in short-snippet,
> >> qucik-switching, as it does in longer term contextual listening and
> >> post-evaluation.
> >
> > It presumes no such thing. Indeed, any such presumption would be
> > absurd. It's actually more plausible to believe that the reason the
> > brain has a harder time distinguishing audible sounds over longer time
> > intervals is precisely because it is processing the sounds differently.
> > But it is demonstrably true that the brain has a harder time
> > distinguishing between audible sounds as the time interval between them
> > increases. There's not even a shred of evidence to the contrary.
> >
>
> Sure it presumes exactly what I mentioned. You are presuming that if
> something "disappears" when the time interval is lengthened, then it is not
> important.
Not only am I not presuming this, but I have never said that *anything*
disappears. Must you misrepresent what I say in order to argue with me?
> You have no evidence for this whatsoever unless you can show
> that the results of the short snippet, quick-switch test give the same
> results as a longer-term, monadic, cross-population, post-listening-analysis
> test.
Why in the world should I have to show that a listening test commonly
used by the leading experts in the field gives the same results as one
that has never, ever been used to test whether two sounds are audibly
different? That's preposterous. It seems to me that the burden of proof
rests with you. Show us that your "test" even works at all.
> Such a test does not rely on one person's aural memory;
Of course it does. You listen to a sample. Then you answer a few
questions about your impressions of that sample. How do you remember
what your impressions were? That's aural memory, Harry. And it works
the same in your "test" as in the real ones.
bob
Harry Lavo
> Harry Lavo wrote:
>> <nab...@hotmail.com> wrote in message
>> news:d9fuo...@news2.newsguy.com...
>> > Harry Lavo wrote:
>> >>
>> >> Of course, this presumes that the brain understands and processes, in
>> >> conjunction with the ear, exactly the same thing in short-snippet,
>> >> qucik-switching, as it does in longer term contextual listening and
>> >> post-evaluation.
>> >
>> > It presumes no such thing. Indeed, any such presumption would be
>> > absurd. It's actually more plausible to believe that the reason the
>> > brain has a harder time distinguishing audible sounds over longer time
>> > intervals is precisely because it is processing the sounds differently.
>> > But it is demonstrably true that the brain has a harder time
>> > distinguishing between audible sounds as the time interval between them
>> > increases. There's not even a shred of evidence to the contrary.
>> >
>>
>> Sure it presumes exactly what I mentioned. You are presuming that if
>> something "disappears" when the time interval is lengthened, then it is
>> not
>> important.
>
> Not only am I not presuming this, but I have never said that *anything*
> disappears. Must you misrepresent what I say in order to argue with me?
>
I'm sorry, but the entire gist of your response has been to this effect.
Othes here can judge your disclaimer for themselves.
>> You have no evidence for this whatsoever unless you can show
>> that the results of the short snippet, quick-switch test give the same
>> results as a longer-term, monadic, cross-population,
>> post-listening-analysis
>> test.
>
> Why in the world should I have to show that a listening test commonly
> used by the leading experts in the field gives the same results as one
> that has never, ever been used to test whether two sounds are audibly
> different? That's preposterous. It seems to me that the burden of proof
> rests with you. Show us that your "test" even works at all.
>
My test is a standard research test, used broadly all over the world in many
fields. The ABX is an audio-specific test. Moreover, it is used, as the
information re: Harmon Kardon's current use of it. It is much less
controversial test on the face of it, since it minimizes disruption of
normal listening patterns. The only thing one might fault it for is
sensitivity, but with this type of test sensitivity is simply a matter of
numbers...need more sensitivity, add more people.
>> Such a test does not rely on one person's aural memory;
>
> Of course it does. You listen to a sample. Then you answer a few
> questions about your impressions of that sample. How do you remember
> what your impressions were? That's aural memory, Harry. And it works
> the same in your "test" as in the real ones.
That is not aural memory, in the sense that you need it in an ABX test.
That is a recall of total impression, involving the brain and the emotions.
And that kind of memory is *very* recallable. It is also subjective. But
that is how we respond to and understand music (and musical reproduction).
It is objectified via the use of statistics. You don't object to that use,
do you?
nab...@hotmail.com
> My test is a standard research test, used broadly all over the world in many
> fields.
But never to test for *difference*. It would be an absolutely terrible
test for difference. And just because one researcher used it one time
for that purpose and got the result he (and you) wanted doesn't change
that.
> The ABX is an audio-specific test. Moreover, it is used, as the
> information re: Harmon Kardon's current use of it.
Not for difference.
> It is much less
> controversial test on the face of it, since it minimizes disruption of
> normal listening patterns.
ABX is "controversial" the way evolution is controversial. There is
science, and then there are peole who wish that science were not so.
> The only thing one might fault it for is
> sensitivity, but with this type of test sensitivity is simply a matter of
> numbers...need more sensitivity, add more people.
As a test for difference, more numbers wouldn't solve the sensitivity
problem. You're talking about a test that couldn't determine that LP
and CD are sonically different! And you think such a test should be the
gold standard for judging the accuracy of all other difference tests?
bob
Mark DeBellis
wrote:
>You can't do a "quick switch" test with two sources that run at
>different speeds because you can't synchronize them, which would be a
>dead giveaway in itself, so that is a bad example.
I don't see why it would be impossible to synchronize them if they are
both files on your computer and there is a program that, when you
switch back and forth, points you to the right place in the other
file.
If on the other hand playback is continuous for either source and it
has to be started once and for all, then, yes, it would be impossible
to synchronize them.
>
>If you want to use that example, you will have to listen first to one,
>then the other, in its entirety, then decide if the speed difference is
>audible. If so, then do a blind series, listening to a known version,
>then to a randomly chosen one, and decide whether it is the same or
>different. In this manner you will eventually arrive at a number for a
>speed differential that is at the audible threshold. That is the basic
>idea of how audio research is done. You may find that speed differences
>of 1.01 will be inaudible to most, but audible to some with perfect
>pitch. If this is interesting enough a question for you, then do the
>research and report it.
Thanks, that is a helpful explanation. But I am not an audio
researcher, just someone who is trying to understand what the research
and the debates are about.
Mark
Harry Lavo
> Harry Lavo wrote:
>> My test is a standard research test, used broadly all over the world in
>> many
>> fields.
>
> But never to test for *difference*. It would be an absolutely terrible
> test for difference. And just because one researcher used it one time
> for that purpose and got the result he (and you) wanted doesn't change
> that.
>
Absolutely it is used to test difference. If it doesn't measure
statistically significant, the difference hypothesis is not supported. Try
telling a drug company it doesn't measure difference from a placebo!
One of its strengths is that it goes beyond difference to get at where and
how and why there is a difference. But only if there is a statistical
difference.
>> The ABX is an audio-specific test. Moreover, it is used, as the
>> information re: Harmon Kardon's current use of it.
>
> Not for difference.
>
Of course for difference. They are profiling speakers against the profile
of know reference speakers. If there is no statistical difference, any
differences in profile are useless (in a scientific sense). Of course, the
Harmon people may set a lower statistical standard for such testing because
they find even directional information helpful. But that is always a
standard development dilemma.
>> It is much less
>> controversial test on the face of it, since it minimizes disruption of
>> normal listening patterns.
>
> ABX is "controversial" the way evolution is controversial. There is
> science, and then there are peole who wish that science were not so.
>
Once again, name calling, however subtle in nature. I am not anti-science.
I am against bad science parading as good science.
>> The only thing one might fault it for is
>> sensitivity, but with this type of test sensitivity is simply a matter of
>> numbers...need more sensitivity, add more people.
>
> As a test for difference, more numbers wouldn't solve the sensitivity
> problem. You're talking about a test that couldn't determine that LP
> and CD are sonically different! And you think such a test should be the
> gold standard for judging the accuracy of all other difference tests?
Believe me Bob, it tests difference. But it tests subjective musical
impression difference, not sound pressure levels. Can *you* see a
difference?
Mark DeBellis
wrote:
>Mark DeBellis wrote:
>> On 23 Jun 2005 00:09:36 GMT, "Buster Mudd" <mr_fu...@mail.com>
>> wrote:
>> >How would you go about *proving* that something was in someone's
>> >"cognitive economy" if that someone was not conscious of that
>> >something? How would you go about *proving* that something was in
>> >someone's "cognitive economy" if that something could not enable that
>> >someone to perform a task, any task?
>>
>> Excellent question. I would say a statement like that is confirmed if
>> it is part of the best explanation we have of the data, in other
>> words, part of the best psychological theory we have.
>
>
>But that sounds suspiciously like "Philosopher's Syndrome": mistaking a
>failure of imagination for an insight into necessity. I'm under the
>impression that no self-respecting scientist would ever claim they'd
>"confirmed" anything if it is simply the best explanation they have of
>the data. All they've done is "postulated" a theory or "presumed" a
>solution, which they would then seek to "confirm" via reproduceable
>experiment.
I don't think we basically disagree, though maybe our terminology is
different. A theory's having the status of being the best explanation
is not prior to experiment. The data include the results of
experiments. And confirmation is always provisional, always open to
revision by further observations. (In addition to the excellent book
you have mentioned already, I would add to the mix The Logic of
Scientific Discovery by Karl Popper, which is relevant to these
particular points.)
>
>> For psychology, the data consists of behavior, as well as
>> other things that are observed.
>>
>> Psychologists postulate unconscious representation all the time. You
>> don't have to be conscious of something in order to have perceived it;
>> so that is not a necessary criterion.
>>
>
>But in order for a psychologist to postulate unconscious representation
>they need to observe something in a subject's behavior that suggests
>that Perceived-But-Not-Brought-To-Consciousness thing *was* affecting
>the subject's cognitive economy. This gets right back to my previous
>question: How would you go about *proving* (confirming? demonstrating?)
>that something was in someone's "cognitive economy" if that something
>could not enable that someone to perform a task?
Well, just as you say, by observing behavior that, together with
everything else that is observed, is best explained by that
hypothesis, in the context of a larger theory.
If by a "task" you mean a behavior that constitutes a "crucial
experiment"--if we see the behavior then we have conclusively proved
the hypothesis, and if we do not then we have conclusively refuted
it--then I think normally there isn't any such thing. A theory in
psychology, like any theory, "faces the tribunal of experience" as a
whole, not statement by statement (Quine, Duhem). Confirmation in
science is holistic.
Mark
Mark DeBellis
wrote:
>Mark, I've noticed a few of your posts in this thread have been
Hi Buster, that's interesting. What do you think the relation is
between these two types of auditory memory? I am not sure whether to
read you as saying that the things being verified by audio tests are
the same things you refer to in the previous paragraph (musical
"content") or different kinds of things.
Mark
jjn...@sonic.net
Your continued insistance that 'loudness' and 'partial loudness' are the same
(because you apparently continue to refuse to study and therefore understand
the subject) really does not do your 'cause' any good whatsoever. If you want
to debunk a theory, you need to know what the theory is. You don't.
Mark DeBellis
identification ("Was that SACD or CD?") test need be the same as those
of a discrimination ("Are A and B the same or different?") test. Does
the research show, in particular, that an identification test (the
kind I undertook) is among the kinds of tests that are reliable for
determining whether two sources sound different?
Mark
nab...@hotmail.com
> <nab...@hotmail.com> wrote in message news:d9mh...@news3.newsguy.com...
> > Harry Lavo wrote:
> >> My test is a standard research test, used broadly all over the world in
> >> many
> >> fields.
> >
> > But never to test for *difference*. It would be an absolutely terrible
> > test for difference. And just because one researcher used it one time
> > for that purpose and got the result he (and you) wanted doesn't change
> > that.
> >
>
> Absolutely it is used to test difference. If it doesn't measure
> statistically significant, the difference hypothesis is not supported. Try
> telling a drug company it doesn't measure difference from a placebo!
Drug companies aren't measuring difference. They're measuring
effectiveness. Does this antibiotic cure infections? Does this cancer
drug reduce the size of tumors? Measuring difference would be: Does
this drug have any physiological effect on the body whatsoever?
And, no, you cannot use a drug test as a difference test, because you
cannot presume that a drug that fails to cure an infection therefore
had no physiological effect.
BTW, now you're talking about monadic tests. Previously, you've touted
what you called "proto-monadic" tests, a la Oohashi. Which is it,
Harry? You say we need a reference against which to compare ABX's
results, but you can't even agree within your own mind about what that
test should be.
> One of its strengths is that it goes beyond difference to get at where and
> how and why there is a difference. But only if there is a statistical
> difference.
>
> >> The ABX is an audio-specific test. Moreover, it is used, as the
> >> information re: Harmon Kardon's current use of it.
> >
> > Not for difference.
> >
>
> Of course for difference.
Oh, this is rich. Now you're touting Harman as the model of your
monadic ideal. Harman doesn't use monadic tests. They use
QUICK-SWITCHING preference tests (not difference tests!). That's why
they built their listening lab, Harry--so they could do quick-switching
tests with speakers. I presume you're now going to tell us that their
findings aren't validated.
> They are profiling speakers against the profile
> of know reference speakers. If there is no statistical difference, any
> differences in profile are useless (in a scientific sense). Of course, the
> Harmon people may set a lower statistical standard for such testing because
> they find even directional information helpful. But that is always a
> standard development dilemma.
>
> >> It is much less
> >> controversial test on the face of it, since it minimizes disruption of
> >> normal listening patterns.
> >
> > ABX is "controversial" the way evolution is controversial. There is
> > science, and then there are peole who wish that science were not so.
> >
>
> Once again, name calling, however subtle in nature. I am not anti-science.
> I am against bad science parading as good science.
Hey, you're the one who started with the religious insults. And anyone
who professes the authority to "validate" science by ignoring the parts
he just doesn't like is anti-science, in my book.
> >> The only thing one might fault it for is
> >> sensitivity, but with this type of test sensitivity is simply a matter of
> >> numbers...need more sensitivity, add more people.
> >
> > As a test for difference, more numbers wouldn't solve the sensitivity
> > problem. You're talking about a test that couldn't determine that LP
> > and CD are sonically different! And you think such a test should be the
> > gold standard for judging the accuracy of all other difference tests?
>
> Believe me Bob, it tests difference. But it tests subjective musical
> impression difference, not sound pressure levels. Can *you* see a
> difference?
Yeah, and that's why ABX is better. ABX doesn't JUST test for "musical
impression," it tests for any difference at all. And you can't identify
a single audible sonic difference that ABX tests can't distinguish.
As for your own "proposed" test, which is it:
1) monadic, like the pharmaceutical companies?
2) proto-monadic, like Oohashi?
3) quick-switching preference, like Harman?
Or does it depend?
Actually, it doesn't. None of those three tests could be used to
confirm an audible difference between LP and CD. An ABX test could do
so easily. So much for validation.
bob
Mark DeBellis
>Can you name some sonic distinction that isn't a partial loudness
>difference?
>The difference between two themes is not a partial loudness difference!
I would infer that on your view the difference between two themes is
not a sonic distinction.
But the difference between two themes is audible, yes? So "sonic
distinction" does not mean the same as "audible difference."
What exactly do you mean by "sonic distinction" and how does it differ
in meaning from "audible difference"?
>You can't understand any of this unless you have a decent grasp of the
>basics of psychoacoustics and the physics of sound, which your recent
>posts indicate that you do not have. So the very first thing you should
>do is pick up a textbook or two ...
That may prove necessary, but we have already begun the dialogue and
my curiosity is piqued. Sometimes it helps to address specific things
that puzzle a person. Please explain it if you like. No pressure.
Mark
Mark DeBellis
>Mark DeBellis wrote:
>nab...@hotmail.com wrote:
>If you want to know how to determine whether two things sound
>different, there are certain tests that are known to be reliable for
>doing this.
>DeBellis:
>> ... the outcome of that particular test
>> fails to demonstrate that there is no difference ...
>
>nab...@hotmail.com:
>NO test can demonstrate that there is no difference. Any such test has
>one of two possible outcomes:
>1) it can demonstrate that there IS a difference;
>2) it can fail to determine whether there is or is not a difference.
>
>Of course, if you keep getting result #2, that should tell you
>something.
>
>(This, by the way, is another very basic concept which you have not yet
>grasped. I don't point this out to belittle you, but to indicate
>further that your lack of background knowledge is hindering your
>ability to understand what people are saying to you. So please take my
>advice and read up a little.)
You write that there are tests that are reliable for determining
"whether two things sound different," so it is you, not I, who fails
to keep your later point firmly in mind.
What I say is that the outcome of the test fails to demonstrate that
there is a difference, and that is consistent with what you say in
(2). It is you who are inconsistent.
Mark
vlad
It was few weeks ago when you described your "monadic" test first time
in this group. Now you are talking about this test as an established
fact.
Even if somebody would go into a hassle and an expense of
implementing your suggestion it is not at all obvious that the test
would produce any results. I think that most likely outcome that it
would find your subjective terms like "warmth", "depth", etc. not
correlated to the sound of the recording. I would bet that the
distribution of particular term would be completely random for
different users.
But of course then you would require not 200 participants but 2000,
etc. or something that again will make a proposed test unfeasible. And
you will continue speculate about validity of your imaginary test.
Please, either provide some proof that you so-called "monadic" test
works or stop speculating about it.
vlad
Mark DeBellis
wrote:
>
>You can't do a "quick switch" test with two sources that run at
>different speeds because you can't synchronize them, which would be a
>dead giveaway in itself, so that is a bad example.
>
>If you want to use that example, you will have to listen first to one,
>then the other, in its entirety, then decide if the speed difference is
>audible. If so, then do a blind series, listening to a known version,
>then to a randomly chosen one, and decide whether it is the same or
>different. In this manner you will eventually arrive at a number for a
>speed differential that is at the audible threshold. That is the basic
>idea of how audio research is done. You may find that speed differences
>of 1.01 will be inaudible to most, but audible to some with perfect
>pitch. If this is interesting enough a question for you, then do the
>research and report it.
>
p.s. Suppose one carried out research such as this and found, for a
given one-minute-long excerpt, what is the audible threshold. So a
given subject could reliably discriminate between the excerpt and a
version that is 1.01 as fast (say). What theoretical reason would we
have to think that, if we did a quick switch test (see my previous
email for a suggestion about how to do it), the subject would be able
to tell the excerpts apart in that test?
I don't understand the point about perfect pitch, because I am
supposing that one version is faster than the other, not that the
speed and pitch are both higher (as would be the case with analog
tape). Maybe I am not seeing your point though.
Mark
Gary Eickmeier
I'm not sure what you mean by "identification test." There is no such
paradigm in what I have read. It is much more difficult to listen to a
randomly selected source and try to "identify" it than to compare two
sources and decide "same" or "different." In an ABX test, for example,
you can listen to the two known sources as long as you want, switch back
and forth between them and listen for differences, see if you can get a
"fix" on just what each sounds like, then go for a test. In the test,
you would select A or B, then let the comparator select X, and decide
whether X is A or B. You usually do this by quick switching between A
and X, then B and X, and deciding same or different. If X is same as A,
then you put A as the identification of it, and press on to trial 2.
If the differences are really audible, the trials will be child's play.
If they sound identical, you will be guessing and probably know it.
Anyway, the task is to decide same or different, not to identify the
source when presented with a single signal.
Gary Eickmeier
Harry Lavo
> Harry Lavo wrote:
>> <nab...@hotmail.com> wrote in message
>> news:d9mh...@news3.newsguy.com...
>> > Harry Lavo wrote:
>> >> My test is a standard research test, used broadly all over the world
>> >> in
>> >> many
>> >> fields.
>> >
>> > But never to test for *difference*. It would be an absolutely terrible
>> > test for difference. And just because one researcher used it one time
>> > for that purpose and got the result he (and you) wanted doesn't change
>> > that.
>> >
>>
>> Absolutely it is used to test difference. If it doesn't measure
>> statistically significant, the difference hypothesis is not supported.
>> Try
>> telling a drug company it doesn't measure difference from a placebo!
>
> Drug companies aren't measuring difference. They're measuring
> effectiveness. Does this antibiotic cure infections? Does this cancer
> drug reduce the size of tumors? Measuring difference would be: Does
> this drug have any physiological effect on the body whatsoever?
>
And how do they measure effectiveness. By the *difference* in physical
phenomenon, and by the *difference* in self-reported behavior, that's how.
When the drug company says this drug reduces the risk of heart attack by
25%, it has found a statistically significant difference between the samples
of a magnitude of at least 25%. Likewise, if 200 people rate amp "A" as a
4.0 on a five-point scale with regard to naturalness in reproducing violins,
and another 200 people rate amp "B" as a 4.3 on this same scale, and the
statistical test for difference indicates that this is statistically
significant at some level (usually 95%), then it can be said that Amp "B"
has a more natural violin sound than Amp "A". The statistical test used is
specific to the scalar technology used.
There is no real difference between the drug comapny tests and the musical
tests, except that the drug tests have objective incidences to meaure (as
well, often, as subjective) whereas the music reproduction test is clearly
all subjective. But that is a result of the fact that music itself is
subjective, and *cannot* be measured objectively. The closest you can come
perhaps is to substitute some kind of psychophysiological measurements.
> And, no, you cannot use a drug test as a difference test, because you
> cannot presume that a drug that fails to cure an infection therefore
> had no physiological effect.
>
Sure you can. If the drug and the placebo (or the control drug) yield the
same incidence of effective cure, there is no difference. The drug doesn't
work (placebo) or at least work any better (control drug). Simple as that.
> BTW, now you're talking about monadic tests. Previously, you've touted
> what you called "proto-monadic" tests, a la Oohashi. Which is it,
> Harry? You say we need a reference against which to compare ABX's
> results, but you can't even agree within your own mind about what that
> test should be.
Monadic tests are the gold standard because their is no test order bias, but
they require the largest sample size and are those fairly impractical except
for deep-pocketed, major studies. Proto monadic *is* a monadic test, with a
*comparative* tagged directly on the end. Thus one can often get by with
smaller sample sizes because there is yet a second measure of difference.
At least some researchers favor it for this reason under some circumstances.
On the other hand, proto-monadic testing has a strong order bias that has
to be controlled. It's a judgement call, but either are preferable to
quick-switch, comparative testing IMO because they intrude less into normal
listening patterns. Both rely on after-the-fact recall and rating.
> One of its strengths is that it goes beyond difference to get at where and
>> how and why there is a difference. But only if there is a statistical
>> difference.
>>
>> >> The ABX is an audio-specific test. Moreover, it is used, as the
>> >> information re: Harmon Kardon's current use of it.
>> >
>> > Not for difference.
>> >
>>
>> Of course for difference.
>
> Oh, this is rich. Now you're touting Harman as the model of your
> monadic ideal. Harman doesn't use monadic tests. They use
> QUICK-SWITCHING preference tests (not difference tests!). That's why
> they built their listening lab, Harry--so they could do quick-switching
> tests with speakers. I presume you're now going to tell us that their
> findings aren't validated.
>
Did you miss John Atkinson's recent posts that upon his recent visit to
Harmon's research facitility, they had switched their speaker testing to
monadic, evaluative testing using rating scales? I'm not holding them up
as a model for anything, simply commenting that they are at least one
company using monadic testing in the audio field. Which refutes your claim
that none are.
>> They are profiling speakers against the profile
>> of know reference speakers. If there is no statistical difference, any
>> differences in profile are useless (in a scientific sense). Of course,
>> the
>> Harmon people may set a lower statistical standard for such testing
>> because
>> they find even directional information helpful. But that is always a
>> standard development dilemma.
>>
>> >> It is much less
>> >> controversial test on the face of it, since it minimizes disruption of
>> >> normal listening patterns.
>> >
>> > ABX is "controversial" the way evolution is controversial. There is
>> > science, and then there are peole who wish that science were not so.
>> >
>>
>> Once again, name calling, however subtle in nature. I am not
>> anti-science.
>> I am against bad science parading as good science.
>
> Hey, you're the one who started with the religious insults. And anyone
> who professes the authority to "validate" science by ignoring the parts
> he just doesn't like is anti-science, in my book.
Religous insults? You mean when I point out that an unwillingness to
consider the underlying premises of the ABX test turns promotion of such a
test (without validation) into a profession akin to religion? As opposed to
true science? If that's your claim, I stand convicted.
>
>> >> The only thing one might fault it for is
>> >> sensitivity, but with this type of test sensitivity is simply a matter
>> >> of
>> >> numbers...need more sensitivity, add more people.
>> >
>> > As a test for difference, more numbers wouldn't solve the sensitivity
>> > problem. You're talking about a test that couldn't determine that LP
>> > and CD are sonically different! And you think such a test should be the
>> > gold standard for judging the accuracy of all other difference tests?
>>
>> Believe me Bob, it tests difference. But it tests subjective musical
>> impression difference, not sound pressure levels. Can *you* see a
>> difference?
>
> Yeah, and that's why ABX is better. ABX doesn't JUST test for "musical
> impression," it tests for any difference at all. And you can't identify
> a single audible sonic difference that ABX tests can't distinguish.
How would you know that? A proper control test has never been done.
>
> As for your own "proposed" test, which is it:
> 1) monadic, like the pharmaceutical companies?
> 2) proto-monadic, like Oohashi?
> 3) quick-switching preference, like Harman?
>
> Or does it depend?
I can't answer because your list is erroneous. I've answered the 1) vs. 2)
above. And 3) simply indicates you missed out on, failed to notice, or for
some reason dismissed John's reporting.
>
> Actually, it doesn't. None of those three tests could be used to
> confirm an audible difference between LP and CD. An ABX test could do
> so easily. So much for validation.
We aren't looking to determine differences, Bob. We're looking to evaluate
audio components sonic signatures and subjective shading of musical
reproduction. And there has been no confimation that ABX or a straight AB
difference test can show up all the various shadings that show up in
longer-term listening evaluations.
Buster Mudd
Definitely two different kind of things, hence the reason for my post
in the first place!
Musical content (of the sort I described) is undeniably "there"; an
auditory memory test might be able to discern whether or not a subject
was concious of prior content while hearing a subsequent passage, but
there's no question that this sort of content exists, and is inherent
in the music. Plus I would think you could make a pretty good arguement
that regardless of whether or not the subject was concious of having
heard this content, that content had an undeniable affect on the
subject's perception of later music.
Contrast that with discernible sonic differences between audio
components: A valid test would have to first determine whether or not
differences (the "content" in this case) even exist before one can move
on to wondering about whether or not auditory memory (concious or not)
of such things affects subsequent listening.
So I think the "relation...between these two types of auditory memory"
is that one *is* a type of audio memory, and the other *might be* --
or, possibly might not be -- a type of audio memory.
Harry Lavo
news:d9nm4...@news4.newsguy.com...
> Harry,
>
> It was few weeks ago when you described your "monadic" test first time
> in this group. Now you are talking about this test as an established
> fact.
>
> Even if somebody would go into a hassle and an expense of
> implementing your suggestion it is not at all obvious that the test
> would produce any results. I think that most likely outcome that it
> would find your subjective terms like "warmth", "depth", etc. not
> correlated to the sound of the recording. I would bet that the
> distribution of particular term would be completely random for
> different users.
>
> But of course then you would require not 200 participants but 2000,
> etc. or something that again will make a proposed test unfeasible. And
> you will continue speculate about validity of your imaginary test.
>
> Please, either provide some proof that you so-called "monadic" test
> works or stop speculating about it.
>
> vlad
>
Well, I guess I can understand why you feel that way. But fact is, Vlad, I
postulated such a test as a key part (the "control" part) of a validation
test here nearly two years ago. I let the matter drop after much
controvery, and only recently brought it up again (in another forum, but it
has spilled over here). I also realized that perhaps understanding of what
I was proposing was buried in the complexity of the overall testing needed
to validate quick-switch testing, so I have tried to make my explainations
as simple as possible.
The reason I say it is a standard test is that it is widely used in the
social sciences, psychological and behavioral sciences, and in the medical
sciences. Audio is a field where it has not traditionally been used, at
least to my knowledge. Partly this may be structural (there are not a lot
of large companies worried about the quality of musical reprodcution, after
all). But more likely it is because the field has been dominated by sound
research conducted by physisists, electircal engineers, and audiologists.
However, more recently scientists have made rapid progress in brain research
with the growing realization that how we hear is very complex, and how we
hear music even more so. There is growing realization that musical
evaluation must be treated as a subjective phenomenon, and that means
treating its measurement using the tools of the social and psychological
scientists, and the medical scientists, not necessarily the physical
scientists.
As difficult as you may find it to believe that ratings of things like
"warmth" or "depth" or "dimensional" have meaning, those kinds of subjective
yet descriptive phrases are widely used in subjective research. Of course,
part of the art of researchers in a given field is determining the best,
most precise, way of asking the question to minimize confusion. You don't
want to say "on a scale of one to five, rate this item on "warmth"". You
doubtless would construct a scale that said " on a scale of one to five,
where 'one' is a relatively cool tone, and 'five' is a relatively warm tone,
where would you place the sound you just heard?". Or something to that
effect.
Part of the research art is developing, and oft-times pretesting, the
questions so that you know they are meaningful and with minimum
misinterpretation. This is all practical "art", and there are commercial
researchers who are quite good at it.
nab...@hotmail.com
If by "identification test," you mean that you listen to a single
signal and decide whether it is CD or SACD, that is extremely
difficult, because you must remember what both SACD and CD sound like.
(And, as I've said before, our aural memory for such small sonic
differences is far too short to do that.)
Whereas, in a proper same-different test (or an ABX test, which is a
variant), you have both signals available to you at all times, and can
switch immediately between them, which allows you to compare directly.
Unfortunately, most home users cannot do that sort of a test, because
it requires you not only to level-match the two (relatively easy) but
also time-sync them (very hard). It would be too easy to tell that the
two were different if one were running even fractionally ahead of the
other.
bob
nab...@hotmail.com
> We aren't looking to determine differences, Bob.
You're the one who started this whole conversation by insisting that an
ABX test was inadequate. Well, the ONLY purpose of an ABX test is to
determine difference. If your argument is that an ABX test is not
adequate for determining something it was not designed to determine,
then you've been wasting our time.
> We're looking to evaluate
> audio components sonic signatures and subjective shading of musical
> reproduction. And there has been no confimation that ABX or a straight AB
> difference test can show up all the various shadings that show up in
> longer-term listening evaluations.
There is no evidence that "various shadings" really do show up (rather
than simply being imagined by the listener) in longer-term listening
evaluations of components that cannot be distinguished in ABX tests.
You are once again assuming your conclusion.
bob
vlad
> "vlad" <v.kuz...@comcast.net> wrote in message
> news:d9nm4...@news4.newsguy.com...
> > Harry,
> >
> > It was few weeks ago when you described your "monadic" test first time
> > in this group. Now you are talking about this test as an established
> > fact.
> >
> > Even if somebody would go into a hassle and an expense of
> > implementing your suggestion it is not at all obvious that the test
> > would produce any results. I think that most likely outcome that it
> > would find your subjective terms like "warmth", "depth", etc. not
> > correlated to the sound of the recording. I would bet that the
> > distribution of particular term would be completely random for
> > different users.
> >
> > But of course then you would require not 200 participants but 2000,
> > etc. or something that again will make a proposed test unfeasible. And
> > you will continue speculate about validity of your imaginary test.
> >
> > Please, either provide some proof that you so-called "monadic" test
> > works or stop speculating about it.
> >
> > vlad
> >
Harry, you did not address my statement about your "monadic" test
procedure. Let me repeat it here -
-- Even if somebody would go into a hassle and an expense of
-- implementing your suggestion it is not at all obvious that the test
-- would produce any results. I think that most likely outcome that it
-- would find your subjective terms like "warmth", "depth", etc. not
-- correlated to the sound of the recording.
Also you are trying to present your test as a mean of "validation"
of ABX/DBT tests. ABX/DBT tests do not need validation. They test
audibility of differences in physical devices (amp, wires, etc) and for
this purpose they work just fine according to experts in this field.
Your 'monadic' testing is designed to measure subjective
differences.
You probably can measure subjective preferences, I will give you that.
For instance after testing of 10000 subjects you can conclude that 52%
favor box a, 45% box B, and 3% are undecided. After all efforts and
money spent on this test will these results have any value? Subjective
is subjective and that is all.
For instance, many people have subjective preference for LPs. But it
does not make LP an accurate reproduction medium? Should we stick to
LPs for music listening? We have much better means now to store and
transfer audio signal. It is the matter of preference for some people,
that's it. Nobody argues with preferences.
>
> Well, I guess I can understand why you feel that way. But fact is, Vlad, I
> postulated such a test as a key part (the "control" part) of a validation
> test here nearly two years ago.
DBT does not need validation by "monadic" tests.
> I let the matter drop after much
> controvery, and only recently brought it up again (in another forum, but it
> has spilled over here). I also realized that perhaps understanding of what
> I was proposing was buried in the complexity of the overall testing needed
> to validate quick-switch testing, so I have tried to make my explainations
> as simple as possible.
>
> The reason I say it is a standard test is that it is widely used in the
> social sciences, psychological and behavioral sciences, and in the medical
> sciences. Audio is a field where it has not traditionally been used, at
> least to my knowledge. Partly this may be structural (there are not a lot
> of large companies worried about the quality of musical reprodcution, after
> all). But more likely it is because the field has been dominated by sound
> research conducted by physisists, electircal engineers, and audiologists.
> However, more recently scientists have made rapid progress in brain research
> with the growing realization that how we hear is very complex, and how we
> hear music even more so. There is growing realization that musical
> evaluation must be treated as a subjective phenomenon, and that means
> treating its measurement using the tools of the social and psychological
> scientists, and the medical scientists, not necessarily the physical
> scientists.
>
Musical perception is subjective phenomenon, and always was.
> As difficult as you may find it to believe that ratings of things like
> "warmth" or "depth" or "dimensional" have meaning, those kinds of subjective
> yet descriptive phrases are widely used in subjective research. Of course,
> part of the art of researchers in a given field is determining the best,
> most precise, way of asking the question to minimize confusion. You don't
> want to say "on a scale of one to five, rate this item on "warmth"". You
> doubtless would construct a scale that said " on a scale of one to five,
> where 'one' is a relatively cool tone, and 'five' is a relatively warm tone,
> where would you place the sound you just heard?". Or something to that
> effect.
No, I think first of all you will find that if you will take two amps
or wires that are undistinguishable in DBT , then results of you
subjective evaluation test will be all over the map. I would expect
that subjective feelings of subject will be very poorly correlated if
correlated at all with particular pieces of equipment.
So before pouring any money or efforts in this kind of testing I would
ask first why you think that this test will give results at all. My
second question would be what you are going to do with results.
Subjective preferences tend to change with the time and can be
influenced by the last review in a Stereophile easily.
I personally don't care about subjective feeling of people that I
don't know.
>
> Part of the research art is developing, and oft-times pretesting, the
> questions so that you know they are meaningful and with minimum
> misinterpretation. This is all practical "art", and there are commercial
> researchers who are quite good at it.
What do you mean by that?
vlad
nab...@hotmail.com
> On 24 Jun 2005 01:09:45 GMT, Gary Eickmeier <geic...@tampabay.rr.com>
> wrote:
>
> >
> >You can't do a "quick switch" test with two sources that run at
> >different speeds because you can't synchronize them, which would be a
> >dead giveaway in itself, so that is a bad example.
> >
> >If you want to use that example, you will have to listen first to one,
> >then the other, in its entirety, then decide if the speed difference is
> >audible. If so, then do a blind series, listening to a known version,
> >then to a randomly chosen one, and decide whether it is the same or
> >different. In this manner you will eventually arrive at a number for a
> >speed differential that is at the audible threshold. That is the basic
> >idea of how audio research is done. You may find that speed differences
> >of 1.01 will be inaudible to most, but audible to some with perfect
> >pitch. If this is interesting enough a question for you, then do the
> >research and report it.
> >
>
> p.s. Suppose one carried out research such as this and found, for a
> given one-minute-long excerpt, what is the audible threshold. So a
> given subject could reliably discriminate between the excerpt and a
> version that is 1.01 as fast (say). What theoretical reason would we
> have to think that, if we did a quick switch test (see my previous
> email for a suggestion about how to do it), the subject would be able
> to tell the excerpts apart in that test?
Because the difference in pitch would be the way you'd be telling them
apart. (You certainly don't think you can tell the difference between a
passage that is 60 seconds long and a passage that is 60.6 seconds
long, do you?) And we know that differences in pitch are much easier to
detect when you can switch directly between the samples.
> I don't understand the point about perfect pitch, because I am
> supposing that one version is faster than the other, not that the
> speed and pitch are both higher (as would be the case with analog
> tape). Maybe I am not seeing your point though.
If one version is faster than the other, then the pitch will be higher,
whatever the medium. The only exception would be if you were to use
digital signal processing to correct for this. In that case, you
probably won't be able to tell them apart without a stopwatch unless
the difference is substantial. Our resident conductor would presumably
do somewhat better, because she is trained to be sensitive to subtle
differences in tempo. But even she would have her limits.
bob
jjn...@sonic.net
> But that is a result of the fact that music itself is
> subjective, and *cannot* be measured objectively. The closest you can come
> perhaps is to substitute some kind of psychophysiological measurements.
Do you <really> believe all that???
Notation?
Music theory?
Tuning systems?
Harmonic series?
Compositional devices?
Just to name a few of the obvious ones.
Mark DeBellis
By an identification test I mean one where you can switch back and
forth between the signals, but where what you have to decide is not
whether they are the same or different, but which one is CD and which
is SACD. This, I think, is difficult for some of the same reasons
that the test you describe above is difficult, so what should be
inferred from a subject's failure to get a high percentage of correct
answers on my kind of identification test is not necessarily the same
as what should be inferred from a subject's failure to get a high
score on a proper "same-different" test.
Mark
Jenn
Hey, I have MANY limitations! :-) I have found that I have sensitivity
in regard to tempi at about 3 beats per min. That is, I can tell that
one performance is slower or fast at about that threshold. I CAN'T pick
specific tempi out of the air with that degree of sensitivity. Others
can come pretty close to that. That's why I don't care for conducting
ballet, for example. Dancers need things REALLY exact in tempo, and I
just don't enjoy working that way; it's anti musical to me.
I'm learning a lot through this discussion, btw. Thanks to the
participants.
nab...@hotmail.com
> So before pouring any money or efforts in this kind of testing I would
> ask first why you think that this test will give results at all.
Because he doesn't like the results we've already got. No other reason.
The problem with using monadic tests for the purpose of determining
whether any difference is discernible between two components is that
the you will get a large (and incalcuable) number of false negatives.
You will get negative results:
1) when subjects really can't distinguish between the two,
2) when they could but didn't in this particular test (the standard
false negative that all such tests face), and
3) when subjects could distinguish between the two, but their
impressions based on whatever criteria you asked them about did not
lean consistently in a single direction. For example, if they could all
hear a difference between LP and CD, but half of them preferred one and
found it more lifelike/musical/etc., and the other half had exactly the
opposite reaction, the results would be inconclusive. And what good is
a test for difference that can't even distinguish between things that
sound as different as LP and CD?
bob
Jenn
I think that what Harry probably means is that the perception of the
sound quality and expressive qualities of music and performance thereof.
Mark DeBellis
wrote:
>Mark DeBellis wrote:
Thank you. That confirms my belief and I appreciate the elegant
description of the testing paradigm.
Mark
Harry Lavo
> Harry Lavo wrote:
>
>> We aren't looking to determine differences, Bob.
>
> You're the one who started this whole conversation by insisting that an
> ABX test was inadequate. Well, the ONLY purpose of an ABX test is to
> determine difference. If your argument is that an ABX test is not
> adequate for determining something it was not designed to determine,
> then you've been wasting our time.
>
It started because an ABX test was proposed as a means of making listening
decisions for audio equipment.
The fact that *difference* is the wrong measure is just one of the problems
with this approach.
>> We're looking to evaluate
>> audio components sonic signatures and subjective shading of musical
>> reproduction. And there has been no confimation that ABX or a straight
>> AB
>> difference test can show up all the various shadings that show up in
>> longer-term listening evaluations.
>
> There is no evidence that "various shadings" really do show up (rather
> than simply being imagined by the listener) in longer-term listening
> evaluations of components that cannot be distinguished in ABX tests.
> You are once again assuming your conclusion.
The shadings can presume to be there, as they are heard by many people,
until proven otherwise. And they can't be proven otherwise except through
something like a monadic control test. The "shadings" are subjective; it
requires a test that can determine if subjective perception is real or not
and that is by ratings among a large cross-section of audiophiles, with
statistical analysis applied.
Harry Lavo
I see your point. Let me correct my statement: the "experiencing" of music
itself is subjective, and *cannot* be measured objectively.
Now hopefully you can agree to that, which is the part that is relevant to a
listening test.
Mark DeBellis
>Mark DeBellis wrote:
>>
>> Suppose one carried out research such as this and found, for a
>> given one-minute-long excerpt, what is the audible threshold. So a
>> given subject could reliably discriminate between the excerpt and a
>> version that is 1.01 as fast (say). What theoretical reason would we
>> have to think that, if we did a quick switch test (see my previous
>> email for a suggestion about how to do it), the subject would be able
>> to tell the excerpts apart in that test?
>
>Because the difference in pitch would be the way you'd be telling them
>apart. (You certainly don't think you can tell the difference between a
>passage that is 60 seconds long and a passage that is 60.6 seconds
>long, do you?) And we know that differences in pitch are much easier to
>detect when you can switch directly between the samples.
>
>> I don't understand the point about perfect pitch, because I am
>> supposing that one version is faster than the other, not that the
>> speed and pitch are both higher (as would be the case with analog
>> tape). Maybe I am not seeing your point though.
>
>If one version is faster than the other, then the pitch will be higher,
>whatever the medium. The only exception would be if you were to use
>digital signal processing to correct for this. In that case, you
>probably won't be able to tell them apart without a stopwatch unless
>the difference is substantial. Our resident conductor would presumably
>do somewhat better, because she is trained to be sensitive to subtle
>differences in tempo. But even she would have her limits.
>
In the example in question, I am supposing that the speed is higher
but not the pitch. If the only way to do this is by digital signal
processing, then so be it. There will be an audible threshold at
which a subject can reliably discriminate between the excerpt and the
sped-up excerpt. My question stands: What theoretical reason would we
have to think that, if we did a quick switch test (see my previous
email for a suggestion about how to do it), the subject would be able
to tell the excerpts apart in that test?
Mark
nab...@hotmail.com
>
> By an identification test I mean one where you can switch back and
> forth between the signals, but where what you have to decide is not
> whether they are the same or different, but which one is CD and which
> is SACD.
That would be slightly easier than just listening to a single one, but
it still requires you to remember the criteria by which you had
previously distinguished (or *thought* you'd distinguished) them. So
it's still harder than a same-different test, or ABX, or similar.
> This, I think, is difficult for some of the same reasons
> that the test you describe above is difficult, so what should be
> inferred from a subject's failure to get a high percentage of correct
> answers on my kind of identification test is not necessarily the same
> as what should be inferred from a subject's failure to get a high
> score on a proper "same-different" test.
Agreed. In particular, there's probably some subset of sonic
differences which you wouldn't detect in an identification test, but
you would in a same-different test.
bob
Harry Lavo
> vlad wrote:
>> So before pouring any money or efforts in this kind of testing I would
>> ask first why you think that this test will give results at all.
>
> Because he doesn't like the results we've already got. No other reason.
Thanks for the gratuitous insult, Bob.
>
> The problem with using monadic tests for the purpose of determining
> whether any difference is discernible between two components is that
> the you will get a large (and incalcuable) number of false negatives.
> You will get negative results:
> 1) when subjects really can't distinguish between the two,
> 2) when they could but didn't in this particular test (the standard
> false negative that all such tests face), and
> 3) when subjects could distinguish between the two, but their
> impressions based on whatever criteria you asked them about did not
> lean consistently in a single direction. For example, if they could all
> hear a difference between LP and CD, but half of them preferred one and
> found it more lifelike/musical/etc., and the other half had exactly the
> opposite reaction, the results would be inconclusive. And what good is
> a test for difference that can't even distinguish between things that
> sound as different as LP and CD?
Basically, Bob, this exposition shows that you have no idea of how scaling
works to measure differences. Please read my current posts before you
*decide* (based on erroneous beliefs) why it doesn't work. If I am to
believe you, I just wasted twenty five years of work and my company(s)
didn't make the hundreds of millions of dollars based on it that they
thought they did.
jjn...@sonic.net
And those were audio tests. Correct?
jjn...@sonic.net
I wouldn't disagree, except that soliciting responses under controlled
conditions is also relevant, which is a bogeyman for you for reasons you
have yet to adequitely explain.
They are not mutually exclusive, despite your assertions otherwise.
Saying the experience of music is subjective is sort of belaboring the obvious.
I think a single word to describe it better, I would use <abstraction>.
nab...@hotmail.com
> >
> Hey, I have MANY limitations! :-) I have found that I have sensitivity
> in regard to tempi at about 3 beats per min.
3 beats per minute out of how many? Three beats of Largo is a lot
longer than 3 beats of Presto.
> That is, I can tell that
> one performance is slower or fast at about that threshold. I CAN'T pick
> specific tempi out of the air with that degree of sensitivity. Others
> can come pretty close to that. That's why I don't care for conducting
> ballet, for example. Dancers need things REALLY exact in tempo, and I
> just don't enjoy working that way; it's anti musical to me.
>
> I'm learning a lot through this discussion, btw. Thanks to the
> participants.
Don't say that. You'll only encourage us.
bob
Harry Lavo
> Harry Lavo wrote:
>snip<
> Harry, you did not address my statement about your "monadic" test
> procedure. Let me repeat it here -
>
> -- Even if somebody would go into a hassle and an expense of
> -- implementing your suggestion it is not at all obvious that the test
> -- would produce any results. I think that most likely outcome that it
> -- would find your subjective terms like "warmth", "depth", etc. not
> -- correlated to the sound of the recording.
>
Well, you thoughts are your thoughts. But I have done a lot of research in
food, where ratings are subjective, and I simply disagree. If one amp, for
example, performs in a way that can be characterized as "cool" and another
as "warm", peoples ratings will reflect that even if they think they are
rating the music rather than the amp. Although there probably is no reason
to deceive them since the test is mondadic. There will be substantial
scatter; they won't march in lockstep. But the averages will reflect the
difference, and if the difference in averages is great enough, they will
reach the 95% significance level. Then you can conclude that amp "A" is
warmer-sounding than Amp "B".
Likewise, you can ask for overall preference and a whole series of ratings
on characteristics. Together they will tell you if and how the two amps
differ.
Keep in mind that this goes beyond measurement. If the "coolness" is a
static frequency response dip, it would also likely be heard in an ABX test.
If it is the way the timbre changes dynamically, heard over an extended
listen, it might not. In this *subjective* test, it really doesn't matter
what is creating the perception; the test simply determines whether the
perception difference is real or not.
> Also you are trying to present your test as a mean of "validation"
> of ABX/DBT tests. ABX/DBT tests do not need validation. They test
> audibility of differences in physical devices (amp, wires, etc) and for
> this purpose they work just fine according to experts in this field.
>
The test differences that are volume-related, since as frequency response,
loudness, and standard distortions. They don't do so well, many of us
believe, on things that are more complex perceptually such as imaging,
transparency, dynamic phase coherence, dimensionality, etc.
> Your 'monadic' testing is designed to measure subjective
> differences.
That is correct.
> You probably can measure subjective preferences, I will give you that.
That's a start. You can also measure differences in perception, believe me.
To say you can't to me, with my background, is like saying you can't measure
harmonic distortion to an EE.
> For instance after testing of 10000 subjects you can conclude that 52%
> favor box a, 45% box B, and 3% are undecided. After all efforts and
> money spent on this test will these results have any value? Subjective
> is subjective and that is all.
>
Well, for starters, such a preference by 10000 people woud be statistically
significant beyond a doubt. So you can say for sure "the two amps sound
different, and "A" is preferred.
Now also suppose that those 10000 subjects also determine that Amp "A"
"sounds less constrained" on dynamic peaks, versus Amp "B" (at the 95%
confidence level), and they also determine that Amp "A" sounds "easier to
listen 'into' on soft passages" than Amp "B", again at the 95% level.
That would give a pretty good indication of why Amp "A" was preferred. It's
value, and what was done with the information, would depend on who and for
what purpose the test was done.
(Incidentally, usually a sample size of 200-300 people per cell is an
acceptable trade off between test cost and statistical sensitivity).
Now in my case I proposed such a test as part of an overall series of tests
to determine if the short-form, quick-switch, comparative tests could give
the same results. If so, their worth would be proven for open-ended
evaluation of audio components. If not they would be misleading for this
use, however valuable for other uses they might be. Or the test technique
might have to be altered slightly. For example, let us hypothesize some
possible results:
a standard ABX test of 20 trials, conducted among ten similar people, fails
to reveal a statistical difference.
a standard ABX test of 20 trials, conducted among ten similar people just
reaches the 95% difference threshold
a standard AB preference test of 20 trials shows roughly the same preference
and significance among 10 people
a standard AB preference test of 20 trials shows no statistical preference
among 10 people, but shows a statistical prefernce when 20 people are
included
All of these would have major implications for those using comparative tests
for the purposes of open-ended evaluation of audio components. But only
once the *benchmark* or control had been established.
> For instance, many people have subjective preference for LPs. But it
> does not make LP an accurate reproduction medium? Should we stick to
> LPs for music listening? We have much better means now to store and
> transfer audio signal. It is the matter of preference for some people,
> that's it. Nobody argues with preferences.
If I were a Sony executive and my testing among 300 people should a
statistically significant 60-40 preference for vinyl over CD, I might think
hard about the product and marketing implications of same. Likewise, if I
had hard evidence that SACD was preferred over CD, I'd certainly be thinking
hard about how to capitalize on that fact.
>
>
>>
>> Well, I guess I can understand why you feel that way. But fact is, Vlad,
>> I
>> postulated such a test as a key part (the "control" part) of a validation
>> test here nearly two years ago.
>
> DBT does not need validation by "monadic" tests.
>
The double-blind technique as a concept certainly does not. However,
qucik-switch comparative testing certainly does for the purpose of
open-ended evaluation of audio components, since these tests were designed
or a whole 'nother purpose.
Then you must use a test that measures this subjective phenomenon in its
fullest. That means the test itself has to be designed to interfere the
least possible way in the actual act of listening and evaluation. This is
where the quick-switch, comparative testing has conceptual weakness, since
it completely alters the listening experience and most likely portions of
the brain involved in this activity.
>
>
>> As difficult as you may find it to believe that ratings of things like
>> "warmth" or "depth" or "dimensional" have meaning, those kinds of
>> subjective
>> yet descriptive phrases are widely used in subjective research. Of
>> course,
>> part of the art of researchers in a given field is determining the best,
>> most precise, way of asking the question to minimize confusion. You
>> don't
>> want to say "on a scale of one to five, rate this item on "warmth"".
>> You
>> doubtless would construct a scale that said " on a scale of one to five,
>> where 'one' is a relatively cool tone, and 'five' is a relatively warm
>> tone,
>> where would you place the sound you just heard?". Or something to that
>> effect.
>
> No, I think first of all you will find that if you will take two amps
> or wires that are undistinguishable in DBT , then results of you
> subjective evaluation test will be all over the map. I would expect
> that subjective feelings of subject will be very poorly correlated if
> correlated at all with particular pieces of equipment.
Au contraire...if their truly is no difference the averages of the two cells
evaluation the amps or wires will be identical from a statistical
standpoint, that is, they would fail to differ at a statistically
significant level. Within each evaluating cell, there would be a lot of
scatter, but the averages are what are used in such a test.
>
> So before pouring any money or efforts in this kind of testing I would
> ask first why you think that this test will give results at all. My
> second question would be what you are going to do with results.
> Subjective preferences tend to change with the time and can be
> influenced by the last review in a Stereophile easily.
>
> I personally don't care about subjective feeling of people that I
> don't know.
I think I've answered all of this above. I've proposed it as a control test
for the short form tests. And if I was a marketing or R&D exec at Sony or
Harman International, I'd consider using it for other purposes, as aparently
Harman has.
>>
>> Part of the research art is developing, and oft-times pretesting, the
>> questions so that you know they are meaningful and with minimum
>> misinterpretation. This is all practical "art", and there are
>> commercial
>> researchers who are quite good at it.
>
> What do you mean by that?
I mean there are firms whose job it is to help companies design, conduct,
and evaluate tests. And one of the skills a company that does this has to
develop is the ability to design and pretest questions that make sense and
increase response coherence. I happened to study under the founder of one
such company while obtaining my MBA from Northwestern back in the early
'60's. Dr. Sidney Levy was a highly regarded leader in the field of
behavioral psychology. And then for twenty-five years as an executive I
helped design and make decisions based on such testing for a major consumer
packaged goods company, working with many such companies.
Mark DeBellis
Yes indeed. Here is another example which may prove useful. Suppose
you have two signals. The first consists of the pattern dot-dot-dee
(where dot and dee are different pitches, say), repeated over and
over, where each dot or dee lasts one second. The second pattern
consists of dot-dot-dot-dee, repeated over and over. Say that these
signals are synchronized to begin at the start of the patterns, and
then each goes on in its own way.
Suppose now that the test consists only in comparing short
corresponding snippets of the two signals, two seconds in length (so
you hear only the snippet, not its surrounding context). If the task
is to say whether the two signals are the same, that will be easy,
because there will be different sounds on at least some of the
samples, assuming enough samples are allowed.
If on the other hand the task is to say which signal is which, it will
be impossible, because the samples are too short. This is an example
of the "forest-for-trees" phenomenon that I worried about in an
earlier post. And the difference between the dot-dot-dee pattern and
the dot-dot-dot-dee pattern is an example of a difference in
properties of temporally extended passages.
That is why I invoked that notion earlier, in order to give at least a
partial explanation of why identification tests might be inadequate.
I want to emphasize, this is a problem for *identification* tests; I
am not saying here that it is a problem for "proper same-different"
tests.
Mark