A perfect demonstration of why AB and ABX tests give false negatives
#1 A perfect demonstration of why AB and ABX tests give false negatives
Saw a version of this on QI last night. Perfect example of why any AB comparison will normally "show" there is no difference between two bits of kit, and so (for example) why all amplifiers sound the same.
https://www.theatlantic.com/technology/ ... ar/373036/
https://www.theatlantic.com/technology/ ... ar/373036/
Whenever an honest man discovers that he's mistaken, he will either cease to be mistaken or he will cease to be honest.
- Dave the bass
- Amstrad Tower of Power
- Posts: 12276
- Joined: Tue May 22, 2007 4:36 pm
- Location: NW Kent, Darn Sarf innit.
#2 Re: A perfect demonstration of why AB and ABX tests give false negatives
Coooooooooooooooool! I like it.
I understand that my brain can make an unintelligible sequence of sounds into a coherent sentence with meaning, that Soundcloud clip just proved it to me. I'm pleased.
But, I don't agree that its "A perfect demonstration of why AB and ABX tests give false negatives" because, when I'm listening to music on bits of kit I'm not trying to unravel a sentence. What I listen to is the music, often with no words so I don't think I'm doing any 'unravelling'.
I'm struggling to get at what I'm trying to explain for the mo. I need to do more word-thinking.
I understand that my brain can make an unintelligible sequence of sounds into a coherent sentence with meaning, that Soundcloud clip just proved it to me. I'm pleased.
But, I don't agree that its "A perfect demonstration of why AB and ABX tests give false negatives" because, when I'm listening to music on bits of kit I'm not trying to unravel a sentence. What I listen to is the music, often with no words so I don't think I'm doing any 'unravelling'.
I'm struggling to get at what I'm trying to explain for the mo. I need to do more word-thinking.
"The fat bourgeois and his doppelganger"
#3 Re: A perfect demonstration of why AB and ABX tests give false negatives
Lets say we have two bits of kit, A and B. AB and ABX testing is used to see if its possible to tell two things apart, if the result is that the testee can't tell them apart, then its declared that the two bits of kit are equivalent so as one is not different to the other it can't be better or preferred.
Lets say B is actually a better less distorted (or in some way better sounding thing), if we hear A first, and then B, we think we have heard a improvement, so we say, ahh, that sounded better, lets go back to A again. But now we can't unhear the improvement, so now going back to A we still hear what we thought was a improvement going from A to B, and we begin to doubt or first thought. Now we go from A to B again, and still hear no improvement, the more we do it the more it becomes clear that we can't hear any difference, and our first thought of a improvement was false.
So even though there was a difference that if we come back to it a few days later, we will hear again, a comparison test shows us that there isnt.
In the video, once you have heard the real clip, you can't hear what you heard first, not matter how hard you try.
Lets say B is actually a better less distorted (or in some way better sounding thing), if we hear A first, and then B, we think we have heard a improvement, so we say, ahh, that sounded better, lets go back to A again. But now we can't unhear the improvement, so now going back to A we still hear what we thought was a improvement going from A to B, and we begin to doubt or first thought. Now we go from A to B again, and still hear no improvement, the more we do it the more it becomes clear that we can't hear any difference, and our first thought of a improvement was false.
So even though there was a difference that if we come back to it a few days later, we will hear again, a comparison test shows us that there isnt.
In the video, once you have heard the real clip, you can't hear what you heard first, not matter how hard you try.
Whenever an honest man discovers that he's mistaken, he will either cease to be mistaken or he will cease to be honest.
- pre65
- Amstrad Tower of Power
- Posts: 21400
- Joined: Wed Aug 22, 2007 11:13 pm
- Location: North Essex/Suffolk border.
#4 Re: A perfect demonstration of why AB and ABX tests give false negatives
As I did not know what ABX testing was I looked it up.
I found this article interesting.
http://hometheaterreview.com/why-do-aud ... x-testing/
I found this article interesting.
http://hometheaterreview.com/why-do-aud ... x-testing/
The only thing necessary for the triumph of evil is for good men to do nothing.
Edmund Burke
G-Popz THE easy listening connoisseur. (Philip)
Edmund Burke
G-Popz THE easy listening connoisseur. (Philip)
#5 Re: A perfect demonstration of why AB and ABX tests give false negatives
That article is a perfect example of what I suggest the video destroys.
Whenever an honest man discovers that he's mistaken, he will either cease to be mistaken or he will cease to be honest.
- Cressy Snr
- Amstrad Tower of Power
- Posts: 10582
- Joined: Wed May 30, 2007 12:25 am
- Location: South Yorks.
#6 Re: A perfect demonstration of why AB and ABX tests give false negatives
Eg,
I listen to amplifier A and it sounds good
I listen to amplifier B and it sounds more detailed "ooh, I haven't heard that triangle before"
I listen to amplifier A and now the triangle I hadn't heard on that amp is there, large as life.
I listen to amplifier B again and now it sounds the same as amplifier A - Bugger!
I listen to amplifier A and it sounds good
I listen to amplifier B and it sounds more detailed "ooh, I haven't heard that triangle before"
I listen to amplifier A and now the triangle I hadn't heard on that amp is there, large as life.
I listen to amplifier B again and now it sounds the same as amplifier A - Bugger!
Sgt. Baker started talkin’ with a Bullhorn in his hand.
- Dave the bass
- Amstrad Tower of Power
- Posts: 12276
- Joined: Tue May 22, 2007 4:36 pm
- Location: NW Kent, Darn Sarf innit.
#7 Re: A perfect demonstration of why AB and ABX tests give false negatives
Got it! Thanks all. OK, its about not being able to undo your hearing recognitions.
"The fat bourgeois and his doppelganger"
#8 Re: A perfect demonstration of why AB and ABX tests give false negatives
Yes, as is said in the video, the brain will use all the information it has including past experience to understand a signal.Dave the bass wrote: ↑Sun Jul 23, 2017 1:16 pm Got it! Thanks all. OK, its about not being able to undo your hearing recognitions.
Whenever an honest man discovers that he's mistaken, he will either cease to be mistaken or he will cease to be honest.
- Dave the bass
- Amstrad Tower of Power
- Posts: 12276
- Joined: Tue May 22, 2007 4:36 pm
- Location: NW Kent, Darn Sarf innit.
#9 Re: A perfect demonstration of why AB and ABX tests give false negatives
Cool, got that.
but here's a non-confrontational 'but'...
Part of what I listen to/for in the sound reproduction overall is the sound images in front of me. Now, I don't think the brain can be tricked here because you either hear/see it in your minds eye or you don't.
but here's a non-confrontational 'but'...
Part of what I listen to/for in the sound reproduction overall is the sound images in front of me. Now, I don't think the brain can be tricked here because you either hear/see it in your minds eye or you don't.
"The fat bourgeois and his doppelganger"
#10 Re: A perfect demonstration of why AB and ABX tests give false negatives
But once you have "seen" it, its very much harder to not see it.
But the meaning (IMHO) of the video is that
Hearing a difference means there is a difference
Not hearing a difference is not proof of the absence of a difference.
But the meaning (IMHO) of the video is that
Hearing a difference means there is a difference
Not hearing a difference is not proof of the absence of a difference.
Whenever an honest man discovers that he's mistaken, he will either cease to be mistaken or he will cease to be honest.
- Dave the bass
- Amstrad Tower of Power
- Posts: 12276
- Joined: Tue May 22, 2007 4:36 pm
- Location: NW Kent, Darn Sarf innit.
#11 Re: A perfect demonstration of why AB and ABX tests give false negatives
Agreed in the case of unravelling and 'oooo... I didn't here that before' kinda moments and perhaps 'tone' too but I don't agree in the case of imaging because thats just it, it either is or isn't there for whatever reason.
I don't think its a brain/memory related 'thing'. It's happening in the 'now', or maybe thats just the way I hear stuff, I think I'm quite sensitive (note to self - "you little Princess DTB!") to imaging. I know we're all different and hear different things.
"The fat bourgeois and his doppelganger"
#12 Re: A perfect demonstration of why AB and ABX tests give false negatives
Interesting, I wonder if that means you are particularly sensitive to phase errors?Dave the bass wrote: ↑Sun Jul 23, 2017 3:53 pmAgreed in the case of unravelling and 'oooo... I didn't here that before' kinda moments and perhaps 'tone' too but I don't agree in the case of imaging because thats just it, it either is or isn't there for whatever reason.
I don't think its a brain/memory related 'thing'. It's happening in the 'now', or maybe thats just the way I hear stuff, I think I'm quite sensitive (note to self - "you little Princess DTB!") to imaging. I know we're all different and hear different things.
Whenever an honest man discovers that he's mistaken, he will either cease to be mistaken or he will cease to be honest.
#13 Re: A perfect demonstration of why AB and ABX tests give false negatives
Thanks for posting the link Nick, it gives me some understanding of my own audio experiences.
Coincidentally, I took myself into Waterstones yesterday and started browsing this book, which seems to give some interesting general insights;
I'll be getting a copy as a birthday present shortly...
Coincidentally, I took myself into Waterstones yesterday and started browsing this book, which seems to give some interesting general insights;
I'll be getting a copy as a birthday present shortly...
Sorry, I couldn't resist!
#14 Re: A perfect demonstration of why AB and ABX tests give false negatives
I'd say the same as Dave, it's the imaging I pick up on first. Maybe explains why I've caught reversed phase on a few occasions?Nick wrote: ↑Sun Jul 23, 2017 6:52 pmInteresting, I wonder if that means you are particularly sensitive to phase errors?Dave the bass wrote: ↑Sun Jul 23, 2017 3:53 pmAgreed in the case of unravelling and 'oooo... I didn't here that before' kinda moments and perhaps 'tone' too but I don't agree in the case of imaging because thats just it, it either is or isn't there for whatever reason.
I don't think its a brain/memory related 'thing'. It's happening in the 'now', or maybe thats just the way I hear stuff, I think I'm quite sensitive (note to self - "you little Princess DTB!") to imaging. I know we're all different and hear different things.
#15 Re: A perfect demonstration of why AB and ABX tests give false negatives
Good example how we all listen slightly differently. For me imaging is not that important, as long as there is good separation of the sounds my head adds the imaging (as it does in mono). So its probably low noise and maybe secondly, low higher order distortion that’s important to me.
Whenever an honest man discovers that he's mistaken, he will either cease to be mistaken or he will cease to be honest.