Here’s a cute example of the benefits of decent testing in which (at the very least) the subjects are blinded to which item they are testing.
The item in this case is the violin, and the hypothesis being tested was that the legendary Stradivarius has a superior sound to violins manufactured more recently.
Twenty one musicians were given six violins to test – three modern, and three old, including two Strads. The researchers dimmed the lights, and even applied perfume to the chin-rests to mask any telltale smells – such as mustiness I guess.
Not only were the musicians not able to identify the Strads, but they preferred the modern instruments.
This finding demolishes decades, and perhaps centuries, of conventional wisdom, that somehow the Stradivarius is superior. Implied in this is that the maker had some secret skill or ingredient, and this has served to keep the price of each instrument up in the millions.
This example fits nicely into a common category of logical fallacies. Just because something is expensive, old, rare, exotic, or blessed by Tibetan monks, does not necessarily make it good, or good value.
It also reinforces the need to continue to ask the question about how we know that various claims are true. When it really matters, we should be asking for the evidence for ourselves, or at least seeking impartial opinions. See this post for some rules of thumb on assessing claims.