Chapter 1: The Importance of Expertise in Testing

It’s fascinating how many people feel equipped to analyze testing strategies, yet most wouldn't dream of changing their own car oil. The takeaway? We should rely on professionals for both oil changes and testing. Allow me to elaborate.

Imagine you're conducting tests on a population to gather specific insights that can only be measured with a certain level of inaccuracy. This could involve examining reactions to treatments—whether medical or commercial—or assessing perceptions, whether visual or emotional, or even determining the presence of a virus linked to a pandemic.

We’re delving into the science of testing, and this field operates under consistent principles, all thanks to the insights of Mr. Bayes! Now, suppose your test boasts an impressive 98% accuracy rate for detecting a reaction, perception, or illness. The logical next step would be to test everyone, right? WRONG.

In this scenario, 2% of the population will be falsely identified as healthy despite actually being ill—this is what we call a false negative. However, it’s crucial to understand that tests can also produce false positives. In this example, let’s set that rate at 8%. Such single-digit error rates are common, particularly with new tests. Over time and with experience, these rates can significantly improve, but that process is often slow and costly in the business realm, and impractical in medicine.

So, should we proceed to test the entire population? WRONG again.

To engage in testing, one must first establish an expected rate of positive reactions, perceptions, or infections. If that rate stands at 1%, conducting widespread testing could be misguided. Let’s break this down.

Assuming a population of 100,000, a 1% positive rate implies there are 1,000 individuals who are truly positive. Your test would successfully identify 980 of these cases, resulting in only 20 false negatives. While 20 out of 100,000 sounds good, consider this:

Out of the 99,000 individuals who are not infected, the test would incorrectly identify 7,920 as positive. This poses a significant problem.

Fortunately, there’s a solution. By screening the population first, you can target individuals with a higher likelihood of being positive. Perhaps you could identify characteristics that increase the probability of a positive result. Imagine if you discovered a group that is seven times more likely to have the condition?

This would lead to a much clearer picture. Now, out of 100,000, you could expect 7,000 true positives. The test would accurately identify 6,860 of these individuals, although it would still miss 140. However, it would also falsely identify an additional 7,440 individuals as positive. While a two-to-one ratio might seem acceptable, particularly according to the CDC, if you're not facing a pandemic, you should aim for much better odds.

This is precisely why preliminary screenings are essential before conducting extensive tests—it all comes down to managing the false positive rate. I would argue this ratio is a key consideration before placing your trust in any test. Moreover, this isn’t just a discourse on pandemics; it applies to various scenarios including credit scores, forensic DNA analysis, and even mammograms. The principles of Bayes are ubiquitous, or at least they should be.

Regrettably, comprehension of these concepts is limited. I often find myself needing to remind others (and myself) of these principles, especially during medical situations where emotions tend to cloud judgment. This underscores the necessity of entrusting testing to professionals—specifically, those who specialize in testing, not merely physicians or individuals with PhDs.

Now, let’s consider a contemporary issue. If you have access to testing but are not part of the targeted group, should you still get tested "just to be cautious"? The answer is NO! In our scenario, you would still have less than a 50% chance of being positive, potentially even as low as 13%. This should be evident, even to a U.S. Senator who is also an ophthalmologist—but alas, that’s a different discussion.

Ultimately, only qualified experts can navigate these complex scenarios correctly. Be it in business or healthcare, emotional biases, logical fallacies, and ignorance of Bayesian principles can undermine even the best efforts. Testing is a discipline—leave it to the experts.

Thank you for reading! Stay safe and, more importantly, stay informed.

Chapter 2: The Consequences of Misunderstanding Testing

In this video, we explore the chaotic world of testing in gaming, examining the mechanics and strategies that lead to both success and failure.

This informative video delves into the Insanity Defense in criminal law, discussing significant tests and legal standards such as the M'Naghten Rule.