How to detect false positive?

Question

When applying statistical hypothesis testing, type I error (false positive) could happen. Often we would not know whether type I error happens. But are there cases otherwise, i.e., we can have the truth later after applying hypothesis testing?

For example, I would like to know if women live longer than men. I set up my hypothesis testing for ages at death under two genders: H0 is equality and H1 is women's death age is larger. Assume the result shows significance - reject null. Also assume later scientific research shows women don't live longer than men, and new data shows insignificance. This would be a type I error, and it's known later after the hypothesis testing.

Where could I find cases like this - type I error is known by other measurements?

Answer 1

One example could be Covid testing, where the null hypothesis is that the individual does not have Covid, and the alternative hypothesis is that the individual has Covid.

When developing Covid test schemes in labs, it is usually the case that we know beforehand whether the individuals have Covid or not (through X-ray or other methods) and assess the probability of Type 1 error of the test by comparing the actual results and the test results.

When applying developed Covid test schemes in practice, we can also detect false positives through repeated sampling/testing of the concerned individuals and see if the test results are consistent throughout. Here is an example (https://medicine.missouri.edu/news/researchers-identify-technique-detect-false-positive-covid-19-results), where individuals who were tested positive went over a quality control protocol for repeat testing to reduce false positives.

Answer 2

If you would like an example that involves repeated formal hypothesis testing, this could be one - suppose you are testing whether males earn more more than females, and you draw a random sample from the population and reject the null hypothesis, and conclude that males earn more than females. And then, you use the same population and draw another random sample, but this time you are not able to reject the null. Or, you use a better income measure (say, by including more income sources, or getting official income data from tax agencies rather than self-reported income) on the same random sample as the first time in the second hypothesis test but fail to reject the null. The inconsistent results across the hypothesis tests can be a flag for possible false positives in the first hypothesis test. And the reason for the possible false positive is sampling variation (each random sample from the same population can be different), or measurement error of income, respectively.

I would not suggest detecting false positives by conducting the same hypothesis test but using a sample from a population later in the time to detect false positives from a hypothesis test based on a sample from an earlier population. It could be the case that the underlying population distribution is actually changing over time, and this will contaminate our conclusion.

If you are finding an example of "first perform statistical hypothesis testing and claim positive, and later the ground truth is given", one example would be - first, we are only able to do hypothesis testing on females' and males' longevity based on a random sample we collect from the population, and later on, say, the national health policy department releases the average longevity of females and males that they calculate base on the entire population we draw our random sample from, then we noticed in our sample we reject the null hypothesis, but the null is actually true with the population information released by the officials. So we are able to confidently conclude that our hypothesis testing has a Type 1 error (false positive).