Is Bayesianism Susceptible to the Mail-Order Prophet Scam? — LessWrong

Comment by ChristianKl - Bayesianism has no rules for what someone priors should be. It has rules about how to progress from a state of having priors.

There's a reason that Einstein did not get his Nobel Prize for the special theory of relativity. At the time the prize was given, the Nobel Prize committee did not believe that the predictions about Mercuries orbit were strong enough evidence for the special theory of relativity to give him the Nobel Prize for it. Besides Mercuries orbit there was also the Michelson-Morley experiment.

To the extend that you were taught in school that the Michelson-Morley and Mercuries orbit provided definite evidence for the special theory of relativity, that's a retrospective accounting from people who already knew it to be true and not the perspective from the physicists that gave Einstein his Nobel Prize.

Whether or not physicists at the time should have updated more strongly into the direction of seeing special relativity as proven depends a lot about what you believe of the merits of alternative explanations for the observations and how well those fitted the data and how likely you consider the measurement for Mercuries orbit to be correct.

[...]

In Bayesanism probabilities are not independent of the model of the observer the same way that frequentism has a notion of observer independent probabilities.

The general mechanism with the mail order scam you talked about is called survivorship bias. If you take the question of how high the existential risk of being nuclear war happens to be, this matters. If you just observe that we now have nuclear weapons for a long time, it might be wrong to update with each passing year into a lower chance of nuclear war because you would not be around to observe reality in case everyone got killed by nuclear war. That's why we need to look at Petrov and Arkhipov to get an understanding about near misses and we treat both of them as heroes on LessWrong.