Re: Bootstrap resampling! -> Randomization test

From: Lewis B Sheiner <lewis@c255.ucsf.edu> Subject: Re: Bootstrap resampling! -> Randomization test Date: Fri, 30 Mar 2001 10:47:55 -0800 I can't give an answer, but I can provide a point of view. The testing paradigm only makes sense when a proponent is advocating a conclusion to a skeptical observer (who can, of course, be the proponent's alter ego). When this is the set-up, then the skeptical observer gets to specify the rules. Currently they are to offer data that are extremely unlikely if the claim does not hold. To be logically tight, the claim-not-holding assertion should be sharp (e.g., the means of two populations are exactly equal), which of course makes the claim rather diffuse (the means are unequal ... but what they are is not addressed). When the paradigm is used to "test" a scientific hypothesis in the service of making sure it is tentatively acceptable, the diffuseness doesn't matter; the experiment being evaluated was set up, presumably, to provide a sharp qualitative challenge. For public acceptance of remedies, there may be some problem with the diffuseness (for example that we wind up unsure of exactly how much tretment benefit to expect). So a simple answer to your question is: we need valid (i.e., correct performance under the null) hypothesis testing procedures whenever we are in a testing situation (as above). As scientists (as opposed to advocators), I believe there is another use of testing, and this is the one that RA Fisher advocated: we are so eager to find patterns in our data that we need some reality check on this tendency. So an hypothesis test of the null hypothesis that "nothing interesting is going on" may be useful, if it is not rejected, to encourage us to abandon fruitless quests to find signal where there is likely only noise. LBS