RE: order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis?

From: harry.mager.hm@bayer-ag.de Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis? Date: 9/29/2003 8:59 AM Dear all, it seems that the decisive problem is that covariate selection and hypothesis testing have to be different processes. If not, the statistical properties of the various estimators will not be known, regardless which selection criterion / criteria or "modern" method is used. Since this prerequisite is too restrictive in most cases in practice, we have to accept that we are relying on approximations to the truth at the very best. Actually, there is no special problem with stepwise procedures that is not inherent in other procedures (genetic algorithms etc., etc.), too. At its very end, stepwise procedures may result into "all regression", examining all possible subsets. With increasing number of covariates to be explored, all regression may result in a computational burden not easy to handle. Most of the modern methods essentially only try to reduce computation time, so the actual problem will remain. If we want to select the "best" model, the very first task would be to define what is "best", i.e., the criterion / criteria to be met have to be defined. It has be shown using a vast amount of simulations, and the results are also supported by theoretical considerations, that whatever criterion is used, the results of a selection process will be overly optimistics with regard to the selection criterion. Harry