order of covariate inclusion

From: pbonate@ilexonc.com Subject: [NMusers] order of covariate inclusion Date: 9/25/2003 12:06 PM Is anyone aware of any studies showing whether the order of covariate inclusion has any impact on the final model using a forward addition model development strategy? For example, suppose you have weight, sex, and age as covariates and then test weight, then age, and then sex in the model using the likelihood ratio test. Does this or can this result in a different model than if you tested sex then age then weight? Thanks, pete bonate Peter L. Bonate, PhD, FCP Director, Pharmacokinetics ILEX Oncology, Inc 4545 Horizon Hill Blvd San Antonio, TX 78229 phone: 210-949-8662 fax: 210-949-8219 email: pbonate@ilexonc.com

From: lgibiansky@emmes.com Subject: Re: [NMusers] order of covariate inclusion Date: 9/25/2003 12:39 PM I do not know specific references, but I am sure this has a great impact. That is why forward addition procedure assumes testing all covariates, and then adding the most significant one. This is not the perfect way to build the model, but at least it gives unique model. As an extreme example, suppose that you have two identical covariates. Then the one that you test first will be significant, and the second one will add nothing. For your example, if sex and weight are confounded, you may end up including into the model the first that you test even if the second one gives you slightly better model. Leonid

From: pbonate@ilexonc.com Subject: Re: [NMusers] order of covariate inclusion Date: 9/25/2003 1:02 PM Leonid, Thanks for your thoughts. I am not sure that order has an impact provided the degree of correlation between covariates is small to moderate. In the example you cite, when you have two covariates that are identical (the most extreme case) or highly correlated what will happen is that when both are entered into the model simultaneously most likely neither will show significance because of ill conditioning. I did a paper on this a few years ago. As the correlation between two variables increases the standard errors of the parameters increase eventually to the point where neither becomes significant. It seems surprising that no one has looked at this. Because if it has "great impact" as you suggest then we need to re-examine all the published pop pk models to date as most published reports fail to include alternative forward addition strategies in their development process. If order is important what guidelines are available or how do modelers choose their order of testing in the model? Any other comments? Anyone? pete

From: lgibiansky@emmes.com Subject: Re: [NMusers] order of covariate inclusion Date: 9/25/2003 1:18 PM Pete, Forward addiction procedure is implemented as follows: If you have covariates A, B, and C, you test all of the models Base + A Base + B Base + C and then choose the best model (say, Base +A, if Base +A is significantly better than Base and better than Base+B) Then you test (Base + A) +B (Base +A) +C and then choose the best one (say, Base +A+B) Then you continue until you exhorted significant covariates. In this way, the procedure is unique. Therefore, all the published models that site "forward addition procedure" uniquely define the model, as above. It seems that you have in mind the following procedure: Let's add A to Base. OK, it is better, then lets add B to (Base + A), etc. This is incorrect for the reasons that you mentioned: the final model will depend on the order. Best, Leonid

From: harry.mager.hm@bayer-ag.de Subject: Re: [NMusers] order of covariate inclusion Date: 9/25/2003 1:23 PM Pete, It seems to be easy to provide a counter example, i.e., if two variables are highly negatively correlated. Of course standard errors will increase with increasing correlations, but this does not necessarily result in insignificance of both variables, what really increases dramatically is model uncertainty, meaning in this case the decision which variable to select is more dictated by random variation / fluctuation than by the underlying relationships. Apart from that, what really counts is the multiple interdependence. The latter may be high even if the pairwise correlations are moderate. If at all, re-examination of Pk/Pd's published has to take into account the entire covariate pool (all those that were originally considered), the approximation methods used [e.g., U. Whlby, N. Jonsson, M. Karlsson published very nice papers on the latter aspects], etc. Harry

From: bachmanw@globomax.com Subject: RE: [NMusers] Re: order of covariate inclusion Date: 9/25/2003 1:38 PM Leonid, are you implying that this can be habit-forming? (...Forward addiction) Bill

From: lgibiansky@emmes.com Subject: RE: [NMusers] Re: order of covariate inclusion Date: 9/25/2003 1:50 PM sorry, I am becoming famous for English misprints and mistakes... Usually Nick picking them up (but now seems to be a night time on that side of the globe)

From: pbonate@ilexonc.com Subject: RE: [NMusers] Re: order of covariate inclusion Date: 9/25/2003 2:01 PM Leonid, I guess I think of forward stepwise modeling buidling more in terms of how I was taught to do it with linear regression. Like with SAS the most significant covariate (based on an F-test) is entered into the model first. If it is significant (based on SLENTRY) it stays in the model. Then the next most significant covariate is entered into the model. This process continues until all covariates are exhausted. Once a covariate is in the model it stays in the model. Again, I guess it depends on the correlations between covariates for whether something stays or goes into the model. Covariates that are uncorrelated should be unaffected by this method. So going back to your example. Suppose two covariates are highly correlated (age and weight). You test base + age base + weight. Both are significant. Then you test base + age + weight and one or both of them turn up as insignificant. What would you do? Thanks, pete

From: Ken.Kowalski@pfizer.com Subject: RE: [NMusers] order of covariate inclusion Date: 9/25/2003 2:04 PM Pete, I agree with Leonid. Stepwise procedures and in particular forward selection can be very sensitive to collinearity (correlation) among the covariates. I'm out of the office the rest of the week so I can't provide you with specific references right now. You should be able to find discussions on the effects of multicollinearity in multiple linear regression in standard statistical texts on linear regression. The effects of multicollinearity on estimation extend to covariate model building with simple and mixed effects nonlinear models even though such effects may not have been discussed in the population modeling literature. If you have a copy of my JPP 2001 paper on the WAM algorithm, I believe I reference a paper by Burk (Technometrics 1977?) that discusses some of the pitfalls with stepwise procedures when there is collinearity among the regressors. Ken

From: alan_xiao@merck.com Subject: RE: [NMusers] Re: order of covariate inclusion Date: 9/25/2003 2:23 PM If you follow the standard approach to let the machine pick the covariates, there should be no problem to duplicate the model by different people. If you pre-select covariates first and then test the model for those non-selected covariates, you could get the problem. Alan.

From: lgibiansky@emmes.com Subject: [NMusers] Re: order of covariate inclusion Date: 9/25/2003 2:25 PM Pete, Let's first define what is "significant". In NONMEM discussions, significant means that the difference in objective function between the nested models is less then significance threshold delta. If you test base (with objective function OFbase) base + age (OFage) base + weight (OFweight) and see that OFbase - OFage > delta, OFbase- OFweight > delta and OFage > OFweight, you include weight into the model (because weight provides you with the lower objective function). Then you test base + weight + age (OFboth) If OFweight - OFboth < delta, you do not include age into the model. If OFweight - OFboth > delta, you include age into the model in addition to weight. End of the forward addition procedure. Note that the end of forward addition procedure may or may not coincide with the end of modeling process. You may found out that although the model with age and weight gives you better OF that weight alone, the standard errors of the estimates (for coefficients that describe covariates) increase dramatically, leading to the model instability due to collaterality of covariates. Then you need to decide (and as far as I know, there are no formal rules how to do it) whether to include extra covariate into the model and accept large uncertainty of the parameter estimates or use fewer covariates with larger confidence in the parameters. Leonid

From: Ken.Kowalski@pfizer.com Subject: RE: [NMusers] order of covariate inclusion Date: 9/25/2003 2:42 PM Pete, As I indicated in my previous message, stepwise procedures have their pitfalls in the presence of collinearity among the covariates. You might try a backward elimination approach starting from a full model (all covariate parameters included simultaneously) and see if you arrive at the same model as the forward selection approach. Of course, if you have a very high degree of collinearity among the covariates you may not be able to fit the full model (rounding errors) but this is useful to know as the modeler may be faced with having to make difficult choices on reducing the set of covariates effects to investigate based on his/her understanding of which covariate effects are the most plausible. One of the problems with stepwise procedures are that they result in selecting a single model and don't give a sense of competing models that might also provide a good fit. Ignoring the effects of collinearity just because a forward selection can find a good fitting model doesn't mean we should feel complacent about the model chosen by forward selection. Because of the order of testing there may be other models that will never be evaluated by the procedure that are as good and maybe better than the one obtained by forward selection. This is one of the reasons that motivated me to develop the WAM algorithm. It provides a ranking of all 2^k possible models (where k is the number of covariate parameters) and the top 10-15 models are then run in NONMEM to give some sense of the competing good-fitting models. Try it...you might like it. Ken

From: sduffull@pharmacy.uq.edu.au Subject: RE: [NMusers] order of covariate inclusion Date: 9/25/2003 6:14 PM Hi Implementation of covariate model building in this manner: >> Forward addiction procedure is implemented as follows: >> If you have covariates A, B, and C, you test all of the >> models Base + A Base + B Base + C and then choose the best >> model (say, Base +A, if Base +A is significantly >> better than Base and better than Base+B) ... does provide initial models that are not nested. The 'sequential' stepwise approach generally avoids this problem (i.e. adding a additional covariates sequentially [in theory] will produce a sequence of nested models). You can of course also consider biological plausibility as an additional guide to choice of initial covariates - for example the effects of adding a covariate describing male/female can often be described completely by consideration of patient weight. Regards Steve ========================================= Stephen Duffull School of Pharmacy University of Queensland Brisbane 4072 Australia Tel +61 7 3365 8808 Fax +61 7 3365 1688 University Provider Number: 00025B Email: sduffull@pharmacy.uq.edu.au www: http://www.uq.edu.au/pharmacy/sduffull/duffull.htm PFIM: http://www.uq.edu.au/pharmacy/sduffull/pfim.htm MCMC PK example: http://www.uq.edu.au/pharmacy/sduffull/MCMC_eg.htm

From: marc.gastonguay@snet.net Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches Date: 9/25/2003 9:10 PM Dear Pete, Leonid, Ken & others, At the risk of complicating this already controversial topic, I'll throw my two cents in... For all the reasons that Ken metioned and more, I think we should be very careful with stepwise approaches (see this website for more reasons "Why stepwise regression is dumb": http://cvu.strath.ac.uk/HyperNews/get/guss-fprt/9.html). Anyway, I've been thinking about this for a while and I'm not sure that we should even bother with stepwise covariate model building. We typically go through this exercise to identify "statistically significant" covariates and the most parsimonious model. When interpreting the modeling results, we usually examine the "significant" covariate effects and make some judgement about the clinical relevance of those covariates, usually explaining away those statistically significant effects that do not produce a clinically relevant change in the parameter(s). We also often state that covariate effects that were not statistically significant have no effect on the parameter in question (which is not entirely accurate). Shouldn't we just focus on clinical relevance and forget about significance of covariate effects? After all, the methods we use to assess significance with the Likelihood ratio test are usually wrong (1 - 3) and inappropriate (due to the multiple comparisons and retrospective nature of the analysis). Let's consider an approach where one builds a full covariate model based on prior scientific knowledge, or particular interest in a set of covariates. This full model must be carefully & thoughtfully constructed to avoid highly correlated/colinear covariates, but it is quite possible to create such a model that will still converge. Inference based on this full model is conducted not via stepwise regression and the likelihood ratio test, but by estimating model parameters and a measure of their uncertainty (bootstrap 95% confidence intervals, for example). The expected clinical impact of covariate effects are then evaluated given the parameter estimates and the uncertainties around these estimates. In addition, conclusions about covariates that had relatively little impact on model parameters can be made with some understanding of how precisely these small ("insignificant") effects were estimated. So instead of saying that a covariate has no effect on a model parameter, one can assess if the lack of effect is actually due to the lack of a relationship, or if the finding is due to insufficient data. I've also read that this full model approach leads to standard errors that are more accurate than a stepwise regression approach, which results in overly optimisitic standard errors (4). The other benefit of this approach is that once the full model has been developed, computations are spent on getting estimates of parameter precision (bootstrap) rather than a lengthy stepwise regression process. Of course there are some practical challenges with this idea, and I have to admit that I still routinely use stepwise backward elimination from a full model as the primary covariate model building tool. I'm working on building a set of case studies to convince myself that the full model/bootstrap approach is sound. Thanks in advance for your thoughts. Marc Gastonguay References: (1) Wahlby U, Jonsson EN, Karlsson MO. Assessment of actual significance levels for covariate effects in NONMEM. J Pharmacokinet Pharmacodyn 2001; 28(3):231-252. (2) Wahlby U, Bouw MR, Jonsson EN, Karlsson MO. Assessment of type I error rates for the statistical sub-model in NONMEM. J Pharmacokinet Pharmacodyn 2002; 29(3):251-269. (3) Gobburu JV, Lawrence J. Application of resampling techniques to estimate exact significance levels for covariate selection during nonlinear mixed effects model building: some inferences. Pharm Res 2002; 19(1):92-98. (4) Altman, D. G. and P. K. Andersen. 1989. Bootstrap investigation of the stability of a Cox regression model. Statistics in Medicine 8: 771-783

From: gary.maier@sepracor.com Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches Date: Friday, September 26, 2003 7:54 AM Hello Marc I think that some of your criticisms of the stepwise approach are right, however as an industrial scientist I believe that we need some sort of acceptable and "easy to implement strategy" in selecting model covariates. If we rely exclusively upon what the modeler believes to be "the relevant set of variables" to examine we introduce too much "art" into the process. The issue then becomes when we send our pop pk reports to FDA there will not be any guideposts for them to follow and the sponsor could end up in endless discussions with regulatory agencies. I think that if stepwise regression is not the answer it is up to the more statistically oriented users to propose an alternative that relies upon some type of statistical guidelines not just art/judgment alone Gary Maier Sepracor gmaier@sepracor.com

From: marc.gastonguay@snet.net Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches Date: 9/26/2003 8:19 AM Gary, Thanks for your comments. As you indicated, we are industrial scientists, and I think that we should allow science to drive our model building - not some formula that is pre-defined by "more statistically oriented users". In fact, some statisticians feel the same way: Here's a quote from Henderson and Velleman's paper "Building multiple regression models interactively" (1981, Biometrics 37: 391-411): "The data analyst knows more than the computer," ..."failure to use that knowledge produces inadequate data analysis". Whatever approach you choose to take, please remeber Box's point of view that the resulting model is still the "wrong" model. If we use science (not art and not statistics) to guide the covariate model I think we'd all agree that the resulting model is a more "useful" one. That's not to say that we should not provide evidence of goodness of fit, predictive performance, and lack of bias with respect to remaining covariates, etc. I just think we should spend more time on evaluating the model and parameter estimates and less time on a prescribed stepwise model building approach that is known to be problematic. Best regards, Marc

From: bachmanw@globomax.com Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis? Date: 9/26/2003 8:59 AM Marc, I don't see any provision for exploratory analysis in your proposal. Do we always know the model? Have we never serendipitously discovered relationships that were contrary to our apriori conceptions or known science? How do you know if something is clinically relevant if you aren't aware of it? The building approach, severely flawed as it is, at least has a shot at uncovering new covariates. I read your approach correctly, you are getting your full model from prior knowledge and covariates of interest. Did I miss something here? Have you abandoned exploratory analysis? Bill

From: Jakob.Ribbing@farmbio.uu.se Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches Date: 9/26/2003 9:33 AM Dear all, A few comments on the recent discussion on stepwise covariate modelling. We have just submitted a paper (Jakob Ribbing and E. Niclas Jonsson, Power, Selection Bias and Predictive Performance of the Population Pharmacokinetic Covariate Model) on a simulation study that investigates the effects of stepwise covariate modelling and in short the conclusions relevant to what has been discussed on NMusers are: 1. Stepwise comparison should NOT be performed on a SMALL DATASET (? 50 subjects) if the purpose is predictive modelling: 1. Weak covariates are heavily biased when selected based on a statistical criterion. Selection bias is caused by the selection procedure used and is not due to the estimation method used. 2. Because of the heavy selection bias a weak covariate could be expected to worsen the predictive performance if selected 3. A weak and clinically insignificant covariate cannot be separated from a clinically significant covariate because of this selection bias. Thus, the covariates which are statistically significant will also most often appear clinically significant even if they arent! 4. Bias correction or other selection criteria than the p-value may allow stepwise regression even on small datasets. 2. Testing correlated covariates for inclusion in the model is not harming the predictive performance of the final model. However, a large dataset is required in order to select, with enough certainty, the better of two highly correlated covariates. To connect to what was said by Marc on this topic, I do agree to that requiring statistical significance of covariates SOMETIMES can be harmful and contra productive if the purpose is predictive modelling. However, even in these cases stepwise regression could be useful for hypothesis generation. Marc suggested selecting the covariate model based purely on prior knowledge, regardless of statistical significance in the dataset analyzed, to estimate the covariate-model parameters. On the other hand, this prior knowledge can be partly elicited from stepwise covariate modelling on a prior dataset. This is an appealing strategy that we will compare to others in a current simulation study, but no results from this are available yet. Best regards, Jakob Jakob Ribbing, MSc Division of Pharmacokinetics and Drug Therapy Department of Pharmaceutical Biosciences Uppsala University Box 591 SE-751 24 Uppsala SWEDEN Phone: +46 18 471 44 37 Mobile phone: +46 70 450 33 77 Fax: +46 18 471 40 03 Email: jakob.ribbing@farmbio.uu.se

From: mark.e.sale@gsk.com Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches Date: 9/26/2003 9:33 AM My own perspective, which many of you have heard already. My concern is usually more with getting the model right than with any statistical test. As I understand it the primary problems with step wise regression are: 1. Inflated type 1 error, and the related inflated Rsquare values, downward bias standard errors for parameters etc. You basically are data dredging, and if you look at enough random effects you'll find one. This concerns me only somewhat, since post-hoc models should always be regarded as hypothesis generation, in a statistical sense. From Frank Harrell - Regression Modeling Strategies: "Step wise variable selection ... if this procedure had just been proposed as a statistical method, it would most likely be rejected because it violates every principle of statistical estimation and hypothesis testing" Also, I typically want the "best" model (whatever that means), so I worry more about type 2 errors than type 1 errors. 2. (More important in my view) - confounding between variables, nicely demonstrated by this paper: Interaction between structural, statistical and covariate models in population pharmacokinetic analysis (Wade JR, Beal SL, Sambol NC J Pharmacokinetics and Biopharmaceutics, 1994 Vol 22 (2) 165-177) Basically, this means that the answer you get depends on how you get there. IMHO, the only way to get the "best" answer is with a formal search of the models that are considered plausible or of interest (either based on previous data or biology). A formal search of the plausible models does not address the issues of inflated type 1 error (in fact it may make it worse). But again, I'm usually not concerned (much) about that, I want the best model. Penalties can introduced to addresses issues of parsimony and Bayesian priors. This method is: Objective Can be predefined (although that only partly helps with inflated type 1) Includes all effects (e.g., compartments, omega terms, residual errors, lag times etc), not just covariates. Robust - it apparently will invariably find the "best" model among those considered. Fast - we are currently running this using distributed computing on 1000 computers, Can be put in a Bayesian framework with prior knowledge (although this isn't currently implemented). This may represent a compromise between Marc Gs comment that we could just build a full covariate model (assumes that we know the model - completely informative prior), and a more traditional (hypothesis testing - uninformative prior) view. Mark Sale M.D. Global Director, Research Modeling and Simulation GlaxoSmithKline 5 Moore Drive RTP NC, 27709 919-483-1808

From: chuanpu.2.hu@gsk.com Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise a pproaches -> abandoning exploratory analysis? Date: 9/26/2003 9:36 AM Marc, I enjoyed reading your posts. Stepwise procedure, as a specific case of model exploration, has awful properties in statistical inference. For this reason, I advocated restricting model explorations, in a scenario where the inference properties are important, at the last PAGE meeting ( http://www.page-meeting.org/page/page2003/Chuanpu.pdf). The essence is that model exploration increases the chance of finding the "right" model (by that I mean "better", I should say), but also making the inference properties worse, hence there is a balance that needs to be maintained. The optimal balance may depend on your goal of the modeling exercise. One particular type of practical challenge comes from estimating a large model that the data could not quite support. As you suggested, one frequently has to make compromises in actual applications. A note: the term you used "not statistics" could be terribly misunderstood. (I think you mean something like "not stepwise regression.") To produce credible results, our procedures must have sound statistical properties, which includes correct standard errors, type I errors, etc. Best regards, Chuanpu -------------------------------------------------------------------------- Chuanpu Hu, Ph.D. Research Modeling and Simulation Clinical Pharmacology Discovery Medicine GlaxoSmithKline P.O. Box 13398 Five Moore Drive Research Triangle Park, NC 27709 Tel: 919-483-8205 Fax: 919-483-6380

From: Ken.Kowalski@pfizer.com Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches Date: 9/26/2003 10:25 AM Marc, I agree with you that the full model is a better basis for making inference such as in constructing confidence intervals. However, I'm not ready to throw away development of parsimonious models either. When the parsimonious model has considerably fewer parameters than the full model, it may provide better predictions by smoothing out some of the noise in the full model predictions since many of the parameter estimates from the full model are just estimating noise. Many view development of a full model as a means to an end...I don't. I like to report out results for the base (no covariates), full, and final parsimonious models and use each of these models for different purposes. The latter, parsimonious models, I typically like to use for predictions. That being said, I think we need to be cautious when using stepwise procedures or any model building procedure (including WAM) particularly when we are dealing with a high degree of collinearity among the covariates. Forward selection procedures can lull one into a false sense of security if the modeler is not cognizant of the collinearity. Whereas building a full model will often require the modeler to deal with it. Ken

From:Ken.Kowalski@pfizer.com Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis? Date: 9/26/2003 10:43 AM Bill, You make a good point. There will often be a need to consider ad hoc covariates that weren't specified during the initial building of a full model. But that is also true of any systematic model building procedure including forward selection. Before beginning a forward selection procedure we still need to list a priori what covariate effects we wish to consider for inclusion in the first step of the procedure. I maintain that incorporating an ad hoc covariate into a final model after using a stepwise procedure is not as informative as incorporating the ad hoc covariate into a previously developed full model. Ken

From: bachmanw@globomax.com Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis? Date: 9/26/2003 10:48 AM Ken, I was not necessarily advocating a stepwise procedure but pointing out the need for an exploratory mechanism in whatever covariate selection/model selection process we use. Finding out things we don't know is the fun part. Bill

From: marc.gastonguay@snet.net Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis? Date: 9/26/2003 10:52 AM Ken, Bill, Jakob, Chuanpu and Mark, Thanks for your feedback. As I indicated, the full model/bootstrap approach is still an idea and there are issues to be worked-out. Perhaps we can find a compromise that addresses all of the issues you've raised. Let me try to address the main issues. First of all, I did not mean to denegrate statistics as a discipline and I should have said stepwise regression. Thanks for pointing this out, Chuanpu. On exploratory analysis and "Knowing" the model: Of course we never really know the model and I do think that we should use the usual goodness of fit diagnostics to compare possible alternatives and guide the development of the structural model, while keeping prior information in mind. In building the full covariate model, I suggested that covariates should be included based on prior scientific knowledge AND your interest in exploring a particular covariate effect. This does not assume that you know the model ahead of time. If you are interested enough to do an exploratory analysis on a particular covariate, you should include it in the full model. I don't think we should proceed with the "kitchen sink" approach, though. As has been mentioned before, you've got to be careful about how you construct the full model so that you avoid problems with correlated/colinear covariates (especially when the data set is small). You may even need a few alternative full models to assess the form of the covariate-parameter relationships (perhaps comparing linear and nonlinear covariate relationships) in order to arrive at a stable full model. This is where graphical exploration of the form of the covariate-parameter relationship can be useful. We don't need stepwise regression to do any of this. On parsimony: I agree that there are certainly advantages to arriving at a parsimonious model. One of the things that is overlooked in a parsimonious model, however, is why a particular covariate was excluded. Was it because the covariate truly has no effect on the parameter of interest or was it excluded because the data are not informative about this potential covariate effect? A full model with point and interval estimates does address this issue. You could envision an approach where the full model is developed and confidence intervals for all parameters are obtained. Then, decisions about moving to a more parsimonious model are made based on the clinical relevance of estimated covariate effects where those covariates having little or no impact are dropped from the model. This preserves the assesment of why a covariate is "insignificant", while allowing a more parsimonious model. I would also suggest that you investigate any remaining trends in covariates that were not included in the full model as part of the model evaluation step. If the model performs poorly with respect to a particular covariate, you may need to go back and pose a new full model. Marc

From: Ken.Kowalski@pfizer.com Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis? Date: 9/26/2003 11:07 AM Marc, Very well said. Ken

From: bachmanw@globomax.com Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis? Date: 9/26/2003 11:21 AM Marc, I know I'm just being a devil's advocate in this discussion, but, there is an argument for (and precedent in other fields) for the "kitchen sink" approach. In the past the argument against it was time consumption. In favor, was the "throwing away data" argument if you didn't use information that was collected at great expense. Now, with the automation of many of these methods (WAM, GAM42, etc) and distributed computing via clusters and grids, time is not as large a factor. Marc, nice segway into my blatant plug - come see our poster at AAPS: Use of a Linux Cluster with PDx-Pop and NONMEM V to Streamline Population Pharmacokinetic Analysis [sorry, the devil made me do it!] Bill ps - It's nice to see a discussion revival on nmuser's, I was beginning to think our listserver was down!

From: mark.e.sale@gsk.com Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis? Date: 9/26/2003 11:48 AM Bill, I have to agree. At the risk of being accused of being a frequentist, the Bayesian view can be taken too far. (collective gasp!!!). But, the reality is that we are often (usually??) wrong about our models, that why we do studies. We shouldn't (yet) complete discard the old "scientific method" of hypothesis generation - data - hypothesis test/new hypothesis generation. The learn/confirm view is frequently more like GenerateHypothesis Convince oneself that hypothesis is true CompletePhD age = 30 Do while not (hypothesis = true) GetData age = age + 1 Data suggests hypothesis isn't true if age > 99 then exit end do end Those in academics should add a call to WriteGrant inside the do loop Mark

From: marc.gastonguay@snet.net Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis? Date: 9/26/2003 12:05 PM Bill, I think the problems with a "kitchen sink" approach go further than time consumption. One of these is the increased chance of obtaining false positive covariates and overfitting along with uninterpretable results. As Mark Sale indicated, even with a broad search approach like the Genetic Algorithm, you need to make rational choices to obtain models that are "considered plausible or of interest (either based on previous data or biology)". Why not use your LINUX cluster to run 1000 bootstrap replicates on the full model? Marc [Thanks for the shameless sales plug.]

From: david_john.garbutt@pharma.novartis.com Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis? Date: Friday, September 26, 2003 12:55 PM Hi, Marc said: 66 You could envision an approach where the full model is developed and confidence intervals for all parameters are obtained. Then, decisions about moving to a more parsimonious model are made based on the clinical relevance of estimated covariate effects where those covariates having little or no impact are dropped from the model. This preserves the assessment of why a covariate is "insignificant", while allowing a more parsimonious model. 99 But there is a big problem here - all those decisions about clinical relevance are as dependent on the data gathered as the results of the automated selection. (eg patient population, concomitant medications, the discovery of a new gene related to metabolism, etc...) So this method cannot be inherently better than the other. To put it another way what make the process (ie selection procedure) more scientific isn't the evidence we find supporting the model it is the fact we can find evidence against our model that can lead us to reject it - paying too much attention to prior knowledge leads us in the opposite direction. Clinical relevance is too shaky to pin everything on - after all before vitamins were discovered no one believed such small quantities of substances in food were important to dietetics. regards, Dave Garbutt DIT, Basel NEW:-> WSJ-310 2.09.22 <-> +41 61 32 49 521

From: lgibiansky@emmes.com Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis? Date: 9/26/2003 12:59 PM Dear All, With all the respect to the other methods that may be more attractive, we cannot rule our forward-addition approach. If you have a model with 5-6 random effects and 20-30 covariates (say, demographics, lab data and concomitant medications, this can easily give you 30), it is unrealistic to fit the full model (30 parameters for each of the random effects). We need to screen the covariates, via diagnostic plots of random effects versus covariates, GAM, significance relative to the base model, etc. If the list of covariates shortens so that you can fit the full model, that is great. But if not, what would you do ? I would add the most significant covariate and continue from that point (forward selection procedure), may be in chunks, adding this covariate to all the parameters at once, and then removing not-significant ones. Mark, is there any alternative to this process if the full model is not converging ? Thanks, Leonid

From: mark.e.sale@gsk.com Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis? Date: 9/26/2003 1:30 PM We (the royal we) think that formal search algorithms work very well in the setting of many possible combinations of covariates, structural and error models. Mark

From: marc.gastonguay@snet.net Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis? Date: 9/26/2003 1:55 PM Hello Dave, You raise an important point. I can see why you'd say that clinical relevance is dependent upon the data gathered in as much that extrapolation of any conclusions from a clinical trial are dependent upon the characteristics of the population studied, etc. Let's assume we're in the case where you can extrapolate results of a trial to the general patient population. Isn't clinical relevance of a model parameter something that must be assessed given some understanding of what is a meaningful change in some endpoint (concentration, response, toxicity, etc.). For example - the clinical relevance of a covariate-induced change in clearance could be assessed by understanding how much exposure can vary before the pharmacodynamic response yeilds unacceptible toxicity or lack of efficacy. If we cannot determine the clinical relevance of a covariate effect, it is very difficult to make use of the model in a "learning" mode (it may still be useful for prediction). When we start with a full model, we can objectively evaluate whether or not a covariate effect is supported by the data at hand by examining point and interval estimates of the covariate parameters. If the original hypothesis is that all of the covariates in the full model are clinically relevant, the data may very well contradict this by resulting in a covariate parameter estimate that is precise and near zero (or the null value for that covariate). The approach I described for covariate modeling does utilize some Bayesian notions in that inference is based on parameter estimates and uncertainties, rather than p-values, but it does not completely rely on informative priors. Not every parameter in the full model has to be based on prior knowledge. The full model could also include covariates that you know nothing about, but are interested in exploring. Marc

From: sduffull@pharmacy.uq.edu.au Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise a pproaches -> abandoning exploratory analysis? Date: 9/26/2003 5:57 PM Mark I don't think that building a model from prior information is necessarily particularly Bayesian. Indeed the use of prior information does not necessarily imply a Bayesian analysis (as shown in the recent JPKPD article using prior in a frequentist sense [I'm at home and don't have the particular details at hand]). You are right though that hypothesis testing is essentially a frequentist concept... nevertheless there are Bayesian alternatives that also do the "same" thing. Consider an hypothesis test (reject/fail to reject) as just a special case of the consideration of the posterior probability that the data arose from a particular model, but with quantal rather than quantitative criterion. Although this is a simplification (and I stand to get many emails about this!) the process of how you go about building models is not necessarily a function of the statistical framework of your analysis ... the method of selection of the best model (however) is. Incidentally I would suggest setting age_max to be << 99 :) Steve

From: marc.gastonguay@snet.net Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis? Date: 9/27/2003 9:07 AM Would it be fair to summarize this lengthy discussion as follows?: 1. Stepwise regression has its problems, but we still use this for covariate model development on a routine basis (in addition to other less commonly employed solutions, such as Wald Approximation Method and Genetic Algorithm). 2. There are different opinions on whether to build a covariate model using stepwise forward addition (or forward/backward) or to start at a full model and proceed with a stepwise backward elimination method. 3. We also have different opinions on how to select the covariate-parameter relationships to be investigated. This ranges from one extreme, where every measured variable should be investigated as a potential effect on every model parameter to the other, where the "best" model is known a priori. The optimal solution is undoubtedly somewhere in the middle. 4. There are different views on how to select the best model, where one approach relies on p-values obtained using the Likelihood Ratio Test and another possibility is based on point and interval estimates (or even full posterior distributions) of model parameters. 5. We need more comparisons and examples to help us work the balance between theoretically appealing approaches and practical implementation. I think the take-home point is that there is NOT just one way to build a model and that whatever you end up with is still the wrong model. Maybe we should spend more time on model evaluation and assessing sensitivity to assumptions once we do arrive at the "best" model. Marc

From: mark.e.sale@gsk.com Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis? Date: 9/29/2003 8:18 AM Nice try at diplomacy Marc, but I'm going to be refractory. We have preliminary data, and Rob Bies has presented some data that show that forward addition/backward elimination rarely (if ever) gives the best answer. Again, my interest is in finding the best answer. The assumption behind any step-wise method is that the search space is monotonically down hill, that there are no interactions between the different effects being considered. That is probably never true. The search spaces we examine are very complicated, not only among covariates, but between structural effects, random effects and covariates. I think that the reason we all use forward addition/backward elimination is related to the metaphor, "If the only tool you have is a hammer, then everything looks like a nail" Lets list a few of the nail-like properties of this problem: 1. If you don't think about it too hard, this algorithm can be used for hypothesis testing. Think about it too hard would include all those pesky statistical assumptions, none of hich are true. 2. If you don't think about it too hard, it can be put in a Bayesian framework. 3. There is no reason to believe that it ever gives the right answer, and we have data to support this. Fortunately, other disciplines have thought about efficient ways to search discrete spaces, and the assumptions required for the different methods. we probably should learn from them. For those interested in learning about modern methods of model selection, we have openings in US (North Carolina and Phil), UK and Italy. Mark Sale M.D. Global Director, Research Modeling and Simulation GlaxoSmithKline 5 Moore Drive RTP NC 27709 919-483-1808

From: harry.mager.hm@bayer-ag.de Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis? Date: 9/29/2003 8:59 AM Dear all, it seems that the decisive problem is that covariate selection and hypothesis testing have to be different processes. If not, the statistical properties of the various estimators will not be known, regardless which selection criterion / criteria or "modern" method is used. Since this prerequisite is too restrictive in most cases in practice, we have to accept that we are relying on approximations to the truth at the very best. Actually, there is no special problem with stepwise procedures that is not inherent in other procedures (genetic algorithms etc., etc.), too. At its very end, stepwise procedures may result into "all regression", examining all possible subsets. With increasing number of covariates to be explored, all regression may result in a computational burden not easy to handle. Most of the modern methods essentially only try to reduce computation time, so the actual problem will remain. If we want to select the "best" model, the very first task would be to define what is "best", i.e., the criterion / criteria to be met have to be defined. It has be shown using a vast amount of simulations, and the results are also supported by theoretical considerations, that whatever criterion is used, the results of a selection process will be overly optimistics with regard to the selection criterion. Harry

From: Ken.Kowalski@pfizer.com Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis? Date: 9/29/2003 9:58 AM Marc, Bill, et al., Nice summary but to me the key point to this discussion based on Pete Bonate's original message in this thread is that we need to be cognizant of the effects of collinearity when we build covariate models regardless of the model building procedure we use. The problem with the 'kitchen sink' approach is that we are putting too much trust in the statistical algorithm to sort out the true covariates from the nuisiance covariates that may be highly correlated with them. The more things we toss into the 'kitchen sink' the more likely we will pick up some of these nuisance covariates in our model. Why is it that we are willing to make very strong mechanistic assumptions regarding the structural model but when it comes to a list of covariates to be investigated we are unwilling to prune the list based on this same mechanistic reasoning? Of course we do try to make the list somewhat plausible (e.g., we don't typically include shoe size as a covariate but its likely to be correlated with weight and may actually explain some of the interindividual variation), we just need to be a little more discriminating (rather than just pruning out the obvious such as shoe size). With regards to building a full model, it is a bit of a 'straw man' argument to say full models are difficult to develop when there are 20-30 covariates. The difficulty is not the number of covariates but the amount of independent information contained in these 20-30 covariates. The full model approach requires the data analyst to deal head-on with the collinearity issue. The issue is not of computational speed. Just because we now have procedures that may be computationally faster than what we had say 10 years ago, doesn't mean that we should blindly (i.e., ignoring collinearity) investigate more covariates just because we can. Ken

From: tgordi@buffalo.edu Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise approaches -> abandoning exploratory analysis? Date: 9/29/2003 11:00 AM Hi! I am not as much experienced with NONMEM (or statistics) as many of the contributors but I guess there are a couple of simple rules one can follow with regard to inclusion of covariates in any model. The first is, of course, just consider those covariates that are of interest. What is of interest might differ from case to case. If it is a exploratory study, one might want to look at many parameters. If one want a more "practical" model, weight and gender might be enough (just examples). A lot of information can be collected in a clinical study but not all of it will be or should be tested. The second simple rule is that the more tests one does, the higher is the risk of finding something that is not there (as Ken mentions below). Although there probably are several methods to deal with the situation, one simple approach would be to decrease the level of significance, let's say use 0.001 instead of 0.05, for a covariate to be deemed influential. What value to choose will be dependent on the number of tests you do, although there are some variations of the rule, which are less conservative. The negative side is, of course, that there will be more difficult to show any significance. However, this might function as a driving force to choose the best candidates only. I understand that this is not the answer to all of the problems but I would say that following these rules helps one to be less "wrong" in the analysis. This discussion also brings me to another issue I have been thinking about. If we agree that conducting several tests lead to inflated TYPE I error, what are the consequences when several different structural models are tried in our routine PK/PD modeling? It is a common practice to accept a drop of 3.8 in the OFV (which is related to a significance level of 0.05) as a "proof" of the superiority of a model over another. My point is not whether using OFV is optimal. The question is whether one should be more restrictive and use 0.01 or even less. Toufigh Gordi

From: lgibiansky@emmes.com Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise a pproaches -> abandoning exploratory analysis? Date: 9/29/2003 11:16 AM Ken, Your message seems to imply that (i) Large number (e.g., 30) of covariates can be found ONLY in problems where we have a lot of collinear covariates, (ii) Thinking through the list of covariates, we ALWAYS can reduce it to the manageable number (and sort them out between different random effects). I would disagree with both statements. If you cannot create manageable full model starting from the base model, you need to screen covariates, that leads us to the forward-addition algorithm (unless you work for Mark Sale with the access to the genetic algorithm and a cluster of 1000+ computers waiting for your input). Unspoken assumption (sufficient condition of convergency) behind the forward addition algorithm is the convexity of the -log(likelihood) as a function of covariates. One can easily create an artificial example where this assumption is violated but it would be interesting to see any real example where this is not true. Sure, I agree that collaterality is the issue that needs to be thought ahead of time, but this is not the only source of the large covariate list. As to the pruning the list, I think it is dangerous to do it too aggressively: this can be very subjective; can prevent us from uncovering new totally unexpected dependencies. I would prefer to do more or less formal search of the explanatory variables, and then interpret them in clinically relevant terms rather than look for clinically relevant dependencies only. This may reduce subjectivity of the analysis, restricting it to the subjective explanation of uncovered dependencies instead of the subjective choice of alternatives to investigate. Leonid

From: GUZY@xoma.com Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise a pproaches -> abandoning exploratory analysis? Date: 9/29/2003 12:22 PM I like the "shoe size" example for covariate. I would like to add my part in this interesting discussion. I am a statistician and therefore aware of how statistics can erroneously lead us to no sense results if we do not pay attention. Suppose we have 100 covariate and let say that there is a 5% chance to get by chance a covariate which exhibit high correlation (we would therefore take it as a part of our model). In average we would have 5 covariate with high correlation that are the result of pure chance. Suppose now that we have 5 other covariate that really exhibit a real correlation and they show up when performing the analysis. If we don't do any statistical adjustment, we will take the 10 covariate and one of them would be for example shoe size. How can we know that shoe size was the result of chance while age for example was the result of a real trend when they both statistically exhibit the same strength of association? Now if you have 100 covariate, you must make a statistical adjustment to prevent inflation of the alfa type error and you can easily reach the situation where you will not be able to accept any of the covariate(you lost completely statistical power). Using purely a machine that would do everything for you without using any prior information (does not mean Bayesian) is not reasonable. I believe that experts should discuss the problem in advance and come up with up to 5 covariate maximum and then performing the analysis. The other issues related to the use of high number of covariate has been addressed before and I will not reiterate (collinearity, other numerical difficulty etc..) Serge Head of Preclinical Statistics and Pharmacometrics, XOMA Corporation President POP-PHARM

From: mark.e.sale@gsk.com Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise a pproaches -> abandoning exploratory analysis? Date: 9/29/2003 12:41 PM Leonid My last email on this, I promise. We have several examples of cases where your convexity assumption (which I called monotonically downhill) is not the case, formally reported by Janet Wade et al. What this means is that the answer you get depends on how you get there, and different users will get different answers for the same data set, with the same prior - not a good result for a scientific analysis. This is not uncommon, in fact it is essentially invariably the case. For my last commercial, all will become clear at Paolo Vicini's symposium at AAPS Wed Oct 29, "What is the right model? Issues surrounding the evaluation of Pharmacostatisical Models in Drug Development" (Well, maybe not completely clear). Mark

From: Ken.Kowalski@pfizer.com Subject: RE: [NMusers] order of covariate inclusion -> avoiding stepwise a pproaches -> abandoning exploratory analysis? Date: 9/29/2003 2:37 PM Leonid, I think you are missing my point. I have nothing against investigating a large number of covariates (e.g., >=30) if they truly provide scientifically relevant and independent information. However, typically what happens when we have a long list of covariates is that many covariates may be collinear and hence are redundant. Such redundancy based on these nuisance covariates that are correlated with the important mechanistic covariates can cause havoc with any model building procedure, masking our ability to discern the true covariate effects. Forward selection procedures are particularly vulnerable because they can be blind to the collinearity issue as they can often find a good fitting model without running into stability/over-parameterization issues that a full model would be confronted with. However, as I've said previously, just because forward selection can find a good fitting model doesn't mean it found the right one. For example, by chance a nuisance covariate that is highly correlated with the true covariate may be selected first by a forward selection procedure and because of the order of testing the true covariate may never get further evaluated in the forward selection procedure and hence will be excluded in favor of the nuisance covariate. I can't tell you how many times that a modeler will say they have a difficult time interpreting covariate effects in the final model selected by a stepwise procedure because they felt certain excluded covariates were more scientifically plausible. I'm merely suggesting that we be a bit more discriminatory with developing our list of plausible covariates to investigate so that we have a set that are the most scientifically plausible and independent. The main reason we use a combination of forward selection/backward elimination with a higher alpha level for inclusion (e.g., alpha = 0.05) is precisely to help mitigate the problem with forward selection alone. By increasing the alpha level for inclusion we allow for "bigger" models to be tested before pruning to a parsimonious model using backward elimination with a smaller alpha level for exclusion (e.g., alpha =0.01 or 0.001). Of course, if we increase the alpha level for inclusion towards 1.0 the combination forward selection/backward elimination procedure will collapse to a purely backward elimination procedure. Moreover, it may only take setting the alpha level for inclusion to 0.20 to begin to develop bigger models using forward selection that will encounter the ill-conditioning problems due to collinearity that we observe with the full model (if we don't become a bit more discriminating in our choice of covariates). I agree that we could err on the other side as well and eliminate important explanatory covariates if we are too discriminatory. Nevertheless, we need to use our best scientific judgement as well as good statistical principles to really uncover the important covariates. Blindly ignoring the limitations of forward selection and just turning the crank to allow the algorithm to identify covariate effects is risky. That is not to say that I'm against forward selection, just that we need to know when it is appropriate to use it and when its not. I still maintain it is better to identify where the redundancies are and eliminate the least plausible covariates when redundancies exist. Certainly if a less plausible covariate is fairly independent of the other covariates then we don't need to eliminate it. I have no problem using forward selection to identify a parsimonious model once we've streamlined the list to those covariates that provide the most independent information where redundancies are removed based on the covariates that are the least plausible. The problem is knowing when we are in a situation where there is a lot of redundancy. Building a full model and looking at the diagnostics from the full model fit (e.g., COV step) will help us to know when we are in a situation where we should deal with the collinearity. Ken _______________________________________________________