covariate selection question

From: Joern Loetsch j.loetsch@em.uni-frankfurt.de Subject: [NMusers] covariate selection question Date: Tue, 17 Jan 2006 13:45:24 +0100 Dear NONMEM users, adding a particular covariate to the basic model does not provide a significant decrease in -2LL. In the course of model building and covariate assignment, adding that particular covariate at a later step did significantly improve -2LL. Deleting that covariate from the final, full model, does significantly increase the objective function. Based on -2LL without regarding now the change in IIV, should that covariate be taken in the final model or not? Thank you in advance. Regards Jrn Ltsch _______________________________________________________ Prof. Dr. med. Jrn Ltsch pharmazentrum frankfurt/ZAFES Institut fr Klinische Pharmakologie Johann Wolfgang Goethe-Universitt Theodor-Stern-Kai 7 D-60590 Frankfurt am Main Tel.:069-6301-4589 Fax.:069-6301-7636 http://www.klinik.uni-frankfurt.de/zpharm/klin/

From: mark.e.sale@gsk.com Subject: RE: [NMusers] covariate selection question Date: Tue, 17 Jan 2006 Joern, It becomes a judgement call (reason #2 to abandon step wise model selection - it isn't objective). Ideally, all of this should be specified prior to analysis - and you don't ask questions that you don't have some a priori reason to believe might be true - don't ask if clearance is a function of astrological sign. In a Bayesian framework, you might have a different criteria for hypotheses that you believe a priori (1 point for wt on volume, but 80 points for hair color on volume) - better statisticians than I could put this in a more rigorous framework. (if you prior probability is 0.8, then delta OBJ = 1, if prior is 0.000001 then delta OBJ = 100, if your prior is 0.0, don't ask the question). It is always better to define subjective criteria prior to the beginning of the analysis, making subjective decisions during an analysis really violates principles of data analysis. (but we do it all the time) Mark Sale M.D. Global Director, Research Modeling and Simulation GlaxoSmithKline 919-483-1808 Mobile 919-522-6668

From: "Joern Loetsch" j.loetsch@em.uni-frankfurt.de Subject: RE: [NMusers] covariate selection question Date: 17-Jan-2006 09:29 Mark, thank you very much indeed. Now, I am still a bit uncertain. One, as you suggest, I could the covariate leave in, which is supported statically. Two, I could it leave out because it does not convincingly improve anything and moreover, it is Weight on F, which is not very helpful anyway. So I tend rather to leave it out. Because I am going to publish the staff, I wanted support to be at ease with the covariates selection that is going to be published and to avoid the taste of randomness or biased selections not supported by statics in the publication. Thank you again Regards Jrn _______________________________________________________ Prof. Dr. med. Jrn Ltsch pharmazentrum frankfurt/ZAFES Institut fr Klinische Pharmakologie Johann Wolfgang Goethe-Universitt Theodor-Stern-Kai 7 D-60590 Frankfurt am Main Tel.:069-6301-4589 Fax.:069-6301-7636 http://www.klinik.uni-frankfurt.de/zpharm/klin/

From: Michael.J.Fossler@gsk.com Subject: RE: [NMusers] covariate selection question Date: Tue, 17 Jan 2006 08:20:24 -0500 What you describe happens frequently. Two (or more) covariates may not have much influence by themselves, but together they influence the fit to a significant extent. However, I would not obsess over the -2LL as a sole criterion by which to judge the fit. Ask yourself the following questions: Does the inclusion of the covariate a) decrease the standard error of the relevant parameter,b) improve the fit (as judged by plots), c) make biologic sense? I would urge you not to rely on the -2LL as the sole criterion - I have seen too many examples where inclusion of a covariate decreased the -2LL but had a negative impact on the fit. Mike ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Michael J. Fossler, Pharm. D., Ph. D., F.C.P. Director Clinical Pharmacokinetics, Modeling & Simulation GlaxoSmithKline (610) 270 - 4797 FAX: (610) 270-5598 Cell: (443) 350-1194 Michael_J_Fossler@gsk.com ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

From: "Jakob Ribbing" Jakob.Ribbing@farmbio.uu.se Subject: RE: [NMusers] covariate selection question Date: Tue, 17 Jan 2006 15:47:48 +0100 Dear Joern, Mike and others, I would agree with Mike. To answer Joerns question on how to interpret the results of the stepwise selection: As far as the p-value/LRT can guide you in selecting the covariate model you should keep this particular covariate in the model. Just be sure to use a p-value/likelihood-ratio which is adjusted for the number of parameter-covariate relations that you have tested (or otherwise explored). To judge if the covariate relation makes biological sense it may be helpful to understand why the covariate first was not significant but later became so. There could be a number of reasons for the covariate-selection behaving this way: 1. Including a very influential covariate-relation may make the picture clearer and other, weaker relations appear from out of the mist due to the reduced random noise. For example, including CRCL on CL for a drug eliminated mainly by renal filtration would reduce the (random) variability in CL so that less important covariate relations could be found 2. One covariate relation could be masking another relation. If the first relation is included in the model the other becomes statistically significant. This behaviour is due to correlation between covariates that both end up influencing the same structural-model parameter (or correlation of estimate between two structural-model parameters). An example of this could be a drug with higher CL for females (compared to males of the same size). This relation may be masked by males generally being larger than females (and size is often an important covariate). Including the one covariate would make inclusion of the other statistically significant. Another example would be model misspecification: Including a linear covariate relation (where another relation would have been more appropriate) could cause a second covariate to compensate for this, eg if WT instead of lean-body weight is included BMI may become statistically significant to compensate for this 3. Random. If the LRT gave almost the same result when including the covariate to the basic and to the latter model (e.g. the nominal p-value changed from 0.011 to 0.099) this could be seen as just a random change. If the p-value required for inclusion were 0.01 the covariate is significant in the latter test but not in the first. This is a problem with all selection methods which either includes a covariate fully (according to the maximum-likelihood estimate) or not at all. On the other hand, getting rid of all the "maybe"-covariates may provide the best big picture of what is important. Further, using the LRT often translates into a p-value - whatever that will tell you :>) Jakob

From: mark.e.sale@gsk.com Subject: RE: [NMusers] covariate selection question Date: Tue, 17 Jan 2006 10:15:58 -0500 Joern, Thanks for the opportunity for me to once again rant on my favourite subjects, the limitations of step wise model building. This behaviour is well documented (see Wade JR. Beal SL. Sambol NC. Interaction between structural, statistical, and covariate models in population pharmacokinetic analysis. Journal of Pharmacokinetics & Biopharmaceutics. 22(2):165-77, 1994 Apr.). First, as you imply, one should clearly not base the final model decision on -2LL alone. Does the covariate addition have any ther good or bad effects (better plots, better PPC, smaller inter individual variances)? Is it biologically plausible or even almost certainly the case? But, on to your question. Imagine, if you will, that you are trying to explain the area of a rectangle. One covariate is a (very) imprecise measure of the length, another is a somewhat less imprecise measure of the width. You put in the length covariate and find a small improvement in ability to explain area - it is a very imprecise measure - or perhaps your structural model is wrong (rather than Area = theta(1)*cov_l x theta(2), you have Area = theta(1)*exp(cov_l) x theta(2), where cov_l is the covariate proportional to length). Next you try cov_w on theta(2) (Area= theta(1) x theta(2)*cov_w) - and this is better. Now, you go back and try cov_l as a predictor of theta(1) - and you find it is helpful, now you have the correct structural and covariate model (with cov_l a predictor of length and cov_w a predictor of width). It can be shown that this can easily occur (the Wade and Beal paper demonstrates it for structural, covariate and variance effects). Hence, my view that step wise searches will only give you the correct answer if all effects are independent - which they never are in complex biological systems. Therefore, step wise searches will never give you the correct answer. so, the answer is, put the covariate in. Mark Sale M.D. Global Director, Research Modeling and Simulation GlaxoSmithKline 919-483-1808 Mobile 919-522-6668

From: Mats Karlsson" mats.karlsson@farmbio.uu.se Subject: RE: [NMusers] covariate selection question Date: 18-Jan-2006 04:21 Hi Mark, Some loose thoughts. Stepwise doesn't equal subjective. Often the stepwise covariate modeling is the least subjective in the entire stepwise procedure of building a population model. It is generally clearer outlined in analysis plans than the stepwise building of the structural or stochastic parts of the model. We know that stepwise model selection has problems, but most of the criticism seems to be focusing on the covariate sub-model. The reason for that may be that none of us would take the time and effort to try a structural or stochastic model that didn't make biological sense. However for covariate model building we do try models that don't make biological sense to everyone. The reason being: (i) it is easier to try too many relations than too few (given that opinions about "biological sense" varies), and/or (ii) it is perceived that regulatory authorities want to have information even about relations that don't make sense (to e.g. to confirm expected non-interactions). I like your point about penalizing decisions based on prior belief. The point that "making subjective decisions during an analysis really violates principles of data analysis" is relevant for confirmatory analyses, but most of the time when we apply biologically rational models we are in learning mode and not making subjective (or data-driven) model building decisions would make the analyses rather useless. The article by Wade et al that you reference, concern mostly the fact that if you get the structural model wrong, other parts of the model can become wrong too (like the covariate model). One would expect that this works the other way around too: If you get your covariate model wrong, the structural model may get wrong too. Similar interactions are likely to occur between other model parts too. Regarding your last comment "step wise searches will never give you the correct answer": (i) the alternative to stepwise searches is to postulate a model before looking at the data - generally not a good idea (ii) no model building procedure will give us the correct answer... I think we all agree that improved model building procedures are valuable, but maybe the part that least needs new methods is the covariate model, we need much more guidance on how to build good structural models. Best regards, Mats -- Mats Karlsson, PhD Professor of Pharmacometrics Div. of Pharmacokinetics and Drug Therapy Dept. of Pharmaceutical Biosciences Faculty of Pharmacy Uppsala University Box 591 SE-751 24 Uppsala Sweden phone +46 18 471 4105 fax +46 18 471 4003 mats.karlsson@farmbio.uu.se

From: Paul Hutson prhutson@pharmacy.wisc.edu Subject: RE: [NMusers] covariate selection question Date: Tue, 17 Jan 2006 22:50:33 -0600 Mark raises an interesting point that may be appropriate for comments from the Wise Ones. I typically consider model building in terms of increasing the number of exponential decay terms, then the covariates and etas trying to lower the objective function. Is it the standard of practice that the covariates are prospectively identified as being collected for inclusion in the model-testing, or are they tested in the model post hoc just because they are available from the exhaustive efficacy & safety testing seen with Phase I and II trials? Most fundamentally, should I declare the covariates to be tested before beginning a trial? but also, Should a higher standard (greater reduction) in the objective function be imposed when doing post hoc data mining because of the effect of repeated measures? I can see where the subsequent use of bootstrap testing tests the robustness of the contribution (value) of the covariate, but should the covariate even be have been accepted in the first place? Part of my concern is the future (maybe current for some of you?) testing of genotypic mutations as covariates of clearance and/or response. Thanks in advance for your wisdom. Paul -- Paul R. Hutson, Pharm.D. Associate Professor UW School of Pharmacy 777 Highland Avenue Madison WI 53705-2222 Tel 608.263.2496 Fax 608.265.5421 Pager 608.265.7000, p7856

From: mark.e.sale@gsk.com Subject: RE: [NMusers] covariate selection question Date: Wed, 18 Jan 2006 08:17:39 -0500 Mats, It isn't the step wise part of the traditional model building that is subjective. Step wise can be completely objective - as in the automated step wise linear and logistic regression algorithms in many stats packages. The subjective part comes when someone is weighing a 3 point decrease in OBJ, with maybe a little better time vs wres (or cwres), but not a compelling biological basis, but now fails a covariance step (which Nick tell us isn't really important anyway - and I'm beginning to agree ....) If all these were specified (and quantifiable?) prior to the analysis, then it would be fine, but they tend to be done real time during the analysis. So .... did the analyst at GSK (I mean Pfizer) decide that the race effect on K21 shouldn't be included because the plot really wasn't any better and it doesn't make any biological sense, or because it might cause problems in discussions about labelling? WRT your comment I think we all agree that improved model building procedures are valuable, but maybe the part that least needs new methods is the covariate model, we need much more guidance on how to build good structural models. I agree, I suspect that the best opportunity for improvement in most models is in the structural model, not the covariate model. Mark Sale M.D. Global Director, Research Modeling and Simulation GlaxoSmithKline 919-483-1808 Mobile 919-522-6668

From: "Gobburu, Jogarao V" GOBBURUJ@cder.fda.gov Subject: RE: [NMusers] covariate selection question Date: Wed, 18 Jan 2006 14:34:41 -0500 Dear Mats, I (and others in our pharmacometrics team) could not help but notice your following remark: "(ii) it is perceived that regulatory authorities want to have information even about relations that don't make sense (to e.g. to confirm expected non-interactions)." The following are my personal comments: In my experience, this perception is unfounded. But then perception is reality, as they say. The exposure-response guidance clearly encourages mechanism-based modeling. In fact, I am unaware of any label where the dosing is based on a prognostic factor that does not make biological sense (and derived using mixed-effects modeling). Statistical inference can (only) provide supportive evidence for mechanism-based covariates. I presume you have had an experience with that type of issue and hence your statement. Unless, the specifics of your experience are known, a meaningful discussion cannot occur. Now, there are cases when modeling found covariates that did not make biological sense - no party involved with the drug wanted dose adjustments based on that covariate. There are cases when the opposite (ie., mechanistic not included, but then included after subsequent discussions) also occurred. On the other hand, there is empirical evidence in few cases where the strong prior beliefs do not hold good - so your 'don't make sense' becomes subjective and depends on prior experience one might have had. All decisions will have some risk (false negatives/positives), the only way I can think of increasing comfort in taking this 'risk' is by adhering to biology. The alternative is prohibitively costly. Joga

From: mark.e.sale@gsk.com Subject: RE: [NMusers] covariate selection question Date: Wed, 18 Jan 2006 15:45:40 -0500 Joga, - the rant continues; Thanks for your insight, the view that you relate is consistent with my personal experience with the FDA. But, I think it is important to point out the risk associated with that view. Not that I disagree, I entirely agree, but think that the risk of this approach needs to be pointed out. The risk is a high degree of inertia in our understanding. If we only ask question that are based on what we already believe, it will greatly impede progress. I certainly agree (as I believe you and Mats are saying), that the "data dredging" can only yield hypotheses, not conclusions. But, it is reasonable to ask the questions, even questions that seem silly, based on our current understanding of biology (may I point out: 1. H pylori and ulcers (silly hypothesis, turned out to be true) 2. PVCs and sudden cardiac death (everyone knew that preventing PVCs would reduce sudden death, turns out not to be true) 3. Beta Carotene and Vitamin E and cancer (lots of retrospectively controlled data, good biological explanation - turned out not to be true) the list of hypotheses that were inconsistent with current understanding of biology - that turned out to be true is very long. A good Bayesian, I think, never accepts a hypothesis - only assigns a probability that it is true - while assigning some non-zero probability to many other hypotheses, even the silly ones. In this way, as data is accumulated, we could, in theory, eventually accept hypotheses that are currently viewed as silly, but in fact are correct. Unfortunately, human being have a remarkably limited ability to entertain multiple hypotheses - in fact, rarely can we really entertain more than one at a time (this has actually been researched - and no one can entertain more than about 3 at once). We have one hypotheses, which decide if it is true (invariable we decide that it is, otherwise we wouldn't have a grant to write). Only if that hypothesis turns out not to be true do we look for another. Importantly, we also have a remarkable ability to dismiss data that is inconsistent with our current view of the world - also documented. (e.g., events over the past few years in certain countries in the Middle East). It is generally thought that Gregor Mendel discarded lots of data that was inconsistent with his hypothesis about genetics - his statistics were far to perfect to be random - every experiment sorted nearly exactly as it should. The result of these two effects is a high degree of persistence of hypotheses/conclusions, regardless of whether they are correct. I don't have a solution, to build models without a basis in understanding of biology is silly, and will without question lead to many wrong conclusions. But, to not ask questions just because our current view of biology would reject it as silly is a problem as well. As usual, Bayesians have the answer, if only we were mentally capable of objectively entertaining 10 competing hypotheses at the same time. In the US at least, the NIH funding system insists on one hypothesis, forcing researchers to decide what they believe, and then defend it to the death, rather than keeping an open mind. Mark Sale M.D. Global Director, Research Modeling and Simulation GlaxoSmithKline 919-483-1808 Mobile 919-522-6668

From: "Kowalski, Ken" Ken.Kowalski@pfizer.com Subject: RE: [NMusers] covariate selection question Date: Thu, 19 Jan 2006 16:49:25 -0500 NMusers, I think blaming the NONMEM OBJ and stepwise procedures when we don't like the result of a covariate model selection process is a bit misplaced. I think there are 3 major factors that contribute to a successful covariate model building strategy. To make my point I'll draw an analogy to building a house. The quality of the house we build depends on 1) the materials, 2) the tools available to work with this material, and 3) the proficiency of the builder in using those materials and tools. In covariate model building the material we have to work with is our data, the tools available to us are NONMEM, stepwise procedures, diagnostic plots, etc., and the builder is the modeler. A successful covariate model building strategy depends on how well the modeler understands the limitations of the data, and how effective they are in using the available tools with the data given the limitations. Of course, there are times when our tools are inadequate for the task at hand, however, I think more often the issue is not fully appreciating the limitations of our data and not tailoring our model building strategies to these limitations. I know I'm treading on old ground but in my opinion the diagnostic output from a successful COV step to help us understand the limitations of our data and how it can be used to guide our model building strategy is under-appreciated. Here is my 2 cents on covariate model building and stepwise procedures. I apologize in advance for the long and rambling message, and for treading on old ground. 1) We generally perform systematic procedures for covariate model building to identify a parsimonious model with the fewest covariates that explain as much of the inter-individual variability as possible. We should not be viewing such procedures as providing an assessment of the statistical significance of each covariate parameter. If we want to assess statistical significance of each and every covariate parameter that we might entertain in a systemate covariate selection procedure (e.g., stepwise procedures) we are better off doing this based on a "full model" (see Point 9c below). 2) Stepwise procedures can routinely find a parsimonious model, however, there is no guarantee that they will find the most parsimonious model nor the most biologically plausible model. There may be several almost equally parsimonious models of which a stepwise procedure may find one. Other parsimonious models not selected by a stepwise procedure may be more biologically plausible. 3) While stepwise procedures and the delta OBJ often cannot be used to find a biologically plausible parsimonious model among a search space of both plausible and non-plausible candidate models, this should not be considered an indictment of stepwise procedures or the delta OBJ. In my opinion it should be considered an indictment of the practice of casting too wide a net searching numerous covariate parameters of which many may have dubious biological relevance. While we try to be mechanistic and guided by biology/pharmacology in postulating structural models, when it comes to specifying covariate submodels we often resort to empiricism. We are willing to investigate numerous covariate effects on several parameter submodels in part because it is easy to use a systematic procedure and just turn the crank with little forethought to the covariate parameters we are evaluating. In so doing, we often cross our fingers and hope that the final model selected by a stepwise procedure is one that can be scientifically justified. When the selected model is not scientifically justifiable, it is easy but misguided to place the blame on the stepwise procedure. In this setting the modeler should ask themselves why they are investigating covariate effects that cannot be scientifically justified. Of course, as Mark has pointed out, we need to be cautious here and recognize that with model building we are generating hypotheses and we must be open-minded to possible hypotheses that may run counter to our prior beliefs. 4) Better upfront planning and judicious selection of covariates and covariate-parameter effects can help steer a stepwise procedure to focus only on biologically plausible models. In specifying the covariate parameters to be evaluated by a stepwise procedure, the modeler should ask themselves upfront, "Am I prepared to accept any model within the search space as being scientfically justifiable?" If the answer to this question is "no" then the modeler should re-think the set of covariate parameters before undertaking the stepwise procedure. 5) There may be degrees of biological plausibility. For example, a gender or sex effect may be interpreted as a surrogate for body size rather than an intrinsic gender/sex effect. In this setting one may question whether gender/sex should be included in the investigation. To be more plausible as well as parsimonious in our search of covariate effects the modeler may wish to choose one body size covariate among several measures of body size (body weight, lean body weight, BMI, BSA, etc.) that they feel is the most plausible and use that in the covariate search. Of course the modeler can and should evaluate diagnostics (graphically) to ensure that any trends in the other body size covariate effects not included in the stepwise procedure can be explained by the one selected for evaluation in the stepwise procedure. 6) To avoid or at least reduce the problems associated with collinearity and selection bias we should try to understand the limitations of our data to provide information on the covariate parameters that we wish to evaluate in a stepwise procedure. This is where I depart from others regarding the value of the COV step. I do agree that a successful COV step should not be used as a "stamp of approval" or down-weight/penalize models when the COV step fails. However, when the COV step runs successfully, there is useful diagnostic information in the COV step output that can help steer us away from some of the pitfalls of stepwise procedures such as those encountered by Joern which initiated this email thread (see Points 7 and 8). 7) During base structural model development it is useful to inspect the COV step output to assess correlation in the parameter estimates before undertaking a stepwise procedure. If two structural parameter estimates are highly correlated the modeler may be faced with a difficult decision as to whether a particular covariate effect is more plausible on one structural parameter or the other as there may be insufficient information in the data to investigate the covariate on both structural parameters. For example, suppose concentration-response data has sufficient curvature to support fitting an Emax model but Emax and EC50 may not be precisely estimated. In this setting the correlation in the estimates of Emax and EC50 may be high. This could lead to potentially unstable covariate model investigations (leading to convergence problems) if we begin to evaluate the same covariate on both parameters. For example, suppose that we are interested in evaluating the effect of sex on both Emax and EC50. Inclusion of a sex effect simultaneously on both Emax and EC50 may exacerbate the instability of the model such that the model may not converge. Because of the correlation in these two structural parameter estimates there may be insufficient information in the data to distinguish whether the sex effect should be on the potency or efficacy or both. In this setting the modeler should question whether it is more plausible to investigate a sex effect on potency or efficacy recognizing the limitations of the data to evaluate it on both. If one is more plausible than the other we should not rely on a stepwise procedure to select among the two as it could by random chance select the one that is less plausible simply due to the collinearity in the parameter estimates. 8) Another place where I use the COV step output to help with covariate model building is in evaluating a "full model". By full model I mean the model in which all of the covariate parameters that one might evaluate in a stepwise procedure are included in the model simultaneously. If the COV step output from this full model suggests that it is stable (i.e., no extremely high correlations or numerous moderately high correlations that would result in a extremely high ratio of the largest to the smallest eigenvalues of the correlation matrix of the estimates...obtained from the PRINT=E option on the $COV step) then we have some diagnostic information to suggest that the data can support evaluation of all the covariate effects. 9) Evaluating a full model has intrinsic value regardless of whether or not the full model is used as part of a systematic covariate model building procedure. Some of the benefits of fitting a full model include: a) COV step output can be used to ensure that the data can support the evaluation of ALL the covariate effects of interest (see Point 8 above). b) Among a class of hierarchical covariate models, the full model represents the best that we can do with respect to OBJ. That is, the delta OBJ between the base and full model is the largest. Thus, the full model can be used to help assess the degree of parsimony of a final model selected by a stepwise procedure. A parsimonious model is one that has an OBJ as close to the full model OBJ but with as few covariate parameters as possible. So, if we use a forward selection procedure in a situation like Joern's where perhaps the combination of the two covariate effects that result in a large drop in OBJ only occurs when both are included simultaneously never gets evaluated by the forward selection procedure, we may very well end up with a final model that is not very parsimonious in comparison to the full model. In this particular setting, it may be advantageous to perform a pure backward elimination procedure beginning with the full model, which by definition, would include both of these covariate effects in the model at the start of the procedure. c) If one is interested in assessing statistical significance of ALL the covariate effects, bootstrapping the full model to construct confidence intervals and/or bootstrap p-values is less likely to be prone to statistical issues regarding the adequacy of the Chi-Square assumption for the likelihood ratio test and the problems associated with multiplicity of testing in using a final model based on a covariate selection procedure to assess statistical significance as both issues can result in the inflation of type I errors. Moreover, the issue of ruling out a DDI effect can be easily incorporated by including it in the full model. d) I'll make a shameless plug for the WAM procedure (see Kowalski & Hutmacher, JPP 2001;28:253-275) which makes use of the COV step output from a full model run to identify a subset of potentially parsimonious models that can then be fit in NONMEM. Unlike stepwise procedures that can only select a single parsimonious model, the WAM procedure can give the modeler a sense of the competing models that may have comparable degrees of parsimony. For those interested, Pfizer in collaboration with Pharsight has developed a freeware version of the WAM software that can be downloaded from the NONMEM repository (ftp:/ftp.globomaxnm.com/Public/nonmem). 10) The benefits of the COV step and full model evaluation are difficult to realize unless we are more judicious in our selection of covariates to be investigated. We need to change our practices to understand the limitations of our data when we perform covariate model building and to apply biological reasoning more effectively in developing our submodels. Ken

From: mark.e.sale@gsk.com Subject: RE: [NMusers] covariate selection question Date: Fri, 20 Jan 2006 11:47:04 -0500 Ken, I agree with (nearly) everything you said. Especially the part about casting too wide a net. Where we disagree is: "Stepwise procedures can routinely find a parsimonious model, however, there is no guarantee that they will find the most parsimonious model nor the most biologically plausible model. There may be several almost equally parsimonious models of which a stepwise procedure may find one. Other parsimonious models not selected by a stepwise procedure may be more biologically plausible." You seem to imply (perhaps you don't mean to), that step wise will typically find the most parsimonious model. Not only is there no guarantee of finding the most parsimonious model, you have no reason to expect that you will. And in fact we have internal data that step wise rarely, if ever finds the best solution (in our, as yet unreported results, stepwise is about 0 for 20 in finding the optimal model - a record that makes the Michigan football team look good - Go Bucks). For stepwise to find the true optimal solution the assumption of independence of effects. The index case in this discussion is strong evidence (and I think we all must believe intuitively) that covariate effects - and probably all effects are highly dependent. Stepwise cannot be expected to given you the most parsimonious model in the presence of dependencies of the effects. Personal opinion: The only reason we use step wise is because we haven't found a better way (with the exception of WAM of course). Textbooks on combinatorial optimization will provide insight into better ways. Mark Sale M.D. Global Director, Research Modeling and Simulation GlaxoSmithKline 919-483-1808 Mobile 919-522-6668

From: "Bill Bachman" bachmanw@comcast.net Subject: RE: [NMusers] covariate selection question Date: Fri, 20 Jan 2006 12:49:21 -0500 Mark, While I agree that stepwise covariate selection is neither the optimal nor most statistically rigorous method of model building, your "0 for 20" results seem contrary to numerous simulation and analysis studies in which the stepwise results are in good, if not perfect, accord with the known simulation model. To me this suggests that while not that the ultimate method, it is not completely ludicrous and could be a part of the modelers toolbox. To totally dismiss it seems a bit drastic in my opinion. Bill

From: mark.e.sale@gsk.com Subject: RE: [NMusers] covariate selection question Date: Fri, 20 Jan 2006 12:57:19 -0500 Bill, I might suggest that simulation studies are based on a model in which there are no complex interactions between effects. Interestingly, even when we have done simulation studies, we sometimes find that the optimal model is not the one used for the simulation, due to random variation, a better model exists. This has only been things like the form of the covariate relationship (exponential rather than linear), or very commonly, the structure of the OMEGA matrix. I would be even more generous than to say that step wise is not completely ludicrous - it actually isn't too bad. And I don't totally dismiss it - I'm using at this very moment. But, the only way to be absolutely positive of finding the optimal solution is an exhaustive search, which we have done a couple of times - takes a lot of computer time. One doesn't however, have to do an exhaustive search to proof that a better model exists - one only has to find a single better model. Mark Sale M.D. Global Director, Research Modeling and Simulation GlaxoSmithKline 919-483-1808 Mobile 919-522-6668

From: "Kowalski, Ken" Ken.Kowalski@pfizer.com Subject: RE: [NMusers] covariate selection question Date: Fri, 20 Jan 2006 13:06:24 -0500 Mark, I certainly did not mean to imply that stepwise procedures can find the most parsimonious model...so I don't think we have any disagreement. Of course we should not equate the most parsimonious model with the true model. Even though we may have sufficient power to detect certain covariate effects, the power to select the true correct model (correct combination of covariates), assuming that it is in within the search space of hierachical models being investigated, is often very poor regardless of the covariate selection procedure (including the WAM). In my paper on the WAM I did a simulation study for a relatively small problem (80 subjects, 400 observations, 9 covariate effects) where the true model was contained within the 2**9=512 possible covariate models. While the power was reasonable to detect indivdidual covariate effects (80-90%) the power to correctly identify the true model out of the 512 possible models was only 11.5%. Of course with larger datasets we may have greater power but if we also substantially increase the search space (number of covariate parameters) this will reduce our power to identify the correct model. Ken

From: Leonid Gibiansky leonidg@metrumrg.com Subject: RE: [NMusers] covariate selection question Date: Fri, 20 Jan 2006 13:59:57 -0500 Just to pick on the word: "Interestingly, even when we have done simulation studies, we sometimes find that the optimal model is not the one used for the simulation, due to random variation, a better model exists." ------ You cannot find better model than the true one (the one that was used for simulations). If you was not able to get it, this either mean that the study design (population selection/sample size in case of the covariate model) cannot support the model or the design has no power to separate very similar models. Leonid

From: "A.J. Rossini" blindglobe@gmail.com Subject: RE: [NMusers] covariate selection question Date: Fri, 20 Jan 2006 21:13:32 +0100 This isn't quite true; it's quite context-laden. To clarify, and I'm nitpicking here: Any finite dataset (datasets!) could be reasonably generated by a number of models, not necessarily the ones you used. The larger the individual dataset (and the more independent datasets taken), the better the chance that you actually rediscover the model that you originally used for generation. Of course, in a sense you are cheating, since you have a good clue on how to restrict the space of potential models in order to "rediscover" it. While we like to simulate, we have to remember that just as the same model can generate many realized datasets, the same dataset can originate from a number of models, and this has implications. And back to the original point: stepwise procedures are notoriously awful, failing to preserve type I error in the final model, i.e. they don't lead to sensible decisions based on the model unless you are lucky. Regularization methods of variable selection (where you slowly increase the amount that covariates contribute and look at the selection paths) seem to do reasonably for automatic variable selection by effect for linear and generalized linear (categorical data) regression, and I thought I'd seen a recent paper on this for nonlinear regression, but not yet for mixed effects. I'm not sure how you'd balance fixed and random effects in this case.

From: Mats Karlssonmats.karlsson@farmbio.uu.se Subject: RE: [NMusers] covariate selection question Date: 24 January 2006 12:27 PM Mark and all, I believe we mainly build models because of their predictive ability. In relation to that, it is hard to see any other model than the model data were simulated from as the "optimal model". Mark, what definition of "optimal model" do you use? The discussion on covariate modeling procedures has gone on for years. We know that all procedures have theoretical deficiencies. However, are the properties of these methods so different that their predictive performances are clinically relevant? When we in Uppsala have applied different covariate methods in parallel on real data and then evaluated the relative predictive performance of the final models on a separate set of real data, we have found only marginal differences between the model building procedures. Does anyone have experience with clinically relevant differences in predictive performance between covariate model building procedures for real data? We need also to consider that the model building procedure itself is only one approximation in the covariate model building. Many other are usually ignored. For example (i) many covariates are measured with error, but this is ignored in the analysis of data, (ii) the time-course of covariates are usually imputed using unrealistic assumptions, (iii) many time-varying covariates are assumed time-constant, (iv) models for the shape of the covariate-parameter relation is often, if at all, assessed using simplistic methods, (v) there is usually assumed that there is no inter-individual variability in covariate relationships, (vi) change in a covariate within a subject is assumed to induce the same parameter value change as the same covariate difference between subjects, (vi) interaction between covariates, that is a parameter-covariate relation is dependent on the value of another covariate, are usually ignored, (vii) missing covariate data are regularly imputed with simplistic procedures, (viii) ... Best regards, Mats -- Mats Karlsson, PhD Professor of Pharmacometrics Div. of Pharmacokinetics and Drug Therapy Dept. of Pharmaceutical Biosciences Faculty of Pharmacy Uppsala University Box 591 SE-751 24 Uppsala Sweden phone +46 18 471 4105 fax +46 18 471 4003 mats.karlsson@farmbio.uu.se _______________________________________________________