Re: What does convergence/covariance show?

> -----Original Message----- > From: [email protected] [mailto:[email protected]] On > Behalf Of Nick Holford > Sent: Tuesday, August 25, 2009 11:54 PM > To: nmusers > Subject: Re: [NMusers] What does convergence/covariance show? > > Mats, > > You are right - I replied before the coffee had started working so I was indeed in a strange world! > > Nevertheless the isolated finding of P<0.05 should not be uncritically interpreted as being of clinical relevance without other considerations such as adequate a priori power and if possible some plausible mechanism even if the P value suggests an increased hazard of death. > > Nick > > Mats Karlsson wrote: > > > Nick, > > > > You're living in a strange world if killing patients is benefit :) > > > > Mats > > > > Mats Karlsson, PhD > > > > Professor of Pharmacometrics > > > > Dept of Pharmaceutical Biosciences > > > > Uppsala University > > > > Box 591 > > > > 751 24 Uppsala Sweden > > > > phone: +46 18 4714105 > > > > fax: +46 18 471 4003 > > > > *From:* Nick Holford [mailto:[email protected]] > > *Sent:* Tuesday, August 25, 2009 11:15 PM > > *To:* Mats Karlsson > > *Subject:* Re: [NMusers] What does convergence/covariance show? > > > > Mats, > > > > If the trial was powered to test the effect of the treatment on survival then I would think that it would be reasonable to consider some practical consequences. However, FDA would not accept one trial alone as evidence of benefit without other strong supporting evidence from a different trial i.e. the OFV alone is not enough to accept clinical importance. > > > > Nick > > > > Mats Karlsson wrote: > > > > Nick, > > > > If the hazard of patients are dying is significantly (p<0.05) higher on > > the > > > new treatment compared to reference, I don't think you need other evidence > > before it has practical consequences. Without mechanistic understanding, > > would you ignore it and move on to the next analysis? > > > > Mats Mats Karlsson, PhD > > > > Professor of Pharmacometrics > > Dept of Pharmaceutical Biosciences > > Uppsala University > > Box 591 > > 751 24 Uppsala Sweden > > phone: +46 18 4714105 > > fax: +46 18 471 4003 > > > > -----Original Message----- > > > > From: [email protected] <mailto:[email protected]> > > [mailto:[email protected]] On > > > Behalf Of Nick Holford > > Sent: Tuesday, August 25, 2009 10:29 PM > > To: nmusers > > Subject: Re: [NMusers] What does convergence/covariance show? > > > > Mats, Thanks for stating more clearly what I tried to say before. Once again -- I agree that OFV is not a measure of clinical importance. But it is correlated with discernible differences in model predictions that may be of clinical importance. A change of OFV of 5 in a survival model may well be useful to reject a null hypothesis and point to some explanatory variable. There are numerous 'statistically significant' findings in the clinical literature like this that have no practical impact. You do not indicate what else in the survival analysis convinced you that the OFV was associated with something of practical consequence. I trust your decision was not based only on the OFV! Nick Mats Karlsson wrote: > > > > Nick, > > > > I agree that small changes (5-10) in OFV often are not practically > > > > important > > > > and and big changes more often are. However, my point is that OFV is > > not > > > the > > > > right scale to judge importance. You should judge it on the > > consequence of > > > you additional complexity to the model (the magnitude of the found > > drug > > > effect/covariate/etc). Just the other day did I analyze survival data > > > > where > > > > a small (5) change in OFV is of practical consequence. > > > > A true treatment effect of a certain size will improve the OFV in > > relation > > > to the size of the dataset. The larger the data set, the larger the > > change > > > in OFV. However, the estimate of the treatment effect does not change > > > > systematically with the size of the data set. The size of the > > treatment > > > effect is what is more appropriate diagnostic for practical > > consequences. > > > OFV we would use only to make sure that we have found the effect by > > > > chance. > > > > Best regards, > > > > Mats > > > > Mats Karlsson, PhD > > > > Professor of Pharmacometrics > > > > Dept of Pharmaceutical Biosciences > > > > Uppsala University > > > > Box 591 > > > > 751 24 Uppsala Sweden > > > > phone: +46 18 4714105 > > > > fax: +46 18 471 4003 > > > > -----Original Message----- > > > > From: [email protected] > > <mailto:[email protected]> [mailto:[email protected]] > > > On > > > > Behalf Of Nick Holford > > > > Sent: Tuesday, August 25, 2009 7:25 AM > > > > To: nmusers > > > > Subject: Re: [NMusers] What does convergence/covariance show? > > > > Mats, > > > > When I referred to a change of 50 being needed to detect something of practical importance I was not saying that was of clinical relevance. That cannot be judged from the OFV alone. But small OFV changes are > > > > rarely if ever indicators of something that is clinically relevant. > > > > I expect you will agree on this point :-) > > > > Nick > > > > Mats Karlsson wrote: > > > > Nick, > > > > I too would use OFV as the most important goodness-of-fit > > diagnostic when > > > comparing models, especially when deeming something to be > > redundant. If > > > adding a component doesn't reduce OFV, I see no reason to include > > it (I > > > think we're agreeing on something!). However, you write > > > > " Small (5-10) changes in OBJ are not of much interest. A change > > of OBJ > > > of > > > > at least 50 is usually needed to detect anything of practical > > > > importance." > > > > Today we use population methods for everything from very rich pop > > pk > > > meta-analyses to very sparsely informative data sets on survival. > > To use > > > OFV > > > > as a measure of goodness-of-fit is central and look at the risk > > something > > > improved the fit by chance, but I would not use it as measure of > > clinical > > > importance. > > > > Best regards, > > > > Mats > > > > Mats Karlsson, PhD > > > > Professor of Pharmacometrics > > > > Dept of Pharmaceutical Biosciences > > > > Uppsala University > > > > Box 591 > > > > 751 24 Uppsala Sweden > > > > phone: +46 18 4714105 > > > > fax: +46 18 471 4003 > > > > -----Original Message----- > > > > From: [email protected] > > <mailto:[email protected]> [mailto:[email protected]] > > > On > > > > Behalf Of Nick Holford > > > > Sent: Tuesday, August 25, 2009 12:14 AM > > > > To: nmusers > > > > Subject: Re: [NMusers] What does convergence/covariance show? > > > > Mats, Leonid, > > > > Thanks for your definitions. I think I prefer that provided by > > Mats but > > > he doesn't say what his test for goodness-of-fit might be. > > > > Leonid already assumes that convergence/covariance are diagnostic > > so it > > > doesnt help at all with an independent definition of > > > > overparameterization. Correlation of random effects is often a > > very > > > important part of a model -- especially for future predictions -- > > so I > > > dont see that as a useful test -- unless you restrict it to > > pathological > > > values eg. |correlation|>0.9?. Even with very high correlations I > > > > sometimes leave them in the model because setting the covariance > > to zero > > > often makes quite a big worsening of the OBJ. > > > > My own view is that "overparameterization" is not a black and > > white > > > entity. Parameters can be estimated with decreasing degrees of > > > > confidence depending on many things such as the design and the > > adequacy > > > of the model. Parameter confidence intervals (preferably by > > bootstrap) > > > are the way i would evaluate how well parameters are estimated. I > > > > usually rely on OBJ changes alone during model development with a > > VPC > > > and boostrap confidence interval when I seem to have extracted all > > I can > > > from the data. The VPC and CIs may well prompt further model > > development > > > and the cycle continues. > > > > Nick > > > > Leonid Gibiansky wrote: > > > > Hi Nick, > > > > I am not sure how you build the models but I am using > > convergence, > > > relative standard errors, correlation matrix of parameter > > estimates > > > (reported by the covariance step), and correlation of random > > effects > > > quite extensively when I decide whether I need extra > > compartments, > > > extra random effects, nonlinearity in the model, etc. For me > > they are > > > very useful as diagnostic of over-parameterization. This is > > the direct > > > evidence (proof?) that they are useful :) > > > > For new modelers who are just starting to learn how to do it, > > or have > > > limited experience, or have problems on the way, I would > > advise to pay > > > careful attention to these issues since they often help me to > > detect > > > problems. You seem to disagree with me; that is fine, I am not > > trying > > > to impose on you or anybody else my way of doing the analysis. > > This is > > > just an advise: you (and others) are free to use it or ignore > > it :) > > > Thanks > > > > Leonid > > > > Mats Karlsson wrote: > > > > <<I would say that if you can remove parameters/model > > components without > > > detriment to goodness-of-fit then the model is > > overparameterized. >> > > > -- > > Nick Holford, Professor Clinical Pharmacology > > Dept Pharmacology & Clinical Pharmacology > > University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New > > Zealand > > > [email protected] <mailto:[email protected]> > > tel:+64(9)923-6730 fax:+64(9)373-7090 > > > mobile: +64 21 46 23 53 > > http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford -- Nick Holford, Professor Clinical Pharmacology Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand [email protected] tel:+64(9)923-6730 fax:+64(9)373-7090 mobile: +64 21 46 23 53 http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford