RE: Confidence intervals of PsN bootstrap output

-----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Nick Holford Sent: Monday, July 11, 2011 7:37 AM To: nmusers Subject: Re: [NMusers] Confidence intervals of PsN bootstrap output Leonid, With regard to discarding runs at the boundary what I had in mind was runs which had reached the maximum number of iterations but I realized later that Jacob was referring to NONMEM's often irritating messages that usually just mean the initial estimate changed a lot or variance was getting close to zero. There are of course some cases where the estimate is truly at a user defined constraint. Assuming that the user has thought carefully about these constraints then I would interpret a run that finished at this constraint boundary as showing NONMEM was stuck in a local minimum (probably because of the constraint boundary) and if the constraint was relaxed then perhaps a more useful estimate would be obtained. In those cases then I think one can make an argument for discarding runs with parameters that are at this kind of boundary as well as those which reached an iteration limit. In general I agree with your remarks (echoing those from Marc Gastonguay) that one needs to think about the way each bootstrap run behaved. But some things like non-convergence and failed covariance are ignorable because they don't influence the bootstrap distribution. There is also the need to recognize that bootstraps can be seriously time consuming and the effort required to understand all the ways that runs might finish is usually not worth it given the purposes of doing a bootstrap. The most important reason for doing a bootstrap is to get more robust estimates of the parameters. This was the main reason why these re-sampling procedures were initially developed. The bootstrap estimate of the parameters will usually be pretty insensitive to the margins of the distribution where the questionable run results are typically located. A secondary semi-quantitative reason is to get a confidence interval which may be helpful for model selection. This may be influenced by the questionable runs but that is just part of the uncertainty that the confidence interval is used to define. Nick On 10/07/2011 11:13 p.m., Leonid Gibiansky wrote: > I thought that the original post was "results at a boundary should NOT > be discarded" and Nick reply was just a typo. If it was not a typo, I > would disagree and argue that all results should be included: > Each data set is a particular realization. We should be able to use > all of them. If some realizations are so special that the model > behaves in an unusual way (with any definition of unusual: > non-convergence, not convergence of the covariance step, parameter > estimates at the boundary, etc.) we either need to accept those as is, > or work with each of those special data sets one by one to push to the > parameter estimates that we can accept, or change the bootstrap > procedure (add stratification by covariates, by dose level, by route > of administration, etc.) so that all data sets behave similarly. > Leonid > > -------------------------------------- > Leonid Gibiansky, Ph.D. > President, QuantPharm LLC > web: www.quantpharm.com > e-mail: LGibiansky at quantpharm.com > tel: (301) 767 5566 > > > > On 7/10/2011 2:57 PM, Stephen Duffull wrote: >> Nick, Jakob, Marc et al >> >>> Thanks for your helpful comments. I agree with you that any results >>> that >>> are at a boundary should be discarded from the bootstrap distribution. >> >> On the whole I the sentiments in this thread align with anecdotal >> findings from my experience. But, I was just wondering how you >> define your boundaries for variance and covariance parameters (e.g. >> OMEGA terms)? >> >> For variance terms, lower boundaries seems reasonably straightforward >> (e.g. 1E-5 seems close to zero). Upper boundaries are of course >> open, for the variance of a log-normal ETA would 1E+5 or 1E+4 be >> large enough to be considered close to a boundary? At what value >> would you discard the result? At what correlation value would you >> discard a result (>0.99,> 0.97...) as being close to 1. Clearly if >> this was for regulatory work you could define these a priori after >> having chosen any arbitrary cut-off. But the devil here lies with >> the non-regulatory work where you may not have defined these >> boundaries a priori. >> >> Steve >> -- >> Professor Stephen Duffull >> Chair of Clinical Pharmacy >> School of Pharmacy >> University of Otago >> PO Box 56 Dunedin >> New Zealand >> E: [email protected] >> P: +64 3 479 5044 >> F: +64 3 479 7034 >> W: http://pharmacy.otago.ac.nz/profiles/stephenduffull >> >> Design software: www.winpopt.com >> >> -- Nick Holford, Professor Clinical Pharmacology Dept Pharmacology& Clinical Pharmacology University of Auckland,85 Park Rd,Private Bag 92019,Auckland,New Zealand tel:+64(9)923-6730 fax:+64(9)373-7090 mobile:+64(21)46 23 53 email: [email protected] http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford