Re: Confidence intervals of PsN bootstrap output

On 7/11/2011 1:37 AM, Nick Holford wrote: > Leonid, > > With regard to discarding runs at the boundary what I had in mind was > runs which had reached the maximum number of iterations but I realized > later that Jacob was referring to NONMEM's often irritating messages > that usually just mean the initial estimate changed a lot or variance > was getting close to zero. > > There are of course some cases where the estimate is truly at a user > defined constraint. Assuming that the user has thought carefully about > these constraints then I would interpret a run that finished at this > constraint boundary as showing NONMEM was stuck in a local minimum > (probably because of the constraint boundary) and if the constraint was > relaxed then perhaps a more useful estimate would be obtained. > > In those cases then I think one can make an argument for discarding runs > with parameters that are at this kind of boundary as well as those which > reached an iteration limit. > > In general I agree with your remarks (echoing those from Marc > Gastonguay) that one needs to think about the way each bootstrap run > behaved. But some things like non-convergence and failed covariance are > ignorable because they don't influence the bootstrap distribution. > > There is also the need to recognize that bootstraps can be seriously > time consuming and the effort required to understand all the ways that > runs might finish is usually not worth it given the purposes of doing a > bootstrap. > > The most important reason for doing a bootstrap is to get more robust > estimates of the parameters. This was the main reason why these > re-sampling procedures were initially developed. The bootstrap estimate > of the parameters will usually be pretty insensitive to the margins of > the distribution where the questionable run results are typically located. > > A secondary semi-quantitative reason is to get a confidence interval > which may be helpful for model selection. This may be influenced by the > questionable runs but that is just part of the uncertainty that the > confidence interval is used to define. > > Nick > > On 10/07/2011 11:13 p.m., Leonid Gibiansky wrote: > > > I thought that the original post was "results at a boundary should NOT > > be discarded" and Nick reply was just a typo. If it was not a typo, I > > would disagree and argue that all results should be included: > > Each data set is a particular realization. We should be able to use > > all of them. If some realizations are so special that the model > > behaves in an unusual way (with any definition of unusual: > > non-convergence, not convergence of the covariance step, parameter > > estimates at the boundary, etc.) we either need to accept those as is, > > or work with each of those special data sets one by one to push to the > > parameter estimates that we can accept, or change the bootstrap > > procedure (add stratification by covariates, by dose level, by route > > of administration, etc.) so that all data sets behave similarly. > > Leonid > > > > -------------------------------------- > > Leonid Gibiansky, Ph.D. > > President, QuantPharm LLC > > web: www.quantpharm.com > > e-mail: LGibiansky at quantpharm.com > > tel: (301) 767 5566 > > > > On 7/10/2011 2:57 PM, Stephen Duffull wrote: > > > > > Nick, Jakob, Marc et al > > > > > > > Thanks for your helpful comments. I agree with you that any results > > > > that > > > > are at a boundary should be discarded from the bootstrap distribution. > > > > > > On the whole I the sentiments in this thread align with anecdotal > > > findings from my experience. But, I was just wondering how you define > > > your boundaries for variance and covariance parameters (e.g. OMEGA > > > terms)? > > > > > > For variance terms, lower boundaries seems reasonably straightforward > > > (e.g. 1E-5 seems close to zero). Upper boundaries are of course open, > > > for the variance of a log-normal ETA would 1E+5 or 1E+4 be large > > > enough to be considered close to a boundary? At what value would you > > > discard the result? At what correlation value would you discard a > > > result (>0.99,> 0.97...) as being close to 1. Clearly if this was for > > > regulatory work you could define these a priori after having chosen > > > any arbitrary cut-off. But the devil here lies with the > > > non-regulatory work where you may not have defined these boundaries a > > > priori. > > > > > > Steve > > > -- > > > Professor Stephen Duffull > > > Chair of Clinical Pharmacy > > > School of Pharmacy > > > University of Otago > > > PO Box 56 Dunedin > > > New Zealand > > > E: [email protected] > > > P: +64 3 479 5044 > > > F: +64 3 479 7034 > > > W: http://pharmacy.otago.ac.nz/profiles/stephenduffull > > > > > > Design software: www.winpopt.com