Re: Confidence intervals of PsN bootstrap output

On 9/07/2011 12:07 a.m., Ribbing, Jakob wrote: > It seems my previous attempt to post this was unsuccessful (either because of the graphs included or because 70 kb was too much?) > > I am resending without graphs and apologize in case of any duplicate postings! > > ------------------------------------------------------------------------ > > *From:*Ribbing, Jakob > *Sent:* 08 July 2011 16:31 > *To:* [email protected] > *Cc:* 'Justin Wilkins'; Norman Z; Ribbing, Jakob > *Subject:* RE: [NMusers] Confidence intervals of PsN bootstrap output > > Dear all, > > I would generally agree with Justin’s comment that one can take any PsN output as is, for internal or external reports. > > However, specifically for the R script used in the PsN bootstrap you can not rely on this, as is. > > There are several issues with this code, of which some are described in the e-mail thread below, from PsN users list. > > You would either have to correct the PsN R-code or else write your own script to get the output that you need for regulatory interaction. > > Regarding what subset of bootstrap sample to use, I do NOT want to open up for a discussion regarding whether there are any differences between bootstrap samples that terminate successfully without or without cov step, and those that terminate with rounding error. This has been discussed previously on nmusers; several times and at length. (There is still a difference in opinion, and as Justin said anyone is free to follow their own preference) > > However, regarding excluding bootstrap samples with terminations at boundary I would strongly discourage doing this by default and without any thought. > > Just as an example, if a portion of your bootstrap samples for an omega element end up at a boundary this is what you would miss out on: > > * If it is a diagonal omega with some frequency of termination at > lower boundary, excluding these would provide a confidence > interval well above zero. By excluding the bootstrap samples > that do not fit with the statistical model that you have > selected, you automatically confirm your selection (i.e. that > data supports the estimation of this IIV or IOV, or whatever the > eta represents), but in my mind the CI based on this subset is > misleading. > * If it is an off-diagonal omega element (representing covariance > between two etas, i.e. on the individual level) with frequent > termination at the upper boundary (correlation of 1) excluding > these bootstrap samples would provide a confidence interval of > the eta correlation that does not include 1. (Correlation is a > secondary parameter calculated based on covariance and IIV > (variance) for the two etas). Again, I would think the CI based > on this subset is misleading, as one automatically confirms the > selection of a BLOCK(3) omega structure, without taking into > consideration a reduction to two parameters that was preferred > by a portion of bootstrap samples. I have included an > illustration of this case in the figure below (I do not know if > postings to nmusers allow including figures, but thought it was > worth a try). > > Obviously, if only using the subset with successful covariance step the exclusion includes bootstrap samples with termination at boundary (if there are any). > > I hope this discussion does not discourage any new users from trying the (non-parameteric) bootstrap. > > In my opinion this is a very powerful method that can provide a huge amount of useful information, beyond the nonmem covariance matrix. > > Next time around the nmusers discussion may be regarding whether the nonmem covariance matrix can be trusted and when a summary of this form is useful, or whether to use the Sandwich or R-matrix; there are many areas where there is no safe ground to tread and no full consensus among users, just as it is sometimes difficult to come up with general advice on what is the most appropriate procedure. > > Best regards > > Jakob > > An illustration of the uncertainty distribution for the correlation between two etas (Notice that full correlation is only available in the subset with boundary problems, as a correlation of one is an implicit boundary condition. Full correlation is also the only reason to the boundary problem among these bootstrap samples): > > */[Jakob] /*Removed > > The original parameterisation is based on covariance between the two etas, rather than correlation, and here the reason to the boundary issue is not at all obvious: > > */[Jakob] /*Removed > > To subscribe to the PsN mailing list: > > http://psn.sourceforge.net/list.php > > Preferably keep any discussion around the specific implementation in PsN to the PsN mailing list, as of little interest to nmusers that are not using PsN. > > The previous discussion on the PsN list, regarding the R-script used in the PsN bootstrap is found below: > > -----Original Message----- > From: fengdubianbian [mailto:[email protected]] > Sent: 15 June 2011 08:30 > To: [email protected] > Subject: [Psn-general] bootstrap problem > > hey all, > > There is .r file auto generated by psn 3.2.4 during bootstraping. > > Some vertical lines will be plot on the distribution of parameters. > > Actually the Median is Mean, the mean is median. > > the R code is: > > if (showmean) { > > legend=paste(legend, "; Mean = ", sp[3], sep="") > > } > > if (showmedian) { > > legend=paste(legend, "; Median = ", sp[4], sep="") > > } > > >sp > > Min. 1st Qu. Median Mean 3rd Qu. Max. > > 0.0001998 0.0002994 0.0002994 0.0002967 0.0002994 0.0004768 > > Kun Wang Ph.D > > -----Original Message----- > From: Kajsa Harling [mailto:[email protected]] > Sent: 23 June 2011 11:52 > To: General Discussion about PsN. > Subject: Re: [Psn-general] bootstrap problem > > Thank you for the error report. This will be fixed in the next release. > > Best regards, > > Kajsa > > -----Original Message----- > From: Ribbing, Jakob > Sent: 24 June 2011 09:25 > To: General Discussion about PsN. > Cc: '[email protected]' > Subject: RE: [Psn-general] bootstrap problem > > Kajsa, > > While you are looking at that R script in PsN; As I recall there are additional bugs. For example, what bootstrap samples to use is hard coded on the script, so no matter what you set in psn.conf or on the command line to bootstrap; for histograms the R script will only use the samples with successful terminations. I almost always want to use all bs samples. > > When you are ready to move bootstrap post-processing into Xpose I can send you an R script that we use at Pfizer for the PsN bootstrap. This provides a full summary of what you may get out of a bootstrap, with nicer graphics, tables summarizing both the nonmem covstep and the non-parametric bootstrap and including optional parameter transformations and bs statistics for secondary parameters. Out script would have to be in Xpose, though, because there are too many options for PsN. > > And I would have to find time to tweak it a bit; I have written the code only for our ePharm environment in LINUX. Unfortunately I will not find the time to do this in 2011, but it is worth waiting for :>) > > Happy summer solstice! > > Jakob > > ------------------------------------------------------------------------ -- Nick Holford, Professor Clinical Pharmacology Dept Pharmacology& Clinical Pharmacology University of Auckland,85 Park Rd,Private Bag 92019,Auckland,New Zealand tel:+64(9)923-6730 fax:+64(9)373-7090 mobile:+64(21)46 23 53 email: [email protected] http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford