Hello everyone,
I am using PsN to do some bootstrap and have some questions regarding the
PsN output.
1. There are two confidence interval (CI) reported in the output file
"bootstrap_results.csv":
standard.error.confidence.intervals
percentile.confidence.intervals
I wonder which one should be used in the publication or report, and what is
the difference between them.
2. When I use excel function
"=PERCENTILE(T5:T505,5%)" and "=PERCENTILE(T5:T505,95%)" to calculate the 5%
and 95% percentile of a parameter from the data "raw_results1.csv" the
result is different from both "standard.error.confidence.intervals" and "
percentile.confidence.intervals".
The same happens to the excel function "=MEDIAN(T5:T505)" result and
the"medians" in the "bootstrap_results.csv".
Does anyone know why it is the case, and which value I should use?
bootstrap_results.csv
medians
423.5635
standard.error.confidence.intervals
5% 419.73239
95% 428.26761
percentile.confidence.intervals
5% 419.56165
95% 427.9239
Excel calculation from raw_results1.csv
Median 423.578
5% percentile 419.593
95% percentile 427.922
Thanks,
Norman
Confidence intervals of PsN bootstrap output
Hi Norman,
I would suggest you rely on you own calculation, rather than the output from
the R script that is used by PsN (but only trust Excel as far as the back-of-an
Envelop). I would do just like you and include all bs samples in calculating
percentiles. Others prefer only to use a subset, based on termination status.
PsN has it's own mailing list (sourceforge) where you can find out more about
bugs and features of PsN programs.
Best
Jakob
Sent from my iPhone
Quoted reply history
On 5 jul 2011, at 22:16, "Norman Z" <[email protected]> wrote:
> Hello everyone,
>
> I am using PsN to do some bootstrap and have some questions regarding the PsN
> output.
>
> 1. There are two confidence interval (CI) reported in the output file
> "bootstrap_results.csv":
> standard.error.confidence.intervals
> percentile.confidence.intervals
> I wonder which one should be used in the publication or report, and what is
> the difference between them.
>
> 2. When I use excel function
> "=PERCENTILE(T5:T505,5%)" and "=PERCENTILE(T5:T505,95%)" to calculate the 5%
> and 95% percentile of a parameter from the data "raw_results1.csv" the
> result is different from both "standard.error.confidence.intervals" and
> "percentile.confidence.intervals".
> The same happens to the excel function "=MEDIAN(T5:T505)" result and the
> "medians" in the "bootstrap_results.csv".
> Does anyone know why it is the case, and which value I should use?
>
> bootstrap_results.csv
> medians
> 423.5635
> standard.error.confidence.intervals
> 5% 419.73239
> 95% 428.26761
> percentile.confidence.intervals
> 5% 419.56165
> 95% 427.9239
>
> Excel calculation from raw_results1.csv
> Median 423.578
> 5% percentile 419.593
> 95% percentile 427.922
>
> Thanks,
>
> Norman
>
Hi Martin,
Thanks for pointing out that the first row is the original estimation. By
exclude this row, I recalculated the value, and the excel calculation is a
lot closer to the PsN output. For most parameters, the difference between
these two calculations could be due to rounding error, yet major difference
can still be found for a few parameters.
The wiki page suggested by Douglas is also very helpful, there we can see
that the underlining mathematics could be different for different software.
Can anyone suggest that what is the normal practice when reporting the
bootstrap value for publication or regulatory submission? Is the PsN output
used directly, or is recalculation by other software a must?
Thanks,
Norman
Quoted reply history
On Tue, Jul 5, 2011 at 7:20 PM, Martin Johnson <[email protected]> wrote:
> Hi Norman,
>
> The first line (row) in the raw_results1.csv is from the original dataset
> (please refer the manual) and as I see from your excel function...there
> are 501 bootstrap samples.
> My guess would be the median calculated from the excel function includes
> the estimates from the original dataset + 500 bootstrap (1+500) which makes
> the difference in the medians.
>
> Additionally, percentile function in excel can give different results than
> other statistical softwares, please try excel function percentile.exc
> (available in excel 2010/2011), that will give your similar (same) results.
>
> Hope this helps a bit
>
> Regards,
> Martin Johnson
> PhD Student
> University of Groningen
> The Netherlands
>
>
> On Jul 5, 2011, at 9:48 PM, Norman Z wrote:
>
> Hello everyone,
>
> I am using PsN to do some bootstrap and have some questions regarding the
> PsN output.
>
> 1. There are two confidence interval (CI) reported in the output file
> "bootstrap_results.csv":
> standard.error.confidence.intervals
> percentile.confidence.intervals
> I wonder which one should be used in the publication or report, and what is
> the difference between them.
>
> 2. When I use excel function
> "=PERCENTILE(T5:T505,5%)" and "=PERCENTILE(T5:T505,95%)" to calculate the
> 5% and 95% percentile of a parameter from the data "raw_results1.csv" the
> result is different from both "standard.error.confidence.intervals" and "
> percentile.confidence.intervals".
> The same happens to the excel function "=MEDIAN(T5:T505)" result and
> the"medians" in the "bootstrap_results.csv".
>
> Does anyone know why it is the case, and which value I should use?
>
> bootstrap_results.csv
> medians
> 423.5635
> standard.error.confidence.intervals
> 5% 419.73239
> 95% 428.26761
> percentile.confidence.intervals
> 5% 419.56165
> 95% 427.9239
>
> Excel calculation from raw_results1.csv
> Median 423.578
> 5% percentile 419.593
> 95% percentile 427.922
>
> Thanks,
>
> Norman
>
>
>
Dear Norman,
As long as you say which bootstrap runs you used in your calculations, and can motivate why, it's really up to you - there's not a gold standard as such. Personally, as a rule of thumb, I only use bootstrap runs that have minimized successfully in calculating the percentile confidence intervals. One could also choose to exclude runs that fail the covariance step, or have estimates are close to boundaries, or have zero gradients (etc) but it really depends on what you would like to show. For example, the proportion of runs that fail covariance can sometimes be a useful indicator of model stability (although opinions vary on the subject).
PsN is configurable in terms of which runs it uses - have a look at the bootstrap documentation (specifically, the various "skip" flags). You could accept its output at face value, provided you know how it was arrived at, or you could choose to do your own calculations from raw_results1.csv.
I would also echo the advice of others regarding Excel - it's not really the best tool for the job. You'd be better off with something like R - it's well worth the time spent getting to know it.
Best
Justin
Quoted reply history
On 7/6/11 4:48 PM, Norman Z wrote:
> Hi Martin,
>
> Thanks for pointing out that the first row is the original estimation. By exclude this row, I recalculated the value, and the excel calculation is a lot closer to the PsN output. For most parameters, the difference between these two calculations could be due to rounding error, yet major difference can still be found for a few parameters.
>
> The wiki page suggested by Douglas is also very helpful, there we can see that the underlining mathematics could be different for different software.
>
> Can anyone suggest that what is the normal practice when reporting the bootstrap value for publication or regulatory submission? Is the PsN output used directly, or is recalculation by other software a must?
>
> Thanks,
>
> Norman
>
> On Tue, Jul 5, 2011 at 7:20 PM, Martin Johnson < [email protected] < mailto: [email protected] >> wrote:
>
> Hi Norman,
>
> The first line (row) in the raw_results1.csv is from the original
> dataset (please refer the manual) and as I see from your excel
> function...there are 501 bootstrap samples.
> My guess would be the median calculated from the excel function
> includes the estimates from the original dataset + 500 bootstrap
> (1+500) which makes the difference in the medians.
>
> Additionally, percentile function in excel can give different
> results than other statistical softwares, please try excel
> function percentile.exc (available in excel 2010/2011), that will
> give your similar (same) results.
>
> Hope this helps a bit
>
> Regards,
> Martin Johnson
> PhD Student
> University of Groningen
> The Netherlands
>
> On Jul 5, 2011, at 9:48 PM, Norman Z wrote:
>
> > Hello everyone,
> >
> > I am using PsN to do some bootstrap and have some questions
> > regarding the PsN output.
> >
> > 1. There are two confidence interval (CI) reported in the output
> > file "bootstrap_results.csv":
> > standard.error.confidence.intervals
> > percentile.confidence.intervals
> > I wonder which one should be used in the publication or report,
> > and what is the difference between them.
> >
> > 2.When I use excel function
> > "=PERCENTILE(T5:T505,5%)" and "=PERCENTILE(T5:T505,95%)" to
> >
> > calculate the 5% and 95% percentile of a parameterfrom the data "raw_results1.csv" the result is different from both
> >
> > "standard.error.confidence.intervals" and
> > "percentile.confidence.intervals".
> > The same happens to the excel function "=MEDIAN(T5:T505)" result
> > and the"medians" in the "bootstrap_results.csv".
> > Does anyone know why it is the case, and which value I should use?
> >
> > bootstrap_results.csv
> > medians
> > 423.5635
> > standard.error.confidence.intervals
> > 5% 419.73239
> > 95% 428.26761
> > percentile.confidence.intervals
> > 5% 419.56165
> > 95% 427.9239
> >
> > Excel calculation from raw_results1.csv
> > Median 423.578
> > 5% percentile 419.593
> > 95% percentile 427.922
> >
> > Thanks,
> >
> > Norman
--
Justin Wilkins
--------------------
Räfiser Feld 10
CH-9470 Buchs
Switzerland
--------------------
[email protected]
It seems my previous attempt to post this was unsuccessful (either
because of the graphs included or because 70 kb was too much?)
I am resending without graphs and apologize in case of any duplicate
postings!
Quoted reply history
________________________________
From: Ribbing, Jakob
Sent: 08 July 2011 16:31
To: [email protected]
Cc: 'Justin Wilkins'; Norman Z; Ribbing, Jakob
Subject: RE: [NMusers] Confidence intervals of PsN bootstrap output
Dear all,
I would generally agree with Justin's comment that one can take any PsN
output as is, for internal or external reports.
However, specifically for the R script used in the PsN bootstrap you can
not rely on this, as is.
There are several issues with this code, of which some are described in
the e-mail thread below, from PsN users list.
You would either have to correct the PsN R-code or else write your own
script to get the output that you need for regulatory interaction.
Regarding what subset of bootstrap sample to use, I do NOT want to open
up for a discussion regarding whether there are any differences between
bootstrap samples that terminate successfully without or without cov
step, and those that terminate with rounding error. This has been
discussed previously on nmusers; several times and at length. (There is
still a difference in opinion, and as Justin said anyone is free to
follow their own preference)
However, regarding excluding bootstrap samples with terminations at
boundary I would strongly discourage doing this by default and without
any thought.
Just as an example, if a portion of your bootstrap samples for an omega
element end up at a boundary this is what you would miss out on:
* If it is a diagonal omega with some frequency of termination at
lower boundary, excluding these would provide a confidence interval well
above zero. By excluding the bootstrap samples that do not fit with the
statistical model that you have selected, you automatically confirm your
selection (i.e. that data supports the estimation of this IIV or IOV, or
whatever the eta represents), but in my mind the CI based on this subset
is misleading.
* If it is an off-diagonal omega element (representing covariance
between two etas, i.e. on the individual level) with frequent
termination at the upper boundary (correlation of 1) excluding these
bootstrap samples would provide a confidence interval of the eta
correlation that does not include 1. (Correlation is a secondary
parameter calculated based on covariance and IIV (variance) for the two
etas). Again, I would think the CI based on this subset is misleading,
as one automatically confirms the selection of a BLOCK(3) omega
structure, without taking into consideration a reduction to two
parameters that was preferred by a portion of bootstrap samples. I have
included an illustration of this case in the figure below (I do not know
if postings to nmusers allow including figures, but thought it was worth
a try).
Obviously, if only using the subset with successful covariance step the
exclusion includes bootstrap samples with termination at boundary (if
there are any).
I hope this discussion does not discourage any new users from trying the
(non-parameteric) bootstrap.
In my opinion this is a very powerful method that can provide a huge
amount of useful information, beyond the nonmem covariance matrix.
Next time around the nmusers discussion may be regarding whether the
nonmem covariance matrix can be trusted and when a summary of this form
is useful, or whether to use the Sandwich or R-matrix; there are many
areas where there is no safe ground to tread and no full consensus among
users, just as it is sometimes difficult to come up with general advice
on what is the most appropriate procedure.
Best regards
Jakob
An illustration of the uncertainty distribution for the correlation
between two etas (Notice that full correlation is only available in the
subset with boundary problems, as a correlation of one is an implicit
boundary condition. Full correlation is also the only reason to the
boundary problem among these bootstrap samples):
[Jakob] Removed
The original parameterisation is based on covariance between the two
etas, rather than correlation, and here the reason to the boundary issue
is not at all obvious:
[Jakob] Removed
To subscribe to the PsN mailing list:
http://psn.sourceforge.net/list.php
Preferably keep any discussion around the specific implementation in PsN
to the PsN mailing list, as of little interest to nmusers that are not
using PsN.
The previous discussion on the PsN list, regarding the R-script used in
the PsN bootstrap is found below:
-----Original Message-----
From: fengdubianbian [mailto:[email protected]]
Sent: 15 June 2011 08:30
To: [email protected]
Subject: [Psn-general] bootstrap problem
hey all,
There is .r file auto generated by psn 3.2.4 during bootstraping.
Some vertical lines will be plot on the distribution of parameters.
Actually the Median is Mean, the mean is median.
the R code is:
if (showmean) {
legend=paste(legend, "; Mean = ", sp[3], sep="")
}
if (showmedian) {
legend=paste(legend, "; Median = ", sp[4], sep="")
}
>sp
Min. 1st Qu. Median Mean 3rd Qu. Max.
0.0001998 0.0002994 0.0002994 0.0002967 0.0002994 0.0004768
Kun Wang Ph.D
-----Original Message-----
From: Kajsa Harling [mailto:[email protected]]
Sent: 23 June 2011 11:52
To: General Discussion about PsN.
Subject: Re: [Psn-general] bootstrap problem
Thank you for the error report. This will be fixed in the next release.
Best regards,
Kajsa
-----Original Message-----
From: Ribbing, Jakob
Sent: 24 June 2011 09:25
To: General Discussion about PsN.
Cc: '[email protected]'
Subject: RE: [Psn-general] bootstrap problem
Kajsa,
While you are looking at that R script in PsN; As I recall there are
additional bugs. For example, what bootstrap samples to use is hard
coded on the script, so no matter what you set in psn.conf or on the
command line to bootstrap; for histograms the R script will only use the
samples with successful terminations. I almost always want to use all bs
samples.
When you are ready to move bootstrap post-processing into Xpose I can
send you an R script that we use at Pfizer for the PsN bootstrap. This
provides a full summary of what you may get out of a bootstrap, with
nicer graphics, tables summarizing both the nonmem covstep and the
non-parametric bootstrap and including optional parameter
transformations and bs statistics for secondary parameters. Out script
would have to be in Xpose, though, because there are too many options
for PsN.
And I would have to find time to tweak it a bit; I have written the code
only for our ePharm environment in LINUX. Unfortunately I will not find
the time to do this in 2011, but it is worth waiting for :>)
Happy summer solstice!
Jakob
________________________________
Dear all,
Below you find the thread from the PsN mailing list that could not be
included with the e-mail I sent just before this one.
Best
Jakob
Preferably keep any discussion around the specific implementation in PsN
to the PsN mailing list, as of little interest to nmusers that are not
using PsN.
The previous discussion on the PsN list, regarding the R-script used in
the PsN bootstrap is found below:
Quoted reply history
-----Original Message-----
From: fengdubianbian [mailto:[email protected]]
Sent: 15 June 2011 08:30
To: [email protected]
Subject: [Psn-general] bootstrap problem
hey all,
There is .r file auto generated by psn 3.2.4 during bootstraping.
Some vertical lines will be plot on the distribution of parameters.
Actually the Median is Mean, the mean is median.
the R code is:
if (showmean) {
legend=paste(legend, "; Mean = ", sp[3], sep="")
}
if (showmedian) {
legend=paste(legend, "; Median = ", sp[4], sep="")
}
>sp
Min. 1st Qu. Median Mean 3rd Qu. Max.
0.0001998 0.0002994 0.0002994 0.0002967 0.0002994 0.0004768
Kun Wang Ph.D
-----Original Message-----
From: Kajsa Harling [mailto:[email protected]]
Sent: 23 June 2011 11:52
To: General Discussion about PsN.
Subject: Re: [Psn-general] bootstrap problem
Thank you for the error report. This will be fixed in the next release.
Best regards,
Kajsa
-----Original Message-----
From: Ribbing, Jakob
Sent: 24 June 2011 09:25
To: General Discussion about PsN.
Cc: '[email protected]'
Subject: RE: [Psn-general] bootstrap problem
Kajsa,
While you are looking at that R script in PsN; As I recall there are
additional bugs. For example, what bootstrap samples to use is hard
coded on the script, so no matter what you set in psn.conf or on the
command line to bootstrap; for histograms the R script will only use the
samples with successful terminations. I almost always want to use all bs
samples.
When you are ready to move bootstrap post-processing into Xpose I can
send you an R script that we use at Pfizer for the PsN bootstrap. This
provides a full summary of what you may get out of a bootstrap, with
nicer graphics, tables summarizing both the nonmem covstep and the
non-parametric bootstrap and including optional parameter
transformations and bs statistics for secondary parameters. Out script
would have to be in Xpose, though, because there are too many options
for PsN.
And I would have to find time to tweak it a bit; I have written the code
only for our ePharm environment in LINUX. Unfortunately I will not find
the time to do this in 2011, but it is worth waiting for :>)
Happy summer solstice!
Jakob
________________________________
Jakob,
Thanks for your helpful comments. I agree with you that any results that are at a boundary should be discarded from the bootstrap distribution.
Although you say you do not wish to re-open the discussion, there is evidence that discarding other non-successful runs makes no difference to the parameter distribution (Gastonguay & El-Tahtawy 2005, Holford et al. 2006) and also makes no difference to model selection results when estimating from simulated data (Byon et al 2008, Ahn et al 2008). While these conclusions may be controversial it is a controversy with evidence only on the side of those who say all bootstrap results may be used to describe the distribution versus speculation from those who say they should be discarded. Perhaps there is some new evidence from the speculators that could further enlighten the discussion?
Nick
Gastonguay G, El-Tahtawy A. Minimization status had minimal impact on the resulting BS [bootstrap] parameter distributions http://metrumrg.com/publications/Gastonguay.BSMin.ASCPT2005.pdf In: ASCPT, 2005. Holford NHG, Kirkpatrick C, Duffull S. NONMEM Termination Status is Not an Important Indicator of the Quality of Bootstrap Parameter Estimates http://www.page-meeting.org/default.asp?abstract=992 . In: PAGE, Bruges, 2006. Byon W, Fletcher CV, Brundage RC. Impact of censoring data below an arbitrary quantification limit on structural model misspecification. J Pharmacokinet Pharmacodyn. 2008;35(1):101-16. Ahn JE, Karlsson MO, Dunne A, Ludden TM. Likelihood based approaches to handling data below the quantification limit using NONMEM VI. J Pharmacokinet Pharmacodyn. 2008;35(4):401-21.
Quoted reply history
On 9/07/2011 12:07 a.m., Ribbing, Jakob wrote:
> It seems my previous attempt to post this was unsuccessful (either because of the graphs included or because 70 kb was too much?)
>
> I am resending without graphs and apologize in case of any duplicate postings!
>
> ------------------------------------------------------------------------
>
> *From:*Ribbing, Jakob
> *Sent:* 08 July 2011 16:31
> *To:* [email protected]
> *Cc:* 'Justin Wilkins'; Norman Z; Ribbing, Jakob
> *Subject:* RE: [NMusers] Confidence intervals of PsN bootstrap output
>
> Dear all,
>
> I would generally agree with Justin’s comment that one can take any PsN output as is, for internal or external reports.
>
> However, specifically for the R script used in the PsN bootstrap you can not rely on this, as is.
>
> There are several issues with this code, of which some are described in the e-mail thread below, from PsN users list.
>
> You would either have to correct the PsN R-code or else write your own script to get the output that you need for regulatory interaction.
>
> Regarding what subset of bootstrap sample to use, I do NOT want to open up for a discussion regarding whether there are any differences between bootstrap samples that terminate successfully without or without cov step, and those that terminate with rounding error. This has been discussed previously on nmusers; several times and at length. (There is still a difference in opinion, and as Justin said anyone is free to follow their own preference)
>
> However, regarding excluding bootstrap samples with terminations at boundary I would strongly discourage doing this by default and without any thought.
>
> Just as an example, if a portion of your bootstrap samples for an omega element end up at a boundary this is what you would miss out on:
>
> * If it is a diagonal omega with some frequency of termination at
> lower boundary, excluding these would provide a confidence
> interval well above zero. By excluding the bootstrap samples
> that do not fit with the statistical model that you have
> selected, you automatically confirm your selection (i.e. that
> data supports the estimation of this IIV or IOV, or whatever the
> eta represents), but in my mind the CI based on this subset is
> misleading.
> * If it is an off-diagonal omega element (representing covariance
> between two etas, i.e. on the individual level) with frequent
> termination at the upper boundary (correlation of 1) excluding
> these bootstrap samples would provide a confidence interval of
> the eta correlation that does not include 1. (Correlation is a
> secondary parameter calculated based on covariance and IIV
> (variance) for the two etas). Again, I would think the CI based
> on this subset is misleading, as one automatically confirms the
> selection of a BLOCK(3) omega structure, without taking into
> consideration a reduction to two parameters that was preferred
> by a portion of bootstrap samples. I have included an
> illustration of this case in the figure below (I do not know if
> postings to nmusers allow including figures, but thought it was
> worth a try).
>
> Obviously, if only using the subset with successful covariance step the exclusion includes bootstrap samples with termination at boundary (if there are any).
>
> I hope this discussion does not discourage any new users from trying the (non-parameteric) bootstrap.
>
> In my opinion this is a very powerful method that can provide a huge amount of useful information, beyond the nonmem covariance matrix.
>
> Next time around the nmusers discussion may be regarding whether the nonmem covariance matrix can be trusted and when a summary of this form is useful, or whether to use the Sandwich or R-matrix; there are many areas where there is no safe ground to tread and no full consensus among users, just as it is sometimes difficult to come up with general advice on what is the most appropriate procedure.
>
> Best regards
>
> Jakob
>
> An illustration of the uncertainty distribution for the correlation between two etas (Notice that full correlation is only available in the subset with boundary problems, as a correlation of one is an implicit boundary condition. Full correlation is also the only reason to the boundary problem among these bootstrap samples):
>
> */[Jakob] /*Removed
>
> The original parameterisation is based on covariance between the two etas, rather than correlation, and here the reason to the boundary issue is not at all obvious:
>
> */[Jakob] /*Removed
>
> To subscribe to the PsN mailing list:
>
> http://psn.sourceforge.net/list.php
>
> Preferably keep any discussion around the specific implementation in PsN to the PsN mailing list, as of little interest to nmusers that are not using PsN.
>
> The previous discussion on the PsN list, regarding the R-script used in the PsN bootstrap is found below:
>
> -----Original Message-----
> From: fengdubianbian [mailto:[email protected]]
> Sent: 15 June 2011 08:30
> To: [email protected]
> Subject: [Psn-general] bootstrap problem
>
> hey all,
>
> There is .r file auto generated by psn 3.2.4 during bootstraping.
>
> Some vertical lines will be plot on the distribution of parameters.
>
> Actually the Median is Mean, the mean is median.
>
> the R code is:
>
> if (showmean) {
>
> legend=paste(legend, "; Mean = ", sp[3], sep="")
>
> }
>
> if (showmedian) {
>
> legend=paste(legend, "; Median = ", sp[4], sep="")
>
> }
>
> >sp
>
> Min. 1st Qu. Median Mean 3rd Qu. Max.
>
> 0.0001998 0.0002994 0.0002994 0.0002967 0.0002994 0.0004768
>
> Kun Wang Ph.D
>
> -----Original Message-----
> From: Kajsa Harling [mailto:[email protected]]
> Sent: 23 June 2011 11:52
> To: General Discussion about PsN.
> Subject: Re: [Psn-general] bootstrap problem
>
> Thank you for the error report. This will be fixed in the next release.
>
> Best regards,
>
> Kajsa
>
> -----Original Message-----
> From: Ribbing, Jakob
> Sent: 24 June 2011 09:25
> To: General Discussion about PsN.
> Cc: '[email protected]'
> Subject: RE: [Psn-general] bootstrap problem
>
> Kajsa,
>
> While you are looking at that R script in PsN; As I recall there are additional bugs. For example, what bootstrap samples to use is hard coded on the script, so no matter what you set in psn.conf or on the command line to bootstrap; for histograms the R script will only use the samples with successful terminations. I almost always want to use all bs samples.
>
> When you are ready to move bootstrap post-processing into Xpose I can send you an R script that we use at Pfizer for the PsN bootstrap. This provides a full summary of what you may get out of a bootstrap, with nicer graphics, tables summarizing both the nonmem covstep and the non-parametric bootstrap and including optional parameter transformations and bs statistics for secondary parameters. Out script would have to be in Xpose, though, because there are too many options for PsN.
>
> And I would have to find time to tweak it a bit; I have written the code only for our ePharm environment in LINUX. Unfortunately I will not find the time to do this in 2011, but it is worth waiting for :>)
>
> Happy summer solstice!
>
> Jakob
>
> ------------------------------------------------------------------------
--
Nick Holford, Professor Clinical Pharmacology
Dept Pharmacology& Clinical Pharmacology
University of Auckland,85 Park Rd,Private Bag 92019,Auckland,New Zealand
tel:+64(9)923-6730 fax:+64(9)373-7090 mobile:+64(21)46 23 53
email: [email protected]
http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
Nick, Jakob, nmusers:
At the risk of re-opening an old discussion, I'd like to expand on Nick's
important point. From our experience the experience of others, including
parameters from all bootstrap runs that report parameter estimates, regardless
of minimization of $COV status, is a reasonable thing to do; provided that the
modeler conducts a careful and thoughtful evaluation of the model and bootstrap
results.
One goal should be to understand why minimization, $COV, or boundary problems
occur. For example, it is useful to create histograms and a scatterplot matrix
of the resulting bootstrap parameter distributions, flagging those runs with
minimization problems. Most of the time, these problematic runs represent the
tails of the distribution for a less precisely defined parameter. Sometimes the
boundary message is just a reflection of the tail of a bootstrap parameter
distribution approaching the null value for that parameter. All of these runs
are informative about the estimation precision and should be included.
This is not always the case, however. In fact, even successful minimization and
$COV do not guarantee that the resulting parameter estimates should be included
in bootstrap confidence intervals. For example, bootstrapping is often a useful
way to identify models with more than 1 unique solution (e.g. flip-flop in PK
rate constants). In addition, some runs may represent a local minimum. Another
problem can arise if the bootstrap resampling is not stratified on the major
covariate(s) or design features of the original data set (e.g. pooled analyses
of adult studies and pediatric studies, or pooled analyses of sparsely sampled
and extensively sampled studies). In all of these cases, the histograms and
scatterplot matrices may reveal a multimodal "posterior" bootstrap distribution
for some of the parameters, which should be addressed before using bootstrap
results for inferential or simulation purposes.
Obviously, it's important for the modeler to think about the problem and to
understand what the results mean. Blindly accepting or rejecting runs based on
a software warning message, or on another scientist's R script, is not
advisable.
Best regards,
Marc
Marc R. Gastonguay, Ph.D. ([email protected])
President & CEO, Metrum Research Group LLC (metrumrg.com)
Scientific Director, Metrum Institute (metruminstitute.org)
Quoted reply history
On Saturday, July 9, 2011 at 9:14 AM, Nick Holford wrote:
> Jakob,
>
> Thanks for your helpful comments. I agree with you that any results that
> are at a boundary should be discarded from the bootstrap distribution.
>
> Although you say you do not wish to re-open the discussion, there is
> evidence that discarding other non-successful runs makes no difference
> to the parameter distribution (Gastonguay & El-Tahtawy 2005, Holford et
> al. 2006) and also makes no difference to model selection results when
> estimating from simulated data (Byon et al 2008, Ahn et al 2008). While
> these conclusions may be controversial it is a controversy with evidence
> only on the side of those who say all bootstrap results may be used to
> describe the distribution versus speculation from those who say they
> should be discarded. Perhaps there is some new evidence from the
> speculators that could further enlighten the discussion?
>
> Nick
>
>
> Gastonguay G, El-Tahtawy A. Minimization status had minimal impact on
> the resulting BS [bootstrap] parameter distributions
> http://metrumrg.com/publications/Gastonguay.BSMin.ASCPT2005.pdf In:
> ASCPT, 2005.
> Holford NHG, Kirkpatrick C, Duffull S. NONMEM Termination Status is Not
> an Important Indicator of the Quality of Bootstrap Parameter Estimates
> http://www.page-meeting.org/default.asp?abstract=992. In: PAGE, Bruges,
> 2006.
> Byon W, Fletcher CV, Brundage RC. Impact of censoring data below an
> arbitrary quantification limit on structural model misspecification. J
> Pharmacokinet Pharmacodyn. 2008;35(1):101-16.
> Ahn JE, Karlsson MO, Dunne A, Ludden TM. Likelihood based approaches to
> handling data below the quantification limit using NONMEM VI. J
> Pharmacokinet Pharmacodyn. 2008;35(4):401-21.
>
Nick, Jakob, Marc et al
> Thanks for your helpful comments. I agree with you that any results that
> are at a boundary should be discarded from the bootstrap distribution.
On the whole I the sentiments in this thread align with anecdotal findings from
my experience. But, I was just wondering how you define your boundaries for
variance and covariance parameters (e.g. OMEGA terms)?
For variance terms, lower boundaries seems reasonably straightforward (e.g.
1E-5 seems close to zero). Upper boundaries are of course open, for the
variance of a log-normal ETA would 1E+5 or 1E+4 be large enough to be
considered close to a boundary? At what value would you discard the result? At
what correlation value would you discard a result (>0.99, > 0.97...) as being
close to 1. Clearly if this was for regulatory work you could define these a
priori after having chosen any arbitrary cut-off. But the devil here lies with
the non-regulatory work where you may not have defined these boundaries a
priori.
Steve
--
Professor Stephen Duffull
Chair of Clinical Pharmacy
School of Pharmacy
University of Otago
PO Box 56 Dunedin
New Zealand
E: [email protected]
P: +64 3 479 5044
F: +64 3 479 7034
W: http://pharmacy.otago.ac.nz/profiles/stephenduffull
Design software: www.winpopt.com
I thought that the original post was "results at a boundary should NOT be discarded" and Nick reply was just a typo. If it was not a typo, I would disagree and argue that all results should be included: Each data set is a particular realization. We should be able to use all of them. If some realizations are so special that the model behaves in an unusual way (with any definition of unusual: non-convergence, not convergence of the covariance step, parameter estimates at the boundary, etc.) we either need to accept those as is, or work with each of those special data sets one by one to push to the parameter estimates that we can accept, or change the bootstrap procedure (add stratification by covariates, by dose level, by route of administration, etc.) so that all data sets behave similarly.
Leonid
--------------------------------------
Leonid Gibiansky, Ph.D.
President, QuantPharm LLC
web: www.quantpharm.com
e-mail: LGibiansky at quantpharm.com
tel: (301) 767 5566
Quoted reply history
On 7/10/2011 2:57 PM, Stephen Duffull wrote:
> Nick, Jakob, Marc et al
>
> > Thanks for your helpful comments. I agree with you that any results that
> > are at a boundary should be discarded from the bootstrap distribution.
>
> On the whole I the sentiments in this thread align with anecdotal findings from
> my experience. But, I was just wondering how you define your boundaries for
> variance and covariance parameters (e.g. OMEGA terms)?
>
> For variance terms, lower boundaries seems reasonably straightforward (e.g. 1E-5
> seems close to zero). Upper boundaries are of course open, for the variance of a
> log-normal ETA would 1E+5 or 1E+4 be large enough to be considered close to a
> boundary? At what value would you discard the result? At what correlation value
> would you discard a result (>0.99,> 0.97...) as being close to 1. Clearly if
> this was for regulatory work you could define these a priori after having chosen any
> arbitrary cut-off. But the devil here lies with the non-regulatory work where you
> may not have defined these boundaries a priori.
>
> Steve
> --
> Professor Stephen Duffull
> Chair of Clinical Pharmacy
> School of Pharmacy
> University of Otago
> PO Box 56 Dunedin
> New Zealand
> E: [email protected]
> P: +64 3 479 5044
> F: +64 3 479 7034
> W: http://pharmacy.otago.ac.nz/profiles/stephenduffull
>
> Design software: www.winpopt.com
Leonid,
With regard to discarding runs at the boundary what I had in mind was runs which had reached the maximum number of iterations but I realized later that Jacob was referring to NONMEM's often irritating messages that usually just mean the initial estimate changed a lot or variance was getting close to zero.
There are of course some cases where the estimate is truly at a user defined constraint. Assuming that the user has thought carefully about these constraints then I would interpret a run that finished at this constraint boundary as showing NONMEM was stuck in a local minimum (probably because of the constraint boundary) and if the constraint was relaxed then perhaps a more useful estimate would be obtained.
In those cases then I think one can make an argument for discarding runs with parameters that are at this kind of boundary as well as those which reached an iteration limit.
In general I agree with your remarks (echoing those from Marc Gastonguay) that one needs to think about the way each bootstrap run behaved. But some things like non-convergence and failed covariance are ignorable because they don't influence the bootstrap distribution.
There is also the need to recognize that bootstraps can be seriously time consuming and the effort required to understand all the ways that runs might finish is usually not worth it given the purposes of doing a bootstrap.
The most important reason for doing a bootstrap is to get more robust estimates of the parameters. This was the main reason why these re-sampling procedures were initially developed. The bootstrap estimate of the parameters will usually be pretty insensitive to the margins of the distribution where the questionable run results are typically located.
A secondary semi-quantitative reason is to get a confidence interval which may be helpful for model selection. This may be influenced by the questionable runs but that is just part of the uncertainty that the confidence interval is used to define.
Nick
Quoted reply history
On 10/07/2011 11:13 p.m., Leonid Gibiansky wrote:
> I thought that the original post was "results at a boundary should NOT be discarded" and Nick reply was just a typo. If it was not a typo, I would disagree and argue that all results should be included: Each data set is a particular realization. We should be able to use all of them. If some realizations are so special that the model behaves in an unusual way (with any definition of unusual: non-convergence, not convergence of the covariance step, parameter estimates at the boundary, etc.) we either need to accept those as is, or work with each of those special data sets one by one to push to the parameter estimates that we can accept, or change the bootstrap procedure (add stratification by covariates, by dose level, by route of administration, etc.) so that all data sets behave similarly.
>
> Leonid
>
> --------------------------------------
> Leonid Gibiansky, Ph.D.
> President, QuantPharm LLC
> web: www.quantpharm.com
> e-mail: LGibiansky at quantpharm.com
> tel: (301) 767 5566
>
> On 7/10/2011 2:57 PM, Stephen Duffull wrote:
>
> > Nick, Jakob, Marc et al
> >
> > > Thanks for your helpful comments. I agree with you that any results that
> > >
> > > are at a boundary should be discarded from the bootstrap distribution.
> >
> > On the whole I the sentiments in this thread align with anecdotal findings from my experience. But, I was just wondering how you define your boundaries for variance and covariance parameters (e.g. OMEGA terms)?
> >
> > For variance terms, lower boundaries seems reasonably straightforward (e.g. 1E-5 seems close to zero). Upper boundaries are of course open, for the variance of a log-normal ETA would 1E+5 or 1E+4 be large enough to be considered close to a boundary? At what value would you discard the result? At what correlation value would you discard a result (>0.99,> 0.97...) as being close to 1. Clearly if this was for regulatory work you could define these a priori after having chosen any arbitrary cut-off. But the devil here lies with the non-regulatory work where you may not have defined these boundaries a priori.
> >
> > Steve
> > --
> > Professor Stephen Duffull
> > Chair of Clinical Pharmacy
> > School of Pharmacy
> > University of Otago
> > PO Box 56 Dunedin
> > New Zealand
> > E: [email protected]
> > P: +64 3 479 5044
> > F: +64 3 479 7034
> > W: http://pharmacy.otago.ac.nz/profiles/stephenduffull
> >
> > Design software: www.winpopt.com
--
Nick Holford, Professor Clinical Pharmacology
Dept Pharmacology& Clinical Pharmacology
University of Auckland,85 Park Rd,Private Bag 92019,Auckland,New Zealand
tel:+64(9)923-6730 fax:+64(9)373-7090 mobile:+64(21)46 23 53
email: [email protected]
http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
Hi all,
Given their usefulness, maybe we should be trying to use nonparametric bootstraps more often, at key decision points in model development, especially now that the ready availability of computing power has made this realistic. (I guess many of us already do this, but it's a point worth making all the same.) Although Nick's point about the amount of time required is well taken, on a large computing grid this is somewhat offset, especially in models with short run times, and as previous posters have pointed out, they can deliver very useful information, especially when used in combination with other useful PsN tools like case-deletion diagnostics.
Before we get too far away from it, one point needs correction - previous posts have implied that PsN uses an R script to generate its results. This is not strictly true. The bundled bootstrap.R script is not run by default, and does little more than generate some very basic histograms of the parameter distributions, using the raw PsN results as input. As Jakob rightly points out, it contains at least one bug (means and medians are flipped), and has some counter-intuitive default settings with respect to the runs it includes in the plots it creates, as well as having compatibility issues with recent R releases. (The cdd.R and llp.R scripts are similarly simplistic.) There are plans to replace them all with Xpose functions later on down the line, as mentioned elsewhere in this thread.
PsN's output table files (bootstrap_results.csv and raw_results1.csv), on the other hand, are created internally and seem reasonably robust. Runs included when generating the bootstrap_results.csv file can (and should) be pre-specified using the "-skip_*" flags passed to the bootstrap command at runtime, or specified in PsN's configuration file, psn.conf (whichever option is chosen, it's important to know - at minimum - what they are and what they imply about the results, linking in with what Marc wrote). Alternatively, the raw bootstrap data in raw_results1.csv can be analyzed by hand if a finer degree of control is called for.
Best
Justin
Quoted reply history
On 7/11/11 7:37 AM, Nick Holford wrote:
> Leonid,
>
> With regard to discarding runs at the boundary what I had in mind was runs which had reached the maximum number of iterations but I realized later that Jacob was referring to NONMEM's often irritating messages that usually just mean the initial estimate changed a lot or variance was getting close to zero.
>
> There are of course some cases where the estimate is truly at a user defined constraint. Assuming that the user has thought carefully about these constraints then I would interpret a run that finished at this constraint boundary as showing NONMEM was stuck in a local minimum (probably because of the constraint boundary) and if the constraint was relaxed then perhaps a more useful estimate would be obtained.
>
> In those cases then I think one can make an argument for discarding runs with parameters that are at this kind of boundary as well as those which reached an iteration limit.
>
> In general I agree with your remarks (echoing those from Marc Gastonguay) that one needs to think about the way each bootstrap run behaved. But some things like non-convergence and failed covariance are ignorable because they don't influence the bootstrap distribution.
>
> There is also the need to recognize that bootstraps can be seriously time consuming and the effort required to understand all the ways that runs might finish is usually not worth it given the purposes of doing a bootstrap.
>
> The most important reason for doing a bootstrap is to get more robust estimates of the parameters. This was the main reason why these re-sampling procedures were initially developed. The bootstrap estimate of the parameters will usually be pretty insensitive to the margins of the distribution where the questionable run results are typically located.
>
> A secondary semi-quantitative reason is to get a confidence interval which may be helpful for model selection. This may be influenced by the questionable runs but that is just part of the uncertainty that the confidence interval is used to define.
>
> Nick
>
> On 10/07/2011 11:13 p.m., Leonid Gibiansky wrote:
>
> > I thought that the original post was "results at a boundary should NOT be discarded" and Nick reply was just a typo. If it was not a typo, I would disagree and argue that all results should be included: Each data set is a particular realization. We should be able to use all of them. If some realizations are so special that the model behaves in an unusual way (with any definition of unusual: non-convergence, not convergence of the covariance step, parameter estimates at the boundary, etc.) we either need to accept those as is, or work with each of those special data sets one by one to push to the parameter estimates that we can accept, or change the bootstrap procedure (add stratification by covariates, by dose level, by route of administration, etc.) so that all data sets behave similarly.
> >
> > Leonid
> >
> > --------------------------------------
> > Leonid Gibiansky, Ph.D.
> > President, QuantPharm LLC
> > web: www.quantpharm.com
> > e-mail: LGibiansky at quantpharm.com
> > tel: (301) 767 5566
> >
> > On 7/10/2011 2:57 PM, Stephen Duffull wrote:
> >
> > > Nick, Jakob, Marc et al
> > >
> > > > Thanks for your helpful comments. I agree with you that any results that
> > > >
> > > > are at a boundary should be discarded from the bootstrap distribution.
> > >
> > > On the whole I the sentiments in this thread align with anecdotal findings from my experience. But, I was just wondering how you define your boundaries for variance and covariance parameters (e.g. OMEGA terms)?
> > >
> > > For variance terms, lower boundaries seems reasonably straightforward (e.g. 1E-5 seems close to zero). Upper boundaries are of course open, for the variance of a log-normal ETA would 1E+5 or 1E+4 be large enough to be considered close to a boundary? At what value would you discard the result? At what correlation value would you discard a result (>0.99,> 0.97...) as being close to 1. Clearly if this was for regulatory work you could define these a priori after having chosen any arbitrary cut-off. But the devil here lies with the non-regulatory work where you may not have defined these boundaries a priori.
> > >
> > > Steve
> > > --
> > > Professor Stephen Duffull
> > > Chair of Clinical Pharmacy
> > > School of Pharmacy
> > > University of Otago
> > > PO Box 56 Dunedin
> > > New Zealand
> > > E: [email protected]
> > > P: +64 3 479 5044
> > > F: +64 3 479 7034
> > > W: http://pharmacy.otago.ac.nz/profiles/stephenduffull
> > >
> > > Design software: www.winpopt.com
--
Justin Wilkins, PhD
Exprimo NV
Tel: +41 (0) 81 599 23 82
Mobile: +41 (0) 76 561 09 49
E-mail: [email protected]
Web: www.exprimo.com
This e-mail is confidential. It is also privileged or otherwise protected by work product immunity or other legal rules. The information is intended to be for use of the individual or entity named above. If you are not the intended recipient, please be aware that any disclosure, copying, distribution or use of the contents of this information is prohibited. You should therefore delete this message from your computer system. If you have received the message in error, please notify us by reply e-mail. The integrity and security of this message cannot be guaranteed on the Internet.
Thank you for your co-operation.
Nick,
" In those cases then I think one can make an argument for discarding runs
with parameters that are at this kind of boundary "
A typical user-defined upper boundary is 1 for fractions (bioavailability,
fraction unbound, etc). In a bootstrap some estimates may well reach this
upper boundary. If, as you say, the analyst has carefully thought through
the boundaries, an estimate at the boundary should represent the global
minimum within the reasonable parameter range. I think one can make a strong
argument for not discarding such runs.
Best regards,
Mats
Mats Karlsson, PhD
Professor of Pharmacometrics
Dept of Pharmaceutical Biosciences
Uppsala University
Sweden
Postal address: Box 591, 751 24 Uppsala, Sweden
Phone +46 18 4714105
Fax + 46 18 4714003
Quoted reply history
-----Original Message-----
From: [email protected] [mailto:[email protected]] On
Behalf Of Nick Holford
Sent: Monday, July 11, 2011 7:37 AM
To: nmusers
Subject: Re: [NMusers] Confidence intervals of PsN bootstrap output
Leonid,
With regard to discarding runs at the boundary what I had in mind was
runs which had reached the maximum number of iterations but I realized
later that Jacob was referring to NONMEM's often irritating messages
that usually just mean the initial estimate changed a lot or variance
was getting close to zero.
There are of course some cases where the estimate is truly at a user
defined constraint. Assuming that the user has thought carefully about
these constraints then I would interpret a run that finished at this
constraint boundary as showing NONMEM was stuck in a local minimum
(probably because of the constraint boundary) and if the constraint was
relaxed then perhaps a more useful estimate would be obtained.
In those cases then I think one can make an argument for discarding runs
with parameters that are at this kind of boundary as well as those which
reached an iteration limit.
In general I agree with your remarks (echoing those from Marc
Gastonguay) that one needs to think about the way each bootstrap run
behaved. But some things like non-convergence and failed covariance are
ignorable because they don't influence the bootstrap distribution.
There is also the need to recognize that bootstraps can be seriously
time consuming and the effort required to understand all the ways that
runs might finish is usually not worth it given the purposes of doing a
bootstrap.
The most important reason for doing a bootstrap is to get more robust
estimates of the parameters. This was the main reason why these
re-sampling procedures were initially developed. The bootstrap estimate
of the parameters will usually be pretty insensitive to the margins of
the distribution where the questionable run results are typically located.
A secondary semi-quantitative reason is to get a confidence interval
which may be helpful for model selection. This may be influenced by the
questionable runs but that is just part of the uncertainty that the
confidence interval is used to define.
Nick
On 10/07/2011 11:13 p.m., Leonid Gibiansky wrote:
> I thought that the original post was "results at a boundary should NOT
> be discarded" and Nick reply was just a typo. If it was not a typo, I
> would disagree and argue that all results should be included:
> Each data set is a particular realization. We should be able to use
> all of them. If some realizations are so special that the model
> behaves in an unusual way (with any definition of unusual:
> non-convergence, not convergence of the covariance step, parameter
> estimates at the boundary, etc.) we either need to accept those as is,
> or work with each of those special data sets one by one to push to the
> parameter estimates that we can accept, or change the bootstrap
> procedure (add stratification by covariates, by dose level, by route
> of administration, etc.) so that all data sets behave similarly.
> Leonid
>
> --------------------------------------
> Leonid Gibiansky, Ph.D.
> President, QuantPharm LLC
> web: www.quantpharm.com
> e-mail: LGibiansky at quantpharm.com
> tel: (301) 767 5566
>
>
>
> On 7/10/2011 2:57 PM, Stephen Duffull wrote:
>> Nick, Jakob, Marc et al
>>
>>> Thanks for your helpful comments. I agree with you that any results
>>> that
>>> are at a boundary should be discarded from the bootstrap distribution.
>>
>> On the whole I the sentiments in this thread align with anecdotal
>> findings from my experience. But, I was just wondering how you
>> define your boundaries for variance and covariance parameters (e.g.
>> OMEGA terms)?
>>
>> For variance terms, lower boundaries seems reasonably straightforward
>> (e.g. 1E-5 seems close to zero). Upper boundaries are of course
>> open, for the variance of a log-normal ETA would 1E+5 or 1E+4 be
>> large enough to be considered close to a boundary? At what value
>> would you discard the result? At what correlation value would you
>> discard a result (>0.99,> 0.97...) as being close to 1. Clearly if
>> this was for regulatory work you could define these a priori after
>> having chosen any arbitrary cut-off. But the devil here lies with
>> the non-regulatory work where you may not have defined these
>> boundaries a priori.
>>
>> Steve
>> --
>> Professor Stephen Duffull
>> Chair of Clinical Pharmacy
>> School of Pharmacy
>> University of Otago
>> PO Box 56 Dunedin
>> New Zealand
>> E: [email protected]
>> P: +64 3 479 5044
>> F: +64 3 479 7034
>> W: http://pharmacy.otago.ac.nz/profiles/stephenduffull
>>
>> Design software: www.winpopt.com
>>
>>
--
Nick Holford, Professor Clinical Pharmacology
Dept Pharmacology& Clinical Pharmacology
University of Auckland,85 Park Rd,Private Bag 92019,Auckland,New Zealand
tel:+64(9)923-6730 fax:+64(9)373-7090 mobile:+64(21)46 23 53
email: [email protected]
http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
All,
This first part is more to clarify and I do not believe this is in
disagreement with what has been said before. The last paragraph is a
question.
The two examples I mentioned regarding boundary conditions are regarding
variance parameters. The second of these, however, is with regards to a
boundary at eta-correlation of one, which is a must rather than just an
irritating NONMEM feature.
I used these examples because they were less controversial and it is
difficult to come up with general statements that apply to all cases.
However, as a third example for a fixed-effects parameter: Imagine a
covariate acting in a linear fashion on a structural parameter that is
bound to be non-negative (e.g. a rate constant, volume, clearance, ED50,
etc). Imagine boundaries on the covariate parameter have been set to
avoid negative values for the structural-model parameter (on the
individual level). For this scenario if a substantial fraction of the
bootstrapped covariate-parameter values end up at one of the boundaries,
one may have to consider two options:
a) Decide that a linear covariate model is inappropriate (at least for
the goal of extrapolating to the whole population with more extreme
covariate values) and change the model into using a different functional
form
b) Dismiss this as random chance, due to small sample/limited
information and a (covariate) slope which "truly" is not far from one of
the boundaries. If this is the case, deleting the bootstrap estimates at
boundary would bias the distribution in an undesirable manner. For that
case the boundary condition is not due to local minimum and we would not
want to discard bootstrap samples at boundary). (Nick's example is of a
different kind, where it is either a local minimum or else not reaching
a minimum at all)
A related question - I am thinking more in terms of simulations with
parameter uncertainty; not just obtaining CI, which was originally what
this thread was about:
There are sometimes situations where a limited set of (clinical-) trial
data gives reasonable point estimates but with huge parameter
uncertainty (regardless nonmem covmaxtrix or bootstrap with appropriate
stratification). The distribution and CI on these parameters may include
unreasonable values, even though there is no obvious physiological
boundary (unreasonable based on prior knowledge that has not been
incorporated into the analysis, e.g. for a certain mechanism and patient
population Typical-Emax beyond 400% or 10 units - depending on if Emax
is parameterised as relative or absolute change). In these situations, a
simplistic option could be to trim one or both ends with regards to the
Emax distribution and discard these bootstrap samples, especially if
only a few values are unreasonable. Alternatively, before running the
bootstrap, one may set the boundary in the control stream (a boundary
that everyone can agree is unreasonable). One would then keep bootstrap
samples that ends up at this boundary for bootstrap distribution, which
is in a way truncated, but so that bootstrap samples indicating linear
concentration/dose-response maintains almost reasonable Emax and
ED50/EC50 values (but as a spike in the distribution at upper Emax).
Notice that re-parameterising the Emax model would not solve the
underlying issue with unreasonable estimates and reducing to a linear
model may be unsuitable, both based on the original dataset and also for
mechanistic reasons). Could you suggest alternative ways of dealing with
this, for these rather general examples (other than the obvious of
applying an informative prior on Emax)? I would be interested in your
solutions both in terms of the non-parametric bootstrap as well as the
parametric bootstrap (based on the nonmem covmatrix).
Much appreciated
Jakob
Quoted reply history
-----Original Message-----
From: [email protected] [mailto:[email protected]]
On Behalf Of Nick Holford
Sent: 11 July 2011 06:37
To: nmusers
Subject: Re: [NMusers] Confidence intervals of PsN bootstrap output
Leonid,
With regard to discarding runs at the boundary what I had in mind was
runs which had reached the maximum number of iterations but I realized
later that Jacob was referring to NONMEM's often irritating messages
that usually just mean the initial estimate changed a lot or variance
was getting close to zero.
There are of course some cases where the estimate is truly at a user
defined constraint. Assuming that the user has thought carefully about
these constraints then I would interpret a run that finished at this
constraint boundary as showing NONMEM was stuck in a local minimum
(probably because of the constraint boundary) and if the constraint was
relaxed then perhaps a more useful estimate would be obtained.
In those cases then I think one can make an argument for discarding runs
with parameters that are at this kind of boundary as well as those which
reached an iteration limit.
In general I agree with your remarks (echoing those from Marc
Gastonguay) that one needs to think about the way each bootstrap run
behaved. But some things like non-convergence and failed covariance are
ignorable because they don't influence the bootstrap distribution.
There is also the need to recognize that bootstraps can be seriously
time consuming and the effort required to understand all the ways that
runs might finish is usually not worth it given the purposes of doing a
bootstrap.
The most important reason for doing a bootstrap is to get more robust
estimates of the parameters. This was the main reason why these
re-sampling procedures were initially developed. The bootstrap estimate
of the parameters will usually be pretty insensitive to the margins of
the distribution where the questionable run results are typically
located.
A secondary semi-quantitative reason is to get a confidence interval
which may be helpful for model selection. This may be influenced by the
questionable runs but that is just part of the uncertainty that the
confidence interval is used to define.
Nick
On 10/07/2011 11:13 p.m., Leonid Gibiansky wrote:
> I thought that the original post was "results at a boundary should NOT
> be discarded" and Nick reply was just a typo. If it was not a typo, I
> would disagree and argue that all results should be included:
> Each data set is a particular realization. We should be able to use
> all of them. If some realizations are so special that the model
> behaves in an unusual way (with any definition of unusual:
> non-convergence, not convergence of the covariance step, parameter
> estimates at the boundary, etc.) we either need to accept those as is,
> or work with each of those special data sets one by one to push to the
> parameter estimates that we can accept, or change the bootstrap
> procedure (add stratification by covariates, by dose level, by route
> of administration, etc.) so that all data sets behave similarly.
> Leonid
>
> --------------------------------------
> Leonid Gibiansky, Ph.D.
> President, QuantPharm LLC
> web: www.quantpharm.com
> e-mail: LGibiansky at quantpharm.com
> tel: (301) 767 5566
>
>
>
> On 7/10/2011 2:57 PM, Stephen Duffull wrote:
>> Nick, Jakob, Marc et al
>>
>>> Thanks for your helpful comments. I agree with you that any results
>>> that
>>> are at a boundary should be discarded from the bootstrap
distribution.
>>
>> On the whole I the sentiments in this thread align with anecdotal
>> findings from my experience. But, I was just wondering how you
>> define your boundaries for variance and covariance parameters (e.g.
>> OMEGA terms)?
>>
>> For variance terms, lower boundaries seems reasonably straightforward
>> (e.g. 1E-5 seems close to zero). Upper boundaries are of course
>> open, for the variance of a log-normal ETA would 1E+5 or 1E+4 be
>> large enough to be considered close to a boundary? At what value
>> would you discard the result? At what correlation value would you
>> discard a result (>0.99,> 0.97...) as being close to 1. Clearly if
>> this was for regulatory work you could define these a priori after
>> having chosen any arbitrary cut-off. But the devil here lies with
>> the non-regulatory work where you may not have defined these
>> boundaries a priori.
>>
>> Steve
>> --
>> Professor Stephen Duffull
>> Chair of Clinical Pharmacy
>> School of Pharmacy
>> University of Otago
>> PO Box 56 Dunedin
>> New Zealand
>> E: [email protected]
>> P: +64 3 479 5044
>> F: +64 3 479 7034
>> W: http://pharmacy.otago.ac.nz/profiles/stephenduffull
>>
>> Design software: www.winpopt.com
>>
>>
--
Nick Holford, Professor Clinical Pharmacology
Dept Pharmacology& Clinical Pharmacology
University of Auckland,85 Park Rd,Private Bag 92019,Auckland,New Zealand
tel:+64(9)923-6730 fax:+64(9)373-7090 mobile:+64(21)46 23 53
email: [email protected]
http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
Hello all,
Sorry to enter the conversation late. (I deleted prior posts to keep from
exceeding the length limit).
I certainly agree with that nonparametric bootstrap procedures need
consideration and interpretation of output. I feel that such procedures
lead to difficulty (as described by many of the previous emails) when the
design is unbalanced (especially when severely so) and only a few
individuals supply data which supports estimation of a covariate or
structural parameter. For example, it might be in a sparse PK setting that
only a few subjects had samples in the absorption phase. Sampling with
replacement might lead some datasets with fewer subjects with absorption
data than in the original dataset. This might lead to erratic behavior (for
example Ka going to large unlikely value) during estimation and hence a
multimodal distribution of the estimates. An example of this for parametric
simulation is in Kimko and Duffull (eds), Simulation for Clinical Design
(2003), Evaluation of Random Sparse Designs for a Population Pharmacokinetic
Study: Assessement of Power and Bias Using Simulation, Hutmacher and
Kowalksi. There, some random sample designs lead to large estimates of Ka -
this did not affect CL or V however - pairwise scatterplots were used to
demonstrate this (as Mark Gastonguay suggested in his thread to do). In
such cases, it might be confidence intervals for the nonparametric bootstrap
are too wide - valid at the nominal level, but inaccurate.
With respect to dealing with boundary constraints and the non-parametric
bootstrap, upfront thought I think can lead to less arbitrariness. Do the
CI's reflect similar findings based on likelihood profiling (LP) or
likelihood ratio tests (LRT)? For example, it might require more thought to
reconcile a bootstrap procedure that yielded 15% of your Emax's be 0 if your
LRT for Emax was > 10 points or the 95% CI based on LP did not include 0,
for example. By allowing the 0 in the constraints an explicit assumption is
made that one is unclear that Emax is greater than 0, and thus the modeler
is allowing a point mass at 0 to exist, which is a difficult distribution
statistically to deal with. One must contemplate whether this makes sense
in the overall clinical interpretation. If it does not, then perhaps EMAX =
exp(theta(X)) should be used to ensure that EMAX is not equal to 0 ever.
Reparameterization can be done for just about any parameter to ensure a
'valid' estimate and I would suggest to do this (a sort of likelihood-based
prior knowledge manifestation) than to arbitrarily pick which estimates from
the bootstrap to use. Even OMEGA matrices can be parameterized to ensure
non-positive semi-definite matrices, which might help in certain situations.
I would also be careful if the nonparametric bootstrap CI's are different
from the COV step CI's as this indicates that something is unknown with
respect to estimation or inference. In the case of small sample size and
non-realistic clinical inference, I would suggest a more formal Bayesian
analysis which pre-specifies the analysts assumptions regarding the
probability or viability of certain estimates (can be influenced by the
prior).
Best regards,
Matt
Hi Nick,
Those "irritating messages that usually just mean the initial estimate changed a lot or variance was getting close to zero" can be removed if you use
NOTHETABOUNDTEST NOOMEGABOUNDTEST NOSIGMABOUNDTEST
at estimation record. I think, these options should always for all bootstrap runs.
Regards,
Leonid
--------------------------------------
Leonid Gibiansky, Ph.D.
President, QuantPharm LLC
web: www.quantpharm.com
e-mail: LGibiansky at quantpharm.com
tel: (301) 767 5566
Quoted reply history
On 7/11/2011 1:37 AM, Nick Holford wrote:
> Leonid,
>
> With regard to discarding runs at the boundary what I had in mind was
> runs which had reached the maximum number of iterations but I realized
> later that Jacob was referring to NONMEM's often irritating messages
> that usually just mean the initial estimate changed a lot or variance
> was getting close to zero.
>
> There are of course some cases where the estimate is truly at a user
> defined constraint. Assuming that the user has thought carefully about
> these constraints then I would interpret a run that finished at this
> constraint boundary as showing NONMEM was stuck in a local minimum
> (probably because of the constraint boundary) and if the constraint was
> relaxed then perhaps a more useful estimate would be obtained.
>
> In those cases then I think one can make an argument for discarding runs
> with parameters that are at this kind of boundary as well as those which
> reached an iteration limit.
>
> In general I agree with your remarks (echoing those from Marc
> Gastonguay) that one needs to think about the way each bootstrap run
> behaved. But some things like non-convergence and failed covariance are
> ignorable because they don't influence the bootstrap distribution.
>
> There is also the need to recognize that bootstraps can be seriously
> time consuming and the effort required to understand all the ways that
> runs might finish is usually not worth it given the purposes of doing a
> bootstrap.
>
> The most important reason for doing a bootstrap is to get more robust
> estimates of the parameters. This was the main reason why these
> re-sampling procedures were initially developed. The bootstrap estimate
> of the parameters will usually be pretty insensitive to the margins of
> the distribution where the questionable run results are typically located.
>
> A secondary semi-quantitative reason is to get a confidence interval
> which may be helpful for model selection. This may be influenced by the
> questionable runs but that is just part of the uncertainty that the
> confidence interval is used to define.
>
> Nick
>
> On 10/07/2011 11:13 p.m., Leonid Gibiansky wrote:
>
> > I thought that the original post was "results at a boundary should NOT
> > be discarded" and Nick reply was just a typo. If it was not a typo, I
> > would disagree and argue that all results should be included:
> > Each data set is a particular realization. We should be able to use
> > all of them. If some realizations are so special that the model
> > behaves in an unusual way (with any definition of unusual:
> > non-convergence, not convergence of the covariance step, parameter
> > estimates at the boundary, etc.) we either need to accept those as is,
> > or work with each of those special data sets one by one to push to the
> > parameter estimates that we can accept, or change the bootstrap
> > procedure (add stratification by covariates, by dose level, by route
> > of administration, etc.) so that all data sets behave similarly.
> > Leonid
> >
> > --------------------------------------
> > Leonid Gibiansky, Ph.D.
> > President, QuantPharm LLC
> > web: www.quantpharm.com
> > e-mail: LGibiansky at quantpharm.com
> > tel: (301) 767 5566
> >
> > On 7/10/2011 2:57 PM, Stephen Duffull wrote:
> >
> > > Nick, Jakob, Marc et al
> > >
> > > > Thanks for your helpful comments. I agree with you that any results
> > > > that
> > > > are at a boundary should be discarded from the bootstrap distribution.
> > >
> > > On the whole I the sentiments in this thread align with anecdotal
> > > findings from my experience. But, I was just wondering how you define
> > > your boundaries for variance and covariance parameters (e.g. OMEGA
> > > terms)?
> > >
> > > For variance terms, lower boundaries seems reasonably straightforward
> > > (e.g. 1E-5 seems close to zero). Upper boundaries are of course open,
> > > for the variance of a log-normal ETA would 1E+5 or 1E+4 be large
> > > enough to be considered close to a boundary? At what value would you
> > > discard the result? At what correlation value would you discard a
> > > result (>0.99,> 0.97...) as being close to 1. Clearly if this was for
> > > regulatory work you could define these a priori after having chosen
> > > any arbitrary cut-off. But the devil here lies with the
> > > non-regulatory work where you may not have defined these boundaries a
> > > priori.
> > >
> > > Steve
> > > --
> > > Professor Stephen Duffull
> > > Chair of Clinical Pharmacy
> > > School of Pharmacy
> > > University of Otago
> > > PO Box 56 Dunedin
> > > New Zealand
> > > E: [email protected]
> > > P: +64 3 479 5044
> > > F: +64 3 479 7034
> > > W: http://pharmacy.otago.ac.nz/profiles/stephenduffull
> > >
> > > Design software: www.winpopt.com
Leonid
> of them. If some realizations are so special that the model behaves in
> an unusual way (with any definition of unusual: non-convergence, not
> convergence of the covariance step, parameter estimates at the boundary,
> etc.) we either need to accept those as is, or work with each of those
> special data sets one by one to push to the parameter estimates that we
> can accept, or change the bootstrap procedure (add stratification by
> covariates, by dose level, by route of administration, etc.) so that all
> data sets behave similarly.
I agree with your comments above and there are times that this might be
possible. But in essence I hear that BS may become overtly time consuming in
some cases if individual BS'd data sets require almost a full modelling
procedure to gain appropriate convergence outcomes. This might make a long BS
into an almost impossible task.
The risk a user then runs is once you know there are runs that don't work as
intended you cannot ignore them. If you can't afford the time resource (say a
single run takes 12-24 hours) to recover each of these (say 20% of 1000 BS
runs) and since you cannot un-know the results you're stuck. Would not a
pragmatic view be appropriate? Report the outcomes of the BS runs as they
occurred but then select those runs that terminated appropriately? Otherwise
BS is too risky...
Steve
PS All models are wrong :-)
--
>
>
Quoted reply history
> On 7/10/2011 2:57 PM, Stephen Duffull wrote:
> > Nick, Jakob, Marc et al
> >
> >> Thanks for your helpful comments. I agree with you that any results that
> >> are at a boundary should be discarded from the bootstrap distribution.
> >
> > On the whole I the sentiments in this thread align with anecdotal findings
> from my experience. But, I was just wondering how you define your boundaries
> for variance and covariance parameters (e.g. OMEGA terms)?
> >
> > For variance terms, lower boundaries seems reasonably straightforward (e.g.
> 1E-5 seems close to zero). Upper boundaries are of course open, for the
> variance of a log-normal ETA would 1E+5 or 1E+4 be large enough to be
> considered close to a boundary? At what value would you discard the result?
> At what correlation value would you discard a result (>0.99,> 0.97...) as
> being close to 1. Clearly if this was for regulatory work you could define
> these a priori after having chosen any arbitrary cut-off. But the devil here
> lies with the non-regulatory work where you may not have defined these
> boundaries a priori.
> >
> > Steve
> > --
> > Professor Stephen Duffull
> > Chair of Clinical Pharmacy
> > School of Pharmacy
> > University of Otago
> > PO Box 56 Dunedin
> > New Zealand
> > E: [email protected]
> > P: +64 3 479 5044
> > F: +64 3 479 7034
> > W: http://pharmacy.otago.ac.nz/profiles/stephenduffull
> >
> > Design software: www.winpopt.com
> >
> >
Hi Matt,
OK, I can certainly see that transformations will be helpful in
bootstrapping; for those persons that throw away samples with
unsuccessful termination or cov step. They would otherwise discard all
bootstrap estimates that indicate Emax is close to zero. Since I most
often use all bootstrap samples that terminate at a minimum I guess in
practice I would virtually have the same distribution of Emax,
regardless of transformation or not?
I fully agree transformations are useful to get convergence and
successful covstep on the original dataset (and I tend to keep the same
transformation when bootstrapping, but only for simplicity). However, I
sometimes use the bootstrap results to which parameters should be
transformed in the first place. From what I have seen, bootstrapping the
transformed model again has never changed the (non-parametric bootstrap)
distribution when boundaries were the same (e.g. both models bound to
positive values of Emax).
Cheers
Jakob
Quoted reply history
-----Original Message-----
From: Matt Hutmacher [mailto:[email protected]]
Sent: 11 July 2011 17:39
To: Ribbing, Jakob; 'nmusers'
Subject: RE: [NMusers] Confidence intervals of PsN bootstrap output
Hi Jakob,
"The 15% bootstrap samples where data suggest a negative drug effect
would
in one case terminate at the zero boundary, in the other case it would
terminate (often unsuccessfully) at highly negative values for log
Emax"...
I have seen that transformation can make the likelihood surface more
stable.
In my experience, when runs terminate using ordinary Emax
parameterization
with 0 lower bounds (note that NONMEM is using a transformation behind
the
scenes to avoid constrained optimization), you can avoid termination and
even get the $COV to run with different parameterizations. The estimate
might be quite negative as you suggest, but I have seen it recovered.
Also,
I have seen termination avoided and COV achieved with Emax=EXP(THETA(X))
and
EC50=EXP(THETA(Y)) when EC50 and EMAX becomes large. I have seen
variance
components that can be estimated in this way but not with traditional
$OMEGA
implementation.
Best,
matt
Hi Jakob,
I agree that in the well-defined cases that reparameterization will not
matter since maximum likelihood is invariant under certain types of
transformation - your bootstrap distribution would be the same. For the
marginal dataset cases (for any dataset bootstrap or otherwise)
transformation will help convergence and COV step estimation by helping
with boundary problems and reducing intrinsic nonlinearity. For
pathological cases such as those in which a few individuals contain
information to estimate key parameters or when there are certain types of
outlying individuals, one should consider the accuracy of the bootstrap
estimate. Just my opinion.
Best Regards,
Matt
Quoted reply history
From: [email protected] [mailto:[email protected]] On
Behalf Of Ribbing, Jakob
Sent: Monday, July 11, 2011 7:43 PM
To: nmusers
Cc: Matt Hutmacher
Subject: RE: [NMusers] Confidence intervals of PsN bootstrap output
Hi Matt,
OK, I can certainly see that transformations will be helpful in
bootstrapping; for those persons that throw away samples with unsuccessful
termination or cov step. They would otherwise discard all bootstrap
estimates that indicate Emax is close to zero. Since I most often use all
bootstrap samples that terminate at a minimum I guess in practice I would
virtually have the same distribution of Emax, regardless of transformation
or not?
I fully agree transformations are useful to get convergence and successful
covstep on the original dataset (and I tend to keep the same transformation
when bootstrapping, but only for simplicity). However, I sometimes use the
bootstrap results to which parameters should be transformed in the first
place. From what I have seen, bootstrapping the transformed model again has
never changed the (non-parametric bootstrap) distribution when boundaries
were the same (e.g. both models bound to positive values of Emax).
Cheers
Jakob
-----Original Message-----
From: Matt Hutmacher [mailto:[email protected]]
Sent: 11 July 2011 17:39
To: Ribbing, Jakob; 'nmusers'
Subject: RE: [NMusers] Confidence intervals of PsN bootstrap output
Hi Jakob,
"The 15% bootstrap samples where data suggest a negative drug effect would
in one case terminate at the zero boundary, in the other case it would
terminate (often unsuccessfully) at highly negative values for log Emax"...
I have seen that transformation can make the likelihood surface more stable.
In my experience, when runs terminate using ordinary Emax parameterization
with 0 lower bounds (note that NONMEM is using a transformation behind the
scenes to avoid constrained optimization), you can avoid termination and
even get the $COV to run with different parameterizations. The estimate
might be quite negative as you suggest, but I have seen it recovered. Also,
I have seen termination avoided and COV achieved with Emax=EXP(THETA(X)) and
EC50=EXP(THETA(Y)) when EC50 and EMAX becomes large. I have seen variance
components that can be estimated in this way but not with traditional $OMEGA
implementation.
Best,
matt