RE: Confidence intervals of PsN bootstrap output
All,
This first part is more to clarify and I do not believe this is in
disagreement with what has been said before. The last paragraph is a
question.
The two examples I mentioned regarding boundary conditions are regarding
variance parameters. The second of these, however, is with regards to a
boundary at eta-correlation of one, which is a must rather than just an
irritating NONMEM feature.
I used these examples because they were less controversial and it is
difficult to come up with general statements that apply to all cases.
However, as a third example for a fixed-effects parameter: Imagine a
covariate acting in a linear fashion on a structural parameter that is
bound to be non-negative (e.g. a rate constant, volume, clearance, ED50,
etc). Imagine boundaries on the covariate parameter have been set to
avoid negative values for the structural-model parameter (on the
individual level). For this scenario if a substantial fraction of the
bootstrapped covariate-parameter values end up at one of the boundaries,
one may have to consider two options:
a) Decide that a linear covariate model is inappropriate (at least for
the goal of extrapolating to the whole population with more extreme
covariate values) and change the model into using a different functional
form
b) Dismiss this as random chance, due to small sample/limited
information and a (covariate) slope which "truly" is not far from one of
the boundaries. If this is the case, deleting the bootstrap estimates at
boundary would bias the distribution in an undesirable manner. For that
case the boundary condition is not due to local minimum and we would not
want to discard bootstrap samples at boundary). (Nick's example is of a
different kind, where it is either a local minimum or else not reaching
a minimum at all)
A related question - I am thinking more in terms of simulations with
parameter uncertainty; not just obtaining CI, which was originally what
this thread was about:
There are sometimes situations where a limited set of (clinical-) trial
data gives reasonable point estimates but with huge parameter
uncertainty (regardless nonmem covmaxtrix or bootstrap with appropriate
stratification). The distribution and CI on these parameters may include
unreasonable values, even though there is no obvious physiological
boundary (unreasonable based on prior knowledge that has not been
incorporated into the analysis, e.g. for a certain mechanism and patient
population Typical-Emax beyond 400% or 10 units - depending on if Emax
is parameterised as relative or absolute change). In these situations, a
simplistic option could be to trim one or both ends with regards to the
Emax distribution and discard these bootstrap samples, especially if
only a few values are unreasonable. Alternatively, before running the
bootstrap, one may set the boundary in the control stream (a boundary
that everyone can agree is unreasonable). One would then keep bootstrap
samples that ends up at this boundary for bootstrap distribution, which
is in a way truncated, but so that bootstrap samples indicating linear
concentration/dose-response maintains almost reasonable Emax and
ED50/EC50 values (but as a spike in the distribution at upper Emax).
Notice that re-parameterising the Emax model would not solve the
underlying issue with unreasonable estimates and reducing to a linear
model may be unsuitable, both based on the original dataset and also for
mechanistic reasons). Could you suggest alternative ways of dealing with
this, for these rather general examples (other than the obvious of
applying an informative prior on Emax)? I would be interested in your
solutions both in terms of the non-parametric bootstrap as well as the
parametric bootstrap (based on the nonmem covmatrix).
Much appreciated
Jakob
Quoted reply history
-----Original Message-----
From: [email protected] [mailto:[email protected]]
On Behalf Of Nick Holford
Sent: 11 July 2011 06:37
To: nmusers
Subject: Re: [NMusers] Confidence intervals of PsN bootstrap output
Leonid,
With regard to discarding runs at the boundary what I had in mind was
runs which had reached the maximum number of iterations but I realized
later that Jacob was referring to NONMEM's often irritating messages
that usually just mean the initial estimate changed a lot or variance
was getting close to zero.
There are of course some cases where the estimate is truly at a user
defined constraint. Assuming that the user has thought carefully about
these constraints then I would interpret a run that finished at this
constraint boundary as showing NONMEM was stuck in a local minimum
(probably because of the constraint boundary) and if the constraint was
relaxed then perhaps a more useful estimate would be obtained.
In those cases then I think one can make an argument for discarding runs
with parameters that are at this kind of boundary as well as those which
reached an iteration limit.
In general I agree with your remarks (echoing those from Marc
Gastonguay) that one needs to think about the way each bootstrap run
behaved. But some things like non-convergence and failed covariance are
ignorable because they don't influence the bootstrap distribution.
There is also the need to recognize that bootstraps can be seriously
time consuming and the effort required to understand all the ways that
runs might finish is usually not worth it given the purposes of doing a
bootstrap.
The most important reason for doing a bootstrap is to get more robust
estimates of the parameters. This was the main reason why these
re-sampling procedures were initially developed. The bootstrap estimate
of the parameters will usually be pretty insensitive to the margins of
the distribution where the questionable run results are typically
located.
A secondary semi-quantitative reason is to get a confidence interval
which may be helpful for model selection. This may be influenced by the
questionable runs but that is just part of the uncertainty that the
confidence interval is used to define.
Nick
On 10/07/2011 11:13 p.m., Leonid Gibiansky wrote:
> I thought that the original post was "results at a boundary should NOT
> be discarded" and Nick reply was just a typo. If it was not a typo, I
> would disagree and argue that all results should be included:
> Each data set is a particular realization. We should be able to use
> all of them. If some realizations are so special that the model
> behaves in an unusual way (with any definition of unusual:
> non-convergence, not convergence of the covariance step, parameter
> estimates at the boundary, etc.) we either need to accept those as is,
> or work with each of those special data sets one by one to push to the
> parameter estimates that we can accept, or change the bootstrap
> procedure (add stratification by covariates, by dose level, by route
> of administration, etc.) so that all data sets behave similarly.
> Leonid
>
> --------------------------------------
> Leonid Gibiansky, Ph.D.
> President, QuantPharm LLC
> web: www.quantpharm.com
> e-mail: LGibiansky at quantpharm.com
> tel: (301) 767 5566
>
>
>
> On 7/10/2011 2:57 PM, Stephen Duffull wrote:
>> Nick, Jakob, Marc et al
>>
>>> Thanks for your helpful comments. I agree with you that any results
>>> that
>>> are at a boundary should be discarded from the bootstrap
distribution.
>>
>> On the whole I the sentiments in this thread align with anecdotal
>> findings from my experience. But, I was just wondering how you
>> define your boundaries for variance and covariance parameters (e.g.
>> OMEGA terms)?
>>
>> For variance terms, lower boundaries seems reasonably straightforward
>> (e.g. 1E-5 seems close to zero). Upper boundaries are of course
>> open, for the variance of a log-normal ETA would 1E+5 or 1E+4 be
>> large enough to be considered close to a boundary? At what value
>> would you discard the result? At what correlation value would you
>> discard a result (>0.99,> 0.97...) as being close to 1. Clearly if
>> this was for regulatory work you could define these a priori after
>> having chosen any arbitrary cut-off. But the devil here lies with
>> the non-regulatory work where you may not have defined these
>> boundaries a priori.
>>
>> Steve
>> --
>> Professor Stephen Duffull
>> Chair of Clinical Pharmacy
>> School of Pharmacy
>> University of Otago
>> PO Box 56 Dunedin
>> New Zealand
>> E: [email protected]
>> P: +64 3 479 5044
>> F: +64 3 479 7034
>> W: http://pharmacy.otago.ac.nz/profiles/stephenduffull
>>
>> Design software: www.winpopt.com
>>
>>
--
Nick Holford, Professor Clinical Pharmacology
Dept Pharmacology& Clinical Pharmacology
University of Auckland,85 Park Rd,Private Bag 92019,Auckland,New Zealand
tel:+64(9)923-6730 fax:+64(9)373-7090 mobile:+64(21)46 23 53
email: [email protected]
http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford