Dear nmusers,
Merry Holiday! (you may change the variable 'Holiday' to an arbitrary value)
First, accept my humble excuse - I can't share any code for this example.
I'm assessing a model where $PRIOR is used to stabilize the fit of an 'old'
model to sparse data using NONMEM. It seems to be the only way forward. Using
the final model the Applicant has then performed a non-parametric bootstrap
(N=1000) to investigate i.a. robustness of the model parameter estimates.
>From a non-scientific literature search (Google Scholar "NONMEM + PRIOR +
>Bootstrap") some examples show up where Bootstrap and PRIOR has been combined.
I'm curious about if any nm-user has investigated how much and in what way the
prior distribution of parameters affect the bootstrap estimates (eg. 2.5th,
50th and 97.5th percentiles)? I would guess that the stronger the prior the
more it would affect the outcome of a bootstrap. Or is that a misconception?
Cheers
Jacob
[cid:[email protected]]
Medical Products Agency
Jacob Brogren
Assessor
Efficacy and Safety 2
P.O. Box 26, SE-751 03 Uppsala, Sweden
Visiting address: Dag Hammarskjölds väg 42
Phone: + 46 (0) 18 17 47 64, switchboard: + 46 (0) 18 17 46 00
Mobile: + 46 (0) xxx xx xx, Fax: + 46 (0) 18 54 85 66
http://www.lakemedelsverket.se
<<inline: image003.jpg>>
Bootstrap in combination with PRIOR?
Jacob,
Bootstrap procedure vary only the new data. The information in priors is still the same. Therefore, results will be as dependent on the priors as the parameter estimates of the final model. When the priors are strong (informative) it would require a lot of new data to move the parameters of the final model and the parameter estimates of each of the bootstrap samples. In the extreme case of the very strong priors (e.g., when the parameters are simply fixed) parameters of all bootstrap samples will also be fixed at the same values. Since the results are dependent of the strength (informative content) of priors, I doubt that any conclusions can be made about each particular case based on the other-people examples. This is likely to be decided on the case-by-case basis.
Nonmem standard errors are usually in a good agreement with the bootstrap results. Therefore, influence of the priors on the parameter estimates (precision) can be evaluated using the standard errors (much quicker than running bootstrap multiple times with different prior values). Extreme test would be to run the model without priors, and see how this would influence the confidence intervals.
Regards,
Leonid
--------------------------------------
Leonid Gibiansky, Ph.D.
President, QuantPharm LLC
web: www.quantpharm.com
e-mail: LGibiansky at quantpharm.com
tel: (301) 767 5566
Quoted reply history
On 12/22/2011 7:58 AM, Brogren Jacob wrote:
> Dear nmusers,
>
> Merry Holiday! (you may change the variable ‘Holiday’ to an arbitrary value)
>
> First, accept my humble excuse – I can’t share any code for this example.
>
> I’m assessing a model where $PRIOR is used to stabilize the fit of an
> ‘old’ model to sparse data using NONMEM. It seems to be the only way
> forward. Using the final model the Applicant has then performed a
> non-parametric bootstrap (N=1000) to investigate i.a. robustness of the
> model parameter estimates.
>
> From a non-scientific literature search (Google Scholar “NONMEM + PRIOR
> + Bootstrap”) some examples show up where Bootstrap and PRIOR has been
> combined.
>
> I’m curious about if any nm-user has investigated how much and in what
> way the prior distribution of parameters affect the bootstrap estimates
> (eg. 2.5^th , 50^th and 97.5^th percentiles)? I would guess that the
> stronger the prior the more it would affect the outcome of a bootstrap.
> Or is that a misconception?
>
> Cheers
>
> Jacob
>
> LV_Mac3
>
> Medical Products Agency
>
> Jacob Brogren
> Assessor
> Efficacy and Safety 2
>
> P.O. Box 26, SE-751 03 Uppsala, Sweden
> Visiting address: Dag Hammarskjölds väg 42
> Phone: + 46 (0) 18 17 47 64, switchboard: + 46 (0) 18 17 46 00
> Mobile: + 46 (0) xxx xx xx, Fax: + 46 (0) 18 54 85 66
> www.lakemedelsverket.se http://www.lakemedelsverket.se
Jacob,
I am guilty of having performed such a bootstrap (but didn't do any of the
testing you describe). Anyway, here's an opinion: By using prior in nonmem
you are trying to get an approximation of what the pooled data fit would be and
get an OFV that is theoretically the same as if you did the pooled analysis.
Similarly I would think that such a bootstrap gives a measure of uncertainty
that is similar as would be found if you had pooled data - so if you increase
uncertainty on the prior element, then I would imagine you would get increase
in bootstrap uncertainty, the magnitude of which will vary depending on the
relative contribution of the data and prior (and thereby tell you whether
information is coming from the data or the prior). For this reason, I think
without some exhaustive simulation exercise with lots of permutations of data
and prior informativeness, there is probably no straight-forward answer as to
how prior uncertainty affects bootstrap uncertainty, and would just try a
couple of bootstraps with increased uncertainty in the priors of parameters
that you think might be overly-influenced to see what happens.
BW,
Joe
Joseph F Standing
MRC Fellow, UCL Institute of Child Health
Antimicrobial Pharmacist, Great Ormond Street Hospital
Honorary Lecturer, London School of Pharmacy
Mobile: +44(0)7970 572435
Quoted reply history
________________________________
From: [email protected] [[email protected]] On Behalf Of
Brogren Jacob [[email protected]]
Sent: 22 December 2011 12:58
To: [email protected]
Subject: [NMusers] Bootstrap in combination with PRIOR?
Dear nmusers,
Merry Holiday! (you may change the variable ‘Holiday’ to an arbitrary value)
First, accept my humble excuse – I can’t share any code for this example.
I’m assessing a model where $PRIOR is used to stabilize the fit of an ‘old’
model to sparse data using NONMEM. It seems to be the only way forward. Using
the final model the Applicant has then performed a non-parametric bootstrap
(N=1000) to investigate i.a. robustness of the model parameter estimates.
>From a non-scientific literature search (Google Scholar “NONMEM + PRIOR +
>Bootstrap”) some examples show up where Bootstrap and PRIOR has been combined.
I’m curious about if any nm-user has investigated how much and in what way the
prior distribution of parameters affect the bootstrap estimates (eg. 2.5th,
50th and 97.5th percentiles)? I would guess that the stronger the prior the
more it would affect the outcome of a bootstrap. Or is that a misconception?
Cheers
Jacob
[cid:[email protected]]
Medical Products Agency
Jacob Brogren
Assessor
Efficacy and Safety 2
P.O. Box 26, SE-751 03 Uppsala, Sweden
Visiting address: Dag Hammarskjölds väg 42
Phone: + 46 (0) 18 17 47 64, switchboard: + 46 (0) 18 17 46 00
Mobile: + 46 (0) xxx xx xx, Fax: + 46 (0) 18 54 85 66
http://www.lakemedelsverket.se
********************************************************************************************************************
This message may contain confidential information. If you are not the intended
recipient please inform the
sender that you have received the message in error before deleting it.
Please do not disclose, copy or distribute information in this e-mail or take
any action in reliance on its contents:
to do so is strictly prohibited and may be unlawful.
Thank you for your co-operation.
NHSmail is the secure email and directory service available for all NHS staff
in England and Scotland
NHSmail is approved for exchanging patient data and other sensitive information
with NHSmail and GSi recipients
NHSmail provides an email address for your career in the NHS and can be
accessed anywhere
For more information and to find out how you can switch, visit
www.connectingforhealth.nhs.uk/nhsmail
********************************************************************************************************************
<<inline: image003.jpg>>
Jacob,
To address your question, I would compare the uncertainty in models parameters
obtained after running a non-parametric bootstrap (NP-BS) with that obtained
after a parametric bootstrap (P-BS).
P-BS allows you to understand (under the null hypothesis) the expected
posterior distribution of model parameters given the design of the new study
and, therefore, you can assess if your new data would contain enough
information to speak about certain parameters.
For instance, if after running a P-BS, you get the same estimate of a certain
parameter in all bootstraps replicates, it means the data available does not
speak about that parameter (regardless of how strong the prior is) and,
consequently, you can fix it before running the PRIOR and, of course, cannot
derive any new conclusion on that parameter based on the new data.
If the new data speaks substantially about model parameters (because an
informative design and/or large sample size), the P-BS would provide you with
the expected value (and uncertainty) of the model parameters (assuming the new
data are arising from the same population used to obtain the priors) and will
give you an idea about the contribution of the new study design in reducing the
uncertainty in model parameter from the priors.
However, if the new data are not arising from the same population used to
obtain the priors, the uncertainty in model parameters might not be reduced
and, therefore, you need to make a decision about the appropriateness of using
PRIOR in that situation. You can make that assessment by comparing the point
estimates (and uncertainty) of the updated parameters with the results of the
P-BS. If the updated parameter estimates falls within the expected updated
values of the model parameters, then you can assume the new data are arising
from the same population used to obtain the priors, provided you have enough
power. However, this exercise tends to be conservative and additional
simulation/estimation exercise might be needed.
Hope it helps. Enjoy the Holiday Season!
Juan J. Perez-Ruixo.
Quoted reply history
-----Original Message-----
From: [email protected] [mailto:[email protected]] On
Behalf Of Leonid Gibiansky
Sent: jueves, 22 de diciembre de 2011 7:08
To: Brogren Jacob
Cc: [email protected]
Subject: Re: [NMusers] Bootstrap in combination with PRIOR?
Jacob,
Bootstrap procedure vary only the new data. The information in priors is
still the same. Therefore, results will be as dependent on the priors as
the parameter estimates of the final model. When the priors are strong
(informative) it would require a lot of new data to move the parameters
of the final model and the parameter estimates of each of the bootstrap
samples. In the extreme case of the very strong priors (e.g., when the
parameters are simply fixed) parameters of all bootstrap samples will
also be fixed at the same values. Since the results are dependent of the
strength (informative content) of priors, I doubt that any conclusions
can be made about each particular case based on the other-people
examples. This is likely to be decided on the case-by-case basis.
Nonmem standard errors are usually in a good agreement with the
bootstrap results. Therefore, influence of the priors on the parameter
estimates (precision) can be evaluated using the standard errors (much
quicker than running bootstrap multiple times with different prior
values). Extreme test would be to run the model without priors, and see
how this would influence the confidence intervals.
Regards,
Leonid
--------------------------------------
Leonid Gibiansky, Ph.D.
President, QuantPharm LLC
web: www.quantpharm.com
e-mail: LGibiansky at quantpharm.com
tel: (301) 767 5566
On 12/22/2011 7:58 AM, Brogren Jacob wrote:
> Dear nmusers,
>
> Merry Holiday! (you may change the variable 'Holiday' to an arbitrary value)
>
> First, accept my humble excuse - I can't share any code for this example.
>
> I'm assessing a model where $PRIOR is used to stabilize the fit of an
> 'old' model to sparse data using NONMEM. It seems to be the only way
> forward. Using the final model the Applicant has then performed a
> non-parametric bootstrap (N=1000) to investigate i.a. robustness of the
> model parameter estimates.
>
> From a non-scientific literature search (Google Scholar "NONMEM + PRIOR
> + Bootstrap") some examples show up where Bootstrap and PRIOR has been
> combined.
>
> I'm curious about if any nm-user has investigated how much and in what
> way the prior distribution of parameters affect the bootstrap estimates
> (eg. 2.5^th , 50^th and 97.5^th percentiles)? I would guess that the
> stronger the prior the more it would affect the outcome of a bootstrap.
> Or is that a misconception?
>
> Cheers
>
> Jacob
>
> LV_Mac3
>
> Medical Products Agency
>
> Jacob Brogren
> Assessor
> Efficacy and Safety 2
>
>
>
> P.O. Box 26, SE-751 03 Uppsala, Sweden
> Visiting address: Dag Hammarskjölds väg 42
> Phone: + 46 (0) 18 17 47 64, switchboard: + 46 (0) 18 17 46 00
> Mobile: + 46 (0) xxx xx xx, Fax: + 46 (0) 18 54 85 66
> www.lakemedelsverket.se http://www.lakemedelsverket.se
>