Hi all,
Creation of VPCs is a way to assess if simulated data generated by the
model is compatible with observed data.
VPCs are usually based on parameter point estimates of the model. Sometimes
parameter uncertainty is also accounted for in the generation of VPCs
(PPCs) where each simulated replicate of the data set is based on a new set
of parameter values representing the uncertainty of the estimates (e.g.
based on a bootstrap).
I wonder if inclusion of uncertainty in this way is really appropriate or
if it just makes the confidence intervals wider and hence easier to qualify
the model. Is it possible based on such an approach, that a model might
look good, when in fact no likely combination of parameter values (based on
parameter uncertainty) would generate data that are compatible with the
observations?
To illustrate my question:
I could generate 100 sets of parameters reflecting parameter uncertainty
(e.g. from a bootstrap). Based on each set of parameters I could then
generate a separate VPC (e.g. showing median, 5 and 95% percentile) to see
if any of the parameter sets are compatible with data. I would then have
100 VPCs, each based on a separate set of parameter values reflecting the
parameter correlations and uncertainty.
If the VPC based on point estimates looks bad, I would (generally) expect
that the other VPCs would be worse (they all have lower likelihood), so
that we have 101 VPCs that does not look good. Some might over predict and
some underpredict, some might describe parts of the relation better than
the VPC based on the point estimates.
By putting the VPCs together from all parameter vectors, the CI becomes
wider, and perhaps now includes the observed data. So based on a set of 100
parameter vectors which individually are not compatible with the observed
data I have now generated a VPC (PPC) where the confidence interval
actually includes the observed metric (e.g median). It seems to me that
based on such an approach it is possible that a model might look good, when
in fact no likely individual set of parameter values would generate data
that are compatible with the observations.
Simulation based on parameter uncertainty is useful when we want to make
inference, but I am unsure of its use for model qualification. In any case
it is confusing that we some times simulate based on point estimates and
sometimes based on parameter uncertainty without any particular rationale
as far as I understand.
Would be interested if someone could shed some light on the inclusion of
uncertainty in simulations for model qualification (VPCs).
Best regards,
Matts Kagedal
Pharmacometrician, Genentech
Should we generate VPCs with or without uncertainty?
Matts,
The way I see the CI's around the point estimates provided in the VPC can
help provide a useful indication of model robustness, especially in regards
to the impact of random effects components, in that portion of your model.
Especially for heterogeneous data (or even all rich data for that matter)
there are a number of binning strategies that can be used, which can impact
the aforementioned intervals.
At the end of the day, we must use our judgement for how the model is being
used to support decisions, and whether information regarding uncertainty
can provide additional support towards the overall evaluation of the key
questions you are trying to address. Eg, if you are dealing with a narrow
therapeutic index drug the value of having a 'feel' for the robustness of
the ability of your model to describe the tails may be valuable
information, even as a qualitative indication of model robustness. On the
other hand, if you are trying to make a decision regarding dose adjustment
between different populations and are looking to normalize large
differences, as well as are constrained to certain oral dosage options,
uncertainty in the point estimates will likely provide very little support
to an argument one way or the other.
Finally, in my opinion, inclusion/exclusion also relies on what the plot is
trying to communicate. Are you trying to personally evaluate model
adequacy, sure, but if using to convey to non- modelers/quantitative people
that your model describes the data - include a visualization of uncertainty
at your own peril :-)
So, for better or worse, I would say - it depends, though I would be highly
concerned if major decisions rode on inclusion/exclusion of parameter
uncertainty, in most cases.
Devin Pastoor
Center for Translational Medicine
University of Maryland, Baltimore
Quoted reply history
On Mon, Jun 8, 2015 at 11:57 AM Matts Kågedal <[email protected]>
wrote:
> Hi all,
>
> Creation of VPCs is a way to assess if simulated data generated by the
> model is compatible with observed data.
> VPCs are usually based on parameter point estimates of the model.
> Sometimes parameter uncertainty is also accounted for in the generation of
> VPCs (PPCs) where each simulated replicate of the data set is based on a
> new set of parameter values representing the uncertainty of the estimates
> (e.g. based on a bootstrap).
>
> I wonder if inclusion of uncertainty in this way is really appropriate or
> if it just makes the confidence intervals wider and hence easier to qualify
> the model. Is it possible based on such an approach, that a model might
> look good, when in fact no likely combination of parameter values (based on
> parameter uncertainty) would generate data that are compatible with the
> observations?
>
> To illustrate my question:
> I could generate 100 sets of parameters reflecting parameter uncertainty
> (e.g. from a bootstrap). Based on each set of parameters I could then
> generate a separate VPC (e.g. showing median, 5 and 95% percentile) to see
> if any of the parameter sets are compatible with data. I would then have
> 100 VPCs, each based on a separate set of parameter values reflecting the
> parameter correlations and uncertainty.
>
> If the VPC based on point estimates looks bad, I would (generally) expect
> that the other VPCs would be worse (they all have lower likelihood), so
> that we have 101 VPCs that does not look good. Some might over predict and
> some underpredict, some might describe parts of the relation better than
> the VPC based on the point estimates.
>
> By putting the VPCs together from all parameter vectors, the CI becomes
> wider, and perhaps now includes the observed data. So based on a set of 100
> parameter vectors which individually are not compatible with the observed
> data I have now generated a VPC (PPC) where the confidence interval
> actually includes the observed metric (e.g median). It seems to me that
> based on such an approach it is possible that a model might look good, when
> in fact no likely individual set of parameter values would generate data
> that are compatible with the observations.
>
> Simulation based on parameter uncertainty is useful when we want to make
> inference, but I am unsure of its use for model qualification. In any case
> it is confusing that we some times simulate based on point estimates and
> sometimes based on parameter uncertainty without any particular rationale
> as far as I understand.
>
> Would be interested if someone could shed some light on the inclusion of
> uncertainty in simulations for model qualification (VPCs).
>
> Best regards,
> Matts Kagedal
>
> Pharmacometrician, Genentech
>
>
>
Hi Devin and Matts,
Of interest, we had similar internal discussion within our department about
this topic quite recently.
I would agree with Devin that it is preferable to include parameters
uncertainty in a VPC, especially if they are used for relevant decision making.
It may result on large CI around your 5th, median and 95th percentiles, however
it means that you likely have limited information to estimate some of the
variability terms. Don’t forget that any “standard VPC” still provides some
uncertainty around your percentiles based on random effects and sample size.
There are several ways to embed params uncertainty, I found quite useful to use
the covariance-matrix (from NM) to embed it in a simulation script – it is
quite straightforward to do that in R with some coding effort (I think Pirana
has R code for that as well), but I’m sure it can be easily replicated with
different tools available.
I guess it would be interesting to see how pharmacometricians would derive the
parameters uncertainty: i.e, covariance matrix vs bootstrap or other means. I
recently saw an elegant and efficient solution of bootstrap VPC done from a
colleague of mine.
Cheers
Stefano
Stefano Zamuner PhD
Senior Director Clinical Pharmacology
GSK
Quoted reply history
From: [email protected] [mailto:[email protected]] On
Behalf Of Devin Pastoor
Sent: 08 June 2015 17:30
To: Matts Kågedal; [email protected]
Subject: Re: [NMusers] Fwd: Should we generate VPCs with or without uncertainty?
Matts,
The way I see the CI's around the point estimates provided in the VPC can help
provide a useful indication of model robustness, especially in regards to the
impact of random effects components, in that portion of your model. Especially
for heterogeneous data (or even all rich data for that matter) there are a
number of binning strategies that can be used, which can impact the
aforementioned intervals.
At the end of the day, we must use our judgement for how the model is being
used to support decisions, and whether information regarding uncertainty can
provide additional support towards the overall evaluation of the key questions
you are trying to address. Eg, if you are dealing with a narrow therapeutic
index drug the value of having a 'feel' for the robustness of the ability of
your model to describe the tails may be valuable information, even as a
qualitative indication of model robustness. On the other hand, if you are
trying to make a decision regarding dose adjustment between different
populations and are looking to normalize large differences, as well as are
constrained to certain oral dosage options, uncertainty in the point estimates
will likely provide very little support to an argument one way or the other.
Finally, in my opinion, inclusion/exclusion also relies on what the plot is
trying to communicate. Are you trying to personally evaluate model adequacy,
sure, but if using to convey to non- modelers/quantitative people that your
model describes the data - include a visualization of uncertainty at your own
peril :-)
So, for better or worse, I would say - it depends, though I would be highly
concerned if major decisions rode on inclusion/exclusion of parameter
uncertainty, in most cases.
Devin Pastoor
Center for Translational Medicine
University of Maryland, Baltimore
On Mon, Jun 8, 2015 at 11:57 AM Matts Kågedal
<[email protected]<mailto:[email protected]>> wrote:
Hi all,
Creation of VPCs is a way to assess if simulated data generated by the model is
compatible with observed data.
VPCs are usually based on parameter point estimates of the model. Sometimes
parameter uncertainty is also accounted for in the generation of VPCs (PPCs)
where each simulated replicate of the data set is based on a new set of
parameter values representing the uncertainty of the estimates (e.g. based on a
bootstrap).
I wonder if inclusion of uncertainty in this way is really appropriate or if it
just makes the confidence intervals wider and hence easier to qualify the
model. Is it possible based on such an approach, that a model might look good,
when in fact no likely combination of parameter values (based on parameter
uncertainty) would generate data that are compatible with the observations?
To illustrate my question:
I could generate 100 sets of parameters reflecting parameter uncertainty (e.g.
from a bootstrap). Based on each set of parameters I could then generate a
separate VPC (e.g. showing median, 5 and 95% percentile) to see if any of the
parameter sets are compatible with data. I would then have 100 VPCs, each based
on a separate set of parameter values reflecting the parameter correlations and
uncertainty.
If the VPC based on point estimates looks bad, I would (generally) expect that
the other VPCs would be worse (they all have lower likelihood), so that we have
101 VPCs that does not look good. Some might over predict and some
underpredict, some might describe parts of the relation better than the VPC
based on the point estimates.
By putting the VPCs together from all parameter vectors, the CI becomes wider,
and perhaps now includes the observed data. So based on a set of 100 parameter
vectors which individually are not compatible with the observed data I have now
generated a VPC (PPC) where the confidence interval actually includes the
observed metric (e.g median). It seems to me that based on such an approach it
is possible that a model might look good, when in fact no likely individual set
of parameter values would generate data that are compatible with the
observations.
Simulation based on parameter uncertainty is useful when we want to make
inference, but I am unsure of its use for model qualification. In any case it
is confusing that we some times simulate based on point estimates and sometimes
based on parameter uncertainty without any particular rationale as far as I
understand.
Would be interested if someone could shed some light on the inclusion of
uncertainty in simulations for model qualification (VPCs).
Best regards,
Matts Kagedal
Pharmacometrician, Genentech
________________________________
This e-mail was sent by GlaxoSmithKline Services Unlimited
(registered in England and Wales No. 1047315), which is a
member of the GlaxoSmithKline group of companies. The
registered address of GlaxoSmithKline Services Unlimited
is 980 Great West Road, Brentford, Middlesex TW8 9GS.
Dear Matts,
For assessing model adequacy, I would use the point estimates. If your best
model is contradicted by the data by showing a poor VPC, there seems little
meaning in trying to include uncertainty. There could be a role for VPCs with
uncertainty though. If you plan to perform simulations with parameter
uncertainty for deciding on trial design etc, I may perform a VPC with
uncertainty and assure myself that the parameter uncertainty does not lead to
unrealistic predictions (indicated by too wide confidence intervals of outer
percentiles). [An alternative is to perform a VPC with every population
parameter vector used in the clinical trial simulation and look for
outrageously poor description of the original data, but that is a bit too much
for most, including me. Better to rely on good methods for parameter
uncertainty).
Best regards,
Mats
Mats Karlsson, PhD
Professor of Pharmacometrics
Dept of Pharmaceutical Biosciences
Faculty of Pharmacy
Uppsala University
Box 591
75124 Uppsala
Phone: +46 18 4714105
Fax + 46 18 4714003
http://www.farmbio.uu.se/research/researchgroups/pharmacometrics/
Quoted reply history
From: [email protected] [mailto:[email protected]] On
Behalf Of Devin Pastoor
Sent: Monday, June 08, 2015 6:30 PM
To: Matts Kågedal; [email protected]
Subject: Re: [NMusers] Fwd: Should we generate VPCs with or without uncertainty?
Matts,
The way I see the CI's around the point estimates provided in the VPC can help
provide a useful indication of model robustness, especially in regards to the
impact of random effects components, in that portion of your model. Especially
for heterogeneous data (or even all rich data for that matter) there are a
number of binning strategies that can be used, which can impact the
aforementioned intervals.
At the end of the day, we must use our judgement for how the model is being
used to support decisions, and whether information regarding uncertainty can
provide additional support towards the overall evaluation of the key questions
you are trying to address. Eg, if you are dealing with a narrow therapeutic
index drug the value of having a 'feel' for the robustness of the ability of
your model to describe the tails may be valuable information, even as a
qualitative indication of model robustness. On the other hand, if you are
trying to make a decision regarding dose adjustment between different
populations and are looking to normalize large differences, as well as are
constrained to certain oral dosage options, uncertainty in the point estimates
will likely provide very little support to an argument one way or the other.
Finally, in my opinion, inclusion/exclusion also relies on what the plot is
trying to communicate. Are you trying to personally evaluate model adequacy,
sure, but if using to convey to non- modelers/quantitative people that your
model describes the data - include a visualization of uncertainty at your own
peril :-)
So, for better or worse, I would say - it depends, though I would be highly
concerned if major decisions rode on inclusion/exclusion of parameter
uncertainty, in most cases.
Devin Pastoor
Center for Translational Medicine
University of Maryland, Baltimore
On Mon, Jun 8, 2015 at 11:57 AM Matts Kågedal
<[email protected]<mailto:[email protected]>> wrote:
Hi all,
Creation of VPCs is a way to assess if simulated data generated by the model is
compatible with observed data.
VPCs are usually based on parameter point estimates of the model. Sometimes
parameter uncertainty is also accounted for in the generation of VPCs (PPCs)
where each simulated replicate of the data set is based on a new set of
parameter values representing the uncertainty of the estimates (e.g. based on a
bootstrap).
I wonder if inclusion of uncertainty in this way is really appropriate or if it
just makes the confidence intervals wider and hence easier to qualify the
model. Is it possible based on such an approach, that a model might look good,
when in fact no likely combination of parameter values (based on parameter
uncertainty) would generate data that are compatible with the observations?
To illustrate my question:
I could generate 100 sets of parameters reflecting parameter uncertainty (e.g.
from a bootstrap). Based on each set of parameters I could then generate a
separate VPC (e.g. showing median, 5 and 95% percentile) to see if any of the
parameter sets are compatible with data. I would then have 100 VPCs, each based
on a separate set of parameter values reflecting the parameter correlations and
uncertainty.
If the VPC based on point estimates looks bad, I would (generally) expect that
the other VPCs would be worse (they all have lower likelihood), so that we have
101 VPCs that does not look good. Some might over predict and some
underpredict, some might describe parts of the relation better than the VPC
based on the point estimates.
By putting the VPCs together from all parameter vectors, the CI becomes wider,
and perhaps now includes the observed data. So based on a set of 100 parameter
vectors which individually are not compatible with the observed data I have now
generated a VPC (PPC) where the confidence interval actually includes the
observed metric (e.g median). It seems to me that based on such an approach it
is possible that a model might look good, when in fact no likely individual set
of parameter values would generate data that are compatible with the
observations.
Simulation based on parameter uncertainty is useful when we want to make
inference, but I am unsure of its use for model qualification. In any case it
is confusing that we some times simulate based on point estimates and sometimes
based on parameter uncertainty without any particular rationale as far as I
understand.
Would be interested if someone could shed some light on the inclusion of
uncertainty in simulations for model qualification (VPCs).
Best regards,
Matts Kagedal
Pharmacometrician, Genentech
Hi All,
I have done it both ways (with and without including parameter uncertainty).
It is important to note that the resulting VPC intervals are degenerate when
you don’t take into account parameter uncertainty. That is, with infinite
sample size these intervals will collapse to the predictions based on the point
estimates (since you’ll essentially be averaging out the sampling variation
(etas and epsilons) when you make a mean/median prediction across a very large
number of subjects). Thus, in situations where you have a very large sample
size for the bins, the VPC intervals can be too narrow and it can be almost
hopeless to demonstrate that the observed values will be contained within these
narrow VPC intervals. This is because one would still expect some
discrepancies between the observed and predicted due to parameter uncertainty.
On the other hand, suppose that you have a relatively small dataset such that
VPC intervals are considerably wider when taking into account parameter
uncertainty. If the observed data (e.g., means, median, etc) are not contained
within the degenerate VPC intervals, but are contained within the VPC intervals
that take into account the parameter uncertainty, this may or may not mean you
have a good predictive model. It may simply mean you just don’t have enough
data to “validate” your model via a VPC given the small sample size and that
you need to collect more data before you can truly evaluate the predictive
performance of your model.
Best,
Ken
Kenneth G. Kowalski
President & CEO
A2PG - Ann Arbor Pharmacometrics Group, Inc.
301 N. Main St., Suite 102
Ann Arbor, MI 48104
Work: 734-274-8255
Cell: 248-207-5082
Fax: 734-913-0230
[email protected]
www.a2pg.com http://www.a2pg.com/
Quoted reply history
From: [email protected] [mailto:[email protected]] On
Behalf Of Mats Karlsson
Sent: Monday, June 08, 2015 1:27 PM
To: Devin Pastoor; Matts Kågedal; [email protected]
Subject: RE: [NMusers] Fwd: Should we generate VPCs with or without uncertainty?
Dear Matts,
For assessing model adequacy, I would use the point estimates. If your best
model is contradicted by the data by showing a poor VPC, there seems little
meaning in trying to include uncertainty. There could be a role for VPCs with
uncertainty though. If you plan to perform simulations with parameter
uncertainty for deciding on trial design etc, I may perform a VPC with
uncertainty and assure myself that the parameter uncertainty does not lead to
unrealistic predictions (indicated by too wide confidence intervals of outer
percentiles). [An alternative is to perform a VPC with every population
parameter vector used in the clinical trial simulation and look for
outrageously poor description of the original data, but that is a bit too much
for most, including me. Better to rely on good methods for parameter
uncertainty).
Best regards,
Mats
Mats Karlsson, PhD
Professor of Pharmacometrics
Dept of Pharmaceutical Biosciences
Faculty of Pharmacy
Uppsala University
Box 591
75124 Uppsala
Phone: +46 18 4714105
Fax + 46 18 4714003
http://www.farmbio.uu.se/research/researchgroups/pharmacometrics/
www.farmbio.uu.se/research/researchgroups/pharmacometrics/
From: [email protected] [mailto:[email protected]] On
Behalf Of Devin Pastoor
Sent: Monday, June 08, 2015 6:30 PM
To: Matts Kågedal; [email protected]
Subject: Re: [NMusers] Fwd: Should we generate VPCs with or without uncertainty?
Matts,
The way I see the CI's around the point estimates provided in the VPC can help
provide a useful indication of model robustness, especially in regards to the
impact of random effects components, in that portion of your model. Especially
for heterogeneous data (or even all rich data for that matter) there are a
number of binning strategies that can be used, which can impact the
aforementioned intervals.
At the end of the day, we must use our judgement for how the model is being
used to support decisions, and whether information regarding uncertainty can
provide additional support towards the overall evaluation of the key questions
you are trying to address. Eg, if you are dealing with a narrow therapeutic
index drug the value of having a 'feel' for the robustness of the ability of
your model to describe the tails may be valuable information, even as a
qualitative indication of model robustness. On the other hand, if you are
trying to make a decision regarding dose adjustment between different
populations and are looking to normalize large differences, as well as are
constrained to certain oral dosage options, uncertainty in the point estimates
will likely provide very little support to an argument one way or the other.
Finally, in my opinion, inclusion/exclusion also relies on what the plot is
trying to communicate. Are you trying to personally evaluate model adequacy,
sure, but if using to convey to non- modelers/quantitative people that your
model describes the data - include a visualization of uncertainty at your own
peril :-)
So, for better or worse, I would say - it depends, though I would be highly
concerned if major decisions rode on inclusion/exclusion of parameter
uncertainty, in most cases.
Devin Pastoor
Center for Translational Medicine
University of Maryland, Baltimore
On Mon, Jun 8, 2015 at 11:57 AM Matts Kågedal <[email protected]> wrote:
Hi all,
Creation of VPCs is a way to assess if simulated data generated by the model is
compatible with observed data.
VPCs are usually based on parameter point estimates of the model. Sometimes
parameter uncertainty is also accounted for in the generation of VPCs (PPCs)
where each simulated replicate of the data set is based on a new set of
parameter values representing the uncertainty of the estimates (e.g. based on a
bootstrap).
I wonder if inclusion of uncertainty in this way is really appropriate or if it
just makes the confidence intervals wider and hence easier to qualify the
model. Is it possible based on such an approach, that a model might look good,
when in fact no likely combination of parameter values (based on parameter
uncertainty) would generate data that are compatible with the observations?
To illustrate my question:
I could generate 100 sets of parameters reflecting parameter uncertainty (e.g.
from a bootstrap). Based on each set of parameters I could then generate a
separate VPC (e.g. showing median, 5 and 95% percentile) to see if any of the
parameter sets are compatible with data. I would then have 100 VPCs, each based
on a separate set of parameter values reflecting the parameter correlations and
uncertainty.
If the VPC based on point estimates looks bad, I would (generally) expect that
the other VPCs would be worse (they all have lower likelihood), so that we have
101 VPCs that does not look good. Some might over predict and some
underpredict, some might describe parts of the relation better than the VPC
based on the point estimates.
By putting the VPCs together from all parameter vectors, the CI becomes wider,
and perhaps now includes the observed data. So based on a set of 100 parameter
vectors which individually are not compatible with the observed data I have now
generated a VPC (PPC) where the confidence interval actually includes the
observed metric (e.g median). It seems to me that based on such an approach it
is possible that a model might look good, when in fact no likely individual set
of parameter values would generate data that are compatible with the
observations.
Simulation based on parameter uncertainty is useful when we want to make
inference, but I am unsure of its use for model qualification. In any case it
is confusing that we some times simulate based on point estimates and sometimes
based on parameter uncertainty without any particular rationale as far as I
understand.
Would be interested if someone could shed some light on the inclusion of
uncertainty in simulations for model qualification (VPCs).
Best regards,
Matts Kagedal
Pharmacometrician, Genentech