RE: VPCs confidence intervals?
This is a great example of the kind of terminology debates that the ASA / ISOP
Statistics and Pharmacometrics special interest group (SxP) is trying to tackle.
As Mats and Bill point out, the common usage within our community is to say
that the percentiles (5th, 95th) are “prediction intervals” and the interval
estimates / uncertainty around these percentiles are “confidence intervals”.
But as Ken points out, these terms do not strictly correspond to the
statistical definition of each if you take into account what the VPC procedure
is actually doing.
The VPC is a model diagnostic procedure for the observed data and provides a
visual check of whether the model is capturing central tendencies and
dispersion in our data. (BTW, I *know* there are debates about the usefulness
or otherwise of VPC plots. I’m not going to address that here and I suggest we
don’t disappear down *that* rabbit hole.) We are NOT trying to make
probabilistic statements about the likelihood of observed percentiles being
within the intervals around these. So if the question arises from some
reviewer based on our use of statistically woolly terms like “prediction
interval” or “confidence interval” we should be ready to put up our hands and
admit that the terms we are using do not imply those statistical properties.
We could advocate changing the terminology used, but that may not have traction
in the community after this length of time. But we *should* be cognizant about
what these things are, what they’re for, what the formal, statistical
terminology implies and what our use (or maybe misuse) is or isn’t implying.
The ASA / ISOP SxP group has had a session accepted at this year’s ACOP meeting
where we hope to surface a few of these thorny issues and debate between our
use of terminology in pharmacometrics, the statistical interpretation of that
terminology and whether it *really* matters. If you’re interested, please come
along and be prepared to engage in the discussion!
Best regards,
Mike
(co-chair of ASA / ISOP SxP SIG)
Quoted reply history
From: [email protected]<mailto:[email protected]>
<[email protected]<mailto:[email protected]>> On Behalf
Of Ken Kowalski
Sent: 14 March 2019 21:02
To: 'Bill Denney'
<[email protected]<mailto:[email protected]>>;
[email protected]<mailto:[email protected]>; Soto, Elena
<[email protected]<mailto:[email protected]>>
Subject: [EXTERNAL] RE: [NMusers] VPCs confidence intervals?
Hi All,
I know there is a lot of confusion about the distinction between a confidence
interval and a prediction interval. Here is a layperson’s way of making the
distinction.
A confidence interval makes inference on a population parameter which is fixed
(never changes) regardless of any sample data that is collected to estimate the
parameter (if you repeatedly sampled an infinite number of observations to
obtain the population value by definition you would get the same population
value for each sample with an infinite sample size) . Thus, the confidence
interval only reflects the uncertainty in the estimate of that parameter.
In contrast, a prediction interval makes inference on a statistic for a future
sample set of data. That statistic will vary from sample to sample and hence
must also take into account the sampling variation as well as the parameter
uncertainty. A prediction interval can be thought of as a confidence interval
of the prediction of some statistic from a future sample. That is, both a
confidence interval and a prediction interval have a confidence level
associated with them. In the case of the confidence interval, the confidence
level is the coverage probability that the interval will contain the true
value of the population parameter if one were to repeat the experiment an
infinite number of times. In the case of the prediction interval, the
confidence level is the coverage probability that the interval will contain the
future sample mean (of a finite sample size) if one were to repeat the
experiment an infinite number of times.
There is another type of statistical interval in addition to confidence and
prediction intervals and that is a tolerance interval. A tolerance interval
can be thought of as a confidence interval that a specified proportion of the
individual responses will be contained within the interval. For example, we
can calculate a 95% tolerance interval to contain 90% of the observed data
(i.e., we are 95% confident that the interval will contain 90% of the
individual observations). Tolerance intervals are more common in a
manufacturing setting where it is important to produce an item to some
specification within some tolerance limits. Nevertheless, there is a certain
VPC plot that we often generate that is somewhat akin to a tolerance interval.
When we summarize our simulated data for VPCs and summarize the 5th and 95th
percentiles of the individual responses this is more akin to a tolerance
interval to contain 90% of the observed individual data. In contrast, when we
summarize the sample mean or median from say 1000 simulated trials and
calculate the 5th and 95th percentiles across the 1000 trials that is more akin
to a prediction interval for that statistic (e.g., sample mean or sample
median). Note however, the intervals obtained as percentiles of a sample
statistic across trials (i.e., prediction interval) or sample observations
across individual subjects (i.e., tolerance interval) don’t have valid coverage
probabilities for repeated experiments unless they take into account parameter
uncertainty.
Kind regards,
Ken