RE: VPCs confidence intervals?

Hi All, I know what Bill is trying to say but it is not quite accurate the way he states it. A prediction interval makes inference on a statistic based on a future sample such as a sample mean of a future set of data. In contrast, a confidence interval makes inference on a parameter such as the population mean which is a fixed number. A prediction interval takes into account both the uncertainty in the existing data used to estimate the population parameter as well as the sampling variation to make inference on a sample statistic (e.g., sample mean for a future trial). A confidence interval only takes into account the uncertainty in the existing data used to estimate the parameter. Based on the Law of Large Numbers, the population mean can be thought of as taking the sample mean of an infinite sample size (i.e., sampling the entire population). For this reason, a prediction interval with an infinite sample size will collapse to a confidence interval. An interval based on VPCs is more akin to a prediction interval since it takes into account the sampling variation based on a finite sample size, however, one cannot assign a valid coverage probability (confidence level) to this interval unless it also takes into account the parameter uncertainty. With VPCs applied to existing data (i.e, an internal VPC) it is customary to not take into account this parameter uncertainty so many refer to such prediction intervals as degenerate as they place 100% certainty on the model parameter estimates used to obtain the VPC predictions. One could potentially call these intervals ‘degenerate prediction intervals’ but I tend to just call them ‘VPC intervals’ (e.g., a 90% VPC interval) so as to avoid misperception that these prediction intervals have a statistically valid coverage probability. However, when VPCs are applied to an independent dataset not used in the development of the model, it is often advised to take into account the parameter uncertainty when performing the VPCs to essentially reflect the trial-to-trial uncertainty of the independent data not used in the estimation of model (i.e., refitting the same model to a new set of trial data will not give the same set of estimates and hence reflects trial-to-trial variation). In this setting, where the VPCs take into account both the parameter uncertainty and sampling variation to predict on an independent (e.g., future) dataset, then one is on more solid ground to refer to these VPC intervals as prediction intervals with valid coverage probabilities. Kind regards, Ken Kenneth G. Kowalski Kowalski PMetrics Consulting, LLC Email: <mailto:[email protected]> [email protected] Cell: 248-207-5082