Hi everybody,
I had a question about the Predictive performance of the POPPK Model.
When I am estimating the precision and bias with the POPPK model I have, am
I supposed to use the individual predictions or the population predictions
???
I am using "Some suggestions for Measuring Predictive Performance" by
Sheiner and Beal : J Pk and Bio Vol (:(4) 1981 :503-512 as reference.
I guess I should be using the population predictions to calculate the
precision and bias as I want to use the model to predict the plasma
concentrations. Or does this choice depend on anything else ??
If I am using the Population predictions then, where else would I be using
the individual Predictions apart from plotting them against the DV to
evaluate the Goodness of Fit?
Thanks in advance
--
--Navin
Predictive Performance
Dear Navin,
If you want to assess the predictive performance of a model,
I would highly recommend using visual predictive checks (VPC,
also called simple predictive checks, or degenerate predictive
checks).
Depending on your study design, VPCs might be easy to
implement or more work intensive. I find VPCs much easier
to interpret than DV vs. PRED or DV vs. IPRED plots. VPCs
are also easily communicated to non-modelers.
If the DV vs. IPRED plot looks biased, a model is often not
flexible enough to describe the data. However, there are
situations when the DV vs. IPRED plot looks almost perfect,
but the DV vs. PRED plot is quite biased and the VPC indicates
a clear bias in model predictions. This might be due to problems
with the parameter variability model.
So in essence, I would look at all three of those plots to assess
the appropriateness of a model. If a model is intended for
simulations, the VPC is a powerful tool to visually assess the
predictive performance and to tell if a potential bias in simulations
might be important for the study objectives or not.
Please find some references below.
Best regards
Juergen
Yano Y, Beal SL, Sheiner LB. Evaluating pharmacokinetic/pharmacodynamic models
using the posterior predictive check.
J Pharmacokinet Pharmacodyn. 2001 Apr;28(2):171-92.
Mentre F, Escolano S. Prediction discrepancies for the evaluation of nonlinear
mixed-effects models.
J Pharmacokinet Pharmacodyn. 2006 Jun;33(3):345-67.
-----------------------------------------------
Juergen Bulitta, PhD, Post-doctoral Fellow
Pharmacometrics, University at Buffalo, NY, USA
Phone: +1 716 645 2855 ext. 281, [EMAIL PROTECTED]
-----------------------------------------------
-----Ursprüngliche Nachricht-----
Von: "navin goyal" <[EMAIL PROTECTED]>
Gesendet: 02.07.07 20:10:56
An: nmusers <[email protected]>
Betreff: [NMusers] Predictive Performance
Hi everybody,
I had a question about the Predictive performance of the POPPK Model.
When I am estimating the precision and bias with the POPPK model I have, am I
supposed to use the
individual predictions or the population predictions ???
I am using "Some suggestions for Measuring Predictive Performance" by Sheiner
and Beal : J Pk and Bio Vol (:(4) 1981 :503-512 as reference.
I guess I should be using the population predictions to calculate the precision
and bias as I want to use the model to predict the plasma concentrations. Or
does this choice depend on anything else ??
If I am using the Population predictions then, where else would I be using the
individual Predictions apart from plotting them against the DV to evaluate the
Goodness of Fit?
Thanks in advance
--
--Navin
Dear Navin,
As Juergen said, VPC is a good graphically tool to indicate a clear bias in
model prediction. One of the limitation of this approach may be when you have a
complex design with diferent doses, different administration schedules and
different covariates (implemented in your population model). In this case you
need to perform several VPC plots splitted by doses, covariates...
Another approach is to compute a metric called Normalized Predictive
Distribution Error (NPDE). NPDE* have been developed to take into account the
full predictive distribution of each individual concentration, and handle
multiple observations within subjects. Under the null hypothesis that a model
under scrutiny describes a validation dataset, the distribution of NPDE should
be the standard normal distribution.
A R package is now available and can be downloaded from WWW.npde.biostat.fr .
Best regards.
Karl
Brendel K., Comets E., Laffont C., Laveille C., Mentré F. Metrics for external
model evaluation with an application to the population pharmacokinetics of
gliclazide. Pharm Res 2006, 23:2036-2049.
-----Message d'origine-----
Quoted reply history
De : [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] la part de Jurgen Bulitta
Envoyé : mardi 3 juillet 2007 02:07
À : navin goyal; [email protected]
Objet : Re: [NMusers] Predictive Performance
Dear Navin,
If you want to assess the predictive performance of a model,
I would highly recommend using visual predictive checks (VPC,
also called simple predictive checks, or degenerate predictive
checks).
Depending on your study design, VPCs might be easy to
implement or more work intensive. I find VPCs much easier
to interpret than DV vs. PRED or DV vs. IPRED plots. VPCs
are also easily communicated to non-modelers.
If the DV vs. IPRED plot looks biased, a model is often not
flexible enough to describe the data. However, there are
situations when the DV vs. IPRED plot looks almost perfect,
but the DV vs. PRED plot is quite biased and the VPC indicates
a clear bias in model predictions. This might be due to problems
with the parameter variability model.
So in essence, I would look at all three of those plots to assess
the appropriateness of a model. If a model is intended for
simulations, the VPC is a powerful tool to visually assess the
predictive performance and to tell if a potential bias in simulations
might be important for the study objectives or not.
Please find some references below.
Best regards
Juergen
Yano Y, Beal SL, Sheiner LB. Evaluating pharmacokinetic/pharmacodynamic models
using the posterior predictive check.
J Pharmacokinet Pharmacodyn. 2001 Apr;28(2):171-92.
Mentre F, Escolano S. Prediction discrepancies for the evaluation of nonlinear
mixed-effects models.
J Pharmacokinet Pharmacodyn. 2006 Jun;33(3):345-67.
-----------------------------------------------
Juergen Bulitta, PhD, Post-doctoral Fellow
Pharmacometrics, University at Buffalo, NY, USA
Phone: +1 716 645 2855 ext. 281, [EMAIL PROTECTED]
-----------------------------------------------
-----Ursprüngliche Nachricht-----
Von: "navin goyal" <[EMAIL PROTECTED]>
Gesendet: 02.07.07 20:10:56
An: nmusers <[email protected]>
Betreff: [NMusers] Predictive Performance
Hi everybody,
I had a question about the Predictive performance of the POPPK Model.
When I am estimating the precision and bias with the POPPK model I have, am I
supposed to use the
individual predictions or the population predictions ???
I am using "Some suggestions for Measuring Predictive Performance" by Sheiner
and Beal : J Pk and Bio Vol (:(4) 1981 :503-512 as reference.
I guess I should be using the population predictions to calculate the precision
and bias as I want to use the model to predict the plasma concentrations. Or
does this choice depend on anything else ??
If I am using the Population predictions then, where else would I be using the
individual Predictions apart from plotting them against the DV to evaluate the
Goodness of Fit?
Thanks in advance
--
--Navin
Now I know that the email address I used before was incorrect... :-)
Quoted reply history
> -----Original Message-----
> From: Kimko, Hui [PRDUS]
> Sent: Tuesday, July 03, 2007 10:48 AM
> To: '[EMAIL PROTECTED]'
> Cc: '[EMAIL PROTECTED]'
> Subject: FW: [NMusers] Predictive Performance
> Dear Navin et al.,
> For external validation, you should use population predictions to calculate
> precision and bias, because you are testing the performance of your model,
> built on a model-building dataset, by predicting a new test data using its
> covariate information only: the IPRED is also based on the observed test data
> set values. As you implied, the diagnostic plot of IPRED vs. DV would be good
> to have. IPRED is used to calculate AIPE.
> Since we are discussing Predictive Performance, I'd like to take this
> opportunity to share what the late Vladmir Piotrovsky was working on, before
> he passed away. He was writing an S+ library, which is called "S speaks
> NONMEM". Below is the S+ code to do external validation, with breaking the
> correlation in residuals within each subject when measures of predictive
> performance are computed. This computation is a modification of Beal and
> Sheiner's MPE and RMSE calculation method. More detailed description can be
> found on page 192 of:
> Vermeulen A, Piotrovsky V, Ludwig E, Population PK of Risperidone and
> 9-hydroxyrisperidone in patients with acute episodes associated with bipolar
> I disorder. JPKPD 34(2):183-206, 2007.
> In remembrance of our beloved Vladmir,
> Hui
> Advanced Modeling & Simulation, J&J Pharmaceutical Research & Development,
> LLC
> ===============================================================
> ################# Piotrovsky's resampling residual method for external
> validation
> ################# Modification of the method by Sheiner and Beal:
> ################# Some suggestions for measuring predictive performance.
> ################# J. Pharmacokinet. Biopharm. 1981; 9(4):503-512
> ################# Author: V. Piotrovsky
> tab <- read.table("C:\\..\run100.txt",header=T,skip=1)
> attach(tab[tab$WRES!=0 ,]) # your table
> z _ split(RES,ID)
> w _ split(RES^2,ID)
> detach()
> mpe _ numeric(0)
> rmse _ numeric(0)
> n _ 1000 # sample size
> for(i in 1:n) mpe[i] _ median(sapply(z,sample,size=1,repl=T))
> for(i in 1:n) rmse[i] _ sqrt(mean(sapply(w,sample,size=1,repl=T)))
> # Plotting
> par(mfrow=c(1,2),mar=c(5,5,2,0))
> hist(mpe,col=0,prob=T,xlab="Median population prediction
> error",ylab="Frequency",cex=1)
> lines(density(mpe,wid=.1,n=100))
> abline(v=quantile(mpe,.025),lty=4)
> abline(v=quantile(mpe,.975),lty=4)
> abline(v=median(mpe),lwd=4)
> box()
> hist(rmse,col=0,prob=T,xlab="Root mean squared prediction
> error",ylab="",cex=1)
> lines(density(rmse,wid=.1,n=100))
> abline(v=quantile(rmse,.025),lty=4)
> abline(v=quantile(rmse,.975),lty=4)
> abline(v=median(rmse),lwd=4)
> box()
> median(mpe)
> mean(mpe)
> quantile(mpe,.025)
> quantile(mpe,.975)
> median(rmse)
> mean(rmse)
> quantile(rmse,.025)
> quantile(rmse,.975)
> -----Ursprüngliche Nachricht-----
> Von: "navin goyal" <[EMAIL PROTECTED]>
> Gesendet: 02.07.07 20:10:56
> An: nmusers <[email protected]>
> Betreff: [NMusers] Predictive Performance
> Hi everybody,
> I had a question about the Predictive performance of the POPPK Model.
> When I am estimating the precision and bias with the POPPK model I have, am I
> supposed to use the
> individual predictions or the population predictions ???
> I am using "Some suggestions for Measuring Predictive Performance" by Sheiner
> and Beal : J Pk and Bio Vol (:(4) 1981 :503-512 as reference.
> I guess I should be using the population predictions to calculate the
> precision and bias as I want to use the model to predict the plasma
> concentrations. Or does this choice depend on anything else ??
> If I am using the Population predictions then, where else would I be using
> the individual Predictions apart from plotting them against the DV to
> evaluate the Goodness of Fit?>
> Thanks in advance
> --
> --Navin
>