FW: Predictive Performance
Now I know that the email address I used before was incorrect... :-)
Quoted reply history
> -----Original Message-----
> From: Kimko, Hui [PRDUS]
> Sent: Tuesday, July 03, 2007 10:48 AM
> To: '[EMAIL PROTECTED]'
> Cc: '[EMAIL PROTECTED]'
> Subject: FW: [NMusers] Predictive Performance
> Dear Navin et al.,
> For external validation, you should use population predictions to calculate
> precision and bias, because you are testing the performance of your model,
> built on a model-building dataset, by predicting a new test data using its
> covariate information only: the IPRED is also based on the observed test data
> set values. As you implied, the diagnostic plot of IPRED vs. DV would be good
> to have. IPRED is used to calculate AIPE.
> Since we are discussing Predictive Performance, I'd like to take this
> opportunity to share what the late Vladmir Piotrovsky was working on, before
> he passed away. He was writing an S+ library, which is called "S speaks
> NONMEM". Below is the S+ code to do external validation, with breaking the
> correlation in residuals within each subject when measures of predictive
> performance are computed. This computation is a modification of Beal and
> Sheiner's MPE and RMSE calculation method. More detailed description can be
> found on page 192 of:
> Vermeulen A, Piotrovsky V, Ludwig E, Population PK of Risperidone and
> 9-hydroxyrisperidone in patients with acute episodes associated with bipolar
> I disorder. JPKPD 34(2):183-206, 2007.
> In remembrance of our beloved Vladmir,
> Hui
> Advanced Modeling & Simulation, J&J Pharmaceutical Research & Development,
> LLC
> ===============================================================
> ################# Piotrovsky's resampling residual method for external
> validation
> ################# Modification of the method by Sheiner and Beal:
> ################# Some suggestions for measuring predictive performance.
> ################# J. Pharmacokinet. Biopharm. 1981; 9(4):503-512
> ################# Author: V. Piotrovsky
> tab <- read.table("C:\\..\run100.txt",header=T,skip=1)
> attach(tab[tab$WRES!=0 ,]) # your table
> z _ split(RES,ID)
> w _ split(RES^2,ID)
> detach()
> mpe _ numeric(0)
> rmse _ numeric(0)
> n _ 1000 # sample size
> for(i in 1:n) mpe[i] _ median(sapply(z,sample,size=1,repl=T))
> for(i in 1:n) rmse[i] _ sqrt(mean(sapply(w,sample,size=1,repl=T)))
> # Plotting
> par(mfrow=c(1,2),mar=c(5,5,2,0))
> hist(mpe,col=0,prob=T,xlab="Median population prediction
> error",ylab="Frequency",cex=1)
> lines(density(mpe,wid=.1,n=100))
> abline(v=quantile(mpe,.025),lty=4)
> abline(v=quantile(mpe,.975),lty=4)
> abline(v=median(mpe),lwd=4)
> box()
> hist(rmse,col=0,prob=T,xlab="Root mean squared prediction
> error",ylab="",cex=1)
> lines(density(rmse,wid=.1,n=100))
> abline(v=quantile(rmse,.025),lty=4)
> abline(v=quantile(rmse,.975),lty=4)
> abline(v=median(rmse),lwd=4)
> box()
> median(mpe)
> mean(mpe)
> quantile(mpe,.025)
> quantile(mpe,.975)
> median(rmse)
> mean(rmse)
> quantile(rmse,.025)
> quantile(rmse,.975)
> -----Ursprüngliche Nachricht-----
> Von: "navin goyal" <[EMAIL PROTECTED]>
> Gesendet: 02.07.07 20:10:56
> An: nmusers <[email protected]>
> Betreff: [NMusers] Predictive Performance
> Hi everybody,
> I had a question about the Predictive performance of the POPPK Model.
> When I am estimating the precision and bias with the POPPK model I have, am I
> supposed to use the
> individual predictions or the population predictions ???
> I am using "Some suggestions for Measuring Predictive Performance" by Sheiner
> and Beal : J Pk and Bio Vol (:(4) 1981 :503-512 as reference.
> I guess I should be using the population predictions to calculate the
> precision and bias as I want to use the model to predict the plasma
> concentrations. Or does this choice depend on anything else ??
> If I am using the Population predictions then, where else would I be using
> the individual Predictions apart from plotting them against the DV to
> evaluate the Goodness of Fit?>
> Thanks in advance
> --
> --Navin
>