VPC appropriateness in complex PK

Dear NMusers: The Visual predictive check (VPC, http://www.page-meeting.org/page/page2005/PAGE2005P105.pdf , and JPKPD, Volume 35, Number 2 / April, 2008) has been touted as a useful tool for assessing the perfomance of population pharmacokinetic models. However I recently came across this abstract from the 2009 PAGE meeting: http://www.page-meeting.org/pdf_assets/4050-Standardized%20Visual%20Predictive%20Check%20in%20Model%20Evaluation%20-%20PAGE2009%20submit.pdf. This abstract states that situations when VPC is not feasible but a "Standardized Visual Predictive Check (SVPC) can be used are as follows: – Patients received individualized dose or there are a small number of patients per dose group and PK or PD is nonlinear, thus observations can not be normalized for dose – There are multiple categorical covariate effects on PK or PD parameters – Covariate is a continuous variable which made stratification impossible – Study design and execution varies among individuals, such as adaptive design, difference in dosing schedule, dose changes and dosing time varies during study, protocol violations – Different concomitant medicines and food intake among individuals when there are drug-drug interactions and food effect on PK However, the original VPC articles seem to suggest that these are the exact situations when the VPC alone is an ideal tool for model validation. Is there any justification for one approach over the other? Has anyone ever seen an SVPC utilized elsewhere, I have found nothing. Are these truly weaknesses of a VPC? Cheers! Dider

Dear Dider, In my opinion the PAGE 2009 abstract by Diane Wang does highlight weaknesses with the standard VPC under certain circumstances. However, I don't think that the SVPC represent the answer to those weaknesses. Prediction corrected VPCs (PC-VPCs) are a better way of addressing these issues and was first mentioned in the Karlsson and Holford tutorial on VPCs at PAGE 2008 ( http://www.page-meeting.org/pdf_assets/8694-Karlsson_Holford_VPC_Tutorial_h ires.pdf). A poster on the PC-VPCs principle and the advantage with these is submitted to the ACoP conference (October 2009). A two page abstract regarding that poster is available already now via the ACoP webpage ( http://www.go-acop.org/acop2009/posters - Title: "Prediction Corrected Visual Predictive Checks" Authors: Martin Bergstrand, Andrew C. Hooker, Johan E. Wallin, Mats O. Karlsson). Please have a look at this abstract and contact me if you have any further questions. Kind regards, Martin Bergstrand, MSc, PhD student ----------------------------------------------- Pharmacometrics Research Group, Department of Pharmaceutical Biosciences, Uppsala University ----------------------------------------------- P.O. Box 591 SE-751 24 Uppsala Sweden ----------------------------------------------- <mailto:[email protected]> [email protected] ----------------------------------------------- Work: +46 18 471 4639 Mobile: +46 709 994 396 Fax: +46 18 471 4003

Hi Dider, VPC is very good when your data set is homogeneous: same or similar dosing, same or similar sampling, same or similar influential covariates that results in similar PK or PD predictions. In cases of diverse data sets, traditional VPC is more difficult to implement, and it may not be useful. To see the problem, consider VPC (without stratification) for the data with two dose groups, 1 and 100 units (with the rest being similar). Obviously, all data that exceed 95% CI would come from the high dose, and all data below 5th percentile would come from the low dose, and overall, VPC plots and stats will not be useful. With two doses, it is easy to fix: just stratify by dose. If you have more diverse groups, you have to either do VPC by group, or find the way to plot all values in one scale. In cases of dose differences and linear kinetics, one can do VPC with all values normalized by dose. In nonlinear cases, it is more difficult. SVPC offers the way out of this problem. In this procedure, each observation is compared with the distribution of observations at the same time point, with the same dosing, and with the same covariate set as in the original data. Position of the observation in the distribution of simulated values is characterized by the percent of simulated values that is above (or below) the observed value. If the model is correct, then percentiles should be uniformly distributed in the range of 0 to 100. This should hold for any PRED value, and dose, any time post-dose etc. It is important not to combine all observed points together (to study overall distribution of the SVPC percentiles): in this case the test in not sensitive. SVPC is useful when these percentile values are plotted versus time, time post dose, or PRED (but not IPRED or DV !!) values. Then, they can be use to see the problems with the model, similar to how WRES vs TIME and WRES vs PRED plots are used. The disadvantage is that you loose visual part: your percentile versus time profiles should look like a square filled with the points rather than like concentration-time profiles. Even in this procedure, it make sense to stratify your plots by dose, influential covariates, etc. to see whether the plots are uniformly good. Dose, covariate, time or PRED dependencies of the SVPC plots may indicate some deficiency of the model. Note that none of these procedures can be used to evaluate the concentration or effect controlled trials, or trials with non-random drop out. In order to use VPC-based procedures for these cases, you need to simulate accordingly: with dosing that depend on simulated values (for concentration or effect controlled trials) or with the drop-out models. Thanks Leonid -------------------------------------- Leonid Gibiansky, Ph.D. President, QuantPharm LLC web: www.quantpharm.com e-mail: LGibiansky at quantpharm.com tel: (301) 767 5566 Dider Heine wrote: > Dear NMusers: > > The Visual predictive check (VPC, http://www.page-meeting.org/page/page2005/PAGE2005P105.pdf , and JPKPD, Volume 35, Number 2 / April, 2008) has been touted as a useful tool for assessing the perfomance of population pharmacokinetic models. However I recently came across this abstract from the 2009 PAGE meeting: http://www.page-meeting.org/pdf_assets/4050-Standardized%20Visual%20Predictive%20Check%20in%20Model%20Evaluation%20-%20PAGE2009%20submit.pdf . This abstract states that situations when VPC is not feasible but a "Standardized Visual Predictive Check (SVPC) can be used are as follows: – Patients received individualized dose or there are a small number of patients per dose group and PK or PD is nonlinear, thus observations can not be normalized for dose > > – There are multiple categorical covariate effects on PK or PD parameters > – Covariate is a continuous variable which made stratification impossible > > – Study design and execution varies among individuals, such as adaptive design, difference in dosing schedule, dose changes and dosing time varies during study, protocol violations – Different concomitant medicines and food intake among individuals when there are drug-drug interactions and food effect on PK > > However, the original VPC articles seem to suggest that these are the exact situations when the VPC alone is an ideal tool for model validation. Is there any justification for one approach over the other? Has anyone ever seen an SVPC utilized elsewhere, I have found nothing. Are these truly weaknesses of a VPC? Cheers! > > Dider

Leonid, Thank you for the explanation. I was writing the response but found your email stated it even better than what I could do myself. :) Basically, VPC is less sensitive, when your data set is not homogeneous, for evaluating random effects, because the 95% percentile interval of predicted concentrations based on the full model reflects not only the random effects (inter- and intra-subject variability) but also fixed effects (difference in study design and covariate effects). SVPC solved this problem as you described. Regarding stratifying the plots by dose and influential covariates when using SVPC, our solution is to group subjects by the covariate of interest (e.g. dose, and influential covariates) using different colors and see if the colors are uniformly distributed in the SVPC plot. This can also be used to identify potential covariates. I have a couple examples in the presentation slides. I am not sure I agree with you that SVPC can not be used for concentration or efficacy controlled trials. In concentration or efficacy controlled trials, patient's dose is adjusted based on concentration or efficacy observed. As long as we have the dosing record, we should be able to get the percentile for each observation of each patient based on predicted PK/PD endpoints using this patient's dosing record, and then pool all observation percentiles together regardless of each patient's dose and dosing schedule. Thanks, Diane Diane D. Wang, Ph.D. Director Clinical Pharmcology Oncology Business Unit Pfizer La Jolla 10555 Science Center Dr. (CB10/2408) San Diego, CA 92121 Office Phone: (858) 622-8021 Cell Phone: (858) 761-3667 email: [email protected]

Dider, There are two types of predictive check that are now recognized as the current state of the art for mixed effect model evaluation. Statistical Predictive Checks (SPC) The SVPC described by Wang and Zhang (2009) is a re-invention (apparently independently) of a method published by Mentré and Escolano in 2006. The method is primarily a numerical method for checking if the distribution of discrepancies between model predictions and observations is uniform (PDE). The primary purpose of the method is to create a statistical test procedure for the distribution of discrepancies. Mentré and her colleagues have refined the original method to remove the correlation between individual observations (Comets et al. 2008) and propose a set of statistical tests based on a normal distribution of discrepancies (NPDE). These standardized/normalized discrepancies (PDE, NPDE) can be plotted as graphs but lose information which might identify the magnitude of model failure because the process of standardization/normalization removes this important clue (as Leonid has also pointed out in his recent email to nmusers). Visual Predictive Checks (VPC) The visual predictive check (Holford 2005) is primarily a graphical method that relies on a subjective evaluation of the pattern of discrepancies between the model predictions and the observations. It preserves the magnitude of the model prediction and can be directly compared to the observation. Examples of how this visual evaluation can lead to recognition of model failure, when the standard 'diagnostic plots' do not, can be found in Holford (2005) and Karlsson & Holford (2008; see Slides 22-26). The original scatterplot VPC may be sufficient when there are small numbers of observations but the distribution of observations cannot be appreciated when scatterplots have many overlapping observation symbols. The percentile VPC (see Karlsson & Holford 2008) solves this problem and allows direct comparison of selected percentiles of the distribution of both predictions and observations. The scatterplot VPC and percentile VPC offer complementary views of the model predictions. The scatterplot VPC is helpful for appreciating the realized design of the study but the percentile VPC is needed to properly compare observations with predictions. A scatterplot VPC by itself is inadequate for model evaluations in almost all cases. Uncritical evaluation of the scatterplot VPC can lead to acceptance of models which may not describe the data well but the failure is often hidden by this naive method used for construction of the plot. Learning and Confirming The statistical predictive checks (SPC) offer a simple way of deciding if a model of arbitrary complexity describes the observations. However, because all models are wrong and the tests may have high power to detect small and possibly unimportant differences this kind of test may reject a model that is in fact useful for its intended purpose. Evaluation of a model in relation to its purpose will usually require an understanding of the magnitude and timing of the discrepancy. Because magnitude is lost with SPC methods they can only be partially helpful as evaluation tools. VPCs can be challenging to construct but the very process of having to think about how to simulate the observations and how to display the results to account for covariates (e.g. dose, disease state) adds more value to the VPC (see Karlsson & Holford 2008 Slide 27/28 for an example where inclusion of a model for dropout related to disease status was helpful in explaining and interpreting the observations). Leonid also echoes this conclusion in his email. The PRED corrected method of VPC construction (Karlsson & Holford 2008, Bergstrand et al. 2009) offers the potential for automated 'correction' for covariate influences but it still requires thought from the user both for construction and evaluation. In the terminology of Sheiner (1997) the VPC can be viewed as an evaluation procedure for learning while the SPC is an evaluation procedure for confirming. They are complementary and have different uses. Unstratified VPCs Leonid has proposed an example which he asserts shows how the VPC is not useful if covariate stratification is not performed. > To see the problem, consider VPC (without stratification) for the data with two dose groups, 1 and 100 units (with the rest being similar). Obviously, all data that exceed 95% CI would come from the high dose, and all data below 5th percentile would come from the low dose, and overall, VPC plots and stats will not be useful. However, I disagree with his conclusion about the usefulness of unstratified VPC plots. If the model is suitable then the percentile VPC should match the observed percentiles. But if the model is unsuitable e.g. a one compartment model has been used but a two compartment model would be better, then the percentile VPC should be able to demonstrate this. Of course if one was interested in finding out if the assumption of dose linearity was satisfied then stratification by dose would be needed in the VPC. Limitations The construction and intepretation of the VPC requires thought. If the data structure and model assumptions are complex then more effort is required. In some situations it may not be possible to simulate the original design (e.g. adaptive designs where the adaptation rules cannot be implemented by a simulation algorithm). In this case then it seems that all simulation based methods (SPC or VPC) would be unhelpful. Best wishes Nick Bergstrand M, Hooker AC, Wallin JE, Karlsson MO. Prediction corrected visual predictive checks http://www.go-acop.org/acop2009/posters ACOP. 2009. Comets E, Brendel K, Mentré F. Computing normalised prediction distribution errors to evaluate nonlinear mixed-effect models: The npde add-on package for R. Comput Methods Programs Biomed. 2008;90(2):154-66. Holford NHG. The visual predictive check – superiority to standard diagnostic (Rorschach) plots [www.page-meeting.org/?abstract=738]. PAGE. 2005;14. Karlsson MO, Holford NHG. A Tutorial on Visual Predictive Checks. PAGE 17 (2008) Abstr 1434 [wwwpage-meetingorg/?abstract=1434]. 2008. Mentré F, Escolano S. Prediction discrepancies for the evaluation of nonlinear mixed-effects models. J Pharmacokinet Pharmacodyn. 2006;33(3):345-67. Sheiner LB. Learning versus confirming in clinical drug development. Clinical Pharmacology & Therapeutics. 1997;61(3):275-91. Wang DD, Zhang S. Standardized visual predictive check – How and when to used it in model evaluation [www.page-meeting.org/?abstract=1501]. PAGE. 2009;18. Leonid Gibiansky wrote: > Hi Dider, > > VPC is very good when your data set is homogeneous: same or similar dosing, same or similar sampling, same or similar influential covariates that results in similar PK or PD predictions. In cases of diverse data sets, traditional VPC is more difficult to implement, and it may not be useful. > > To see the problem, consider VPC (without stratification) for the data with two dose groups, 1 and 100 units (with the rest being similar). Obviously, all data that exceed 95% CI would come from the high dose, and all data below 5th percentile would come from the low dose, and overall, VPC plots and stats will not be useful. With two doses, it is easy to fix: just stratify by dose. If you have more diverse groups, you have to either do VPC by group, or find the way to plot all values in one scale. In cases of dose differences and linear kinetics, one can do VPC with all values normalized by dose. In nonlinear cases, it is more difficult. > > SVPC offers the way out of this problem. In this procedure, each observation is compared with the distribution of observations at the same time point, with the same dosing, and with the same covariate set as in the original data. Position of the observation in the distribution of simulated values is characterized by the percent of simulated values that is above (or below) the observed value. If the model is correct, then percentiles should be uniformly distributed in the range of 0 to 100. This should hold for any PRED value, and dose, any time post-dose etc. > > It is important not to combine all observed points together (to study overall distribution of the SVPC percentiles): in this case the test in not sensitive. SVPC is useful when these percentile values are plotted versus time, time post dose, or PRED (but not IPRED or DV !!) values. Then, they can be use to see the problems with the model, similar to how WRES vs TIME and WRES vs PRED plots are used. The disadvantage is that you loose visual part: your percentile versus time profiles should look like a square filled with the points rather than like concentration-time profiles. Even in this procedure, it make sense to stratify your plots by dose, influential covariates, etc. to see whether the plots are uniformly good. Dose, covariate, time or PRED dependencies of the SVPC plots may indicate some deficiency of the model. > > Note that none of these procedures can be used to evaluate the concentration or effect controlled trials, or trials with non-random drop out. In order to use VPC-based procedures for these cases, you need to simulate accordingly: with dosing that depend on simulated values (for concentration or effect controlled trials) or with the drop-out models. > > Thanks > Leonid > > -------------------------------------- > Leonid Gibiansky, Ph.D. > President, QuantPharm LLC > web: www.quantpharm.com > e-mail: LGibiansky at quantpharm.com > tel: (301) 767 5566 > > Dider Heine wrote: > > > Dear NMusers: > > > > The Visual predictive check (VPC, http://www.page-meeting.org/page/page2005/PAGE2005P105.pdf , and JPKPD, Volume 35, Number 2 / April, 2008) has been touted as a useful tool for assessing the perfomance of population pharmacokinetic models. However I recently came across this abstract from the 2009 PAGE meeting: http://www.page-meeting.org/pdf_assets/4050-Standardized%20Visual%20Predictive%20Check%20in%20Model%20Evaluation%20-%20PAGE2009%20submit.pdf . This abstract states that situations when VPC is not feasible but a "Standardized Visual Predictive Check (SVPC) can be used are as follows: – Patients received individualized dose or there are a small number of patients per dose group and PK or PD is nonlinear, thus observations can not be normalized for dose – There are multiple categorical covariate effects on PK or PD parameters – Covariate is a continuous variable which made stratification impossible – Study design and execution varies among individuals, such as adaptive design, difference in dosing schedule, dose changes and dosing time varies during study, protocol violations – Different concomitant medicines and food intake among individuals when there are drug-drug interactions and food effect on PK > > > > However, the original VPC articles seem to suggest that these are the exact situations when the VPC alone is an ideal tool for model validation. Is there any justification for one approach over the other? Has anyone ever seen an SVPC utilized elsewhere, I have found nothing. Are these truly weaknesses of a VPC? Cheers! > > > > Dider -- Nick Holford, Professor Clinical Pharmacology Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand [email protected] tel:+64(9)923-6730 fax:+64(9)373-7090 mobile: +64 21 46 23 53 http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford

Thank you Martin, It sounds to me as if PC-VPCs trump any benefits of an SVPC. Look forward to your presentation at ACoP. Best! Dider

Diane, I probably worded it incorrectly. I was going to say that for concentration or effect controlled trails you cannot use straightforward VPC simulation based on the actual dosing history; you have to be more careful. Let me show the example that illustrates how VPC/SVPC behaves in the concentration or effect controlled trials. Assume that we conduct the two-dose study. The first dose (same for all subjects) is given to learn the kinetics. The second dose is adjusted (based on the previous data) in order to get the same Cmax for all subjects. For simplicity, assume that the world is nearly perfect: no or small residual variability, no or small inter-occasion variability. Then the dose adjustment can be perfect, and the second-dose Cmax for all subjects would hit the target. No or small second-dose Cmax variability would be observed. Now, let's do VPC. If you simulate based on the actual dosing history (even from the from the true model), your first-dose Cmax will be distributed similar to the observed data. However, your second-dose Cmax will vary significantly (even more than the first-dose Cmax) because you use the actual dose, rather than adjust the dose based on the individual parameters. Thus, standard VPC/SVPC/NPDE/PC-VPC/etc. will be misleading. One needs to simulate using the same dose adjustment algorithm as in the actual study. Only for these simulations predictive check plots can be used. Thanks Leonid -------------------------------------- Leonid Gibiansky, Ph.D. President, QuantPharm LLC web: www.quantpharm.com e-mail: LGibiansky at quantpharm.com tel: (301) 767 5566 Wang, Diane wrote: > Leonid, > > Thank you for the explanation. I was writing the response but found > your email stated it even better than what I could do myself. :) > Basically, VPC is less sensitive, when your data set is not homogeneous, > for evaluating random effects, because the 95% percentile interval of > predicted concentrations based on the full model reflects not only the > random effects (inter- and intra-subject variability) but also fixed > effects (difference in study design and covariate effects). SVPC solved > > this problem as you described. > > Regarding stratifying the plots by dose and influential covariates when > using SVPC, our solution is to group subjects by the covariate of > interest (e.g. dose, and influential covariates) using different colors > and see if the colors are uniformly distributed in the SVPC plot. This > can also be used to identify potential covariates. I have a couple > > examples in the presentation slides. > > I am not sure I agree with you that SVPC can not be used for > concentration or efficacy controlled trials. In concentration or > efficacy controlled trials, patient's dose is adjusted based on > concentration or efficacy observed. As long as we have the dosing > record, we should be able to get the percentile for each observation of > each patient based on predicted PK/PD endpoints using this patient's > dosing record, and then pool all observation percentiles together > > regardless of each patient's dose and dosing schedule. > > Thanks, > > Diane > > Diane D. Wang, Ph.D. > Director > > Clinical Pharmcology Oncology Business Unit > > Pfizer La Jolla > 10555 Science Center Dr. (CB10/2408) > San Diego, CA 92121 > Office Phone: (858) 622-8021 > Cell Phone: (858) 761-3667 > email: [email protected] >

Leonid, I agree with you that VPC can not be used for concentration, effect controlled trials or trials with adaptive design. However SVPC does work in all these situations. The dosing record for each patient obtained from CRF is the adjusted dose based on the individual parameters. Therefore, 95 and 5 percentiles of the simulated concentrations based on this individual's information should only reflect random effect as all fixed effects are the same. Compared to PC-VPC, SVPC doesn't have the disadvantage Martin indicated in his Acop abstract "Prediction Corrected Visual Predictive Checks" that "PC-VPC only accounts for differences in typical subject predictions; there may also be differences in expected variability around this prediction". Rather than correcting for typical subject's predictions, SVPC uses each individual's exact design template without approximation and is not affected by the variability/uncertainty of the predictions. Diane

Diane, Martin, I think you are correct that SVPC (percentiles plotted versus time) and PC-VPC should work in the adaptive-study example that I provided. Still, conceptually, the simulation process should mimic the actual study. If adaptation was used in the actual study, it is better to do VPC-type simulations using adaptation rules, the same as in the original study. For adaptive trials, individual dose depends on individual parameters while in the non-adaptive simulations, simulated random effect distributions are dose-independent. It is not obvious why the resulting simulated distributions should always be similar to the observed distributions even if the model is correct. Thanks Leonid -------------------------------------- Leonid Gibiansky, Ph.D. President, QuantPharm LLC web: www.quantpharm.com e-mail: LGibiansky at quantpharm.com tel: (301) 767 5566 Wang, Diane wrote: > Leonid, > > I agree with you that VPC can not be used for concentration, effect > controlled trials or trials with adaptive design. However SVPC does > work in all these situations. The dosing record for each patient > obtained from CRF is the adjusted dose based on the individual > parameters. Therefore, 95 and 5 percentiles of the simulated > concentrations based on this individual's information should only > > reflect random effect as all fixed effects are the same. > > Compared to PC-VPC, SVPC doesn't have the disadvantage Martin indicated > in his Acop abstract "Prediction Corrected Visual Predictive Checks" > that "PC-VPC only accounts for differences in typical subject > predictions; there may also be differences in expected variability > around this prediction". Rather than correcting for typical subject's > predictions, SVPC uses each individual's exact design template without > approximation and is not affected by the variability/uncertainty of the > predictions. > > Diane >

Hi, Like Leonid, I am having trouble understanding how trials originally conducted with adaptive designs can be used for predictive checks if the simulation dose regimen is not based on the randomly assigned individual PK parameters. If the original adapted doses ("obtained from the CRF") are used then the simulated concentrations will not approach the adaptive design target as they would have done in the original data. Thus the distribution of simulated concentrations will be wider than the distribution of observed concentrations (see Bergstrand et al 2009 Example 3 left hand plot). Traditional visual predictive checks using the original doses will clearly show that the distributions of observations and simulated concentrations are different and would wrongly reject an adequate PK model. I would expect methods based on statistical predictive checks (PDE (including SVPC), NPDE) would detect that the distribution of prediction discrepancies is not as expected (uniform for PDE; normal for NPDE) and also wrongly reject an adequate PK model. PRED-corrected VPCs will not detect a difference between the PRED-corrected simulated concentrations and the PRED-corrected observations. This is because the PRED correction process is equivalent to normalizing all subjects to the same dose at each time point. For a linear PK model the variability in concs will have all the dose information removed and thus the adaptive changes in dose become irrelevant. Note that the PRED-corrected 'observations' will be quite different from the original observations and the trend of the PRED-corrected 'observations' variability will be quite unlike that seen in the data (see Bergstrand et al 2009 Example 3 right hand plot). This could be confusing but it should not lead to wrongly rejecting an adequate model. If the simulations are done using an adaptive dosing algorithm that is similar to that used in the original study then the statistical predictive checks and visual predictive checks (without or with PRED-correction) should not reject an adequate PK model. A non-PRED-corrected visual predictive check (NPC-VPC) should also correctly represent the actual observations and the simulated distributions if it used an adaptive dosing model. I think this is a key difference between the empirical PRED-corrected and mechanism based adaptive dose model approaches to a VPC. The mechanism based approach gives more visual reassurance that the combined models i.e. the PK model and the adaptive dosing model, can describe the data. This will give visual support for using the model combination for future trial simulations. The empirical PRED-corrected VPC does not give this kind of support for future use of the PK model under an adaptive design scenario. Nick Bergstrand M, Hooker AC, Wallin JE, Karlsson MO. Prediction corrected visual predictive checks http://www.go-acop.org/acop2009/posters ACOP. 2009. Leonid Gibiansky wrote: > Diane, Martin, > > I think you are correct that SVPC (percentiles plotted versus time) and PC-VPC should work in the adaptive-study example that I provided. Still, conceptually, the simulation process should mimic the actual study. If adaptation was used in the actual study, it is better to do VPC-type simulations using adaptation rules, the same as in the original study. > > For adaptive trials, individual dose depends on individual parameters while in the non-adaptive simulations, simulated random effect distributions are dose-independent. It is not obvious why the resulting simulated distributions should always be similar to the observed distributions even if the model is correct. > > Thanks > Leonid > > -------------------------------------- > Leonid Gibiansky, Ph.D. > President, QuantPharm LLC > web: www.quantpharm.com > e-mail: LGibiansky at quantpharm.com > tel: (301) 767 5566 > > Wang, Diane wrote: > > > Leonid, > > > > I agree with you that VPC can not be used for concentration, effect > > controlled trials or trials with adaptive design. However SVPC does > > work in all these situations. The dosing record for each patient > > obtained from CRF is the adjusted dose based on the individual > > parameters. Therefore, 95 and 5 percentiles of the simulated > > concentrations based on this individual's information should only > > > > reflect random effect as all fixed effects are the same. Compared to PC-VPC, SVPC doesn't have the disadvantage Martin indicated > > > > in his Acop abstract "Prediction Corrected Visual Predictive Checks" > > that "PC-VPC only accounts for differences in typical subject > > predictions; there may also be differences in expected variability > > around this prediction". Rather than correcting for typical subject's > > predictions, SVPC uses each individual's exact design template without > > approximation and is not affected by the variability/uncertainty of the > > predictions. > > > > Diane > > -----Original Message----- > > > > From: Leonid Gibiansky [ mailto: [email protected] ] Sent: Friday, September 18, 2009 5:28 PM > > > > To: Wang, Diane > > Cc: Dider Heine; [email protected] > > Subject: Re: [NMusers] VPC appropriateness in complex PK > > > > Diane, > > > > I probably worded it incorrectly. I was going to say that for concentration or effect controlled trails you cannot use straightforward > > > > VPC simulation based on the actual dosing history; you have to be more careful. Let me show the example that illustrates how VPC/SVPC behaves in the concentration or effect controlled trials. > > > > Assume that we conduct the two-dose study. The first dose (same for all subjects) is given to learn the kinetics. The second dose is adjusted (based on the previous data) in order to get the same Cmax for all subjects. For simplicity, assume that the world is nearly perfect: no or > > > > small residual variability, no or small inter-occasion variability. Then > > > > the dose adjustment can be perfect, and the second-dose Cmax for all subjects would hit the target. No or small second-dose Cmax variability would be observed. > > > > Now, let's do VPC. If you simulate based on the actual dosing history (even from the from the true model), your first-dose Cmax will be distributed similar to the observed data. However, your second-dose Cmax > > > > will vary significantly (even more than the first-dose Cmax) because you > > > > use the actual dose, rather than adjust the dose based on the individual > > > > parameters. Thus, standard VPC/SVPC/NPDE/PC-VPC/etc. will be misleading. > > > > One needs to simulate using the same dose adjustment algorithm as in the > > > > actual study. Only for these simulations predictive check plots can be used. > > > > Thanks > > Leonid > > > > -------------------------------------- > > Leonid Gibiansky, Ph.D. > > President, QuantPharm LLC > > web: www.quantpharm.com > > e-mail: LGibiansky at quantpharm.com > > tel: (301) 767 5566 > > > > Wang, Diane wrote: > > > > > Leonid, > > > > > > Thank you for the explanation. I was writing the response but found > > > your email stated it even better than what I could do myself. :) > > > Basically, VPC is less sensitive, when your data set is not > > > > homogeneous, > > > > > for evaluating random effects, because the 95% percentile interval of > > > predicted concentrations based on the full model reflects not only the > > > random effects (inter- and intra-subject variability) but also fixed > > > effects (difference in study design and covariate effects). SVPC > > > > solved > > > > > this problem as you described. > > > Regarding stratifying the plots by dose and influential covariates > > > > when > > > > > using SVPC, our solution is to group subjects by the covariate of > > > interest (e.g. dose, and influential covariates) using different > > > > colors > > > > > and see if the colors are uniformly distributed in the SVPC plot. > > > > This > > > > > can also be used to identify potential covariates. I have a couple > > > > > > examples in the presentation slides. I am not sure I agree with you that SVPC can not be used for > > > > > > concentration or efficacy controlled trials. In concentration or > > > efficacy controlled trials, patient's dose is adjusted based on > > > concentration or efficacy observed. As long as we have the dosing > > > record, we should be able to get the percentile for each observation > > > > of > > > > > each patient based on predicted PK/PD endpoints using this patient's > > > dosing record, and then pool all observation percentiles together > > > regardless of each patient's dose and dosing schedule. > > > Thanks, > > > > > > Diane > > > Diane D. Wang, Ph.D. > > > Director > > > Clinical Pharmcology Oncology Business Unit > > > Pfizer La Jolla > > > 10555 Science Center Dr. (CB10/2408) > > > San Diego, CA 92121 > > > Office Phone: (858) 622-8021 > > > Cell Phone: (858) 761-3667 > > > email: [email protected] > > > > > > -----Original Message-----

Dear Martin and Uppsala group, Maybe I’m wrong, but I guess PC-VPC is not available in the current version of PsN. Did you plan to implement it in the new PsN version? Many thanks Kind Regards Marco ------------------------------------------------------------------------------ Marco Campioni, PhD Modelling & Simulations Exploratory Medicine Merck Serono S.A. - Geneva 9, Chemin des Mines 1202 Geneva, Switzerland Location: B1.4 Phone: +41 22 414 4554 Fax: +41 22 414 3059 Email: marco.campioni "Martin Bergstrand" <martin.bergstrand Sent by: owner-nmusers 18/09/2009 18:50 To "'Dider Heine'" <ddrheine cc Subject RE: [NMusers] VPC appropriateness in complex PK Dear Dider, In my opinion the PAGE 2009 abstract by Diane Wang does highlight weaknesses with the standard VPC under certain circumstances. However, I don’t think that the SVPC represent the answer to those weaknesses. Prediction corrected VPCs (PC-VPCs) are a better way of addressing these issues and was first mentioned in the Karlsson and Holford tutorial on VPCs at PAGE 2008 ( http://www.page-meeting.org/pdf_assets/8694-Karlsson_Holford_VPC_Tutorial_hires.pdf ). A poster on the PC-VPCs principle and the advantage with these is submitted to the ACoP conference (October 2009). A two page abstract regarding that poster is available already now via the ACoP webpage ( http://www.go-acop.org/acop2009/posters - Title: “Prediction Corrected Visual Predictive Checks” Authors: Martin Bergstrand, Andrew C. Hooker, Johan E. Wallin, Mats O. Karlsson). Please have a look at this abstract and contact me if you have any further questions. Kind regards, Martin Bergstrand, MSc, PhD student ----------------------------------------------- Pharmacometrics Research Group, Department of Pharmaceutical Biosciences, Uppsala University ----------------------------------------------- P.O. Box 591 SE-751 24 Uppsala Sweden ----------------------------------------------- martin.bergstrand e ----------------------------------------------- Work: +46 18 471 4639 Mobile: +46 709 994 396 Fax: +46 18 471 4003

Dear Martin and Uppsala group, Maybe I’m wrong, but I guess PC-VPC is not available in the current version of PsN. Did you plan to implement it in the new PsN version? Many thanks Kind Regards Marco ------------------------------------------------------------------------------ Marco Campioni, PhD Modelling & Simulations Exploratory Medicine Merck Serono S.A. - Geneva 9, Chemin des Mines 1202 Geneva, Switzerland Location: B1.4 Phone: +41 22 414 4554 Fax: +41 22 414 3059 Email: [email protected] "Martin Bergstrand" <[email protected]> Sent by: [email protected] 18/09/2009 18:50 To "'Dider Heine'" <[email protected]>, <[email protected]> cc Subject RE: [NMusers] VPC appropriateness in complex PK Dear Dider, In my opinion the PAGE 2009 abstract by Diane Wang does highlight weaknesses with the standard VPC under certain circumstances. However, I don’t think that the SVPC represent the answer to those weaknesses. Prediction corrected VPCs (PC-VPCs) are a better way of addressing these issues and was first mentioned in the Karlsson and Holford tutorial on VPCs at PAGE 2008 ( http://www.page-meeting.org/pdf_assets/8694-Karlsson_Holford_VPC_Tutorial_hires.pdf ). A poster on the PC-VPCs principle and the advantage with these is submitted to the ACoP conference (October 2009). A two page abstract regarding that poster is available already now via the ACoP webpage ( http://www.go-acop.org/acop2009/posters - Title: “Prediction Corrected Visual Predictive Checks” Authors: Martin Bergstrand, Andrew C. Hooker, Johan E. Wallin, Mats O. Karlsson). Please have a look at this abstract and contact me if you have any further questions. Kind regards, Martin Bergstrand, MSc, PhD student ----------------------------------------------- Pharmacometrics Research Group, Department of Pharmaceutical Biosciences, Uppsala University ----------------------------------------------- P.O. Box 591 SE-751 24 Uppsala Sweden ----------------------------------------------- [email protected] ----------------------------------------------- Work: +46 18 471 4639 Mobile: +46 709 994 396 Fax: +46 18 471 4003

Dear NMusers, For some reason my last message to NMusers got lost in www-space. Since both Leonid and Nick have responded to my initial message I repost this message so that you can follow the discussion (see email below). In addition to this message I would also like to comment on the messages by Diane, Leonid and Nick. Nick and Leonid: I agree that it would be useful if one could also simulate that adaptive design (e.g. dose adaptations) and show the observations on the non transformed scale. However this will in many cases be very hard since dos adaptations are often done not according to a strict algorithm and/or all information supporting the dose alterations is not available. It is to my experience quite commonly written I study protocols that dose adjustments can be done “by the discretion of the investigator”. Diane and Leonid: If I understood the SVPC procedure correctly from Diane’s presentation it utilizes a principle similar to that behind Numerical Predictive Check (NPC). Most of all SVPC seem to have a striking similarity to the first version of the prediction discrepancies as described by Metré et al (1). The prediction discrepancies have been further developed into the normalised prediction distribution errors (NPDE) (2). From my experience both NPC and NPDE are useful diagnostic tools but not applicable to data from studies with adaptive dos adjustments (correlation between ETAs and design). What is the unique feature with SVPC that sets it apart from the prediction discrepancies and makes it applicable to studies with adaptive dos adjustments? Nick: Regarding this sentence “The empirical PRED-corrected VPC does not give this kind of support for future use of the PK model under an adaptive design scenario”. Why is this? If the PC-VPC can verify that you have an acceptable structure model and unbiased parameter estimates you can then simulate any type of adaptive design scenario. Best regards, Martin 1. Prediction discrepancies for the evaluation of … Mentré F, Escolano S. JPKPD. 2006 2. Computing normalised prediction distribution errors ... Comets E, Brendel K, Mentré F. CMPB. 2008

So to summarize, 1. The VPC (Holford NHG. PAGE. 2005;14) is appropriate for screening of all pk models, and is complementary to: 2. The SPC (original method by Mentre et. al J Pharmacokinet Pharmacodyn. 2006;33(3):345-67, subsequently re-presented by Wang et al, PAGE, 2009) which should be used to identify if the distribution of discrepancies between model predictions and observations is uniform. 3. The PC-VPC (implemented from within PSN) offers similar advantages to 1 and 2 above (combined) with the additional advantage of being automated. 4. In the case of adaptive trial desings, a VPC with an adaptive algorithm should be implemented for model evaluation. Agreed? Cheers! Dider

In my opinion, the concentration data after dose adaptation should not be simulated based on the adaptive algorithm implemented in the original design. Using the simple but extreme case proposed by Leonid, i.e. all patients were targeting a common concentration level with no or small residual variability and no or small inter-occasion variability. For example, under iv infusion, the target is Css of 100ng/ml. All patients start with the same infusion rate and the infusion rate will be adjusted after steady state is achieved. For patient A, if the Css under the initial infusion rate is 50ng/ml, then double the infusion rate. For patient B, if the Css under the initial infusion rate is 200ng/ml, reduce the infusion rate by half. After dose adaption, every one should hit 100ng/ml. Then no matter what the model is, all the simulated concentrations after dose adaptation will hit the same target (matching exactly the observed data) if the original adaptive algorithm is used in simulation. In this case, there is no way the structure model can be wrong because it is very simple (R/CLi). But imagine we fit an exponential between-subject variability model to a log-normally distributed CLi. Then using the exponential distribution to simulate CLi and then adjust individual infusion rate accordingly. All patients will still hit 100ng/ml after dose adjustment. Of course, one can argue the pre-adaptation concentration should pick up the mis-specification. The point is that post-adaption data cannot be simulated based on the adaptive algorithm. On the other hand, simulation can be done based on original adapted doses ("obtained from the CRF") assuming the whole set of simulated individuals will also take the exact dose sequence that was taken by one specific individual in the trial. The only difference among these individuals will be due to the random between-subject variation on PK parameters. SVPC or PDE should still show uniform distribution even for those post-adaption data (even if all observed data are 100ng/ml). Yaning Wang, Ph.D. Team Leader, Pharmacometrics Office of Clinical Pharmacology Office of Translational Science Center for Drug Evaluation and Research U.S. Food and Drug Administration Phone: 301-796-1624 Email: yaning.wang "The contents of this message are mine personally and do not necessarily reflect any position of the Government or the Food and Drug Administration."

Nick and Martin, Thank you for pointing out similarity between SVPC and PDE method published by Mentré and Escolano. The end result of these two approaches is the same but the simulation process is a little different as they were developed independently and from a different perspective. In SVPC, we simulated 1000 (100 is actually enough as shown in my presentation)individual's concentrations (including both between and within subject variability) for each individual based on its own study template to computer percentile of each observation. This allows us to fix all the covariate effects including dose to evaluate random effect and structure model. The paper by M Mentré et al simulated 1000 or more individual PK parameters (only between subject variability) and used the normal cumulative distribution function (within subject variability) to computer the percentile. Mentre's paper focused on statistics of predictive discrepancy and was not discussed in the context of VPC. PDE is not proposed as a solution or an alternative approach for VPC when VPC is not feasible or can not be performed correctly. This might be why no one has used it for the purpose of predictive check since its publication. In a way, SVPC can be viewed as an application of PDE although the simulation process is easier. One can simply use the original dataset as simulation dataset and set SUBPROBLEMS=100. The main purpose of my PAGE presentation is to raise awareness of the inadequacy of VPC in many situations. Among published population PK/PD papers, VPC was often conducted regardless of presence of covariate effect, individualized dosing and other fixed effects. As to which approach to use, as long as it is conducted correctly and fit the purpose, it is an individual's choice. It is always good to have options. Diane

In my opinion, the concentration data after dose adaptation should not be simulated based on the adaptive algorithm implemented in the original design. Using the simple but extreme case proposed by Leonid, i.e. all patients were targeting a common concentration level with no or small residual variability and no or small inter-occasion variability. For example, under iv infusion, the target is Css of 100ng/ml. All patients start with the same infusion rate and the infusion rate will be adjusted after steady state is achieved. For patient A, if the Css under the initial infusion rate is 50ng/ml, then double the infusion rate. For patient B, if the Css under the initial infusion rate is 200ng/ml, reduce the infusion rate by half. After dose adaption, every one should hit 100ng/ml. Then no matter what the model is, all the simulated concentrations after dose adaptation will hit the same target (matching exactly the observed data) if the original adaptive algorithm is used in simulation. In this case, there is no way the structure model can be wrong because it is very simple (R/CLi). But imagine we fit an exponential between-subject variability model to a log-normally distributed CLi. Then using the exponential distribution to simulate CLi and then adjust individual infusion rate accordingly. All patients will still hit 100ng/ml after dose adjustment. Of course, one can argue the pre-adaptation concentration should pick up the mis-specification. The point is that post-adaption data cannot be simulated based on the adaptive algorithm. On the other hand, simulation can be done based on original adapted doses ("obtained from the CRF") assuming the whole set of simulated individuals will also take the exact dose sequence that was taken by one specific individual in the trial. The only difference among these individuals will be due to the random between-subject variation on PK parameters. SVPC or PDE should still show uniform distribution even for those post-adaption data (even if all observed data are 100ng/ml). Yaning Wang, Ph.D. Team Leader, Pharmacometrics Office of Clinical Pharmacology Office of Translational Science Center for Drug Evaluation and Research U.S. Food and Drug Administration Phone: 301-796-1624 Email: [email protected] "The contents of this message are mine personally and do not necessarily reflect any position of the Government or the Food and Drug Administration."

Yaning, I am not understanding something in your example which you use to assert that adaptive dosing should not be used in simulations for a VPC. if you used adaptive dosing and the correct PK structural and random effects model (as you seem to describe) then if there was no error in the measured conc/timing then of course the simulated concs would exactly match the observed concs in each subject. What is wrong with that? Your example uses the correct simulation model for prediction of the original data distribution and so the predictive check should of course confirm the model. That is what a predictive check is about. It checks that the model is adequate (i.e. good enough to to be the true model). In reality adaptive designs based on measured concs will always have errors in the measured conc or timing of concs in addition to the between subject variability in PK parameters so using the nominal adaptive design algorithm would always add some additional noise to the predicted distribution. Another real life feature is that clinical study staff dont follow the protocol (or there may not even be a protocol!). If one can devise a clinically reasonable adaptive design protocol then this is a useful step in trying to perform a predictive check of a trial with an unfollowed or unknown protocol. If the adaptive design protocol matches the data using a VPC then one can have some confidence in using the PK model and the adaptive design algorithm to simulate future trials which are not expected to have a strictly followed adaptive protocol. The need for this kind of auxiliary model to account for real life variations in design has been recognized for dropouts due to informative missingness mechanisms. It not only allows a predictive check to evaluate the adequacy of the model but also gives a mechanism for future clinical trial simulation. With regard to your second point -- it has made me think more carefully about the assumptions of the SPC method and I think I can now convince myself that you are correct that the original dosing design needs to be used for simulation. Here is my rationale which I hope you will be willing to try to follow and see if the logic is reasonable. If this logic is OK then I will retract my previous statement that SPCs would fail if they did not simulate with an adaptive design. My understanding of how SPCs work involves the basic idea of comparing a measured conc to a distribution of simulated concs that arise from the structural and random effects model under the original design that gave rise to the measured conc. If the original design (i.e. "CRF" dosing regimen) that gave rise to the measured concs is used then the distribution of simulated concs will depend on the structural and random effects model conditional on the original design. It is clear that the mean simulated concentration for an individual will not match the observed concentration in a particular individual because the random PK parameters for that individual will be randomly different from the actual PK parameters. This is why the CRF dose cannot be used for the VPC (unless both the predictions and observations are 'normalized' so that the actual dosing regimen does not matter as is done in the PC-VPC). But it is plausible that the discrepancy of a single observed concentration from the simulated distribution of concs at that time for that individual will be random across individuals and thus the prediction discrepancy would be uniform (if the PK structural and random effects model is correct). However, the within subject correlation of these discrepancies due to the PK parameter between subject differences (which will affect all concentrations in that individual) will violate the assumption of independent discrepancies and this is what the NPDE method tries to achieve by removing the within subject correlation. Thus I think we should not rely on PDE methods for SPC unless there is only one observation per subject. The technology for doing uncorrelated PDEs (e.g. NPDE) is now freely available ( http://www.npde.biostat.fr/index.php ) so there is really no justification to use correlated PDEs . Nick Wang, Yaning wrote: > In my opinion, the concentration data after dose adaptation should not > be simulated based on the adaptive algorithm implemented in the original > design. Using the simple but extreme case proposed by Leonid, i.e. all > patients were targeting a common concentration level with no or small > residual variability and no or small inter-occasion variability. For > example, under iv infusion, the target is Css of 100ng/ml. All patients > start with the same infusion rate and the infusion rate will be adjusted > after steady state is achieved. For patient A, if the Css under the > initial infusion rate is 50ng/ml, then double the infusion rate. For > patient B, if the Css under the initial infusion rate is 200ng/ml, > reduce the infusion rate by half. After dose adaption, every one should > hit 100ng/ml. Then no matter what the model is, all the simulated > concentrations after dose adaptation will hit the same target (matching > exactly the observed data) if the original adaptive algorithm is used in > simulation. In this case, there is no way the structure model can be > wrong because it is very simple (R/CLi). But imagine we fit an > exponential between-subject variability model to a log-normally > distributed CLi. Then using the exponential distribution to simulate CLi > and then adjust individual infusion rate accordingly. All patients will > still hit 100ng/ml after dose adjustment. Of course, one can argue the > pre-adaptation concentration should pick up the mis-specification. The > point is that post-adaption data cannot be simulated based on the > adaptive algorithm. > > On the other hand, simulation can be done based on original adapted > doses ("obtained from the CRF") assuming the whole set of simulated > individuals will also take the exact dose sequence that was taken by one > specific individual in the trial. The only difference among these > individuals will be due to the random between-subject variation on PK > parameters. SVPC or PDE should still show uniform distribution even for > > those post-adaption data (even if all observed data are 100ng/ml). > > Yaning Wang, Ph.D. Team Leader, Pharmacometrics Office of Clinical Pharmacology Office of Translational Science Center for Drug Evaluation and Research U.S. Food and Drug Administration Phone: 301-796-1624 Email: [email protected] > > "The contents of this message are mine personally and do not necessarily > reflect any position of the Government or the Food and Drug > Administration." >

Dear Nick and NMusers, Thank you for your very insightful comments on these matters. I in principle agree with you all the way. As I have said before I think that it is very useful if you can simulate dose adaptations and use these for VPCs. However even if you can do so I think that the PC-VPC is a useful complement. Especially since it will be unaffected by the performance of the dos adaptation model and better diagnose the random effect components of the model. Furthermore the PC-VPC is much simpler to implement since you don't not have to develop and include a dos adaptation model before simulating. After evaluating the NPDEs coming from NONMEM7 for my adaptive dose trial example I can see that these are unaffected (mean=0, variance=1). I still don't understand exactly why this is so but I have obviously been doing something wrong before. My humble apology to the community for spreading this false information. Best regards, Martin

Hello all, I am sorry to revive this thread after a few days, but I have to say, I am really confused - both by the discussion and issues. I must admit that I do not really have much experience with concentration controlled or random dose-adapted trials (not fixed titration), so please let me know if I am not thinking about this clearly or correctly. The first thing that confuses me is this. If have a true model for the response and dose adaption and simulate one trial. Then if I simulate 100 trials from the same model, shouldn't these look stochastically similar? Shouldn't I be able to find a plot that shows that the first simulation is compatible (comparable) with the 100? (Perhaps this is what Nick was saying early on in this thread and sorry if I am miss-paraphrasing you here Nick - please correct me if so). If I fit the true model to data with a reasonable sample size, shouldn't the estimates be close to the true values of the parameters and put me in a situation like described in the paragraph immediately above (compatible data and simulations)? If I have a true model and observed dosing history, shouldn't the empirical distribution of the dosing history be compatible with simulated dosing histories such that they look stochastically similar? If so, then should it matter if we use the observed dosing history or the random rule in the simulations as long as the correlations between etas and doses are preserved? It seems to me that the simulation model is readily constructed and easy to think about. If the above 3 paragraphs are reasonable, then what I am struggling with is that perhaps this suggests the analysis model may not correct. That is, the "compatibility" of the responses and doses may not be adequately addressed when constructing the likelihood. Is the correlation between doses and etas being appropriately expressed in the likelihood? This thread reminded me of the article: Beal SL. Conditioning on certain random events associated with statistical variability in PK/PD. J Pharmacokinet Pharmacodyn. 2005 Apr;32(2):213-43. Therein he discussed dose titration; and if memory serves, the likelihood for conditioning on the observed doses is pretty complicated in order to keep the responses and etas and doses compatible (perhaps). Perhaps his article will shed some light on the situation?.... Kind regards, Matt