OFV or Diagnostic Plot ?? Which one rules...

Hi Everyone, I am fitting two compartment PK model to Marijuana (THC) concentrations. When I apply proportional error (or proportional plus additive) residual model, I get pretty good fits (except 15% of subjects) at all time points. However, when I apply only additive error residual model, I get perfect fits in all subjects but objective functional value is increased by about 20 units. DV vs IPRED reveal all concentrations on line of unity. My question is: should I go with additive error model which gives me perfect fit but higher OFV or should I go with proportional error model which gives me lower OFV but not so good fit in couple of subjects? Regards, Sumeet Singla Graduate Student Dpt. of Pharmaceutics & Translational Therapeutics College of Pharmacy- University of Iowa

Hi Sumeet, The OFV is often king. But what does the VPC look like? That is the gold standard. Diagnostic plots are equivalent to Rorschach blots. If you find a way to let us see the VPCs then we can all enjoy the view. Best wishes, Nick Holford NHG. The visual predictive check - superiority to standard diagnostic (Rorschach) plots www.page-meeting.org/?abstract=738. Last accessed 13 Feb 2019. PAGE. 2005;14. Nguyen TH, Mouksassi MS, Holford N, Al-Huniti N, Freedman I, Hooker AC, et al. Model Evaluation of Continuous Data Pharmacometric Models: Metrics and Graphics. CPT: pharmacometrics & systems pharmacology. 2017;6(2):87-109.

Sumeet, DV vs IPRED is only one, and the least helpful plot. You may want to look on DV vs PRED, both in original scale and on log log scale, CWRES vs time, PRED, distributions and correlation of random effects, etc. and only then one can decide which of the models is better. Based on the description, I would guess that model with proportional error provides better fit at very low concentrations, visible in log scale plots. So you may also factor this in in the decision process. If max concentrations are more important, additive error may help but if low concentrations are more important, you may want to use combined or proportional error. Regards, Leonid

Hi Sumeet, For the VPC I would suggest (assuming you are using NONMEM) that you install Perl-speaks-NONMEM (PsN, https://uupharmacometrics.github.io/PsN/index.html), which can be used to run a VPC without a lot of additional coding. The results can easily be viewed using Xpose 4 ( https://cran.r-project.org/package=xpose4). Other tools like Monolix and nlmixr have VPC functionality baked in. One would usually need to evaluate and balance all the diagnostic evidence available before coming to a conclusion. There’s no one magic plot or number, although how you weight the various diagnostics could depend on the purpose of the model (the question being asked). For instance, if you want PK predictions for driving a PD model, perhaps having good individual predictions of the subjects you have is more interesting than the model’s ability to predict new subjects (although being able to do so is still important). On the other hand, if you’re wanting to describe the PK at the population level or use the model for simulations, the population predictions and the residuals (and VPCs) are usually the most important things to look at – in this scenario individual fits, while still relevant, are not the primary objective. At the end of the day, though, all diagnostics should look OK unless something is wrong somewhere (with some rare exceptions, which you can check using mirror plots in PsN and Xpose). In addition to the diagnostics Nick and Leonid have mentioned, how precisely estimated are your parameters? What is the estimated shrinkage on your random-effects parameters? How big is your condition number (ratio of lowest and highest eigenvalues)? Together with the other diagnostics, these snippets of information can also add to the overall picture. Best Justin — Justin Wilkins, PhD Occams Kirchnerstraße 22 59457 Werl Germany www.occams.com +49 2922 927 8843 [email protected]<mailto:[email protected]> https://www.linkedin.com/in/justinjwilkins/ [cid:[email protected]]

Hi Sumeet, OFV is an objective fit of the model to the data, so generally that should be your leading criteria. Outside of that you often have to deal with subjectivity. However, there are quite a few caveats to relying too strongly on OFV. You should try to avoid "chasing OFV" by testing too many models or those in which the theoretical justification is lacking. Which error model best agrees with other information you have about the concentrations you have? You might also consider whether poor diagnostics you see as part of the residual error model really originate from structural model-misspecification. It can happen that you "hide" the shortcomings of the structural model by putting too much flexibility into the residual error model. It then becomes very hard to improve the structural model when the information is "swallowed up" by the residual error model. You cant fix what you cant see. I often try to think about how the model will be used outside of the development process. In its intended application does the model need to predict high concentrations or low concentrations more accurately? A proportional error model lets low concentrations play a stronger role in the model likelihood compared to proportional+additive. Basically, getting OFV 20 points lower with prop+add compared to prop means that the model can fit higher concentrations better if a little bit worse fit can be tolerated in low concentrations. You have to decide which one is most appropriate. It depends on how the model is intended to be used and how your structural model compares to what you think the "true" model might be. Warm regards, Douglas Eleveld