Re: Problems with an apparent compiler-senstive model

From: Nick Holford n.holford@auckland.ac.nz Subject: Re: [NMusers] Problems with an apparent compiler-senstive model Date: Fri, 04 Aug 2006 09:30:41 +1200 Manoj, I was careful in my comments to say 'predictive check' (PC) NOT 'posterior predictive check' (PPC). The PPC, as described by Gelman et al. (1996), involves simulation using the posterior distribution of the parameter estimates. The PPC is tricky to do using NONMEM (Yano et al. 2001) even if one has an estimate of the variance-covariance of the estimate (from $COV or bootstrap). Yano et al. demonstrated that a degenerate PC (DPC) could be equivalent to a PPC (with their specific example). The DPC involves simulation using the point estimates of the model without consideration of uncertainty. My (limited) experience of using a DPC for visual evaluation of model perfomance (the visual predictive check or VPC) has shown that a relatively quick and simple check can be done which can reveal major problems with a model (Holford 2006). More sophisticated methods have been described by Mentre et al (2006). These simulation based methods of evaluating model performance do not necessarily require the availability of the variance-covariance matrix of the estimate. They can be performed with any model regardless of NONMEM's termination status. I consider standard errors to be almost worthless for evaluating model performance. They can give some crude clue for parameters that are not well identified by the design but do not help diagnose model deficiencies for predictions. Indeed having a few poorly identified parameters may not harm the predictive performance. It is only a desire for parsimony that is affected in this situation. If you are really and truly interested in the parameter estimate itself (rather than the model predictions) then bootstrap estimates of parameter uncertainty are probably more reliable than predictions of confidence intervals using asymptotic estimates of standard errors and the often invalid assumption of normality (see Matthews et al 2004 Table 5 for an example of the discrepancies). Nick Gelman A, Meng X-L, Stern H. Posterior predictive assessment of model fitness via realized discrepancies. Statist Sinica. 1996;6:733-807. Holford NHG. The Visual Predictive Check Superiority to Standard Diagnostic (Rorschach) Plots http://www.page-meeting.org/default.asp?abstract=738 PAGE; 2005; Pamplona; 2005. Matthews I, Kirkpatrick C, Holford NHG. Quantitative justification for target concentration intervention - Parameter variability and predictive performance using population pharmacokinetic models for aminoglycosides. British Journal of Clinical Pharmacology. 2004;58(1):8-19. Mentre F, Escolano S. Prediction discrepancies for the evaluation of nonlinear mixed-effects models. J Pharmacokinet Pharmacodyn. 2006 Jun;33(3):345-67. Yano Y, Beal SL, Sheiner LB. Evaluating pharmacokinetic/pharmacodynamic models using the posterior predictive check. J Pharmacokinet Pharmacodyn. 2001 Apr;28(2):171-92.