Re: Model selection criteria?

From: "Nick Holford" n.holford@auckland.ac.nz Subject: Re: [NMusers] Model selection criteria? Date: Sat, July 2, 2005 6:29 am Catherine, Selection of a model depends on the purpose of modelling. If the goal is to predict response time course then a predictive check procedure can be used to reassure yourself that your model is adequate (e.g. http://www.health.auckland.ac.nz/pharmacology/staff/nholford/workshops/PAGE/). If the goal is to identify sources of between subject variability (BSV) which might be helpful in predicting doses in individuals then you might want to look at how much the estimated BSV is reduced (e.g. Matthews et al. 2004). In most cases I rely on changes in objective function to find the model that gives the best overall fit then use graphical methods to confirm that the predictions make sense. Any reduction in OBJ means a better fit but not necessarily a better model for the intended purpose (see above). There are hypothesis testing criteria based on the change in OBJ but these are usually not of much interest in themselves because modellers aren't typically interested in P values. If you really want to know the P value associated with a given change in OBJ it can take quite a bit of work to get a reliable estimate (e.g. http://wfn.sourceforge.net/wfnrt.htm). In most cases if you use the FOCE estimation method you can reasonably assume changes in OBJ are approximately chi-squared distributed to get an idea of the Type I error. The AIC and SC are not typically used by the NONMEM community for model selection. This is in part because there is no easy way to obtain the WRSS term but also because the change in objective function is easily obtained and gives almost the same kind of information. Precision of a parameter estimate might be a model selection criterion if you the purpose of modelling is to estimate a particular parameter with less than some degree of uncertainty. I think this is a very rarely applied purpose of modelling but if you do it then you should not rely on NONMEM's standard error estimates to compute confidence intervals but use a numerical procedure such as bootstrap or likelihood profiling (e.g. see http://wfn.sourceforge.net/wfnbs.htm). It is only possible to determine bias when you know the true parameter value which means for any practical parameter of interest you must be using simulation. Simulation is a very helpful method to understand how the combination of your experimental design, model and estimation method interact to produce parameter estimates. You can use (Monte Carlo) simulation to determine RMSE and bias but you cannot determine bias for models applied to data with a priori unknown parameter values. Nick Matthews I, Kirkpatrick C, Holford NHG. Quantitative justification for target concentration intervention - Parameter variability and predictive performance using population pharmacokinetic models for aminoglycosides. British Journal of Clinical Pharmacology 2004;58(1):8-19. -- Nick Holford, Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556 http://www.health.auckland.ac.nz/pharmacology/staff/nholford/