From: "Catherine Sherwin" catherine.sherwin@stonebow.otago.ac.nz
Subject: [NMusers] Model selection criteria?
Date: Fri, July 1, 2005 4:34 am
Hi NMusers,
I would like your help and advice in regards to correct model selection
when using NONMEM.
I have looked at using the Akaike Information Criterion. The equation I
used was the following: AIC= N x log (WRSS) + 2P.
I have also looked at using the Schwartz Criterion. The equation was: SC =
N x log(WRSS)+ log(N) x P.
In both equations, N is the number of observations, WRSS is the weighted
residual sum of squares and P is the number of parameters.
Can you tell me if these are appropriate guides to model selection for
NONMEM?
Also can you comment on the use of the following in determining selection
of the correct model?
1) Precision (root mean squared prediction error)?
2) Accuracy or bias (mean prediction error)?
3) And reduction in objective function of more than 5.02 (p<0.01)?
Regards
Catherine Sherwin
Model selection criteria?
From: "Nick Holford" n.holford@auckland.ac.nz
Subject: Re: [NMusers] Model selection criteria?
Date: Sat, July 2, 2005 6:29 am
Catherine,
Selection of a model depends on the purpose of modelling. If the goal is to predict
response time course then a predictive check procedure can be used to reassure
yourself that your model is adequate (e.g.
http://www.health.auckland.ac.nz/pharmacology/staff/nholford/workshops/PAGE/). If
the goal is to identify sources of between subject variability (BSV) which might be
helpful in predicting doses in individuals then you might want to look at how much
the estimated BSV is reduced (e.g. Matthews et al. 2004).
In most cases I rely on changes in objective function to find the model that gives
the best overall fit then use graphical methods to confirm that the predictions make
sense. Any reduction in OBJ means a better fit but not necessarily a better model
for the intended purpose (see above). There are hypothesis testing criteria based on
the change in OBJ but these are usually not of much interest in themselves because
modellers aren't typically interested in P values. If you really want to know the P
value associated with a given change in OBJ it can take quite a bit of work to get a
reliable estimate (e.g. http://wfn.sourceforge.net/wfnrt.htm). In most cases if you
use the FOCE estimation method you can reasonably assume changes in OBJ are
approximately chi-squared distributed to get an idea of the Type I error.
The AIC and SC are not typically used by the NONMEM community for model selection.
This is in part because there is no easy way to obtain the WRSS term but also
because the change in objective function is easily obtained and gives almost the
same kind of information.
Precision of a parameter estimate might be a model selection criterion if you the
purpose of modelling is to estimate a particular parameter with less than some
degree of uncertainty. I think this is a very rarely applied purpose of modelling
but if you do it then you should not rely on NONMEM's standard error estimates to
compute confidence intervals but use a numerical procedure such as bootstrap or
likelihood profiling (e.g. see http://wfn.sourceforge.net/wfnbs.htm).
It is only possible to determine bias when you know the true parameter value which
means for any practical parameter of interest you must be using simulation.
Simulation is a very helpful method to understand how the combination of your
experimental design, model and estimation method interact to produce parameter
estimates. You can use (Monte Carlo) simulation to determine RMSE and bias but you
cannot determine bias for models applied to data with a priori unknown parameter
values.
Nick
Matthews I, Kirkpatrick C, Holford NHG. Quantitative justification for target
concentration intervention - Parameter variability and predictive performance using
population pharmacokinetic models for aminoglycosides. British Journal of Clinical
Pharmacology 2004;58(1):8-19.
--
Nick Holford, Dept Pharmacology & Clinical Pharmacology
University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand
email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556
http://www.health.auckland.ac.nz/pharmacology/staff/nholford/
From: "Catherine Sherwin" catherine.sherwin@stonebow.otago.ac.nz
Subject: RE: [NMusers] Model selection criteria?
Date: Tue, July 5, 2005 10:09 pm
Hi Nick,
Thank you very much for the information and the references.
Catherine
*****************************************************
Catherine M. T. Sherwin
PhD Candidate
Dept. of Paediatrics & Child Health
Dunedin School of Medicine
University of Otago
P.O Box 913
Dunedin, New Zealand
Phone: (03) 474 7836
Fax: (03) 474 7817
_______________________________________________________