Re: LLR test, AIC, BIC

From: Nick Holford n.holford@auckland.ac.nz Subject: Re: [NMusers] LLR test, AIC, BIC Date: 10/22/2003 5:21 PM Matt, Thanks for your comments. I am aware of the 'true' randomization test method that you refer to (e.g. see http://wfn.sourceforge.net/wfnrt.htm). I agree that a method based on randomization of the actual data is what Fisher originally proposed e.g. the 'Fisher exact test'. However, I do not know of a way to perform this kind of randomization to investigate the performance the LLR to distinguish a one vs two cpt model. The parametric bootstrap method I described to create an empirical null distribution has been called by one group of authors the 'simulation hypothesis test' (Gisleskog PO, Karlsson MO, Beal SL. Use of Prior Information to Stabilize a Population Data Analysis. Journal of Pharmacokinetics & Biopharmaceutics 2003;29(5/6):473-505). A randomization test, based strictly on the original data, is used to estimate the probability of rejecting the null for *that specific set of data*. It is not a generalized result. The 'simulation hypothesis test' more closely resembles the asymptotic flavour of conventional statistical testing because it is obtained by considering a large sample from the proposed distribution of typical data generated from the null model. I prefer not to use the term 'simulation hypothesis test' because it focusses attention on hypothesis testing. The procedure can be seen in a broader context. It is an algorithm for generating the null distribution of a (test) statistic. This null distribution has several uses other than strictly doing an hypothesis test with some arbitrary alpha criterion e.g. it can be used to estimate the true probability of the data arising under the null, it can be used to create a table of lookup values for doing hypothesis testing, it can be used to teach and learn about the shape of distributions that are are widely assumed to have certain shapes (but these assumptions may be wrong). The NONMEM community has been exposed over the last couple of years to the problems of assuming the chi-square distribution for the null distribtion of LLR (especially with FO but also with FOCE). If you need to get involved in making important modelling decisions using hypothesis testing with the LLR then I would encourage you to verify by experiment what null distribution is required for your decision. The other diagnostics you mention are of course valuable and I would typically rely more on a visual examination of the time course of observed and predicted concs to make a decision on an individual data set. However, some tasks e.g. using clinical trial simulation to examine the power of designs, require an automatable, objective decision criterion. I have been using the randomization test to get better critical values for rejecting the null when doing clinical trial simulation. This has had a major impact on the estimates of power -- critical values for LLR changes are often much larger than expected even using FOCE. Nick -- Nick Holford, Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556 http://www.health.auckland.ac.nz/pharmacology/staff/nholford/