Re: LLR test, AIC, BIC

From: "Hutmacher, Matt" mhutmach@amgen.com Subject: Re: [NMusers] LLR test, AIC, BIC Date: 10/22/2003 3:12 PM Hello all, I certainly agree that what Nick has proposed below is one way to proceed. However, I would like to comment on his suggestion. The algorithm that Nick proposes below, in my mind, is not technically a randomization test. In my experience, the randomization test randomly permutes the "actual data" to establish the distribution of the null hypothesis test. What Nick is doing is "simulating data" to assess what the LRT would look like if the data were generated by the 1 compartment model that he describes. Since the data are simulated from a model that was based on a fit to the observed data, I would argue that what is being simulated is not the true null distribution but an approximation to it. Thus the cut-off value that you select is a prediction. While I believe that this is one way to proceed, the method is not as absolute (in my mind) as Nick suggests. The assessment of type 1 error of 5% is a prediction not a truth. Perhaps a simpler way to proceed here is to look at a few diagnostics. Check the model's condition number to see that the model is not over-parameterized. Check to make sure the theta estimates don't indicate the 2-cmt model is wanting to collapse into a 1-cmt. Look at the WRES and IWRES plots versus time to make sure the 1-cmt model is adequately characterizing the peak concentrations and the tail (no time trends). If you are still unsure, simulate from the 2-cmt and 1-cmt models and see which can better reproduce the data (small posterior predictive check). Matt