RE: $OMEGA blocks and log-likelihood profiling

From: Nick Holford n.holford@auckland.ac.nz Subject: RE:[NMusers] $OMEGA blocks and log-likelihood profiling Date: Wed, 30 Jun 2004 15:51:23 +1200 Ken, Thanks for keeping the ball rolling... I think we need to keep clear a distinction between statistical theory and the assumptions required by the theory when applied to NONMEM. Please note this sentence in my response to Matt (below) "These data based experimental tests have forced me to think harder about the assumptions we make when applying statistical theory in this area." Thanks for the suggestions for improving my education. My own preference for a textbook is Seber GAF, Wild CJ. Non-linear regression. New York: John Wiley & Sons; 1989 (the authors are academic colleagues at the University of Auckland <grin>). I have no quibble with the theories (at least to the extent I understand them) but as you note below they often involve assumptions of normality (e.g. for predicting CIs from SEs) or that the likelihood is correct when applying the chi-square distribution to the likelihood ratio test. Both of these assumptions are dubious when applied to results from NONMEM. Bob Leary provided some additional evidence of the problems of NONMEM's Maximum Approximate Likelihood at the PAGE meeting 2 weeks ago. http://www.page-meeting.org/page/page2004/Leary.pdf On matters that are not based on statistical theory like 'reliability' and 'model stability' which you justify by 'good statistical practice' I can only say that I prefer data based SOPs. If NONMEM's $COV results are only approximately correct then just how valuable is the inspection of the entrails for evidence of model stability? I note that Bates & Watts offer no guidance on either reliability or stability (in the index) but Seber & Wild do offer a discussion of parameter stability but only give a practical example in the case of a model with 2 parameters. There is no mention of model or parameter reliability in the index. My confidence in non-parametric bootstrap results is not simply on the basis of a plausible model but also from consideration of a reasonable design confirmed by simulation based tests. I consider your example of a reductio ad absurdam design to be a straw man in this context. With regard to your point about simulation -- You seem to have forgotten that I reported to you already results of using simulation with this problem ( http://www.cognigencorp.com/nonmem/nm/99jul152003.html). There was no evidence of bias or important differences in CIs in runs that failed compared with those that were successful, or those that also completed the covariance step. Your recent suggestion to examine Q-Q plots of the distributions of estimates obtained with varying termination/minimization conditions has been very helpful - thank you. I ended up (with some assistance from Christoffer Turnoe) creating CDFs of the empirical distribution of the non-parametric bootstrap parameters. There are some minor systematic differences between these plots when I compare the worst case (terminated due to proximity to infinite objective function value) with the best (covariance step completed). But the inferences about coverage are not importantly different in the context of the application (Matthews et al.2004). I am currently doing some more runs to increase the total number of $COV successful estimates to refine the evaluation of the distributions. I would be happy to send you the data for you to perform your own examinations. NONMEM parameter estimation involves calculations that are often at the limits of computational precision. I know that Stuart Beal has put a lot of effort into trying to make this as platform independent as possible but it is widely known that NONMEM results depend on compiler (and options) and CPU type. I believe that Stuart fine-tuned NONMEM on a specific platform (Sun workstation and compiler, I think). Heuristic decisions were no doubt made on the basis of performance on this platform. Even if other compilers and processors provide superior numerics NONMEM may not perform so well because of the Sun specific pragmatic implementation. Thus NONMEM running with an AMD Athlon CPU and the Compaq Visual Fortran compiler (my own platform) may fail despite being very close to a solution that would be successful on a Sun. It is my current hypothesis that it is this numerical dice throwing that gives rise to the high minimization failure rate rather than any major deficiency in the data or model that I have been using. My data based experiments are currently aimed at testing this hypothesis. If minimization and/or covariance step success are dependent on pseudo-random numerical issues then I would predict that the distribution of parameter estimates would be very similar irrespective of success of failure. Results to date do not provide evidence to reject this hypothesis. Nick Matthews I, Kirkpatrick C, Holford NHG. Quantitative justification for target concentration intervention - Parameter variability and predictive performance using population pharmacokinetic models for aminoglycosides. British Journal of Clinical Pharmacology 2004;58(1):8-19. -- Nick Holford, Dept Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand email:n.holford@auckland.ac.nz tel:+64(9)373-7599x86730 fax:373-7556 http://www.health.auckland.ac.nz/pharmacology/staff/nholford/