T matrix

From: Leonid Gibiansky Subject:Re: [NMusers] T matrix Date:Fri, March 8, 2002 5:21 pm Just to pick up on the issue: >You should understand that the SEs provided by NONMEM are not worth much. If you want to use them as measure of the confidence interval for your parameters then they rely on at least two untenable asssumptions 1) you have an infinite amount of data 2) the confidence intervals are symmetrical around the parameter estimate. Alternative empirical methods e.g. bootstrap or log likelihood profile are better way of understanding the confidence you have in your parameters. > In my experience, SEs provided by NONMEM worth a lot. We compared them with the bootstrap or log likelihood profile SEs and found out that in most cases the results are similar with the exception of the parameters with intrinsically non-symmetric distribution (e.g., bounded by zero, defined with large error and with the estimate close to zero). Moreover, log likelihood profile SEs with FO are less reliable (too narrow) compared to NONMEM estimates. Bootstrap is reasonable, but with fitting of 1000 or so problems you cannot be sure that your tails of the bootstrap distributions are not the local minimums (not to mention extra time and efforts that you need to invest in bootstrap). Summarizing, we found that NONMEM SEs provide quick and in most cases reliable information which is rarely corrected by more computer-intensive techniques. Leonid

From:Nick Holford Subject:Re: [NMusers] T matrix Date:Sat, March 9, 2002 4:09 pm Leonid, Thanks for these helpful comments. Leonid Gibiansky wrote: > In my experience, SEs provided by NONMEM worth a lot. We compared them with > the bootstrap or log likelihood profile SEs and found out that Is this work published yet? In Jul 2000 you said you were planning to publish. http://www.boomer.org/pkin/PK00/PK2000103.html I would be keen on seeing the details of your results. > in most > cases the results are similar with the exception of the parameters with > intrinsically non-symmetric distribution (e.g., bounded by zero, defined > with large error and with the estimate close to zero). It is exactly these parameters with an asymmetric estimation confidence that are the main reason I am personally interested in obtaining a confidence interval e.g. if I estimate Emax as part of a model to test if a drug is effective or not then I want to know how reliable the Emax value is. Typically Emax will be bounded by zero and defined with large error and may well have an estimate close to zero for many drugs in early drug development. > Moreover, log > likelihood profile SEs with FO are less reliable (too narrow) compared to > NONMEM estimates. Bootstrap is reasonable, but with fitting of 1000 or so > problems you cannot be sure that your tails of the bootstrap distributions > are not the local minimums (not to mention extra time and efforts that you > need to invest in bootstrap). Summarizing, we found that NONMEM SEs provide > quick and in most cases reliable information which is rarely corrected by > more computer-intensive techniques. I accept your point that FO may be misleading unless one has taken the time to establish the log-likelihood difference required to define the desired confidence interval e.g. Assessment of Actual Significance Levels for Covariate Effects in NONMEM Whlby U.; Jonsson E.N.; Karlsson M.O. Journal of Pharmacokinetics and Pharmacodynamics, June 2001, vol. 28, no. 3, pp. 231-252(22). Note that I would not use the log-likelihood profile to estimate the parameter SE (which IMHO is of no intrinsic merit) but would use it to define a confidence interval for the parameter. I also accept that bootstrap methods may be impractical for routine use but if the parameter is really critical e.g. Emax during drug development, then resources spent on bootstrap may be more cost-effective than relying on asymptotic SEs with the unlikely assumption that the confidence interval is symmetrical. My comments about SEs not being worth much should be interpreted in the context of the application of using the SE and what value arises from that. For most modelling projects I have been involved in it is very unusual for the the project owner to have demonstrated any value from knowing the SEs.

From:Leonid Gibiansky Subject:Re: [NMusers] T matrix Date:Mon, March 11, 2002 9:11 am Nick, We published these observations (concerning NONMEM, bootstrap, profiling and jackknife SE and CI) in three posters on the recent AAPS meeting: L. Gibiansky, E. Gibiansky, Parameter estimates and confidence intervals for population PK model, American Association of Pharmaceutical Scientists, Annual Meeting, Denver, CO, 2001. E. Gibiansky, L. Gibiansky, S. Bramer, Comparison of NONMEM, bootstrap, jackknife and profiling parameter estimates and confidence intervals for the aripiprazole population PK model, American Association of Pharmaceutical Scientists, Annual Meeting, Denver, CO, 2001. L. Gibiansky, E. Gibiansky, R.Z. Yu, and R.S. Geary, ISIS 2302: Validation of the population PK model and PK/PD analysis, American Association of Pharmaceutical Scientists, Annual Meeting, Denver, CO, 2001. Leonid

From:Nick Holford Subject:Re: [NMusers] T matrix Date:Mon, March 11, 2002 5:56 pm Leonid, Thanks for posting these references and thank you also for sending me separately PPT copies of your poster material. One of the posters (aripiprazole) has a table that allows numerical comparison of the symmetry and coverage produced by 95% confidence intervals derived from log likelihood profiling (LLP) and NONMEM standard errors (SE). Using your own numbers I find that the LLP method in several cases has substantially different coverage and is often asymmetrical. Parameter Sym Coverage q1 CL 84% 92% q2 V 107% 97% q3 Ka 107% 200% q4 CAT1 CL 102% 103% q5 LBW CL 103% 106% q6 AGE_V 92% 138% q7 WT_V 81% 158% q8 omega1 100% 86% q9 omega2 126% 77% q10 omega3 88% 103% q11 sigma 100% 40% Sym=(abs((Hi-Est)/(Lo-Est)) LLP)/(abs((Hi-Est)/(Lo-Est)) SE) Coverage=((Hi-Lo) LLP)/((Hi-Lo) SE) Est=parameter estimate; Lo=95% CI lower; Hi=95% CI upper bound This poster concludes "NONMEM standard errors and CI of the parameter estimates were similar to the ones obtained by more computer-intensive methods.". I would not draw the same conclusion. It depends on what you want to call similar but I don't consider the coverage on KA, AGE_V, WT_V to be similar. The asymmetry of CL and WT_V is also more than trivial. Nick TSR Matrices thread From:Leonid Gibiansky Subject:[NMusers] NONMEM SE and CI Date:Tue, March 12, 2002 9:06 am Nick, Let's put this discussion into the prospective: 1. We wrote "similar", but not identical, so I would not expect a one-to-one agreement. 2. Asymmetry of CL: if you look on the plots, asymmetry of profiling CI is in the different direction compared to the asymmetry of bootstrap CI. If so, it would be hard to argue which of them to trust. 3. Asymmetry of WT_V: yeas, I agree, there is some asymmetry that is not recognizable by NONMEM. But is it really important difference: 0.489-0.953 (profiling), 0.599-0.895 (NONMEM) with the estimate equal to 0.746 ? Is it really "not similar" when you see 20% difference in the symmetry of the confidence interval for the parameter if the covariate model ? 4. Coverage on KA: yes, difference is large here, but bootstrap CI are 1/3 way between the NONMEM CI and profiling CI. So NONMEM is not so bad. 5. Coverage on AGE_V, WT_V: again, there is a difference, but bootstrap CI are between NONMEM and profiling. So again NONMEM is reasonable. Let me also add that for the parameters that describe random effect variances, omega1-omega3, NONMEM CI are approximately half-way between the profiling and bootstrap CI, with a rather significant gap between these "better" methods. For the variance of the error term, sigma, the NONMEM CI coincides with the bootstrap CI whereas the profiling CI are twice more narrow. So if you would be forced to trust the results of just one method, I would trust NONMEM for this particular problem, with the other methods giving similar although not identical results. Another peace of information: this is FOCEI, so bootstrap runs took about 2 weeks of computer time if I remember correctly (500 out of 1000 runs converged). With profiling, there were many-many runs that were interrupted by numerical errors, and we started them again and again with the new initial values (I would estimate, 250 more runs were made). It took a lot of time even with our automated routines. I am not sure that this is an adequate price for the 20% improvement in the CI interval for WT_V, or even 40% improvement that we would get for KA CI. From CI, I can get the following qualitative information: - parameter is well-defined; - parameter is defined but not too well; - the NONMEM converged but the parameter value cannot be trusted. This qualitative information and also quite reliable quantitative information can be readily extracted from the NONMEM SE, and none of the more elaborate techniques will change it. It is surely sufficient during the model development and for most of the final models as well. If one need specific, up to 20-30% precise CI, then bootstrap, profiling, something else, can help sometimes. This may be needed if you plan to use the model for simulations. Even then, posterior predictive checks methods would be a reasonable alternative to the bootstrap or profiling. I am not trying to diminish the significance of the profiling or the bootstrap methods: I routinely did them on my projects, and they bring new more detailed information about the model. My main objection was the comment that NONMEM SE does not worth much. Actually, they allow you immediately and correctly capture "the big picture" and even most of the fine print, with the few details that can be studied, if really needed, by some more elaborate methods. Leonid

From:Nick Holford Subject:Re: [NMusers] NONMEM SE and CI Date:Tue, March 12, 2002 4:15 pm Leonid, Thanks for your remarks about perspective. My perspective is that the NONMEM SEs are not reliable for making CI predictions. When you really want a CI then you do not know if you can trust the CI predicted from SE (e.g. in your aripiprazole example, you would not have known that you could predict CL but not KA). Because of this uncertainty I maintain they are of little worth for making CI predictions *when you really need* this prediction. Remember also that your models were quite straightforward with essentially linear parameters. I would be reluctant to generalize your observations comparing the performance of SE, profiling and bootstrap to other models e.g. when estimating Emax. There is another use of SE that I sometimes hear about. It is claimed that if the $COV step runs then this somehow means that the model is alright. But I have had runs finish with a successful $COV but then re-running with slightly different initial estimates the run terminates or minimizes successfully at a lower obj function value but unable to run $COV. In this case I dont think the initial $COV success is any confirmation of the model. Nick -- Nick Holford, Divn Pharmacology & Clinical Pharmacology University of Auckland, 85 Park Rd, Private Bag 92019, Auckland, New Zealand email:n.holford@auckland.ac.nz tel:+64(9)373-7599x6730 fax:373-7556 http://www.health.auckland.ac.nz/pharmacology/staff/nholford/