Re: Change of NSIG or R matrix
Bob,
NM-Help defines what it means by the SIGDIG estimation option.
SIGDIGITS=n
Number of significant digits required in the final parameter
estimate. SIGDIGITS is not used by the Monte-Carlo methods.
Default: 3. May also be coded NSIGDIGITS.
SIGL=n
n is used to calculate the step-size for finite difference deriv-
atives independent of the SIGDIGITS value. If n=0 or n=100 then
SIGL is ignored and SIGDIGITS is used as in versions prior to
NONMEM 7. SIGL should usually be 2 to 3 times the value of NSIG.
The number of significant digits reported is the
number of significant digits in the least-well-determined element.
The report "MINIMIZATION SUCCESSFUL" is issued when this number is no
less than the number of significant digits requested using the SIGDIG-
ITS option of the $ESTIMATION record.
NONMEM 7 has an additional estimation option (SIGL) that it used to provide additional control for finite-difference derivatives. Unless Bob Bauer can clarify the meaning further I still believe that the meaning of SIGDIG when used as a convergence criterion refers to the number of significant digits in the parameter value. What you describe is more like the meaning of TOL (which is used to control the local error for the DEQ solver).
I have no particular interest in the accuracy of the calculation of the number of significant digits in the parameter estimate but there is a large body of empirical experience using SIGDIG of 3 (or more). Furthermore, NONMEM reports parameter estimates with 3 significant digits which I am prepared to believe more if convergence was achieved with SIGDIG of 3.
As noted many times before on this list there is no empirical evidence to support the idea that the calculation of the asymptotic covariance matrix is associated with more reliable OFV or parameter estimates. What you say may be true as a mathematical description of the variance-covariance matrix properties but it does not mean the OFV and parameter estimates are correlated with this description and there is no evidence to support a correlation that I know of.
Leonid,
Thanks for telling me of your experience with SIGDIG=2 suggesting that this does not change the OFV or parameter estimates at least for some kinds of problem. I think this empirical knowledge is valuable but more examples are needed. Quicker run times because of fewer function evaluations are nice to have but only if the OFV and parameter estimates are not inferior. There are no free lunches so there must be some point where doing less work means that the OFV and parameter estimates are less reliable.
I tried a non-parametric bootstrap of a published model describing tumour growth (Tham LS, Wang L, Soo RA, Lee SC, Lee HS, Yong WP, et al. A pharmacodynamic model for the time course of tumor shrinkage by gemcitabine + carboplatin in non-small cell lung cancer patients. Clin Cancer Res. 2008;14(13):4213-8). The model has 2 differential equations to describe the amount of drug and the size of the tumour. I used the bootstrap average parameter estimates from SIGDIG=3 as the reference and calculated the bias in estimates obtained from SIGDIG=2. The absolute bias ranged from 2.5 to +67% for the fixed effect parameters and 4.7% to 100.8% for the random effects. So I would conclude that perhaps using SIGDIG=2 is problematic in terms of parameter estimates. On the other hand the OFV values were very similar with only 2 cases where the SIGDIG=2 OFV was more than 0.05 units worse (1.001 and 1.34 units worse).
You note that I hold an extreme opinion in the distribution of those who have offered opinions about the importance/non-importance of successful convergence and execution of the $COV step: "But it is safe to mention that Nick's view is on the extreme of the observed distribution of point of views on this subject "
A key point is that my extreme view is supported by experimental data (which has been independently confirmed by Marc Gastonguay and colleagues) and is not based on asymptotically derived speculations :-P.
Nick
Quoted reply history
On 23/10/2013 2:15 a.m., Bob Leary wrote:
> Nick -
> a) The usual definition of 'number of significant digits' is -log10(relative
> precision). Thus a sigdig of 3 is a precision of 1 part in 1000, and a sigdig
> of 2 corresponds to 1% precision, not 10% as in your example.
> b) that being said, the sigdigs in the parameters reported by NONMEM need to be
> taken with a grain of salt - they probably represent best case, 'speed of
> light' type numbers where the real precision may be considerably worse. I do
> not know specifically how they are computed, but my guess is that it is based
> on the fact that in a converged problem, the relative gradient has been driven
> below some specified tolerance. One can then infer precision from the
> condition number of the Hessian of the overall objective function and the
> actual relative gradients. But NONMEM uses a quasi-Newton method -
> there is no Hessian available to the method, but only a stand-in accumulated
> curvature matrix (a 'pseudo Hessian') that is usually much better conditioned
> than the actual Hessian. The only thing that can really be concluded is that,
> at the moment the top level iteration is stopped and convergence declared based
> on the relative gradient, the next iteration , if it were done, would not
> change the parameter estimates by more than the reported sigdig value. This is
> quite a different conclusion than reported parameter estimates are with
> sigdigits of the 'true' values.
>
> c) I know you have often argued that the failure of a covariance step has
> little or no evidential value for determining whether the minimization step was
> 'successful', and I generally agree with you.
> But the failure of the covariance step does mean that the Hessian could not be
> numerically estimated at all (failed the positive definiteness test). This
> does provide some additional evidence that one should be even more skeptical of
> the reported sigdig values of the parameter estimates.
>
> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On
> Behalf Of Nick Holford
> Sent: Tuesday, October 22, 2013 4:45 AM
> To: [email protected]
> Subject: Re: [NMusers] Change of NSIG or R matrix
>
> Xinting,
>
> First of all 'successful minimization' has nothing to do with a good model.
> NONMEM's internal decision to declare success or termination is often a
> pseudo-random choice. If you look at the sigdigs of the estimate you will
> typically find that the lowest value is 2.9 and many others are greater than 5.
> This gives you a clue to which parameters are well determined and which are
> less well known. It is a not a YES/NO decision.
>
> Second, NSIG determines the number of significant digits in the parameter estimates. If
> you choose a number less than 3 then it means you don't care if the answer is 10.1 or
> 10.9. They both have 2 sig digs but the estimates differ by nearly 10%. There is a large
> body of empirical literature that has relied on NSIG=3 (or more). I do not see any reason
> to ignore this in order to get a meaningless "minimization successful" message
> from a random number generator.
>
> I look forward to hearing from "many" to understand why they believe that
> "minimization successful" indicates that the model results are somehow better even though
> the parameter estimates have hardly any significant digits.
>
> Nick
>
> On 22/10/2013 9:17 p.m., Xinting Wang wrote:
>
> > Dear Nick,
> >
> > Thank you very much for your suggestion. Could you explain a little
> > bit about the statement regarding NSIG < 3? I seem to remember that
> > many suggested to use a smaller NSIG to get a successful minimization.
> >
> > Dear Leonid,
> >
> > I read about the recommendation of SIGL, NSIG and TOL, but I am not
> > quite familiar with the use of these options in subroutine ADVAN4. If
> > I set SIGL a fixed value, let's say 12, and NSIG 3, does this mean I
> > also have to identify a value for TOL in $subroutine? I appreciate
> > your help very much.
> >
> > Thank you both.
> >
> > Regards
> >
> > On 8 October 2013 21:59, Leonid Gibiansky <[email protected]
> > <mailto:[email protected]>> wrote:
> >
> > Yes, it should be fine to use S matrix if you cannot get default
> > to run, and use NSIG larger or smaller than default value of 3
> > (although this is not guaranteed, usually NSIG does not change the
> > OF value or parameter estimates in any significant way). Note that
> > Nonmem manual recommends that SIGL >= 3*NSIG, TOL >= SIGL.
> > Separate SIGL can be set on COV step, and it is recommended that
> > SIGL >= 4*NSIG on COV step. In real life I've seen many examples
> > where larger NSIG and SIGL resulted in successful COV step, and
> > also many examples when default values were better (in getting COV
> > step). UNCONDITIONAL on COV step allows you to run COV even when
> > minimization ended with some error.
> >
> > Contrary to Nick's experience, I found that COV step is useful as
> > it reveals which of the model parameters are poorly estimated, and
> > that CI based on SE are usually quite good and are in a general
> > agreement with the bootstrap CI, but it may depend on the problem.
> >
> > Leonid
> >
> > --------------------------------------
> > Leonid Gibiansky, Ph.D.
> > President, QuantPharm LLC
> > web: www.quantpharm.com http://www.quantpharm.com
> > e-mail: LGibiansky at quantpharm.com http://quantpharm.com
> > tel: (301) 767 5566 <tel:%28301%29%20767%205566>
> >
> > On 10/8/2013 3:57 AM, Xinting Wang wrote:
> >
> > Dear all,
> >
> > I have a naive question regarding the modeling building process in
> > NONMEM. With more and more covariates added in the model, I
> > often come
> > across an error message saying that "ERROR 134", or R MATRIX
> > SINGULAR.
> >
> > After searching from the internet, I learned that changing NSIG in
> > $ESTIMATION and MATRIX=S in $COV would be helpful for both
> > problems
> > respectively. And from my own experience, it dose help with
> > the modeling
> > building.
> >
> > However, my concern is, I used different NSIG and MATRIX in
> > the previous
> > steps. Is it proper to use different NSIGs and MATRICE in a
> > single model
> > building? If not, could you please explain this a little bit?
> >
> > Thank you in advance!
> >
> > Best Regards
> > --
> > Xinting
> > Wang
> >
> > --
> > Xinting
>
> --
> Nick Holford, Professor Clinical Pharmacology Dept Pharmacology & Clinical
> Pharmacology, Bldg 503 Room 302A University of Auckland,85 Park Rd,Private Bag
> 92019,Auckland,New Zealand
> office:+64(9)923-6730 mobile:NZ +64(21)46 23 53
> email: [email protected]
> http://holford.fmhs.auckland.ac.nz/
>
> Holford NHG. Disease progression and neuroscience. Journal of Pharmacokinetics
> and Pharmacodynamics. 2013;40:369-76
> http://link.springer.com/article/10.1007/s10928-013-9316-2
> Holford N, Heo Y-A, Anderson B. A pharmacokinetic standard for babies and
> adults. J Pharm Sci. 2013:
> http://onlinelibrary.wiley.com/doi/10.1002/jps.23574/abstract
> Holford N. A time to event tutorial for pharmacometricians. CPT:PSP. 2013;2:
> http://www.nature.com/psp/journal/v2/n5/full/psp201318a.html
> Holford NHG. Clinical pharmacology = disease progression + drug action. British
> Journal of Clinical Pharmacology. 2013:
> http://onlinelibrary.wiley.com/doi/10.1111/bcp.12170/abstract
>
>