Dear Nmusers,
we have very rich data from MRI concentration measurements, with 11
compartments and multiple compartments observed. The model is fit via SAEM
(nburn=2000), and followed by an IMPMAP as in the described in the 7.1.2
manual. OMEGA is band with pair-wise block correlations in the following
style:
$OMEGA BLOCK(2)
.02 ;CL
0.01 0.06 ; VC
$OMEGA BLOCK(2)
5.4 ; QMVP
0.001 0.05 ;VMVP
$OMEGA BLOCK(2)
0.06 ; QTVP
0.001 0.25 ;VTPV
$EST PRINT=1 METHOD=SAEM INTERACTION NBURN=2000 NITER=200 CTYPE=2 NSIG=2
FILE=SAEM.EXT
$EST METHOD=IMPMAP EONLY = 1 INTERACTION ISAMPLE=1000 NITER=5 FILE=IMP.EXT
$COV PRINT=E UNCONDITIONAL
Fits and CWRES diagnostics are perfect, and VPC checks are good.
However, we have negative eigenvalues (the following example has been edited
by removing digits)
ETAPval = 0.2 0.2 0.3 0.04 0.8 0.95 0.003 0.1 0.6 0.4 0.9 0.1 0.5 0.4 0.2
0.8 0.3 0.3 0.4 0.01 0.8
ETAshr% = 13. 0.4 38 20 23 33 46 30 18 41 54 22 2. 26. 49. 12. 0.07 24. 18.
35. 2.5
EPSshr% = 7.5 8.1
Number of Negative Eigenvalues in Matrix= 7
Most negative value= -65339.
Most positive value= 88796185.9
Forcing positive definiteness
Root mean square deviation of matrix from original= 1.37E-003
My question: can we trust this fit?
Dieter Menne
Menne Biomed/University Hospital of Zürich
How serious are negative eigenvalues?
Dieter:
You can trust the fit. The negative eignvalue diagnostic arises from
evaluating the information matrix of the estimates evaluated after the fit.
Because this was constructed with Monte Carlo components, on occasion the
slight imprecision from calculating obscure off-diagonal elements results in a
matrix that is not positive definite. An algorithm is used to shift the
eigenvalues slightly, and the following diagnostic
Root mean square deviation of matrix from original= 1.37E-003
tells you how far the non-adjusted information matrix, multiplied by the
positive definite adjusted inverse matrix, differs from the Identity matrix.
Since this differs from the identity matrix only by 0.137% in your case, the
reported variance-covariance matrix, which is now positive definite, still
serves as a reasonable representation of the inverse of the information matrix,
despite the positive definiteness adjustment required. You can also visually
inspect the standard errors to see if these are reasonably sized.
Alternatively, you can repeat the step:
$EST METHOD=IMPMAP EONLY = 1 INTERACTION ISAMPLE=1000 NITER=5 FILE=IMP.EXT
And because of slight Monte Carlo fluctuations is likely to give you a slightly
different result each time. If your root mean square deviation varies, or on
occasion you obtain a result without requiring the positive definiteness
adjustment, then this is another indicator that the negative eigenvalue result
is only the result of Monte Carlo fluctuations.
Another method is to increase ISAMPLE to 3000, to reduce Monte Carlo
fluctuations.
Robert J. Bauer, Ph.D.
Vice President, Pharmacometrics
ICON Development Solutions
Tel: (215) 616-6428
Mob: (925) 286-0769
Email: [email protected]
Web: www.icondevsolutions.com
Quoted reply history
-----Original Message-----
From: [email protected] [mailto:[email protected]] On
Behalf Of Dieter Menne
Sent: Monday, September 06, 2010 12:11 PM
To: [email protected]
Subject: [NMusers] How serious are negative eigenvalues?
Dear Nmusers,
we have very rich data from MRI concentration measurements, with 11
compartments and multiple compartments observed. The model is fit via SAEM
(nburn=2000), and followed by an IMPMAP as in the described in the 7.1.2
manual. OMEGA is band with pair-wise block correlations in the following
style:
$OMEGA BLOCK(2)
.02 ;CL
0.01 0.06 ; VC
$OMEGA BLOCK(2)
5.4 ; QMVP
0.001 0.05 ;VMVP
$OMEGA BLOCK(2)
0.06 ; QTVP
0.001 0.25 ;VTPV
$EST PRINT=1 METHOD=SAEM INTERACTION NBURN=2000 NITER=200 CTYPE=2 NSIG=2
FILE=SAEM.EXT
$EST METHOD=IMPMAP EONLY = 1 INTERACTION ISAMPLE=1000 NITER=5 FILE=IMP.EXT
$COV PRINT=E UNCONDITIONAL
Fits and CWRES diagnostics are perfect, and VPC checks are good.
However, we have negative eigenvalues (the following example has been edited
by removing digits)
ETAPval = 0.2 0.2 0.3 0.04 0.8 0.95 0.003 0.1 0.6 0.4 0.9 0.1 0.5 0.4 0.2
0.8 0.3 0.3 0.4 0.01 0.8
ETAshr% = 13. 0.4 38 20 23 33 46 30 18 41 54 22 2. 26. 49. 12. 0.07 24. 18.
35. 2.5
EPSshr% = 7.5 8.1
Number of Negative Eigenvalues in Matrix= 7
Most negative value= -65339.
Most positive value= 88796185.9
Forcing positive definiteness
Root mean square deviation of matrix from original= 1.37E-003
My question: can we trust this fit?
Dieter Menne
Menne Biomed/University Hospital of Zürich
I think, if at least 7 eigenvalues are nearly zero (up to the numerical precision) it means that the model is greatly over parametrized. While one can trust the model predictions, one may need to investigate whether to trust the model parameter estimates. Results indicate that there is a 7 (or more) -dimensional sub-space of parameters where any point provides the same predictions. If you printed out iteration history, you may plot scatter plot matrix of parameters versus parameters to see which of the parameters appear to be strongly correlated. Parameter estimates of the correlated parameters cannot be trusted.
Leonid
--------------------------------------
Leonid Gibiansky, Ph.D.
President, QuantPharm LLC
web: www.quantpharm.com
e-mail: LGibiansky at quantpharm.com
tel: (301) 767 5566
Quoted reply history
On 9/6/2010 1:27 PM, Bauer, Robert wrote:
> Dieter:
> You can trust the fit. The negative eignvalue diagnostic arises from
> evaluating the information matrix of the estimates evaluated after the fit.
> Because this was constructed with Monte Carlo components, on occasion the
> slight imprecision from calculating obscure off-diagonal elements results in a
> matrix that is not positive definite. An algorithm is used to shift the
> eigenvalues slightly, and the following diagnostic
>
> Root mean square deviation of matrix from original= 1.37E-003
>
> tells you how far the non-adjusted information matrix, multiplied by the
> positive definite adjusted inverse matrix, differs from the Identity matrix.
> Since this differs from the identity matrix only by 0.137% in your case, the
> reported variance-covariance matrix, which is now positive definite, still
> serves as a reasonable representation of the inverse of the information matrix,
> despite the positive definiteness adjustment required. You can also visually
> inspect the standard errors to see if these are reasonably sized.
>
> Alternatively, you can repeat the step:
>
> $EST METHOD=IMPMAP EONLY = 1 INTERACTION ISAMPLE=1000 NITER=5 FILE=IMP.EXT
>
> And because of slight Monte Carlo fluctuations is likely to give you a slightly
> different result each time. If your root mean square deviation varies, or on
> occasion you obtain a result without requiring the positive definiteness
> adjustment, then this is another indicator that the negative eigenvalue result
> is only the result of Monte Carlo fluctuations.
>
> Another method is to increase ISAMPLE to 3000, to reduce Monte Carlo
> fluctuations.
>
> Robert J. Bauer, Ph.D.
> Vice President, Pharmacometrics
> ICON Development Solutions
> Tel: (215) 616-6428
> Mob: (925) 286-0769
> Email: [email protected]
> Web: www.icondevsolutions.com
>
> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On
> Behalf Of Dieter Menne
> Sent: Monday, September 06, 2010 12:11 PM
> To: [email protected]
> Subject: [NMusers] How serious are negative eigenvalues?
>
> Dear Nmusers,
>
> we have very rich data from MRI concentration measurements, with 11
> compartments and multiple compartments observed. The model is fit via SAEM
> (nburn=2000), and followed by an IMPMAP as in the described in the 7.1.2
> manual. OMEGA is band with pair-wise block correlations in the following
> style:
>
> $OMEGA BLOCK(2)
> .02 ;CL
> 0.01 0.06 ; VC
> $OMEGA BLOCK(2)
> 5.4 ; QMVP
> 0.001 0.05 ;VMVP
> $OMEGA BLOCK(2)
> 0.06 ; QTVP
> 0.001 0.25 ;VTPV
>
> $EST PRINT=1 METHOD=SAEM INTERACTION NBURN=2000 NITER=200 CTYPE=2 NSIG=2
> FILE=SAEM.EXT
> $EST METHOD=IMPMAP EONLY = 1 INTERACTION ISAMPLE=1000 NITER=5 FILE=IMP.EXT
> $COV PRINT=E UNCONDITIONAL
>
> Fits and CWRES diagnostics are perfect, and VPC checks are good.
>
> However, we have negative eigenvalues (the following example has been edited
> by removing digits)
>
> ETAPval = 0.2 0.2 0.3 0.04 0.8 0.95 0.003 0.1 0.6 0.4 0.9 0.1 0.5 0.4 0.2
> 0.8 0.3 0.3 0.4 0.01 0.8
> ETAshr% = 13. 0.4 38 20 23 33 46 30 18 41 54 22 2. 26. 49. 12. 0.07 24. 18.
> 35. 2.5
> EPSshr% = 7.5 8.1
> Number of Negative Eigenvalues in Matrix= 7
> Most negative value= -65339.
> Most positive value= 88796185.9
> Forcing positive definiteness
> Root mean square deviation of matrix from original= 1.37E-003
>
> My question: can we trust this fit?
>
> Dieter Menne
> Menne Biomed/University Hospital of Zürich
>
>
Dieter,
You ask:
> My question: can we trust this fit?
The answer depends on why you are doing the modelling.
If your goal is to describe the time course of concentrations then the overall ability of the model to describe what you saw depends on the totality of the model and its parameters. The model may be overparameterized but it may still do what you want it to do i.e. describe (and predict) the time course of concentrations in each compartment. If you are satisfied with the VPC showing that simulations from the model appropriately describe the observed concentrations then I think the answer to your question is yes.
Quoted reply history
On the other hand if the goal is to estimate the size of one or more critical parameters then you will need to pay attention to how well these parameters are estimated. As Leonid has pointed out it seems that at least some of the model parameters are not well identified. This may be unimportant if the parameters you want to describe are robustly estimated.
For example, if you had a simple PK model with samples mainly taken at steady state with few observations during absorption then you may get a good estimate of clearance but a rather poor estimate of KA. You cannot simply remove a parameter such as KA (you have to describe the sparse absorption somehow) but it will have little impact on the clearance estimate. Thus the model can be trusted for the purpose of estimating clearance but not absorption rate.
Nick
On 7/09/2010 12:11 a.m., Dieter Menne wrote:
> Dear Nmusers,
>
> we have very rich data from MRI concentration measurements, with 11
> compartments and multiple compartments observed. The model is fit via SAEM
> (nburn=2000), and followed by an IMPMAP as in the described in the 7.1.2
> manual. OMEGA is band with pair-wise block correlations in the following
> style:
>
> $OMEGA BLOCK(2)
> .02 ;CL
> 0.01 0.06 ; VC
> $OMEGA BLOCK(2)
> 5.4 ; QMVP
> 0.001 0.05 ;VMVP
> $OMEGA BLOCK(2)
> 0.06 ; QTVP
> 0.001 0.25 ;VTPV
>
> $EST PRINT=1 METHOD=SAEM INTERACTION NBURN=2000 NITER=200 CTYPE=2 NSIG=2
> FILE=SAEM.EXT
> $EST METHOD=IMPMAP EONLY = 1 INTERACTION ISAMPLE=1000 NITER=5 FILE=IMP.EXT
> $COV PRINT=E UNCONDITIONAL
>
> Fits and CWRES diagnostics are perfect, and VPC checks are good.
>
> However, we have negative eigenvalues (the following example has been edited
> by removing digits)
>
> ETAPval = 0.2 0.2 0.3 0.04 0.8 0.95 0.003 0.1 0.6 0.4 0.9 0.1 0.5 0.4 0.2
> 0.8 0.3 0.3 0.4 0.01 0.8
> ETAshr% = 13. 0.4 38 20 23 33 46 30 18 41 54 22 2. 26. 49. 12. 0.07 24. 18.
> 35. 2.5
> EPSshr% = 7.5 8.1
> Number of Negative Eigenvalues in Matrix= 7
> Most negative value= -65339.
> Most positive value= 88796185.9
> Forcing positive definiteness
> Root mean square deviation of matrix from original= 1.37E-003
>
> My question: can we trust this fit?
>
> Dieter Menne
> Menne Biomed/University Hospital of Zürich
--
Nick Holford, Professor Clinical Pharmacology
Dept Pharmacology& Clinical Pharmacology
University of Auckland,85 Park Rd,Private Bag 92019,Auckland,New Zealand
tel:+64(9)923-6730 fax:+64(9)373-7090 mobile:+64(21)46 23 53
email: [email protected]
http://www.fmhs.auckland.ac.nz/sms/pharmacology/holford
Thanks for your comments, Leonid.
To paraphrase your main argument: 7 negative eigenvalues mean 7 values close
to zero, so we have a highly over-parameterized system. While I fear it's
correct (I could not get untrendy CWRES otherwise), let's take Robert's
argument to the extreme:
Simplified, from an SAEM fit we have
Most negative value= -6.
Most positive value= 8879
I used the original number divided by 10000 for easier reading. As Robert
argues, the negative value is the result of the statistical approach. So by
a quirk of the procedure, the mode of the distribution could be 0.01, 0.1,
but also +10. If the latter value were true, we would be close to a
reasonably conditioned matrix.
Is this argument valid, just in theory?
I know it's not valid for my data, because I run several repeats this night
and things tend to the worse.
Dieter
(working together with Andreas Steingötter, Zürich, and Rickmer Braren,
Munich)