Dear NONMEM Users
Does anyone have a view on the relative merits/reliability/accuracy of
NONMEM ($COV step) vs SPSS (NLR) with respect to their derived values of the
parameter standard errors and parameter correlation matrices?
The data I am analysing are single subject (not population). Parameter
estimates from the two programs are, to all intents and purposes, identical.
However, the SE values from NONMEM $COV are consistently smaller by
1.5-2.0-fold.
Any thoughts?
Gavin
__________________________________________________
Dr Gavin E Jarvis MA PhD VetMB MRCVS
University Lecturer in Veterinary Anatomy
Department of Physiology, Development & Neuroscience
Physiological Laboratory
Downing Street
Cambridge
CB2 3EG
Tel: +44 (0) 1223 333745
Fellow and College Lecturer in Pharmacology
Selwyn College
Cambridge
CB3 9DQ
Tel: +44 (0) 1223 761303
Email: <mailto:[email protected]> [email protected]
Web: http://www.pdn.cam.ac.uk/staff/jarvis www.pdn.cam.ac.uk/staff/jarvis
Twit: @GavinEJarvis
NONMEM vs SPSS
Dear Gavin,
Perhaps an idea is to compare the different MATRIX options of the covariance
step of NONMEM, and with the bootstrap, to assess their relative properties.
Erik
Quoted reply history
________________________________
From: [email protected] [[email protected]] on behalf of
Gavin Jarvis [[email protected]]
Sent: Saturday, March 29, 2014 5:55 PM
To: [email protected]
Subject: [NMusers] NONMEM vs SPSS
Dear NONMEM Users
Does anyone have a view on the relative merits/reliability/accuracy of NONMEM
($COV step) vs SPSS (NLR) with respect to their derived values of the parameter
standard errors and parameter correlation matrices?
The data I am analysing are single subject (not population). Parameter
estimates from the two programs are, to all intents and purposes, identical.
However, the SE values from NONMEM $COV are consistently smaller by
1.5-2.0-fold.
Any thoughts?
Gavin
__________________________________________________
Dr Gavin E Jarvis MA PhD VetMB MRCVS
University Lecturer in Veterinary Anatomy
Department of Physiology, Development & Neuroscience
Physiological Laboratory
Downing Street
Cambridge
CB2 3EG
Tel: +44 (0) 1223 333745
Fellow and College Lecturer in Pharmacology
Selwyn College
Cambridge
CB3 9DQ
Tel: +44 (0) 1223 761303
Email: [email protected]<mailto:[email protected]>
Web: http://www.pdn.cam.ac.uk/staff/jarvis
Twit: @GavinEJarvis
Gavin,
NONMEM has been noted (Senn et al 2012) to produce smaller SE (R-1 S R-1 method) compared to estimates from Mathcad, SAS, GenStat and R. The Mathcad estimates were identical to SAS, Genstat and R when using numerical derivatives and larger when based on the expected Fisher information matrix. So there is clearly some kind of between and within program standard error describing the uncertainty in standard error estimates obtained from different methods.
Comparisons of NONMEM asymptotic SE to those estimated by non-parametric bootstraps have shown they are often similar for relatively 'linear' (in the mathematical not PK sense) models but are different (usually smaller) as the model becomes more 'non-linear'. The 95% confidence intervals also tend to become asymmetrical which invalidates the use of an asymptotic SE for calculation of confidence intervals.
I prefer to use bootstrap derived confidence intervals to describe uncertainty of parameter estimates. The use of the SE is the standard error of traditional approaches.
Best wishes,
Nick
Senn, S., et al. (2012). "The ghosts of departed quantities: approaches to dealing with observations below the limit of quantitation." Stat Med 31(30): 4280-4295.
Quoted reply history
On 30/03/2014 5:55 a.m., Gavin Jarvis wrote:
> Dear NONMEM Users
>
> Does anyone have a view on the relative merits/reliability/accuracy of NONMEM ($COV step) vs SPSS (NLR) with respect to their derived values of the parameter standard errors and parameter correlation matrices?
>
> The data I am analysing are single subject (not population). Parameter estimates from the two programs are, to all intents and purposes, identical. However, the SE values from NONMEM $COV are consistently smaller by 1.5-2.0-fold.
>
> Any thoughts?
>
> Gavin
>
> __________________________________________________
>
> *Dr Gavin E Jarvis MA PhD VetMB MRCVS*
>
> University Lecturer in Veterinary Anatomy
>
> Department of Physiology, Development & Neuroscience
>
> Physiological Laboratory
>
> Downing Street
>
> Cambridge
>
> CB2 3EG
>
> Tel: +44 (0) 1223 333745
>
> Fellow and College Lecturer in Pharmacology
>
> Selwyn College
>
> Cambridge
>
> CB3 9DQ
>
> Tel: +44 (0) 1223 761303
>
> Email: [email protected] <mailto:[email protected]>
>
> Web: www.pdn.cam.ac.uk/staff/jarvis < http://www.pdn.cam.ac.uk/staff/jarvis >
>
> Twit: @GavinEJarvis
--
Nick Holford, Professor Clinical Pharmacology
Dept Pharmacology & Clinical Pharmacology, Bldg 503 Room 302A
University of Auckland,85 Park Rd,Private Bag 92019,Auckland,New Zealand
office:+64(9)923-6730 mobile:NZ +64(21)46 23 53
email: [email protected]
http://holford.fmhs.auckland.ac.nz/
McCune JS, Bemer MJ, Barrett JS, Scott Baker K, Gamis AS and Holford NH (2014)
Busulfan in infant to adult hematopoietic cell transplant recipients: a
population pharmacokinetic model for initial and bayesian dose personalization.
Clin Cancer Res 20:754-763.
Størset E, Holford N, Hennig S, Bergmann TK, Bergan S, Bremer S, Åsberg A,
Midtvedt K and Staatz CE (2014) Improved prediction of tacrolimus
concentrations early after kidney transplantation using theory-based
pharmacokinetic modelling. Br J Clin Pharmacol Accepted online 20 Feb 2014
DOI:10.1111/bcp.12361
Dear Gavin,
This is most likely because most nonlinear regression programs invert the
Hessian (second derivative matrix of the model with respect to the
parameters) to obtain the covariance matrix. This corresponds to the R
matrix in NONMEM. However, the default method that NONMEM uses is a
sandwich estimator involving both the Hessian (R) and the square of the
first derivatives matrix (S). I suspect that if you use the MATRIX=R option
on the $COV step you will find that the standard errors will now be in
agreement with SPSS (NLR). I know Stu Beal made the sandwich estimator the
default as it is supposed to be more robust to non-normality but I would
have preferred the MATRIX=R option to be the default to be more consistent
with other nonlinear regression software implementations.
Ken
Quoted reply history
From: [email protected] [mailto:[email protected]] On
Behalf Of Gavin Jarvis
Sent: Saturday, March 29, 2014 12:55 PM
To: [email protected]
Subject: [NMusers] NONMEM vs SPSS
Dear NONMEM Users
Does anyone have a view on the relative merits/reliability/accuracy of
NONMEM ($COV step) vs SPSS (NLR) with respect to their derived values of the
parameter standard errors and parameter correlation matrices?
The data I am analysing are single subject (not population). Parameter
estimates from the two programs are, to all intents and purposes, identical.
However, the SE values from NONMEM $COV are consistently smaller by
1.5-2.0-fold.
Any thoughts?
Gavin
__________________________________________________
Dr Gavin E Jarvis MA PhD VetMB MRCVS
University Lecturer in Veterinary Anatomy
Department of Physiology, Development & Neuroscience
Physiological Laboratory
Downing Street
Cambridge
CB2 3EG
Tel: +44 (0) 1223 333745
Fellow and College Lecturer in Pharmacology
Selwyn College
Cambridge
CB3 9DQ
Tel: +44 (0) 1223 761303
Email: <mailto:[email protected]> [email protected]
Web: http://www.pdn.cam.ac.uk/staff/jarvis www.pdn.cam.ac.uk/staff/jarvis
Twit: @GavinEJarvis
I concur with Ken's statement, and I also prefer to use MATRIX=R as the first
choice for covariance assessment. On occasion, MATRIX=S can be used if there
are numerical difficulties in assessing the R matrix, and if there are enough
subjects relative to the dimension size (number of total parameters estimated)
of the variance-covariance matrix to be estimated.
Robert J. Bauer, Ph.D.
Vice President, Pharmacometrics, R&D
ICON Development Solutions
7740 Milestone Parkway
Suite 150
Hanover, MD 21076
Tel: (215) 616-6428
Mob: (925) 286-0769
Email: [email protected]<mailto:[email protected]>
Web: http://www.iconplc.com/
Quoted reply history
From: [email protected] [mailto:[email protected]] On
Behalf Of Ken Kowalski
Sent: Saturday, March 29, 2014 3:44 PM
To: 'Gavin Jarvis'; [email protected]
Subject: RE: [NMusers] NONMEM vs SPSS
Dear Gavin,
This is most likely because most nonlinear regression programs invert the
Hessian (second derivative matrix of the model with respect to the parameters)
to obtain the covariance matrix. This corresponds to the R matrix in NONMEM.
However, the default method that NONMEM uses is a sandwich estimator involving
both the Hessian (R) and the square of the first derivatives matrix (S). I
suspect that if you use the MATRIX=R option on the $COV step you will find that
the standard errors will now be in agreement with SPSS (NLR). I know Stu Beal
made the sandwich estimator the default as it is supposed to be more robust to
non-normality but I would have preferred the MATRIX=R option to be the default
to be more consistent with other nonlinear regression software implementations.
Ken
From: [email protected]<mailto:[email protected]>
[mailto:[email protected]] On Behalf Of Gavin Jarvis
Sent: Saturday, March 29, 2014 12:55 PM
To: [email protected]<mailto:[email protected]>
Subject: [NMusers] NONMEM vs SPSS
Dear NONMEM Users
Does anyone have a view on the relative merits/reliability/accuracy of NONMEM
($COV step) vs SPSS (NLR) with respect to their derived values of the parameter
standard errors and parameter correlation matrices?
The data I am analysing are single subject (not population). Parameter
estimates from the two programs are, to all intents and purposes, identical.
However, the SE values from NONMEM $COV are consistently smaller by
1.5-2.0-fold.
Any thoughts?
Gavin
__________________________________________________
Dr Gavin E Jarvis MA PhD VetMB MRCVS
University Lecturer in Veterinary Anatomy
Department of Physiology, Development & Neuroscience
Physiological Laboratory
Downing Street
Cambridge
CB2 3EG
Tel: +44 (0) 1223 333745
Fellow and College Lecturer in Pharmacology
Selwyn College
Cambridge
CB3 9DQ
Tel: +44 (0) 1223 761303
Email: [email protected]<mailto:[email protected]>
Web: http://www.pdn.cam.ac.uk/staff/jarvis
Twit: @GavinEJarvis
Thanks for the discussion! Some literature indeed would suggest not too focus
too much on the sandwich estimator
( http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.8.5787&rep=rep1&type=pdf)
certainly not in situations where the sample size is ‘small’ and when leverage
points are encountered. The notions about bias
( http://www.stat.berkeley.edu/~census/mlesan.pdf) may be even more important
and the suggestion by http://www.polmeth.wustl.edu/media/Paper/robust.pdf seems
worthwhile to try (I at least never did – anyone?): they suggest to compare the
sandwich estimator with the Hessian (classical) one. A difference between the
two could be used as a diagnostic pointing to some model misspecification.
Jeroen
Quoted reply history
From: [email protected] [mailto:[email protected]] On
Behalf Of Bauer, Robert
Sent: Saturday, March 29, 2014 21:46
To: Kowalski, Ken; 'Gavin Jarvis'; [email protected]
Subject: RE: [NMusers] NONMEM vs SPSS
I concur with Ken’s statement, and I also prefer to use MATRIX=R as the first
choice for covariance assessment. On occasion, MATRIX=S can be used if there
are numerical difficulties in assessing the R matrix, and if there are enough
subjects relative to the dimension size (number of total parameters estimated)
of the variance-covariance matrix to be estimated.
Robert J. Bauer, Ph.D.
Vice President, Pharmacometrics, R&D
ICON Development Solutions
7740 Milestone Parkway
Suite 150
Hanover, MD 21076
Tel: (215) 616-6428
Mob: (925) 286-0769
Email: [email protected]<mailto:[email protected]>
Web: http://www.iconplc.com/
From: [email protected]<mailto:[email protected]>
[mailto:[email protected]] On Behalf Of Ken Kowalski
Sent: Saturday, March 29, 2014 3:44 PM
To: 'Gavin Jarvis'; [email protected]<mailto:[email protected]>
Subject: RE: [NMusers] NONMEM vs SPSS
Dear Gavin,
This is most likely because most nonlinear regression programs invert the
Hessian (second derivative matrix of the model with respect to the parameters)
to obtain the covariance matrix. This corresponds to the R matrix in NONMEM.
However, the default method that NONMEM uses is a sandwich estimator involving
both the Hessian (R) and the square of the first derivatives matrix (S). I
suspect that if you use the MATRIX=R option on the $COV step you will find that
the standard errors will now be in agreement with SPSS (NLR). I know Stu Beal
made the sandwich estimator the default as it is supposed to be more robust to
non-normality but I would have preferred the MATRIX=R option to be the default
to be more consistent with other nonlinear regression software implementations.
Ken
From: [email protected]<mailto:[email protected]>
[mailto:[email protected]] On Behalf Of Gavin Jarvis
Sent: Saturday, March 29, 2014 12:55 PM
To: [email protected]<mailto:[email protected]>
Subject: [NMusers] NONMEM vs SPSS
Dear NONMEM Users
Does anyone have a view on the relative merits/reliability/accuracy of NONMEM
($COV step) vs SPSS (NLR) with respect to their derived values of the parameter
standard errors and parameter correlation matrices?
The data I am analysing are single subject (not population). Parameter
estimates from the two programs are, to all intents and purposes, identical.
However, the SE values from NONMEM $COV are consistently smaller by
1.5-2.0-fold.
Any thoughts?
Gavin
__________________________________________________
Dr Gavin E Jarvis MA PhD VetMB MRCVS
University Lecturer in Veterinary Anatomy
Department of Physiology, Development & Neuroscience
Physiological Laboratory
Downing Street
Cambridge
CB2 3EG
Tel: +44 (0) 1223 333745
Fellow and College Lecturer in Pharmacology
Selwyn College
Cambridge
CB3 9DQ
Tel: +44 (0) 1223 761303
Email: [email protected]<mailto:[email protected]>
Web: http://www.pdn.cam.ac.uk/staff/jarvis
Twit: @GavinEJarvis
Dear All
Thank you for all the very helpful comments.
In reply:
1. MATRIX=R does make the standard error and correlation values much more
similar to SPSS(NLR)
2. The residual error model is additive, homoscedastic (just ETA(1)). The
data are extremely tight (R^2 >99.9%) – almost perfect! The purpose of my
analysis is to assess structural models for analysing asymmetric dose-response
curves. The problem is that some models produces parameters that lose empirical
meaning and are very highly correlated.
3. I tried the bootstrap option using WFN. However, the parameter
estimates all came out identical – probably because the data is so tight – this
makes it tricky to evaluate standard errors!
Gavin
Quoted reply history
From: Bauer, Robert [mailto:[email protected]]
Sent: 29 March 2014 20:46
To: Ken Kowalski; 'Gavin Jarvis'; [email protected]
Subject: RE: [NMusers] NONMEM vs SPSS
I concur with Ken’s statement, and I also prefer to use MATRIX=R as the first
choice for covariance assessment. On occasion, MATRIX=S can be used if there
are numerical difficulties in assessing the R matrix, and if there are enough
subjects relative to the dimension size (number of total parameters estimated)
of the variance-covariance matrix to be estimated.
Robert J. Bauer, Ph.D.
Vice President, Pharmacometrics, R&D
ICON Development Solutions
7740 Milestone Parkway
Suite 150
Hanover, MD 21076
Tel: (215) 616-6428
Mob: (925) 286-0769
Email: [email protected]
Web: www.iconplc.com http://www.iconplc.com/
From: [email protected] [mailto:[email protected]] On
Behalf Of Ken Kowalski
Sent: Saturday, March 29, 2014 3:44 PM
To: 'Gavin Jarvis'; [email protected]
Subject: RE: [NMusers] NONMEM vs SPSS
Dear Gavin,
This is most likely because most nonlinear regression programs invert the
Hessian (second derivative matrix of the model with respect to the parameters)
to obtain the covariance matrix. This corresponds to the R matrix in NONMEM.
However, the default method that NONMEM uses is a sandwich estimator involving
both the Hessian (R) and the square of the first derivatives matrix (S). I
suspect that if you use the MATRIX=R option on the $COV step you will find that
the standard errors will now be in agreement with SPSS (NLR). I know Stu Beal
made the sandwich estimator the default as it is supposed to be more robust to
non-normality but I would have preferred the MATRIX=R option to be the default
to be more consistent with other nonlinear regression software implementations.
Ken
From: [email protected] [mailto:[email protected]] On
Behalf Of Gavin Jarvis
Sent: Saturday, March 29, 2014 12:55 PM
To: [email protected]
Subject: [NMusers] NONMEM vs SPSS
Dear NONMEM Users
Does anyone have a view on the relative merits/reliability/accuracy of NONMEM
($COV step) vs SPSS (NLR) with respect to their derived values of the parameter
standard errors and parameter correlation matrices?
The data I am analysing are single subject (not population). Parameter
estimates from the two programs are, to all intents and purposes, identical.
However, the SE values from NONMEM $COV are consistently smaller by
1.5-2.0-fold.
Any thoughts?
Gavin
__________________________________________________
Dr Gavin E Jarvis MA PhD VetMB MRCVS
University Lecturer in Veterinary Anatomy
Department of Physiology, Development & Neuroscience
Physiological Laboratory
Downing Street
Cambridge
CB2 3EG
Tel: +44 (0) 1223 333745
Fellow and College Lecturer in Pharmacology
Selwyn College
Cambridge
CB3 9DQ
Tel: +44 (0) 1223 761303
Email: <mailto:[email protected]> [email protected]
Web: http://www.pdn.cam.ac.uk/staff/jarvis www.pdn.cam.ac.uk/staff/jarvis
Twit: @GavinEJarvis
Hi Jeroen,
The BOOTSTRAP option of $SIMULATION gives different results when N=1 (each
measurement is treated as having a different ID). Could that perhaps be useful?
Erik
Quoted reply history
________________________________
From: [email protected] [[email protected]] on behalf of
Elassaiss - Schaap, J (Jeroen) [[email protected]]
Sent: Monday, March 31, 2014 12:23 PM
To: Gavin Jarvis; [email protected]
Subject: RE: [NMusers] NONMEM vs SPSS
Dear Gavin,
Reading back your original post, if your data are really N=1 and you have this
perfect fit phenomenon there is probably little value in reporting the SEs. But
on the other hand your new reply suggests that you are doing a simulation
exercise… in which case a regression on aggregated data may be less productive.
Perhaps you could consider doing an analysis with SSE (psn.sf.net, not sure
whether WfN has similar tools) to figure out which design would support the
models you are considering with little additional effort.
Jeroen
PS: bootstrap on N=1 does not work, the nonmem approaches use all sampling over
subjects. (There are other ways of doing a bootstrap)
From: [email protected] [mailto:[email protected]] On
Behalf Of Gavin Jarvis
Sent: Monday, March 31, 2014 11:33
To: [email protected]
Subject: RE: [NMusers] NONMEM vs SPSS
Dear All
Thank you for all the very helpful comments.
In reply:
1. MATRIX=R does make the standard error and correlation values much more
similar to SPSS(NLR)
2. The residual error model is additive, homoscedastic (just ETA(1)). The
data are extremely tight (R^2 >99.9%) – almost perfect! The purpose of my
analysis is to assess structural models for analysing asymmetric dose-response
curves. The problem is that some models produces parameters that lose empirical
meaning and are very highly correlated.
3. I tried the bootstrap option using WFN. However, the parameter
estimates all came out identical – probably because the data is so tight – this
makes it tricky to evaluate standard errors!
Gavin
From: Bauer, Robert [mailto:[email protected]]
Sent: 29 March 2014 20:46
To: Ken Kowalski; 'Gavin Jarvis';
[email protected]<mailto:[email protected]>
Subject: RE: [NMusers] NONMEM vs SPSS
I concur with Ken’s statement, and I also prefer to use MATRIX=R as the first
choice for covariance assessment. On occasion, MATRIX=S can be used if there
are numerical difficulties in assessing the R matrix, and if there are enough
subjects relative to the dimension size (number of total parameters estimated)
of the variance-covariance matrix to be estimated.
Robert J. Bauer, Ph.D.
Vice President, Pharmacometrics, R&D
ICON Development Solutions
7740 Milestone Parkway
Suite 150
Hanover, MD 21076
Tel: (215) 616-6428
Mob: (925) 286-0769
Email: [email protected]<mailto:[email protected]>
Web: http://www.iconplc.com/
From: [email protected]<mailto:[email protected]>
[mailto:[email protected]] On Behalf Of Ken Kowalski
Sent: Saturday, March 29, 2014 3:44 PM
To: 'Gavin Jarvis'; [email protected]<mailto:[email protected]>
Subject: RE: [NMusers] NONMEM vs SPSS
Dear Gavin,
This is most likely because most nonlinear regression programs invert the
Hessian (second derivative matrix of the model with respect to the parameters)
to obtain the covariance matrix. This corresponds to the R matrix in NONMEM.
However, the default method that NONMEM uses is a sandwich estimator involving
both the Hessian (R) and the square of the first derivatives matrix (S). I
suspect that if you use the MATRIX=R option on the $COV step you will find that
the standard errors will now be in agreement with SPSS (NLR). I know Stu Beal
made the sandwich estimator the default as it is supposed to be more robust to
non-normality but I would have preferred the MATRIX=R option to be the default
to be more consistent with other nonlinear regression software implementations.
Ken
From: [email protected]<mailto:[email protected]>
[mailto:[email protected]] On Behalf Of Gavin Jarvis
Sent: Saturday, March 29, 2014 12:55 PM
To: [email protected]<mailto:[email protected]>
Subject: [NMusers] NONMEM vs SPSS
Dear NONMEM Users
Does anyone have a view on the relative merits/reliability/accuracy of NONMEM
($COV step) vs SPSS (NLR) with respect to their derived values of the parameter
standard errors and parameter correlation matrices?
The data I am analysing are single subject (not population). Parameter
estimates from the two programs are, to all intents and purposes, identical.
However, the SE values from NONMEM $COV are consistently smaller by
1.5-2.0-fold.
Any thoughts?
Gavin
__________________________________________________
Dr Gavin E Jarvis MA PhD VetMB MRCVS
University Lecturer in Veterinary Anatomy
Department of Physiology, Development & Neuroscience
Physiological Laboratory
Downing Street
Cambridge
CB2 3EG
Tel: +44 (0) 1223 333745
Fellow and College Lecturer in Pharmacology
Selwyn College
Cambridge
CB3 9DQ
Tel: +44 (0) 1223 761303
Email: [email protected]<mailto:[email protected]>
Web: http://www.pdn.cam.ac.uk/staff/jarvis
Twit: @GavinEJarvis