Hi NMusers,
I believe below is a very common question, but I could not find a
clear answer in literature.
Sometimes, we want to find out which algorithm offers the better model
fitting for a given dataset.
Is it possible to use the objective function value (OFV) to compare
the model fitting computed by various algorithms (e.g. FOCE, IMP, and
SAEM) ? Put it into another way, the same input dataset with the same
fitted model/estimates will lead to similar OFVs among FOCE, IMP, and
SAEM?
Thanks,
Mark
OFV from different algorithms
Mark
I posed a variant of this question to Stu Beal (Sheiner's statistician) > 20 years ago. He answered that one cannot compare ADVAN's, e.g.,
between ADVAN4 and ADVAN6 for the identical model.
I verified this today when I ran a model with those two ADVAN's -- both converged, they yielded quite similar parameter estimates, but the OF differed by 60 units.
I will be interested to hear Bob Bauer's reply to this issue.
Dennis
Dennis Fisher MD
P < (The "P Less Than" Company)
Phone / Fax: 1-866-PLessThan (1-866-753-7784)
www.PLessThan.com http://www.plessthan.com/
Quoted reply history
> On Sep 23, 2020, at 5:09 PM, Mark Tepeck <mark.tepeck
wrote:
>
> Hi NMusers,
>
> I believe below is a very common question, but I could not find a
> clear answer in literature.
>
> Sometimes, we want to find out which algorithm offers the better model
> fitting for a given dataset.
>
> Is it possible to use the objective function value (OFV) to compare
> the model fitting computed by various algorithms (e.g. FOCE, IMP, and
> SAEM) ? Put it into another way, the same input dataset with the same
> fitted model/estimates will lead to similar OFVs among FOCE, IMP, and
> SAEM?
>
>
> Thanks,
>
>
> Mark
>
Dear Dennis:
Gosh, that is super interesting. I would guess it was the differences in the first derivative between the methods. ADVAN4 will be closed form, and ADVAN6 will be (I believe) numerically calculated.
Steve
Quoted reply history
From: owner-nmusers
Of Dennis Fisher
Sent: Wednesday, September 23, 2020 5:36 PM
To: Mark Tepeck <mark.tepeck
Cc: nmusers
Subject: Re: [NMusers] OFV from different algorithms
Mark
I posed a variant of this question to Stu Beal (Sheiner's statistician) > 20 years ago. He answered that one cannot compare ADVAN's, e.g., between ADVAN4 and ADVAN6 for the identical model.
I verified this today when I ran a model with those two ADVAN's -- both converged, they yielded quite similar parameter estimates, but the OF differed by 60 units.
I will be interested to hear Bob Bauer's reply to this issue.
Dennis
Dennis Fisher MD
P < (The "P Less Than" Company)
Phone / Fax: 1-866-PLessThan (1-866-753-7784)
http://www.plessthan.com/
On Sep 23, 2020, at 5:09 PM, Mark Tepeck <mark.tepeck
.tepeck
Hi NMusers,
I believe below is a very common question, but I could not find a
clear answer in literature.
Sometimes, we want to find out which algorithm offers the better model
fitting for a given dataset.
Is it possible to use the objective function value (OFV) to compare
the model fitting computed by various algorithms (e.g. FOCE, IMP, and
SAEM) ? Put it into another way, the same input dataset with the same
fitted model/estimates will lead to similar OFVs among FOCE, IMP, and
SAEM?
Thanks,
Mark
Within one method, FOCEI or IMP models can be compared by OF, but not
between methods. FOCEI and IMP OF are similar by the order of magnitude,
but should not be used for comparison between methods.
SAEM OF should not be used for model comparison at all. Among two models
that differ by SAEM OF, the one with the higher OF may have better fit
(meaning, lower FOCEI or IMP OF than the other model with lower SAEM OF).
So if you would like to compare models obtained by different methods,
you may re-run both of them with fixed parameters using the same method
(not SAEM), and then compare obtained OF values.
Leonid
Quoted reply history
On 9/23/2020 8:09 PM, Mark Tepeck wrote:
> Hi NMusers,
>
> I believe below is a very common question, but I could not find a
> clear answer in literature.
>
> Sometimes, we want to find out which algorithm offers the better model
> fitting for a given dataset.
>
> Is it possible to use the objective function value (OFV) to compare
> the model fitting computed by various algorithms (e.g. FOCE, IMP, and
> SAEM) ? Put it into another way, the same input dataset with the same
> fitted model/estimates will lead to similar OFVs among FOCE, IMP, and
> SAEM?
>
>
> Thanks,
>
>
> Mark
>
Dennis,
I would be curious if you get the same difference in OF value if you do the following for the ADVAN6 version:
Same initial estimates between the 2 ADVANs
$ESTIMATION use the options SORT, SIGDIG=3, SIGL=9
$SUBROUTINE ADVAN6 TOL=9
If $ERROR is %CV or Log error model, include
DFLAG=0
IF(CMT.EQ.DOSING CMT) DFLAG=1
For the %CV set IPRED=F+DFLAG
For Log error set IPRED=LOG(F+DFLAG)
This helps handle the first oral dose record more consistently between advan4 and advan6
Over the years, I've seen differences for the same model between 2 advans become more similar.
Luann Phillips
Distinguished Scientist, Pharmacometrics
Cognigen Corporation, a Simulations Plus company
Quoted reply history
From: owner-nmusers
Of Dennis Fisher
Sent: Wednesday, September 23, 2020 8:36 PM
To: Mark Tepeck <mark.tepeck
Cc: nmusers
Subject: Re: [NMusers] OFV from different algorithms
Mark
I posed a variant of this question to Stu Beal (Sheiner's statistician) > 20 years ago. He answered that one cannot compare ADVAN's, e.g., between ADVAN4 and ADVAN6 for the identical model.
I verified this today when I ran a model with those two ADVAN's -- both converged, they yielded quite similar parameter estimates, but the OF differed by 60 units.
I will be interested to hear Bob Bauer's reply to this issue.
Dennis
Dennis Fisher MD
P < (The "P Less Than" Company)
Phone / Fax: 1-866-PLessThan (1-866-753-7784)
http://www.plessthan.com/
On Sep 23, 2020, at 5:09 PM, Mark Tepeck <mark.tepeck
.tepeck
Hi NMusers,
I believe below is a very common question, but I could not find a
clear answer in literature.
Sometimes, we want to find out which algorithm offers the better model
fitting for a given dataset.
Is it possible to use the objective function value (OFV) to compare
the model fitting computed by various algorithms (e.g. FOCE, IMP, and
SAEM) ? Put it into another way, the same input dataset with the same
fitted model/estimates will lead to similar OFVs among FOCE, IMP, and
SAEM?
Thanks,
Mark
Hi NMusers,
I believe below is a very common question, but I could not find a
clear answer in literature.
Sometimes, we want to find out which algorithm offers the better model
fitting for a given dataset.
Is it possible to use the objective function value (OFV) to compare
the model fitting computed by various algorithms (e.g. FOCE, IMP, and
SAEM) ? Put it into another way, the same input dataset with the same
fitted model/estimates will lead to similar OFVs among FOCE, IMP, and
SAEM?
Thanks,
Mark
Mark
I posed a variant of this question to Stu Beal (Sheiner's statistician) > 20
years ago. He answered that one cannot compare ADVAN's, e.g., between ADVAN4
and ADVAN6 for the identical model.
I verified this today when I ran a model with those two ADVAN's -- both
converged, they yielded quite similar parameter estimates, but the OF differed
by 60 units.
I will be interested to hear Bob Bauer's reply to this issue.
Dennis
Dennis Fisher MD
P < (The "P Less Than" Company)
Phone / Fax: 1-866-PLessThan (1-866-753-7784)
www.PLessThan.com http://www.plessthan.com/
Quoted reply history
> On Sep 23, 2020, at 5:09 PM, Mark Tepeck <[email protected]> wrote:
>
> Hi NMusers,
>
> I believe below is a very common question, but I could not find a
> clear answer in literature.
>
> Sometimes, we want to find out which algorithm offers the better model
> fitting for a given dataset.
>
> Is it possible to use the objective function value (OFV) to compare
> the model fitting computed by various algorithms (e.g. FOCE, IMP, and
> SAEM) ? Put it into another way, the same input dataset with the same
> fitted model/estimates will lead to similar OFVs among FOCE, IMP, and
> SAEM?
>
>
> Thanks,
>
>
> Mark
>
Dear Dennis:
Gosh, that is super interesting. I would guess it was the differences in the
first derivative between the methods. ADVAN4 will be closed form, and ADVAN6
will be (I believe) numerically calculated.
Steve
Quoted reply history
From: [email protected] <[email protected]> On Behalf Of
Dennis Fisher
Sent: Wednesday, September 23, 2020 5:36 PM
To: Mark Tepeck <[email protected]>
Cc: [email protected]
Subject: Re: [NMusers] OFV from different algorithms
Mark
I posed a variant of this question to Stu Beal (Sheiner's statistician) > 20
years ago. He answered that one cannot compare ADVAN's, e.g., between ADVAN4
and ADVAN6 for the identical model.
I verified this today when I ran a model with those two ADVAN's -- both
converged, they yielded quite similar parameter estimates, but the OF differed
by 60 units.
I will be interested to hear Bob Bauer's reply to this issue.
Dennis
Dennis Fisher MD
P < (The "P Less Than" Company)
Phone / Fax: 1-866-PLessThan (1-866-753-7784)
http://www.plessthan.com/
On Sep 23, 2020, at 5:09 PM, Mark Tepeck
<[email protected]<mailto:[email protected]>> wrote:
Hi NMusers,
I believe below is a very common question, but I could not find a
clear answer in literature.
Sometimes, we want to find out which algorithm offers the better model
fitting for a given dataset.
Is it possible to use the objective function value (OFV) to compare
the model fitting computed by various algorithms (e.g. FOCE, IMP, and
SAEM) ? Put it into another way, the same input dataset with the same
fitted model/estimates will lead to similar OFVs among FOCE, IMP, and
SAEM?
Thanks,
Mark
Within one method, FOCEI or IMP models can be compared by OF, but not between methods. FOCEI and IMP OF are similar by the order of magnitude, but should not be used for comparison between methods.
SAEM OF should not be used for model comparison at all. Among two models that differ by SAEM OF, the one with the higher OF may have better fit (meaning, lower FOCEI or IMP OF than the other model with lower SAEM OF).
So if you would like to compare models obtained by different methods, you may re-run both of them with fixed parameters using the same method (not SAEM), and then compare obtained OF values.
Leonid
Quoted reply history
On 9/23/2020 8:09 PM, Mark Tepeck wrote:
> Hi NMusers,
>
> I believe below is a very common question, but I could not find a
> clear answer in literature.
>
> Sometimes, we want to find out which algorithm offers the better model
> fitting for a given dataset.
>
> Is it possible to use the objective function value (OFV) to compare
> the model fitting computed by various algorithms (e.g. FOCE, IMP, and
> SAEM) ? Put it into another way, the same input dataset with the same
> fitted model/estimates will lead to similar OFVs among FOCE, IMP, and
> SAEM?
>
> Thanks,
>
> Mark
Dennis,
I would be curious if you get the same difference in OF value if you do the
following for the ADVAN6 version:
Same initial estimates between the 2 ADVANs
$ESTIMATION use the options SORT, SIGDIG=3, SIGL=9
$SUBROUTINE ADVAN6 TOL=9
If $ERROR is %CV or Log error model, include
DFLAG=0
IF(CMT.EQ.DOSING CMT) DFLAG=1
For the %CV set IPRED=F+DFLAG
For Log error set IPRED=LOG(F+DFLAG)
This helps handle the first oral dose record more consistently between advan4
and advan6
Over the years, I've seen differences for the same model between 2 advans
become more similar.
Luann Phillips
Distinguished Scientist, Pharmacometrics
Cognigen Corporation, a Simulations Plus company
Quoted reply history
From: [email protected] <[email protected]> On Behalf Of
Dennis Fisher
Sent: Wednesday, September 23, 2020 8:36 PM
To: Mark Tepeck <[email protected]>
Cc: [email protected]
Subject: Re: [NMusers] OFV from different algorithms
Mark
I posed a variant of this question to Stu Beal (Sheiner's statistician) > 20
years ago. He answered that one cannot compare ADVAN's, e.g., between ADVAN4
and ADVAN6 for the identical model.
I verified this today when I ran a model with those two ADVAN's -- both
converged, they yielded quite similar parameter estimates, but the OF differed
by 60 units.
I will be interested to hear Bob Bauer's reply to this issue.
Dennis
Dennis Fisher MD
P < (The "P Less Than" Company)
Phone / Fax: 1-866-PLessThan (1-866-753-7784)
http://www.plessthan.com/
On Sep 23, 2020, at 5:09 PM, Mark Tepeck
<[email protected]<mailto:[email protected]>> wrote:
Hi NMusers,
I believe below is a very common question, but I could not find a
clear answer in literature.
Sometimes, we want to find out which algorithm offers the better model
fitting for a given dataset.
Is it possible to use the objective function value (OFV) to compare
the model fitting computed by various algorithms (e.g. FOCE, IMP, and
SAEM) ? Put it into another way, the same input dataset with the same
fitted model/estimates will lead to similar OFVs among FOCE, IMP, and
SAEM?
Thanks,
Mark
Colleagues,
In principle, the log likelihood can be used for such comparisons, however an L2 or peak Signal to Noise ratio or cross entropy is widely used in the signal processing literature for such comparisons.
In NONNEM, the problem relates to numerical stability and accuracy of the approximations by which the OFV is estimated. The closed form exact derivative often results in far less numerical noise. This is also true for ADVAN6 when analytic derivatives (not numerical) are utilized.
The signal processing and machine learning fields have evolved methods to handle this and these questions of comparing structures and covariance matrix have good practical solutions.
It would be interesting to make the same comparisons in e.g., torsten, nlmixr or Pumas.
Regards,
Immanuel
Quoted reply history
> On September 23, 2020 8:49 PM Steven L Shafer <
>
> [email protected]
>
> > wrote:
>
> Dear Dennis:
>
> Gosh, that is super interesting. I would guess it was the differences in the first derivative between the methods. ADVAN4 will be closed form, and ADVAN6 will be (I believe) numerically calculated.
>
> Steve
>
> From:
>
> [email protected]
>
> <
>
> [email protected]
>
> >
>
> On Behalf Of
>
> Dennis Fisher
>
> Sent:
>
> Wednesday, September 23, 2020 5:36 PM
>
> To:
>
> Mark Tepeck <
>
> [email protected]
>
> >
>
> Cc:
>
> [email protected]
>
> Subject:
>
> Re: [NMusers] OFV from different algorithms
>
> Mark
>
> I posed a variant of this question to Stu Beal (Sheiner's statistician) > 20 years ago. He answered that one cannot compare ADVAN's, e.g., between ADVAN4 and ADVAN6 for the identical model.
>
> I verified this today when I ran a model with those two ADVAN's -- both converged, they yielded quite similar parameter estimates, but the OF differed by 60 units.
>
> I will be interested to hear Bob Bauer's reply to this issue.
>
> Dennis
>
> Dennis Fisher MD P < (The "P Less Than" Company) Phone / Fax: 1-866-PLessThan (1-866-753-7784) www.PLessThan.com
>
> > On Sep 23, 2020, at 5:09 PM, Mark Tepeck <
> >
> > [email protected]
> >
> > > wrote:
> >
> > Hi NMusers, I believe below is a very common question, but I could not find a clear answer in literature. Sometimes, we want to find out which algorithm offers the better model fitting for a given dataset. Is it possible to use the objective function value (OFV) to compare the model fitting computed by various algorithms (e.g. FOCE, IMP, and SAEM) ? Put it into another way, the same input dataset with the same fitted model/estimates will lead to similar OFVs among FOCE, IMP, and SAEM? Thanks, Mark
Colleagues,
I believe that the ODE solutions may be different both in the predictions
and the gradients (which are computed and added for focei). Hence the
difference that Dennis saw.
Leonid is right that this can be done in NONMEM by adding an additional
estimation step, but I'm unsure if you can change the ODE solver in the
extra step and I'm unsure if it will estimate the ETAs when calculating the
objective function (like a POSTHOC step). If it re-estimates the ETAs,
then the objective function does not necessarily reflect the true solution
of the other method (FOCEi is conditioned on the individual estimates so
changing the ETAs will change the objective function).
Another way to compare and compare across software as Immanuel suggested
is by using nlmixr's objective function.
If you recast the model in nlmixr you can use the nlmixr ODE solver to
compare the NONMEM estimation methods with pre-specifying the ETAs (which
can be fixed) and then set maximum outer and inner evaluations to zero
However, this still requires the nlmixr ODE solving method to give the same
solutions as NONMEM (ie ADVAN6 or ADVAN13 may be different).
However, if you linearize the system and recast the linearized system in
nlmixr, you can use the NONMEM predictions coupled with nlmixr's objective
to get a FOCEi objective function based on any software's solutions.
Automatic linearization in nlmixr is not yet supported, though.
Matt.
Quoted reply history
On Thu, Sep 24, 2020 at 1:07 AM Immanuel Freedman <
[email protected]> wrote:
> Colleagues,
>
> In principle, the log likelihood can be used for such comparisons, however
> an L2 or peak Signal to Noise ratio or cross entropy is widely used in the
> signal processing literature for such comparisons.
>
> In NONNEM, the problem relates to numerical stability and accuracy of the
> approximations by which the OFV is estimated. The closed form exact
> derivative often results in far less numerical noise. This is also true
> for ADVAN6 when analytic derivatives (not numerical) are utilized.
>
> The signal processing and machine learning fields have evolved methods to
> handle this and these questions of comparing structures and covariance
> matrix have good practical solutions.
>
> It would be interesting to make the same comparisons in e.g., torsten,
> nlmixr or Pumas.
>
> Regards,
>
> Immanuel
>
> On September 23, 2020 8:49 PM Steven L Shafer <[email protected]>
> wrote:
>
>
> Dear Dennis:
>
>
>
> Gosh, that is super interesting. I would guess it was the differences in
> the first derivative between the methods. ADVAN4 will be closed form, and
> ADVAN6 will be (I believe) numerically calculated.
>
>
>
> Steve
>
>
>
> *From:* [email protected] <[email protected]> *On
> Behalf Of *Dennis Fisher
> *Sent:* Wednesday, September 23, 2020 5:36 PM
> *To:* Mark Tepeck <[email protected]>
> *Cc:* [email protected]
> *Subject:* Re: [NMusers] OFV from different algorithms
>
>
> Mark
>
>
> I posed a variant of this question to Stu Beal (Sheiner's statistician) >
> 20 years ago. He answered that one cannot compare ADVAN's, e.g., between
> ADVAN4 and ADVAN6 for the identical model.
>
> I verified this today when I ran a model with those two ADVAN's -- both
> converged, they yielded quite similar parameter estimates, but the OF
> differed by 60 units.
>
>
> I will be interested to hear Bob Bauer's reply to this issue.
>
>
> Dennis
>
>
> Dennis Fisher MD
> P < (The "P Less Than" Company)
> Phone / Fax: 1-866-PLessThan (1-866-753-7784)
> www.PLessThan.com http://www.plessthan.com/
>
>
>
>
>
>
> On Sep 23, 2020, at 5:09 PM, Mark Tepeck <[email protected]> wrote:
>
>
> Hi NMusers,
>
> I believe below is a very common question, but I could not find a
> clear answer in literature.
>
> Sometimes, we want to find out which algorithm offers the better model
> fitting for a given dataset.
>
> Is it possible to use the objective function value (OFV) to compare
> the model fitting computed by various algorithms (e.g. FOCE, IMP, and
> SAEM) ? Put it into another way, the same input dataset with the same
> fitted model/estimates will lead to similar OFVs among FOCE, IMP, and
> SAEM?
>
>
> Thanks,
>
>
> Mark
>
>
>
It seems there have been two different questions posed on the same subject
line. I will deal with each one separately.
1) The original question from Mark Tepeck has been answered by Leonid
Gibiansky, that generally when covariate model comparisons are performed, the
OFV from the same estimation algorithm should be used (Imp with IMP, FOCEI with
FOCEI, etc.)
2) The question about how the absorption-two compartment model modeled in
ADVAN6 (or the other ODE ADVANs) compares to the in-line analytical ADVAN4
version, can be answered as follows: If the models built using ADVAN4 and
ADVAN6 are identical in every way with respect to the user's setup (residual
variance, OMEGA layout lag times, bioavailability, etc.), then the OFV between
these two should be very close, within less than one OFV. The difference
between finite difference evaluation of derivatives versus analytical
derivatives, should not cause a difference of more than that. There may be
exceptions to this if a model is quite complex with potential numerical issues.
Dennis's particular experience of a difference of 60 units suggests that the
models were not set up identically in ADVN4 versus ADVAN6. I believe Luann
Philipps in her response was addressing this possibility. By the way, many
modelers favor ADVAN13 as the go-to ODE solver.
Robert J. Bauer, Ph.D.
Senior Director
Pharmacometrics R&D
ICON Early Phase
820 W. Diamond Avenue
Suite 100
Gaithersburg, MD 20878
Office: (215) 616-6428
Mobile: (925) 286-0769
[email protected]
www.iconplc.com