Re: Covariate model building with SAEM andIMP
Dear Joanna,
One option would be to use 95%CI of the covariate parameter estimates. IMP method should provide you with the estimate and SEs that can be used to compute those 95%CI (if you would like to test at 0.05 level). You may say that the effect is significant if 95% CI does not include the null value. Alternatively, you may discuss it in terms of the effect value and uncertainty of the estimate.
If you would like to use OF directly, you may increase the ISAMPLE number to 3000 or even higher: this should decrease stochastic fluctuation of the OF. If you open the file root.cvn where root is the name of the control stream, you will find there the OBJ value and SD (averaged over the iterations). Those values can be used for model comparison.
Leonid
--------------------------------------
Leonid Gibiansky, Ph.D.
President, QuantPharm LLC
web: www.quantpharm.com
e-mail: LGibiansky at quantpharm.com
tel: (301) 767 5566
Quoted reply history
On 9/16/2013 4:36 PM, Lewis, Joanna wrote:
> Dear NMusers,
>
> I have been trying to test for covariate effects in a complex model
> described by stiff ODEs, and with poorly-defined parameter values.
>
> I have found by trial and error that FOCE has difficulty fitting my
> model when covariate effects are included so I have been using SAEM for
> parameter estimation, followed by an importance sampling step for
> evaluating the objective function. I have then compared OFVs obtained by
> importance sampling, comparing the difference in OFV between two models
> to a chi-squared distribution.
>
> Initially, I used the following commands for estimation and OFV
> evaluation, with five importance sampling iterations as suggested in the
> NONMEM 7.2.0 user guide:
> $ESTIMATION METHOD=SAEM INTER PRINT=1 NBURN=1000 ISAMPLE=2 NITER=500 CTYPE=3
> $ESTIMATION METHOD=IMP EONLY=1 ISAMPLE=1000 NITER=5
> However, when I examined the OFV at each of the six iterations 1-5 I
> found that the OFV seemed still to be decreasing so I increased NITER in
> the second $ESTIMATION step to 150:
> $ESTIMATION METHOD=SAEM INTER PRINT=1 NBURN=1000 ISAMPLE=2 NITER=500 CTYPE=3
> $ESTIMATION METHOD=IMP EONLY=1 ISAMPLE=1000 NITER=150
>
> I found that the objective function drifts over approximately the first
> 10 iterations and then settles down to values around a constant level,
> with a standard deviation of around 3-4 OFV units. Averaged over samples
> 50-150, some covariate models had lower OFVs than the basic model with
> no covariates, but others had higher.
>
> There are two questions I would like to ask the community. Firstly, are
> these results what you would expect, or does anything in my description
> suggest a problem with my model or the way it has been coded? Secondly,
> if OFVs may vary randomly by 3-4 units within a model, and p=0.05
> corresponds to 3.84 units if one additional degree of freedom has been
> introduced, how can a genuinely better model be distinguished from an
> importance sampling step which happened to have a lower OFV?
>
> I would be interested and grateful to hear your thoughts on either or
> both questions.
>
> With best wishes
> Joanna Lewis
>
> 2020 Science Research Fellow
> +44 (0)20 7679 5300
> http://www.2020science.net/people/joanna-lewis