Covariate model building with SAEM andIMP

Dear NMusers, I have been trying to test for covariate effects in a complex model described by stiff ODEs, and with poorly-defined parameter values. I have found by trial and error that FOCE has difficulty fitting my model when covariate effects are included so I have been using SAEM for parameter estimation, followed by an importance sampling step for evaluating the objective function. I have then compared OFVs obtained by importance sampling, comparing the difference in OFV between two models to a chi-squared distribution. Initially, I used the following commands for estimation and OFV evaluation, with five importance sampling iterations as suggested in the NONMEM 7.2.0 user guide: $ESTIMATION METHOD=SAEM INTER PRINT=1 NBURN=1000 ISAMPLE=2 NITER=500 CTYPE=3 $ESTIMATION METHOD=IMP EONLY=1 ISAMPLE=1000 NITER=5 However, when I examined the OFV at each of the six iterations 1-5 I found that the OFV seemed still to be decreasing so I increased NITER in the second $ESTIMATION step to 150: $ESTIMATION METHOD=SAEM INTER PRINT=1 NBURN=1000 ISAMPLE=2 NITER=500 CTYPE=3 $ESTIMATION METHOD=IMP EONLY=1 ISAMPLE=1000 NITER=150 I found that the objective function drifts over approximately the first 10 iterations and then settles down to values around a constant level, with a standard deviation of around 3-4 OFV units. Averaged over samples 50-150, some covariate models had lower OFVs than the basic model with no covariates, but others had higher. There are two questions I would like to ask the community. Firstly, are these results what you would expect, or does anything in my description suggest a problem with my model or the way it has been coded? Secondly, if OFVs may vary randomly by 3-4 units within a model, and p=0.05 corresponds to 3.84 units if one additional degree of freedom has been introduced, how can a genuinely better model be distinguished from an importance sampling step which happened to have a lower OFV? I would be interested and grateful to hear your thoughts on either or both questions. With best wishes Joanna Lewis 2020 Science Research Fellow +44 (0)20 7679 5300 http://www.2020science.net/people/joanna-lewis