IMPMAP behavior question

On 8/23/2018 1:54 PM, Mark Sale wrote: > I have a model that seems to be behaving strangely, looking for interpretation > help > > in model building, the OBJ is usually ~20900. Until this model, where, on the > covariance step (IMPMAP method) the OBJ drops 9000 points (20798 to 11837), > monitoring from output file below > > iteration 70 OBJ= 20798.6782833867 eff.= 5530. Smpl.= 10000. > Fit.= 0.99524 > Convergence achieved > iteration 70 OBJ= 11837.9045704476 eff.= 5475. Smpl.= 10000. > Fit.= 0.99522 > > Parameters don't change much (edited .ext file below). > > 50 1.35E+01 9.96E-01 4.42E-02 9.41E-01 3.05E+01 1.29E-01 20799.68932 > 60 1.35E+01 9.67E-01 4.45E-02 9.43E-01 3.05E+01 1.29E-01 20792.90665 > 70 1.35E+01 9.73E-01 4.44E-02 9.44E-01 3.05E+01 1.29E-01 20798.67828 > 70 1.35E+01 9.73E-01 4.44E-02 9.44E-01 3.05E+01 1.29E-01 11837.90457 > > Plots don't look particularly different than other model (and look pretty > good), p values for ETAs are very reasonable, it converges, condition # is > good. Only two issues: > RSE for 2 OMEGAs is a little large (0.5) > an interoccasion variability term (on V) is very large (~4, exponential). This > is, I think, related to many subjects with data only at steady state. > > Further, when I advance this model, add another covariate, or another IOV on > CL, to address the issue with SS data, cannot identify Volume uniquely (using > the final parameter from this model as the initial in the next model), I cannot > reproduce these results - the OBJ goes back to ~20,800, with essentially the > same parameter estimates. So I end up rejected all additional covariates in > this model (at least by LRT). > > other details, running on Windows, 64 bit, Intel compiler, NONMEM version 7.3. > > Can I believe this OBJ value? Should I base an additional hypotheses on the > SEE, rather than the LRT? > > But, basically, why is this happening? > > thanks > > Mark Sale M.D. > Senior Vice President, Pharmacometrics > Nuventra Pharma Sciences, Inc. > 2525 Meridian Parkway, Suite 200 > Durham, NC 27713 > Phone (919)-973-0383 > [email protected]<[email protected]> > > CONFIDENTIALITY NOTICE The information in this transmittal (including > attachments, if any) may be privileged and confidential and is intended only > for the recipient(s) listed above. Any review, use, disclosure, distribution or > copying of this transmittal, in any form, is prohibited except by or on behalf > of the intended recipient(s). If you have received this transmittal in error, > please notify me immediately by reply email and destroy all copies of the > transmittal.

From: [email protected] [mailto:[email protected]] On Behalf Of Mark Sale Sent: Thursday, August 23, 2018 11:37 AM To: Leonid Gibiansky; [email protected] Subject: Re: [NMusers] IMPMAP behavior question thanks Leonid, I looked at that, and ossilation/sampling doesn't seem to be the issue, mean was 20804.0136597062, SD = 2.19872089635881. And the OBJ is very stable in the minimzation part, up until the last iteration/covariance iteration. It was run parallel, and I guess that your comment about not waiting for all the workers concerns me. There is a timeout event in the log file: ITERATION 70 STARTING SUBJECTS 1 TO 8 ON MANAGER: OK STARTING SUBJECTS 9 TO 17 ON WORKER1: OK STARTING SUBJECTS 18 TO 32 ON WORKER2: OK STARTING SUBJECTS 33 TO 85 ON WORKER3: OK COLLECTING SUBJECTS 1 TO 8 ON MANAGER COLLECTING SUBJECTS 18 TO 32 ON WORKER2 COLLECTING SUBJECTS 9 TO 17 ON WORKER1 COLLECTING SUBJECTS 33 TO 85 ON WORKER3 ITERATION 70 STARTING SUBJECTS 1 TO 8 ON MANAGER: OK STARTING SUBJECTS 9 TO 17 ON WORKER1: OK STARTING SUBJECTS 18 TO 32 ON WORKER2: OK STARTING SUBJECTS 33 TO 85 ON WORKER3: OK COLLECTING SUBJECTS 1 TO 8 ON MANAGER TIMEOUT FROM WORKER1 RESUBMITTING JOB TO LOCAL STARTING SUBJECTS 9 TO 17 ON MANAGER: OK COLLECTING SUBJECTS 18 TO 32 ON WORKER2 and no mention of collecting subjects 33 to 85 on worker 3, or subjects 9 to 17 on worker 1. so, that could be the problem. Bob - thoughts? Mark Sale M.D. Senior Vice President, Pharmacometrics Nuventra Pharma Sciences, Inc. 2525 Meridian Parkway, Suite 200 Durham, NC 27713 Phone (919)-973-0383 [email protected]<[email protected]> CONFIDENTIALITY NOTICE The information in this transmittal (including attachments, if any) may be privileged and confidential and is intended only for the recipient(s) listed above. Any review, use, disclosure, distribution or copying of this transmittal, in any form, is prohibited except by or on behalf of the intended recipient(s). If you have received this transmittal in error, please notify me immediately by reply email and destroy all copies of the transmittal. ________________________________ From: Leonid Gibiansky <[email protected]<mailto:[email protected]>> Sent: Thursday, August 23, 2018 11:14:51 AM To: Mark Sale; [email protected]<mailto:[email protected]> Subject: Re: [NMusers] IMPMAP behavior question Mark, IMPMAP procedure produces run.cnv file. There you can find mean and SD of OF (over the last few iterations that were considered for convergence stop). I use these numbers for covariate assessment as iteration-to-iteration numbers oscillate and cannot be reliably compared. Concerning the last iteration OF drop, cannot tell for sure but I've seen OF drops in some cases when the main manager do not wait for the slaves to return OF of their portion of the data. prn file has parameters TIMEOUTI and TIMEOUT, and I would try to increase them and see whether this fixes the problem Thanks Leonid On 8/23/2018 1:54 PM, Mark Sale wrote: > I have a model that seems to be behaving strangely, looking for > interpretation help > > > in model building, the OBJ is usually ~20900. Until this model, where, on the > covariance step (IMPMAP method) the OBJ drops 9000 points (20798 to 11837), > monitoring from output file below > > > > iteration 70 OBJ= 20798.6782833867 eff.= 5530. Smpl.= 10000. > Fit.= 0.99524 > Convergence achieved > iteration 70 OBJ= 11837.9045704476 eff.= 5475. Smpl.= > 10000. Fit.= 0.99522 > > Parameters don't change much (edited .ext file below). > > 50 1.35E+01 9.96E-01 4.42E-02 9.41E-01 3.05E+01 1.29E-01 20799.68932 > 60 1.35E+01 9.67E-01 4.45E-02 9.43E-01 3.05E+01 1.29E-01 20792.90665 > 70 1.35E+01 9.73E-01 4.44E-02 9.44E-01 3.05E+01 1.29E-01 20798.67828 > 70 1.35E+01 9.73E-01 4.44E-02 9.44E-01 3.05E+01 1.29E-01 11837.90457 > > > Plots don't look particularly different than other model (and look pretty > good), p values for ETAs are very reasonable, it converges, condition # is > good. Only two issues: > RSE for 2 OMEGAs is a little large (0.5) > an interoccasion variability term (on V) is very large (~4, exponential). > This is, I think, related to many subjects with data only at steady state. > > Further, when I advance this model, add another covariate, or another IOV on > CL, to address the issue with SS data, cannot identify Volume uniquely (using > the final parameter from this model as the initial in the next model), I > cannot reproduce these results - the OBJ goes back to ~20,800, with > essentially the same parameter estimates. So I end up rejected all additional > covariates in this model (at least by LRT). > > > other details, running on Windows, 64 bit, Intel compiler, NONMEM version 7.3. > > > Can I believe this OBJ value? Should I base an additional hypotheses on the > SEE, rather than the LRT? > > But, basically, why is this happening? > > thanks > > > > Mark Sale M.D. > Senior Vice President, Pharmacometrics > Nuventra Pharma Sciences, Inc. > 2525 Meridian Parkway, Suite 200 > Durham, NC 27713 > Phone (919)-973-0383 > [email protected]<[email protected]<mailto:[email protected]%[email protected]>> > > CONFIDENTIALITY NOTICE The information in this transmittal (including > attachments, if any) may be privileged and confidential and is intended only > for the recipient(s) listed above. Any review, use, disclosure, distribution > or copying of this transmittal, in any form, is prohibited except by or on > behalf of the intended recipient(s). If you have received this transmittal in > error, please notify me immediately by reply email and destroy all copies of > the transmittal. > > >

From: [email protected] <[email protected]> On Behalf Of Bauer, Robert Sent: Thursday, August 23, 2018 2:49 PM To: Mark Sale <[email protected]>; Leonid Gibiansky <[email protected]>; [email protected] Subject: RE: [NMusers] IMPMAP behavior question Mark: You would also likely see in the .phi file that the OBJ values may be 0 for those subjects not collected. The solution is as Leonid said, increase TIMEOUT in the pnm file. Robert J. Bauer, Ph.D. Senior Director Pharmacometrics R&D ICON Early Phase 820 W. Diamond Avenue Suite 100 Gaithersburg, MD 20878 Office: (215) 616-6428 Mobile: (925) 286-0769 <mailto:[email protected]> [email protected] http://www.iconplc.com/ www.iconplc.com From: [email protected] <mailto:[email protected]> [mailto:[email protected]] On Behalf Of Mark Sale Sent: Thursday, August 23, 2018 11:37 AM To: Leonid Gibiansky; [email protected] <mailto:[email protected]> Subject: Re: [NMusers] IMPMAP behavior question thanks Leonid, I looked at that, and ossilation/sampling doesn't seem to be the issue, mean was 20804.0136597062, SD = 2.19872089635881. And the OBJ is very stable in the minimzation part, up until the last iteration/covariance iteration. It was run parallel, and I guess that your comment about not waiting for all the workers concerns me. There is a timeout event in the log file: ITERATION 70 STARTING SUBJECTS 1 TO 8 ON MANAGER: OK STARTING SUBJECTS 9 TO 17 ON WORKER1: OK STARTING SUBJECTS 18 TO 32 ON WORKER2: OK STARTING SUBJECTS 33 TO 85 ON WORKER3: OK COLLECTING SUBJECTS 1 TO 8 ON MANAGER COLLECTING SUBJECTS 18 TO 32 ON WORKER2 COLLECTING SUBJECTS 9 TO 17 ON WORKER1 COLLECTING SUBJECTS 33 TO 85 ON WORKER3 ITERATION 70 STARTING SUBJECTS 1 TO 8 ON MANAGER: OK STARTING SUBJECTS 9 TO 17 ON WORKER1: OK STARTING SUBJECTS 18 TO 32 ON WORKER2: OK STARTING SUBJECTS 33 TO 85 ON WORKER3: OK COLLECTING SUBJECTS 1 TO 8 ON MANAGER TIMEOUT FROM WORKER1 RESUBMITTING JOB TO LOCAL STARTING SUBJECTS 9 TO 17 ON MANAGER: OK COLLECTING SUBJECTS 18 TO 32 ON WORKER2 and no mention of collecting subjects 33 to 85 on worker 3, or subjects 9 to 17 on worker 1. so, that could be the problem. Bob - thoughts? Mark Sale M.D. Senior Vice President, Pharmacometrics Nuventra Pharma Sciences, Inc. 2525 Meridian Parkway, Suite 200 Durham, NC 27713 Phone (919)-973-0383 [email protected] CONFIDENTIALITY NOTICE The information in this transmittal (including attachments, if any) may be privileged and confidential and is intended only for the recipient(s) listed above. Any review, use, disclosure, distribution or copying of this transmittal, in any form, is prohibited except by or on behalf of the intended recipient(s). If you have received this transmittal in error, please notify me immediately by reply email and destroy all copies of the transmittal. _____ From: Leonid Gibiansky <[email protected] <mailto:[email protected]> > Sent: Thursday, August 23, 2018 11:14:51 AM To: Mark Sale; [email protected] <mailto:[email protected]> Subject: Re: [NMusers] IMPMAP behavior question Mark, IMPMAP procedure produces run.cnv file. There you can find mean and SD of OF (over the last few iterations that were considered for convergence stop). I use these numbers for covariate assessment as iteration-to-iteration numbers oscillate and cannot be reliably compared. Concerning the last iteration OF drop, cannot tell for sure but I've seen OF drops in some cases when the main manager do not wait for the slaves to return OF of their portion of the data. prn file has parameters TIMEOUTI and TIMEOUT, and I would try to increase them and see whether this fixes the problem Thanks Leonid On 8/23/2018 1:54 PM, Mark Sale wrote: > I have a model that seems to be behaving strangely, looking for > interpretation help > > > in model building, the OBJ is usually ~20900. Until this model, where, on the > covariance step (IMPMAP method) the OBJ drops 9000 points (20798 to 11837), > monitoring from output file below > > > > iteration 70 OBJ= 20798.6782833867 eff.= 5530. Smpl.= 10000. > Fit.= 0.99524 > Convergence achieved > iteration 70 OBJ= 11837.9045704476 eff.= 5475. Smpl.= > 10000. Fit.= 0.99522 > > Parameters don't change much (edited .ext file below). > > 50 1.35E+01 9.96E-01 4.42E-02 9.41E-01 3.05E+01 1.29E-01 20799.68932 > 60 1.35E+01 9.67E-01 4.45E-02 9.43E-01 3.05E+01 1.29E-01 20792.90665 > 70 1.35E+01 9.73E-01 4.44E-02 9.44E-01 3.05E+01 1.29E-01 20798.67828 > 70 1.35E+01 9.73E-01 4.44E-02 9.44E-01 3.05E+01 1.29E-01 11837.90457 > > > Plots don't look particularly different than other model (and look pretty > good), p values for ETAs are very reasonable, it converges, condition # is > good. Only two issues: > RSE for 2 OMEGAs is a little large (0.5) > an interoccasion variability term (on V) is very large (~4, exponential). > This is, I think, related to many subjects with data only at steady state. > > Further, when I advance this model, add another covariate, or another IOV on > CL, to address the issue with SS data, cannot identify Volume uniquely (using > the final parameter from this model as the initial in the next model), I > cannot reproduce these results - the OBJ goes back to ~20,800, with > essentially the same parameter estimates. So I end up rejected all additional > covariates in this model (at least by LRT). > > > other details, running on Windows, 64 bit, Intel compiler, NONMEM version 7.3. > > > Can I believe this OBJ value? Should I base an additional hypotheses on the > SEE, rather than the LRT? > > But, basically, why is this happening? > > thanks > > > > Mark Sale M.D. > Senior Vice President, Pharmacometrics > Nuventra Pharma Sciences, Inc. > 2525 Meridian Parkway, Suite 200 > Durham, NC 27713 > Phone (919)-973-0383 > [email protected] <mailto:[email protected]%[email protected]> > <[email protected]> > > CONFIDENTIALITY NOTICE The information in this transmittal (including > attachments, if any) may be privileged and confidential and is intended only > for the recipient(s) listed above. Any review, use, disclosure, distribution > or copying of this transmittal, in any form, is prohibited except by or on > behalf of the intended recipient(s). If you have received this transmittal in > error, please notify me immediately by reply email and destroy all copies of > the transmittal. > > >