freezes during FOCE+I search with nm743 on Linux
Dear NMusers,
Recently we are encountering freezes during FOCE+I searches on our Linux
system. The version of NONMEM used is 7.4.3. We set $EST PRINT=1 and observed
that for some datasets (we are working on a large number of simulated datasets)
the search would halt indefinitely at certain iteration no. without any further
progress (so, never terminated). The only file being constantly updated was the
.cpu file.
We resorted to strace and observed that at such iteration no., the following
block popped up repeatedly:
wait4(5956, 0x7ffc2e280874, WNOHANG, NULL) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
nanosleep({1, 0}, {1, 0}) = 0
But we are not sure what to do with it.
Other observations:
* The exact same model file and datafile terminated successfully with nm743
on a Windows 10 system.
* Whenever this problem occured there seemed to be zero gradients going on.
* We suspected handling of arithmetic errors such as div/0 error and
implemented something like this under $ERROR, but it did not resolve the halt:
FLAG=0
IF (F.EQ.0) FLAG=1
Y=((1-FLAG)*F+FLAG*0.001)*(1+EPS(1))
* Supplying lower and upper bounds on relevant theta('s) seems to remove
the halt.
We are wondering if this is a bug in the Linux version. Even if convergence
cannot be achieved, we think that the search process needs to timeout itself.
Any suggestion?
P.S. Please let me know if an example set of control stream and datafile is
desired.
Thanks and regards,
Matthew