RE: Funny behaviour with MCETA>1 and parallel computation

From: Paolo Denti [mailto:[email protected]] Sent: Wednesday, July 30, 2014 3:54 AM To: Bauer, Robert; nmusers Subject: Re: [NMusers] Funny behaviour with MCETA>1 and parallel computation Hi Bob, thanks for the prompt response and suggestions. I have tried to implement with RANMETHOD=P and increasing MCETA to 1000, but unfortunately without success. Even using MCETA=1000, the result is the same: when I use the parallel computation feature, it performs worse than using MCETA=0. With one processor, things work as expected. Maybe I did not explain the situation well. The issue is not that the optimisation takes a slightly different path and it reaches a different minimum, that would not surprise me, as I know different rounding and other random factor can influence that. The problem here is that when using parallel computation even on the first iteration (using MAXEVAL=0) MCETA>1 gives a worse OFV that MCETA=0. This still does not make any sense to me, irrespectively of random number generators, numerical approximation,etc. My understanding is that, in each individual, NONMEM will try 0 and other initial estimates to find the optimal ETAs, and then it will choose the solution giving the lowest individual OFV. Even if this is done with different seeds and on different CPUs, they will all try 0, so whatever MCETA=0 gives out, it should be the upper bound for the OFV for that individual. Then all these individual OFVs are summed together to find the total OFV. Since NONMEM is trying 0 in each subject - plus other random values that may vary - it should at least be able to use those results. In each individual it can only do better by trying extra values, and if all the individual OFVs are lower or at worst the same as the ones provided by MCETA=0, then the total can only be better. Am I missing something? Is NONMEM maybe minimising only some other individual likelihood in the MAP step, and that does not coincide with the individual OFV? To better understand I have looked at the values of individual OFVs in the two runs (MCETA=1000 and MCETA=0, both with parallel computaion) and all the tables are exactly the same cell by cell (to the 5th digit or so), except for the records of that outlying individual. In spite of trying 1000 initial estimates (including 0) MCETA=1000 gets the individual ETA for that subject wrong, and gives a worse iOFV than MCETA=0. And it's not a matter of numerical noise, the individual OFV is 100 points worse and the individual parameters are very different.. MCETA=0 is a sub-case of MECTA>1, so MCETA>1 should not do any worse. I have tried and re-tried, and it is unlikely that the estimator was unlucky all the times with that subject, even trying 1000 initial estimates.. I don't know how to explain this. But maybe I am misunderstanding how this works? Any explanation? Thank you, Paolo On 2014/07/29 22:45, Bauer, Robert wrote: Paolo: MCETA=5 is a small number of random samples of etas to test, and when you parallelize, your random seed pattern changes, unless, you include $EST RANMETHOD=P. Yes, sometimes a worse result follows if a lower OBJ is chosen, yet those etas that gave a lower OBJ offered a worse path than when eta=0. I suggest the following: $EST METHOD=1 INTERACTION MAXEVAL=0 MCETA=1000 RANMETHOD=P ... ; spend a lot of random samples doing a thorough search with MCETA=1000, but it takes time, so do it just for the first iteration. $EST METHOD=1 INTERACTION MAXEVAL=9999 MCETA=5 RANMETHOD=P ... ; remaining iterations use MCETA more lightly. Robert J. Bauer, Ph.D. Vice President, Pharmacometrics, R&D ICON Development Solutions 7740 Milestone Parkway Suite 150 Hanover, MD 21076 Tel: (215) 616-6428 Mob: (925) 286-0769 Email: [email protected]<mailto:[email protected]> Web: http://www.iconplc.com -----Original Message----- From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Paolo Denti Sent: Tuesday, July 29, 2014 3:54 PM To: nmusers Subject: [NMusers] Funny behaviour with MCETA>1 and parallel computation Dear all, I am working with FOCE-INTERACTION and a PK model with some issues in optimising the individual ETAs. A couple of subjects in the dataset are very fast absorbers and sometimes NONMEM fails to find the optimal value for their individual ETAs. This results in the OFV randomly jumping up a lot (~100 points) when I restart the model from the final estimates of a previous run. The model may sometimes find its way back to the same minimum, but not always. So I decided to use the new feature MCETA, which is supposed to address these kinds of issues by trying several randomly generated sets of initial ETAs, increasing the chance of finding the best value of individual ETA for each subject. I first tested the new feature on a single CPU and everything worked as expected: using MCETA=0 (the default) the OFV jumps up after restarting with the final values, while with MCETA=5 the problem is solved and the OFV is stable. I then ran the same test using the parallel computation feature to speed it up, but I noticed that here the OFV was jumping up anyway. Not only, using MCETA=5 was actually giving a WORSE OFV than the default value of MCETA=0, which is indeed giving the same result irrespectively of whether I use the parallel computation or not. I have retried several times to make sure I hadn't made any mistakes and when I use MCETA>1 and the parallel computation feature, it sometimes perform worse than the default MCETA=0. I tried other values of MCETA too, but same story. When I say that the OFV goes up, I am referring to the first iteration, so I have replicated these results using MAXEVAL=0. Any suggestion about what may be going on? My understanding is that using MCETA>1 CANNOT be any worse than the default behaviour (MECTA=0) of just using 0 as initial value. The guide says NONMEM is supposed to try both 0 AND some other values as initial estimates, and choose in every subject the option that minimises the OFV. So in the worst case scenario NONMEM will just use 0 and I will have wasted some electrons. Am I misunderstanding something? Sorry for the lengthy email and thanks for any input on this. Paolo -- ------------------------------------------------ Paolo Denti, PhD Pharmacometrics Group Division of Clinical Pharmacology Department of Medicine University of Cape Town K45 Old Main Building Groote Schuur Hospital Observatory, Cape Town 7925 South Africa phone: +27 21 404 7719 fax: +27 21 448 1989 email: [email protected]<mailto:[email protected]> ------------------------------------------------ ________________________________ UNIVERSITY OF CAPE TOWN This e-mail is subject to the UCT ICT policies and e-mail disclaimer published on our website at http://www.uct.ac.za/about/policies/emaildisclaimer/ or obtainable from +27 21 650 9111. This e-mail is intended only for the person(s) to whom it is addressed. If the e-mail has reached you in error, please notify the author. If you are not the intended recipient of the e-mail you may not use, disclose, copy, redirect or print the content. If this e-mail is not related to the business of UCT it is sent by the sender in the sender's individual capacity.