RE: NONMEM/PsN benchmark for SGE expansion

Dear Julia, I am very happy to see that Merck is leveraging parallelized computing so significantly. We did some systematic testing on parallelized modeling and published this recently. AAPS Journal 2011, DOI: http://dx.doi.org/10.1208/s12248-011-9258-9 http://www.springerlink.com/content/c215433172002281/ The results on the efficiency of parallelization were generally in good agreement with the testing that Chee and Bob presented for parallelized S-ADAPT earlier. This was using the Importance Sampling EM algorithm (pmethod=4 in S-ADAPT; equivalent to method IMPMAP in NONMEM). For this example, parallelizing on 8 threads yielded a 6.9 times faster estimation and parallelizing on 48 threads yielded a 23 times faster estimation. As the datasets had 48 subjects, each thread received one subject in the latter case and about 50% of the estimation time was distributing data through the network. The benefit of parallelizing increases significantly: 1) If the algorithm has a large (>99%) parallelizable fraction that can be distributed among worker nodes. (IMPMAP is very well suited for this, MCMC is not, FOCE should have a smaller parallelizable fraction than IMPMAP). Example: A program with 50% parallelizable fraction can only be accelerated to 2-fold the single threaded speed, no matter how many cores one has. 2) If the dataset has many subjects. This is most critical for industry. 3) If the model is complex and requires differential equations. (Parallelizing a one compartment model is unlikely to yield much benefit due to network traffic). 4) Bootstrap analyses are ideal to be distributed on the network. Bootstraps are best run in single threaded though, as one can parallelize with 100% efficiency across the 1000 bootstrap replicates. Some additional thoughts: a) The larger your cluster, the more important it is to invest in the model code and dataset debugger before a model is compiled, since you do not want to manually shut-down the 2000 simultaneously running exe-files that might not have closed properly. This is one of the key reasons why we invested significant time in developing a free pre-processor for S-ADAPT. b) If you have 2000 nodes, it may be worth to consider launching jobs from several master nodes. You can run into trouble both with the available RAM and with the network traffic, if everything needs to funnel through one master node, even if you use 4x Infiniband networking, for example. c) Creating a cuing system to prioritize jobs from different users and projects may help. Your computational chemistry group must have a system like this. d) Saving and analyzing intermediary results is most critical for large parallelized jobs. Hope this provides some useful ideas. Overall, I think for (complex) models that require differential equations, parallelizing will decide whether a project is feasible or not in the time available. This is why we almost always parallelize. Best wishes Juergen Jürgen B. Bulitta, Ph.D., Senior Scientist, Ordway Research Institute, 150 New Scotland Avenue, Albany, NY 12208, USA Phone: +1 (518) 641-6418, Fax: +1 (518) 641-6304 Email: [email protected] http://www.ordwayresearch.org/profile_bulitta.html