setup of parallel processing and supporting software - help wanted

Hi Pavel, In general, parallelization discussions always revolve around the following question: “Can you create independent blocks of work?” You should make a clear distinction here between parallelizing nonmem, and running several nonmem runs in parallel. Let’s talk about estimating of a single model, doing a covariate search and doing a bootstrap. Very roughly speaking, Nonmem works as follows: 1) Pick a THETA 2) Estimate the probability curve of all ETA’s for all subjects 3) Compute the integral over all probability curves to find a probability for THETA 4) Pick a more likely THETA, rinse and repeat Parallelizing NONMEM means parallelizing step #2. Step #1, #3 and #4 cannot be parallelized. In practice, we simply split up the subjects in N groups. Each worker calculates the probability curve for all subject in its group and sends the results back to the main worker, who can then calculate step #3 and step #4. This works well for very complex models with a considerable estimation step: ODE systems. If you have a very fast model (e.g. simple $PRED section) and a huge dataset, it might be faster to run all of this locally on a single node. Note that Nonmem 7.3 does not parallelize anything other than the subject ETA estimation step! Nonmem 7.4 will parallelize also the TABLE and/or COVARIANCE estimation step. Conclusion: Parallelizing nonmem only works well in specific cases. You can execute the parallelization by specifying a parafile. If your system administrator specified a parametric parafile, you can also choose the number of CPU’s to parallelize over using [nodes]=x in nmfe, or with -nodes=xxx in PsN execute. Let’s now talk about a covariate search. In this case, we want to evaluate 12 models; we can evaluate them concurrently, as there is no dependence between them. PsN works wonderfully here: you can configure the amount of parallel runs using the -threads=xxx switch. 1) Estimate the base model 2) Create 12 instances of the base model, adding a single covariate to each instance. Launch all of these instances in parallel. 3) Once these 12 instances completed, select the most significant covariate. 4) Create 11 instances of the model from step #3, adding a single covariate to each instance. Launch all of these instances in parallel. 5) etc. As you can see, there is still some dependence: we need all results from step #2 to evaluate step #3. On top of that, parallelizing step #2 means you will have to collect all of those results back over the network to do step #3 (I/O impact). If you do not parallelize, they will already be sitting in main memory. In practice, we use the following calculation: · #threads = #max_covariate_steps · #CPU_available / #max_covariate_steps = #nodes So for a cluster of 20 CPU’s: · A covariate search with 5 covariates would be launched using: scm myModel.scm -threads=5 -nodes=4 · A covariate search with 20 covariates would be launched using: scm myModel.scm -threads -nodes=1 Remember that running multiple nonmem runs concurrently should always be preferred over parallelizing a single nonmem run. Finally, let’s talk about bootstrapping: In this case, there is no dependence between the results. This problem can be perfectly parallelized. In this case, always prefer to run multiple nonmem runs concurrently, instead of parallelizing a single nonmem run. bootstrap myModel -samples 00 -threads 00 -nodes=1 Final summary: For a single nonmem run, parallelization may or may not work, depending on how complex your model $PRED code is. For a covariate search, try to prefer running multiple runs at the same time, rather than parallelizing single runs. For bootstraps, always run multiple runs at the same time. Never parallelize a single nonmem run. Kind regards, Ruben

Mark a) I have to disagree with you that the efficiency of MPI implementation does not depend on the size of the data set for a single desktop SMP machine with multiple processors - larger data sets mean higher granularity and more cpu-bound work between stoppages for communication. This assumes the NLME MPI implementation is done efficiently - I don't know the details of the NONMEM MPI implementation, particularly those of how communications are handled. b) your I/O timings seem horrendously large (if by msec you mean milliseconds) I/O times of 40 milliseconds per function evaluation (assuming 1 function evaluation is a single sweep through all Nsub subjects, evaluating and summing the likelihood contribution from each subject) seem very high. I have been running MPI since its original release in 1994 (I was a member of the committee that designed the first release of MPI during 1992-1994 ) - these communications timings would seem more appropriate for machines from that era. I/O timings for MPI are usually modeled by a latency (startup time - typically on current SMP single desktop machines on the order of 1 microsecond) , and a bandwidth (on the order of 10's of gigabytes/sec for current era SMPs, but much lower for clusters). Based on the latency/bandwidth model, the conventional wisdom is to manage the message processing so as to favor a few large messages as opposed to many small messages to minimize the latency contribution. If possible, small messages should be concatenated into larger messages. I don't know the details of the MPI implementation in NONMEM, but for FOCE-like NLME algorithms, it is possible to limit the number of messages to just a few per function evaluation. If the data set size is expanded by adding more subjects, then more work (more subjects processed) will be done between stoppages for communication at the function evaluation boundaries. In the MPI implementation for Phoenix NLME, I find it almost impossible to find a model where I/O dominates to such an extent that the MPI version runs slower than the single processor version on a 4-processor Intel i7 desktop. For example, I just tested (FOCE) the classic simple closed form Emax model used in the INSERM estimation method comparison exercise from 2004 (Girard and Mentre', PAGE 2005, abstract 234) with Phoenix NLME. It would be hard to find a simpler model - E + EMAX*DOSE/(ED50 + DOSE) +EPS, with random effects on each of the three parameters E0, EMAX, and ED50, and three observations per subject. If I expand the data set to around 1600 from the original 100 subjects and run on a four processor i7, the internally reported cpu time is 72 sec for four processors vs 18 sec for one processor (a speedup of 4). Wall clock times were a few seconds longer for each run. If I make the data set smaller, down to the original size of 100, the speedup clearly suffers a decrease but I still observe a reported cpu time speedup of 2.5x for the four processors (times are well under 1 sec, so reliable wall clock times are not available). (this was done on a relatively old i7 desktop, so more current machines may do better). c) It is not always necessary to parallelize over function evaluations (i.e. over subjects). In importance sampling EM methods, (IMP in NONMEM, QRPEM in Phoenix NLME), in principle the parallelization can be done over the sample points used in the monte carlo or quasi-monte carlo integral evaluations - there are usually many more of these than processors available. In PHX QRPEM, we actually do it this way and it works fine. Now all processors are working on the same subject at the same time, so load balancing problems tend to go away, but communications overhead increases since now you have to pass separate messages for each subject, whereas in FOCE-like algorithms you only have to pass messages at the end of a sweep through all the subjects. One thing we have noticed is that QRPEM parallelized this way is much more reproducible - single processor results almost always match multiprocessor results exactly, which is not always the case with some of the FOCE-like methods. Bob Leary Fellow, Pharsight Corporation