Re: setup of parallel processing and supporting software - help wanted

________________________________ From: owner-nmusers_at_globomaxnm.com <owner-nmusers_at_globomaxnm.com> on behalf of Faelens, Ruben (Belgium) <Ruben.Faelens_at_sgs.com> Sent: Wednesday, December 9, 2015 5:42 AM To: Pavel Belo; nmusers_at_globomaxnm.com Subject: RE: [NMusers] setup of parallel processing and supporting software - help wanted Hi Pavel, In general, parallelization discussions always revolve around the following question: Can you create independent blocks of work? You should make a clear distinction here between parallelizing nonmem, and running several nonmem runs in parallel. Lets talk about estimating of a single model, doing a covariate search and doing a bootstrap. Very roughly speaking, Nonmem works as follows: 1) Pick a THETA 2) Estimate the probability curve of all ETAs for all subjects 3) Compute the integral over all probability curves to find a probability for THETA 4) Pick a more likely THETA, rinse and repeat Parallelizing NONMEM means parallelizing step #2. Step #1, #3 and #4 cannot be parallelized. In practice, we simply split up the subjects in N groups. Each worker calculates the probability curve for all subject in its group and sends the results back to the main worker, who can then calculate step #3 and step #4. This works well for very complex models with a considerable estimation step: ODE systems. If you have a very fast model (e.g. simple $PRED section) and a huge dataset, it might be faster to run all of this locally on a single node. Note that Nonmem 7.3 does not parallelize anything other than the subject ETA estimation step! Nonmem 7.4 will parallelize also the TABLE and/or COVARIANCE estimation step. Conclusion: Parallelizing nonmem only works well in specific cases. You can execute the parallelization by specifying a parafile. If your system administrator specified a parametric parafile, you can also choose the number of CPUs to parallelize over using [nodes]=x in nmfe, or with -nodes=xxx in PsN execute. Lets now talk about a covariate search. In this case, we want to evaluate 12 models; we can evaluate them concurrently, as there is no dependence between them. PsN works wonderfully here: you can configure the amount of parallel runs using the -threads=xxx switch. 1) Estimate the base model 2) Create 12 instances of the base model, adding a single covariate to each instance. Launch all of these instances in parallel. 3) Once these 12 instances completed, select the most significant covariate. 4) Create 11 instances of the model from step #3, adding a single covariate to each instance. Launch all of these instances in parallel. 5) etc. As you can see, there is still some dependence: we need all results from step #2 to evaluate step #3. On top of that, parallelizing step #2 means you will have to collect all of those results back over the network to do step #3 (I/O impact). If you do not parallelize, they will already be sitting in main memory. In practice, we use the following calculation: #threads = #max_covariate_steps #CPU_available / #max_covariate_steps = #nodes So for a cluster of 20 CPUs: A covariate search with 5 covariates would be launched using: scm myModel.scm -threads=5 -nodes=4 A covariate search with 20 covariates would be launched using: scm myModel.scm -threads -nodes=1 Remember that running multiple nonmem runs concurrently should always be preferred over parallelizing a single nonmem run. Finally, lets talk about bootstrapping: In this case, there is no dependence between the results. This problem can be perfectly parallelized. In this case, always prefer to run multiple nonmem runs concurrently, instead of parallelizing a single nonmem run. bootstrap myModel -samples 00 -threads 00 -nodes=1 Final summary: For a single nonmem run, parallelization may or may not work, depending on how complex your model $PRED code is. For a covariate search, try to prefer running multiple runs at the same time, rather than parallelizing single runs. For bootstraps, always run multiple runs at the same time. Never parallelize a single nonmem run. Kind regards, Ruben From: owner-nmusers_at_globomaxnm.com [mailto:owner-nmusers_at_globomaxnm.com] On Behalf Of Pavel Belo Sent: dinsdag 8 december 2015 22:54 To: nmusers_at_globomaxnm.com Subject: [NMusers] setup of parallel processing and supporting software - help wanted Hello The Team, We hear different opinions about effectiveness of parallel processing with NONMEM from very helpful to less helpful. It can be task dependent. How useful is it in phase 3 for basic and covariate models, as well as for bootstrapping? We reached a non-exploratory (production) point when popPK is on a critical path and sophisticated but slow home-made utilities may be insufficient. Are there efficient/quick companies/institutions, which setup parallel processing, supporting software and, possibly, some other utilities (cloud computing, ...)? A group which used to helped us a while ago disappeared somewhere... Thanks, Pavel Information in this email and any attachments is confidential and intended solely for the use of the individual(s) to whom it is addressed or otherwise directed. Please note that any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the Company. Finally, the recipient should check this email and any attachments for the presence of viruses. The Company accepts no liability for any damage caused by any virus transmitted by this email. All SGS services are rendered in accordance with the applicable SGS conditions of service available on request and accessible at http://www.sgs.com/en/Terms-and-Conditions.aspx