Simulation with uncertainty

Dear NMusers, I would like to get your thoughts on some common used techniques for simulation with uncertainty. If one is interested in simulating the expected mean response, there are two methods that are can be employed: (1) Use the variance-covariance matrix (2) Use of bootstrap results. What assumptions are we making when using each of the methods? What are the respective prose and cons? Do you have any preference in terms of when to use method 1 over 2 or vice versa? Thanks and kind regards //Dinko

Hi Dinko, I focused the answer on uncertainty in population parameters, but obviously there are other uncertainties like uncertainty in covariate values (in the same population, or in a new and often wider/more severe population of a prospective study), uncertainty in model space and how the model(s) will work for extrapolation into a different population, duration, or other study setting. Likewise, there are many ways of deriving a var-covar matrix (most commonly from the nonmem covstep, but there are other methods, like sse (in PsN), multivariate llp[1], SIR[2], etc. - Simulation using $PRIOR NWPRI in nonmem generally not recommended before nm8, but TNPRI is OK and the most automatic way of using the covmatrix). I try to keep the answer within scope of the question, though. With regards to uncertainty in parameter space my opinion is that most often both var-covar matrix (e.g. from nonmem covstep) and (non-parametric) bootstrap work fine. · The bootstrap is more computer intensive, but often requires less work for the analyst. · The bootstrap requires sufficient subjects speaking to each parameter. There were some preliminary results on this for population models, presented at this year's PAGE[3] · Among the bootstrap samples there are sometimes a few that are WAY OUT, and in that cause you may need to deal with it (if the parameter in question would highly affect the outcome you are after with your simulations). In many cases it would be necessary to scrutinize what parameter values you are getting from the var-covar matrix in much the same way as for the (non-parametric) bootstrap, but for simpler cases it may be sufficient to look at the numerics of point estimates and the var-covar matrix to get the picture. For a more elaborate model, or where uncertainty is high (maybe for several parameters of interest), using var-covar matrix becomes more cumbersome for the analyst. If still pursuing this approach I would generally run the bootstrap to understand how the model must be re-parameterised for the var-covar matrix to be useful e.g.: · A parameter with a lower boundary of zero and that has high uncertainty generally should be estimated on log scale. o However, a caution on that if you log-transform, assuming that Emax is higher than zero, then obviously any dose will produce an effect that is statistically different from zero, because of this assumption. o Notice that we estimate on the transformed scale to handle uncertainty in population parameters appropriately, and that the model fit should otherwise be identical (i.e. identical OFV for point estimates, but changes in the nonmem covmatrix). · I have seen a few examples where the drug effect model has been a rather simplistic Emax model, but where ED50 (or EC50) was highly uncertain and with high correlation between the estimates of ED50 and Emax. In these situations for the var-covar matrix to be useful it may not be enough only to log transform and one may have to re-parameterise, e.g. so that primary parameters are ED50 and TVEfficacy for the reference dose (instead of TVEmax as primary parameter - a primary parameter is what is actually estimated e.g. represented as a theta). With this parameterisation, the median (across the draws from var-covar matrix) of mean effect of the reference dose has agreed with the mean effect based on point estimates (of course, simulations based on point estimates still includes IIV, residual errors, etc.). I am not saying these two would always have to agree, but for these cases the agreement has been there for the non-parametric bootstrap (both before and after the re-parameterisation). In such a situation I say that the initial results from the var-covar matrix were not reliable. o This is the major benefit with the bootstrap; that it avoids the assumption that comes with the multi-variate normal and therefore does not require these types of re-parameterisations for simulations with uncertainty (in population parameters) o Notice that even for these examples of re-parameterisation, I am not suggesting that you change the actual model. If you previously had IIV on Emax, then keep it like that, even though TVEmax now is a secondary parameter (i.e. a parameter that is not estimated directly, since the theta now represent the efficacy for the reference dose) § OFV for point estimates will not change with these types of re-parameterisations, since the model is the same, much like estimating CL and V, instead of K and V - it is the same model, just re-parameterised (if, on the other hand you move IIV from acting on K and V, to acting on CL and V, you would get different parameter values and different OFV) § The distribution of e.g. Emax and ED50 based on (non-parametric) bootstrap will not change with these re-parameterisation, since bootstrap samples are without assumption of multi-variate normal uncertainty § The distribution of Emax and ED50 based on var-covar matrix WILL change - this is why we have to make these types of re-parameterisations Today there are efficient tools and functionality for nonmem simulations both based on (non-parametric) bootstrap estimates and based on var-covar matrix. PsN is highly efficient and flexible for this purpose. Most importantly, you should use a tool that simulate each replicate study with a different set of population parameters. Simulation with different sets of population parameters for each subject is only useful if you want to make inferences on a single individual (but you ask about mean response, so I take it is mean across subjects in a population or prospective study/program). Best regards Jakob References: [1] PAGE 21 (2012) Abstr 2594 [www.page-meeting.org/?abstract=2594] [2] PAGE 22 (2013) Abstr 2907 [www.page-meeting.org/?abstract=2907] [3] PAGE 22 (2013) Abstr 2899 [www.page-meeting.org/?abstract=2899]

Dear Dinko, As Jakob already mentioned, the assumptions we are making when simulating with parameter uncertainty using the variance-covariance matrix or the bootstrap are the same as when we use these techniques to get confidence intervals around model parameters. For the covariance matrix, we assume the parameter vectors arise from a multivariate normal distribution given by the asymptotic covariance matrix. For the bootstrap, parameter vectors arise from each bootstrapped dataset, so there is no assumption about a global parameter distribution. Both of these methods can have drawbacks, in particular when one is far from asymptotic conditions (problematic for covariance matrices) or when datasets have few individuals, many stratas, or when the bootstrap is not problematic (see Niebecker et al. PAGE 2013). Another alternative to simulate with parameter uncertainty is to use Sampling Importance Resampling, which I presented at PAGE this year. The principle is to simulate parameter vectors from the covariance matrix, but add a step where they are evaluated on the original data. Weights are then assigned to each parameter vector representing how likely they are given the data at hand, and based on these weights you can sample parameter vectors to use for simulation with uncertainty. Best regards, Anne-Gaëlle ------------------------------------------------------- Anne-Gaëlle Dosne, PharmD, PhD student Pharmacometrics Research Group, Department of Pharmaceutical Biosciences, Uppsala University PO Box 591 - 751 24 Uppsala - Sweden Mobile: +46 725 859 870 Email: [email protected] --------------------------------------------------------