Right skewness in bootstrap distribution

From: [email protected] [mailto:[email protected]] On Behalf Of Felipe Hurtado Sent: 23 April 2013 19:57 To: [email protected] Subject: [NMusers] Right skewness in bootstrap distribution Dear NONMEM users, I am modeling some PK data using a linear 3-compartment model, in which drug concentrations were measured in two of these compartments simultaneously after i.v. dose. The model fits the data reasonably well, and all parameters seem reasonable except for V1 (volume of the central compartment, which occurs to be the dosing compartment). Estimate for V1 is very small, what does not make sense considering the average dose given and the mean Cp0 calculated by NCA. This result suggests drug distribution is restricted to plasma, however it was observed extensive distribution to tissues. IIV for V1 is relatively small (19.6%, n=8 subjects). The histogram for V1 (nonparametric bootstrap with 100 replicates) shows a right skewed distribution with the presence of a subpopulation and broad confidence interval (5th percentile tends to zero). I tried to solve this by fixing V1 to a reasonable value, running the model to calculate all other parameters, and then changing the initial estimates to these parameters in order to recalculate V1, but it turns out to the same small estimate. Any suggestions will be appreciated! Thanks in advance. Felipe

From: [email protected] [mailto:[email protected]] On Behalf Of Sebastian Frechen Sent: Wednesday, 24 April 2013 1:29 AM To: Felipe Hurtado; [email protected] Subject: AW: [NMusers] Right skewness in bootstrap distribution Dear Felipe, I totally argree with Jakob. Maybe just some more comments on the bootstrap to support this. Each estimator comes up with its own sampling distribution reflecting the uncertainty for the obtained estimate given your data and model. You can do assumption on this distribution, for example the arithemtic mean as an estimate for the "true average in a population" follows in general a normal distribution if the sample size is suffiently large enough. However, this does not apply for every estimator! One of the basic ideas of the bootstrap is now that you do not know the underlying sampling distribution of your parameter estimate. But using the non-parametric bootstrap method (sample from you dataset with replacement), you construct this distribution (and it is not necessarily Gaussian) by estimating your parameter in each of the generated sample. This in turn gives you a fairly good feeling of how precise your estimate is given your model and the sample size. With respect to your volume: Have you tried fitting the data to one- or two-compartment models? How does the volume behave then? Why are you using a three-compartment model? Best regards, Sebastian ------------------------------------------------------------------------------------ Dr. Sebastian Frechen Department of Pharmacology, Clinical Pharmacology Cologne University Hospital ----------------------- Gleueler Str. 24 50931 Cologne Germany ________________________________ Von: [email protected]<mailto:[email protected]> [[email protected]]" im Auftrag von "Ribbing, Jakob [[email protected]] Gesendet: Dienstag, 23. April 2013 22:30 An: Felipe Hurtado; [email protected]<mailto:[email protected]> Betreff: RE: [NMusers] Right skewness in bootstrap distribution Dear Felipe, The distribution obtained from the (nonparametric) bootstrap represents uncertainty in the population parameters, and the histogram for V1 should not be interpreted as a distribution of individual parameter values. There are issues with relying on the nonparametric distribution based on only eight subjects. The tail to the right may be just due to one or two subjects with a larger central volume. Otherwise (disregarding too few subjects in this specific example); there is nothing wrong with a right-tailing uncertainty distribution. In fact, it may even be expected when uncertainty is high and parameter is restricted to positive values. You would obtain a similar uncertainty distribution from the nonmem covmatrix by estimating (typical) central volume on log scale. This should not change OFV, but will alter the covmatrix. It is difficult to comment on whether the Vc estimate is unreasonable or not. If early observations are well predicted by the model, then what amount is located in central compartment, and what amount is available in the two peripheral compartment at these early time points? If you do not understand how the model may describe the observed data you could output these amounts in a table and investigate disposition at these early time points. NCA extrapolations to time zero may not agree, but that to me is mostly a theoretical issue - it would be pointless to measure concentrations at the same time as a (bolus) dose. Best regards Jakob From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Felipe Hurtado Sent: 23 April 2013 19:57 To: [email protected]<mailto:[email protected]> Subject: [NMusers] Right skewness in bootstrap distribution Dear NONMEM users, I am modeling some PK data using a linear 3-compartment model, in which drug concentrations were measured in two of these compartments simultaneously after i.v. dose. The model fits the data reasonably well, and all parameters seem reasonable except for V1 (volume of the central compartment, which occurs to be the dosing compartment). Estimate for V1 is very small, what does not make sense considering the average dose given and the mean Cp0 calculated by NCA. This result suggests drug distribution is restricted to plasma, however it was observed extensive distribution to tissues. IIV for V1 is relatively small (19.6%, n=8 subjects). The histogram for V1 (nonparametric bootstrap with 100 replicates) shows a right skewed distribution with the presence of a subpopulation and broad confidence interval (5th percentile tends to zero). I tried to solve this by fixing V1 to a reasonable value, running the model to calculate all other parameters, and then changing the initial estimates to these parameters in order to recalculate V1, but it turns out to the same small estimate. Any suggestions will be appreciated! Thanks in advance. Felipe

From: Sebastian Frechen [mailto:[email protected]] Sent: Wednesday, April 24, 2013 8:34 PM To: Felipe Hurtado; [email protected] Subject: AW: [NMusers] Right skewness in bootstrap distribution Dear All, let me also add some more thoughts. I do not think that it is a matter of a low value of the bootstrap itself considering a sample size of only 8 subjects. In this case, every estimate of a parameter and its related variability have to be treated with caution - that's clear. However, here is a very simple example that the bootstrap also may work for small sample sizes from a pure mathematical point of view: Let's say, we would like to estimate the true average weight in a population of interest. Our sample consists of n=8 observations, for example: X = (64, 68 ,72 ,75 ,76 ,83 ,91 ,93). Of course, the mean is a reasonable estimate (77.8) and we easily obtain a (slightly biased [<5%]) estimated standard error of sd(X)/sqrt(n)=3.7 Now let's run a bootstrap with 1000 samples by resampling X with replacement; I estimated in this case a standard error of 3.6 given by the standard deviation of my estimates for the mean in each bootstrap sample. At least in this example, this value corresponds to the value obtained by the ordinary method. The same applies for estimated confidence intervals. In our case, I would assume an overall normal distribution for weight. Thus, computing for the mean an exact 95%-CI based on the t-distribution with 7 degrees of freedom, we obtain (69.0 ; 86.5). Constructing a 95%-CI based on the percentiles of my bootstrap sample distribution, the CI is (70.8 ; 84.9) - slightly too narrow but still not too bad and still giving us a reasonable impression for the precision. Of course, this is a very easy example where the bootstrap does not really make sense since exact methods exist and also there may be constellations where the bootstrap does not produce sound results at all. It's clear that larger sample sizes significantly improve the bootstrap. But still, in my opinion especially in cases of small numbers, a bootstrap analysis might be even of greater value than results from asymptotic computations. Best regards, Sebastian ---------------------------------------------------------------------------- -------- Dr. Sebastian Frechen Department of Pharmacology, Clinical Pharmacology Cologne University Hospital ----------------------- Gleueler Str. 24 50931 Cologne Germany _____ Von: De Ridder, Filip [JRDBE] [[email protected]] Gesendet: Mittwoch, 24. April 2013 09:54 An: Sebastian Frechen; Felipe Hurtado; [email protected] Betreff: RE: [NMusers] Right skewness in bootstrap distribution Dear All, A few more additional thoughts regarding the use of the bootstrap here. Although I agree with what Sebastian writes, I think the value of the bootstrap is very limited when you have a sample size of only 8 subjects. Re-sampling with replacement from such a small number can easily result in replicates in which a certain subjects appears 2 or 3 even three times. In this way, the bootstrap distribution becomes quite discrete in nature and can easily start to look weird. I suspect the "subpopulation" Felipe mentions corresponds to bootstrap samples in which a subject with an extreme V1 occurs repeatedly. On a related note, did all of your bootstrap replicates yield a converged NONMEM run? Kind regards, Filip De Ridder Model-based Drug Development Janssen R&D From: [email protected] [mailto:[email protected]] On Behalf Of Sebastian Frechen Sent: Wednesday, 24 April 2013 1:29 AM To: Felipe Hurtado; [email protected] Subject: AW: [NMusers] Right skewness in bootstrap distribution Dear Felipe, I totally argree with Jakob. Maybe just some more comments on the bootstrap to support this. Each estimator comes up with its own sampling distribution reflecting the uncertainty for the obtained estimate given your data and model. You can do assumption on this distribution, for example the arithemtic mean as an estimate for the "true average in a population" follows in general a normal distribution if the sample size is suffiently large enough. However, this does not apply for every estimator! One of the basic ideas of the bootstrap is now that you do not know the underlying sampling distribution of your parameter estimate. But using the non-parametric bootstrap method (sample from you dataset with replacement), you construct this distribution (and it is not necessarily Gaussian) by estimating your parameter in each of the generated sample. This in turn gives you a fairly good feeling of how precise your estimate is given your model and the sample size. With respect to your volume: Have you tried fitting the data to one- or two-compartment models? How does the volume behave then? Why are you using a three-compartment model? Best regards, Sebastian ---------------------------------------------------------------------------- -------- Dr. Sebastian Frechen Department of Pharmacology, Clinical Pharmacology Cologne University Hospital ----------------------- Gleueler Str. 24 50931 Cologne Germany _____ Von: [email protected] [[email protected]]" im Auftrag von "Ribbing, Jakob [[email protected]] Gesendet: Dienstag, 23. April 2013 22:30 An: Felipe Hurtado; [email protected] Betreff: RE: [NMusers] Right skewness in bootstrap distribution Dear Felipe, The distribution obtained from the (nonparametric) bootstrap represents uncertainty in the population parameters, and the histogram for V1 should not be interpreted as a distribution of individual parameter values. There are issues with relying on the nonparametric distribution based on only eight subjects. The tail to the right may be just due to one or two subjects with a larger central volume. Otherwise (disregarding too few subjects in this specific example); there is nothing wrong with a right-tailing uncertainty distribution. In fact, it may even be expected when uncertainty is high and parameter is restricted to positive values. You would obtain a similar uncertainty distribution from the nonmem covmatrix by estimating (typical) central volume on log scale. This should not change OFV, but will alter the covmatrix. It is difficult to comment on whether the Vc estimate is unreasonable or not. If early observations are well predicted by the model, then what amount is located in central compartment, and what amount is available in the two peripheral compartment at these early time points? If you do not understand how the model may describe the observed data you could output these amounts in a table and investigate disposition at these early time points. NCA extrapolations to time zero may not agree, but that to me is mostly a theoretical issue - it would be pointless to measure concentrations at the same time as a (bolus) dose. Best regards Jakob From: [email protected] [mailto:[email protected]] On Behalf Of Felipe Hurtado Sent: 23 April 2013 19:57 To: [email protected] Subject: [NMusers] Right skewness in bootstrap distribution Dear NONMEM users, I am modeling some PK data using a linear 3-compartment model, in which drug concentrations were measured in two of these compartments simultaneously after i.v. dose. The model fits the data reasonably well, and all parameters seem reasonable except for V1 (volume of the central compartment, which occurs to be the dosing compartment). Estimate for V1 is very small, what does not make sense considering the average dose given and the mean Cp0 calculated by NCA. This result suggests drug distribution is restricted to plasma, however it was observed extensive distribution to tissues. IIV for V1 is relatively small (19.6%, n=8 subjects). The histogram for V1 (nonparametric bootstrap with 100 replicates) shows a right skewed distribution with the presence of a subpopulation and broad confidence interval (5th percentile tends to zero). I tried to solve this by fixing V1 to a reasonable value, running the model to calculate all other parameters, and then changing the initial estimates to these parameters in order to recalculate V1, but it turns out to the same small estimate. Any suggestions will be appreciated! Thanks in advance. Felipe

From: [email protected] [mailto:[email protected]] On Behalf Of Sebastian Frechen Sent: den 25 april 2013 01:34 To: Felipe Hurtado; [email protected] Subject: AW: [NMusers] Right skewness in bootstrap distribution Dear All, let me also add some more thoughts. I do not think that it is a matter of a low value of the bootstrap itself considering a sample size of only 8 subjects. In this case, every estimate of a parameter and its related variability have to be treated with caution - that's clear. However, here is a very simple example that the bootstrap also may work for small sample sizes from a pure mathematical point of view: Let's say, we would like to estimate the true average weight in a population of interest. Our sample consists of n=8 observations, for example: X = (64, 68 ,72 ,75 ,76 ,83 ,91 ,93). Of course, the mean is a reasonable estimate (77.8) and we easily obtain a (slightly biased [<5%]) estimated standard error of sd(X)/sqrt(n)=3.7 Now let's run a bootstrap with 1000 samples by resampling X with replacement; I estimated in this case a standard error of 3.6 given by the standard deviation of my estimates for the mean in each bootstrap sample. At least in this example, this value corresponds to the value obtained by the ordinary method. The same applies for estimated confidence intervals. In our case, I would assume an overall normal distribution for weight. Thus, computing for the mean an exact 95%-CI based on the t-distribution with 7 degrees of freedom, we obtain (69.0 ; 86.5). Constructing a 95%-CI based on the percentiles of my bootstrap sample distribution, the CI is (70.8 ; 84.9) - slightly too narrow but still not too bad and still giving us a reasonable impression for the precision. Of course, this is a very easy example where the bootstrap does not really make sense since exact methods exist and also there may be constellations where the bootstrap does not produce sound results at all. It's clear that larger sample sizes significantly improve the bootstrap. But still, in my opinion especially in cases of small numbers, a bootstrap analysis might be even of greater value than results from asymptotic computations. Best regards, Sebastian ---------------------------------------------------------------------------- -------- Dr. Sebastian Frechen Department of Pharmacology, Clinical Pharmacology Cologne University Hospital ----------------------- Gleueler Str. 24 50931 Cologne Germany _____ Von: De Ridder, Filip [JRDBE] [[email protected]] Gesendet: Mittwoch, 24. April 2013 09:54 An: Sebastian Frechen; Felipe Hurtado; [email protected] Betreff: RE: [NMusers] Right skewness in bootstrap distribution Dear All, A few more additional thoughts regarding the use of the bootstrap here. Although I agree with what Sebastian writes, I think the value of the bootstrap is very limited when you have a sample size of only 8 subjects. Re-sampling with replacement from such a small number can easily result in replicates in which a certain subjects appears 2 or 3 even three times. In this way, the bootstrap distribution becomes quite discrete in nature and can easily start to look weird. I suspect the “subpopulation” Felipe mentions corresponds to bootstrap samples in which a subject with an extreme V1 occurs repeatedly. On a related note, did all of your bootstrap replicates yield a converged NONMEM run? Kind regards, Filip De Ridder Model-based Drug Development Janssen R&D From: [email protected] [mailto:[email protected]] On Behalf Of Sebastian Frechen Sent: Wednesday, 24 April 2013 1:29 AM To: Felipe Hurtado; [email protected] Subject: AW: [NMusers] Right skewness in bootstrap distribution Dear Felipe, I totally argree with Jakob. Maybe just some more comments on the bootstrap to support this. Each estimator comes up with its own sampling distribution reflecting the uncertainty for the obtained estimate given your data and model. You can do assumption on this distribution, for example the arithemtic mean as an estimate for the "true average in a population" follows in general a normal distribution if the sample size is suffiently large enough. However, this does not apply for every estimator! One of the basic ideas of the bootstrap is now that you do not know the underlying sampling distribution of your parameter estimate. But using the non-parametric bootstrap method (sample from you dataset with replacement), you construct this distribution (and it is not necessarily Gaussian) by estimating your parameter in each of the generated sample. This in turn gives you a fairly good feeling of how precise your estimate is given your model and the sample size. With respect to your volume: Have you tried fitting the data to one- or two-compartment models? How does the volume behave then? Why are you using a three-compartment model? Best regards, Sebastian