RE: $OMEGA blocks and log-likelihood profiling

From: "Kowalski, Ken" Ken.Kowalski@pfizer.com Subject: RE:[NMusers] $OMEGA blocks and log-likelihood profiling Date: Tue, June 8, 2004 5:27 pm Nick and all, I can be convinced of your assertion that it is OK to pool bootstrap results for all the runs including those with failed convergence but I would need additional information. The table below summarizing the means and SDs don't provide sufficient information. If you are planning to use these bootstrap results to report out confidence intervals then it is important to compare the tails of the distributions with and without the failed convergence runs. At a minimum you should calculate bootstrap CIs with and without the failed runs to see how they compare. However, since only 28% of your runs had successful convergence, you might be faced with very poor precision to estimate the tail percentiles for the bootstrap CIs based on the successful convergence runs alone. In which case you might consider performing quantile-quantile (Q-Q) plots comparing the order statistics between the empirical distributions of the bootstrap estimates for the failed versus successful runs. If these Q-Q plots are fairly concordant between say the 10th and 90th percentiles (80% of the distribution) then I would be inclined to believe that the empirical distribution of the bootstrap estimates is independent of convergence status and that any breakdown in the tails is probably due to poor precision. In this setting I would then conclude it is OK to pool both the successful and failed convergence results in reporting bootstrap CIs. Assuming such information supports your assertion, I would be careful not to over-generalize these results to suggest that one can always pool failed convergence results. I believe every bootstrap simulation where you encounter a high convergence failure rate has to be dealt with on a case-by-case basis and a similar exercise as above would have to be performed if you wanted to pool these results. Moreover, if for a particular example, the empirical distributions are different between the failed and successful runs then the analyst has to go back to the drawing board to try and diagnose the reason for the high convergence failure rate. Here is where we seem to be in disagreement regarding the value of the COV step. Equating success of the COV step with good luck suggests that success or failure of the COV step is a purely random event outside of our control...I strongly disagree. I agree with you that success or failure of the COV step alone provides insufficient information regarding the reliability of the estimates, and in general it is not a good idea to perform formal inference with CIs based on the COV step std errors, but we appear to disagree on the value of the COV step output as a diagnostic tool to help assess instability which may be the reason for the high convergence failure rate. I agree with Mats that the COV step is an imperfect diagnostic and certainly we can get COV step and convergence failures that are unrelated to instability (e.g., over-parameterization) but that doesn't mean it has no value as you seem to suggest. Ken