Obtaining RSE%

From: [email protected] <[email protected]> On Behalf Of Santosh Sent: Friday, July 26, 2024 3:38 AM To: [email protected] Subject: [NMusers] Obtaining RSE% Dear esteemed experts! When using one or more estimation methods & covariance step in a NONMEM control stream, the resulting ext file contains final estimate (for all estimation steps) & standard error (only for the last estimation step). Is there a way that standard error is generated for every estimation step? TIA Santosh

On Sat, Jul 27, 2024, 12:47 PM <[email protected]> wrote: > Dear Santosh, > > > > There is a good reason for this. Wald (1943) has shown that the inverse > of the Hessian (R matrix) evaluated at the maximum likelihood estimates is > a consistent estimator of the covariance matrix. It is based on Wald’s > approximation that the likelihood surface locally near the maximum > likelihood estimates can be approximated by a quadratic function in the > parameters. This theory does not hold for any set of parameter estimates > along the algorithm’s search path prior to convergence to the maximum > likelihood estimates. Moreover, inverting the Hessian evaluated at an > interim step prior to convergence would likely be a poor approximation > especially early in the search path where the gradients are large (i.e., > large changes in OFV for a given change in the parameters would probably > have substantial curvature and not be well approximated by a quadratic > model in the parameters). > > > > Thus, the COV step in NONMEM is only applied once convergence is obtained > during the EST step. > > > > Wald, A. “Tests of statistical hypotheses concerning several parameters > when the number of observations is large.” *Trans. Amer. Math. Soc.* > 1943;54:426. > > > > Best, > > > > Ken > > > > Kenneth G. Kowalski > > President > > Kowalski PMetrics Consulting, LLC > > Email: [email protected] > > Cell: 248-207-5082 > > > > > > *From:* [email protected] <[email protected]> *On > Behalf Of *Santosh > *Sent:* Friday, July 26, 2024 3:38 AM > *To:* [email protected] > *Subject:* [NMusers] Obtaining RSE% > > > > Dear esteemed experts! > > When using one or more estimation methods & covariance step in a NONMEM > control stream, the resulting ext file contains final estimate (for all > estimation steps) & standard error (only for the last estimation step). > > > > Is there a way that standard error is generated for every estimation step? > > > > TIA > > Santosh >

From: Leonid Gibiansky <[email protected]> Sent: Saturday, July 27, 2024 4:18 PM To: Ken Kowalski <[email protected]> Cc: Santosh <[email protected]>; nmusers <[email protected]> Subject: Re: [NMusers] Obtaining RSE% It can be done if you add extra $cov statement after each estimation method record Thank you Leonid On Sat, Jul 27, 2024, 12:47 PM <[email protected] <mailto:[email protected]> > wrote: Dear Santosh, There is a good reason for this. Wald (1943) has shown that the inverse of the Hessian (R matrix) evaluated at the maximum likelihood estimates is a consistent estimator of the covariance matrix. It is based on Wald’s approximation that the likelihood surface locally near the maximum likelihood estimates can be approximated by a quadratic function in the parameters. This theory does not hold for any set of parameter estimates along the algorithm’s search path prior to convergence to the maximum likelihood estimates. Moreover, inverting the Hessian evaluated at an interim step prior to convergence would likely be a poor approximation especially early in the search path where the gradients are large (i.e., large changes in OFV for a given change in the parameters would probably have substantial curvature and not be well approximated by a quadratic model in the parameters). Thus, the COV step in NONMEM is only applied once convergence is obtained during the EST step. Wald, A. “Tests of statistical hypotheses concerning several parameters when the number of observations is large.” Trans. Amer. Math. Soc. 1943;54:426. Best, Ken Kenneth G. Kowalski President Kowalski PMetrics Consulting, LLC Email: [email protected] <mailto:[email protected]> Cell: 248-207-5082 From: [email protected] <mailto:[email protected]> <[email protected] <mailto:[email protected]> > On Behalf Of Santosh Sent: Friday, July 26, 2024 3:38 AM To: [email protected] <mailto:[email protected]> Subject: [NMusers] Obtaining RSE% Dear esteemed experts! When using one or more estimation methods & covariance step in a NONMEM control stream, the resulting ext file contains final estimate (for all estimation steps) & standard error (only for the last estimation step). Is there a way that standard error is generated for every estimation step? TIA Santosh

On Sat, Jul 27, 2024 at 6:44 PM <[email protected]> wrote: > Aah – I see that I misunderstood Santosh’s question. I thought Santosh > was asking about reporting standard errors at each iteration step within > the estimation algorithm. > > > > Best, > > > > Ken > > > > *From:* Leonid Gibiansky <[email protected]> > *Sent:* Saturday, July 27, 2024 4:18 PM > *To:* Ken Kowalski <[email protected]> > *Cc:* Santosh <[email protected]>; nmusers <[email protected]> > *Subject:* Re: [NMusers] Obtaining RSE% > > > > It can be done if you add extra $cov statement after each estimation > method record > > Thank you > > Leonid > > > > > > On Sat, Jul 27, 2024, 12:47 PM <[email protected]> wrote: > > Dear Santosh, > > > > There is a good reason for this. Wald (1943) has shown that the inverse > of the Hessian (R matrix) evaluated at the maximum likelihood estimates is > a consistent estimator of the covariance matrix. It is based on Wald’s > approximation that the likelihood surface locally near the maximum > likelihood estimates can be approximated by a quadratic function in the > parameters. This theory does not hold for any set of parameter estimates > along the algorithm’s search path prior to convergence to the maximum > likelihood estimates. Moreover, inverting the Hessian evaluated at an > interim step prior to convergence would likely be a poor approximation > especially early in the search path where the gradients are large (i.e., > large changes in OFV for a given change in the parameters would probably > have substantial curvature and not be well approximated by a quadratic > model in the parameters). > > > > Thus, the COV step in NONMEM is only applied once convergence is obtained > during the EST step. > > > > Wald, A. “Tests of statistical hypotheses concerning several parameters > when the number of observations is large.” *Trans. Amer. Math. Soc.* > 1943;54:426. > > > > Best, > > > > Ken > > > > Kenneth G. Kowalski > > President > > Kowalski PMetrics Consulting, LLC > > Email: [email protected] > > Cell: 248-207-5082 > > > > > > *From:* [email protected] <[email protected]> *On > Behalf Of *Santosh > *Sent:* Friday, July 26, 2024 3:38 AM > *To:* [email protected] > *Subject:* [NMusers] Obtaining RSE% > > > > Dear esteemed experts! > > When using one or more estimation methods & covariance step in a NONMEM > control stream, the resulting ext file contains final estimate (for all > estimation steps) & standard error (only for the last estimation step). > > > > Is there a way that standard error is generated for every estimation step? > > > > TIA > > Santosh > >

From: [email protected] <[email protected]> Sent: Saturday, July 27, 2024 12:36 PM To: 'Santosh' <[email protected]>; [email protected] Subject: RE: [NMusers] Obtaining RSE% Dear Santosh, There is a good reason for this. Wald (1943) has shown that the inverse of the Hessian (R matrix) evaluated at the maximum likelihood estimates is a consistent estimator of the covariance matrix. It is based on Wald’s approximation that the likelihood surface locally near the maximum likelihood estimates can be approximated by a quadratic function in the parameters. This theory does not hold for any set of parameter estimates along the algorithm’s search path prior to convergence to the maximum likelihood estimates. Moreover, inverting the Hessian evaluated at an interim step prior to convergence would likely be a poor approximation especially early in the search path where the gradients are large (i.e., large changes in OFV for a given change in the parameters would probably have substantial curvature and not be well approximated by a quadratic model in the parameters). Thus, the COV step in NONMEM is only applied once convergence is obtained during the EST step. Wald, A. “Tests of statistical hypotheses concerning several parameters when the number of observations is large.” Trans. Amer. Math. Soc. 1943;54:426. Best, Ken Kenneth G. Kowalski President Kowalski PMetrics Consulting, LLC Email: <mailto:[email protected]> [email protected] Cell: 248-207-5082 From: <mailto:[email protected]> [email protected] < <mailto:[email protected]> [email protected]> On Behalf Of Santosh Sent: Friday, July 26, 2024 3:38 AM To: <mailto:[email protected]> [email protected] Subject: [NMusers] Obtaining RSE% Dear esteemed experts! When using one or more estimation methods & covariance step in a NONMEM control stream, the resulting ext file contains final estimate (for all estimation steps) & standard error (only for the last estimation step). Is there a way that standard error is generated for every estimation step? TIA Santosh

On 29-07-2024 14:41, [email protected] wrote: > Dear NMusers, > > It was recently pointed out to me by a statistical colleague that my recent NMusers post about the inverse Hessian (R matrix) evaluated at the maximum likelihood estimates is a consistent estimator of the covariance matrix (i.e., converges to the true value with large N) is only true for linear models. For nonlinear models, the standard errors produced by NONMEM and other nonlinear estimation software are not only asymptotic but also approximate. Moreover, how well that approximation works will also depend on the parameterization. This I believe is one of the motivations behind “mu referencing” in NONMEM and the use of log transformations of the parameters to help improve Wald-based approximations. I thank Alan Maloney for pointing this out to me. > > Kind regards, > > Ken > > *From:*[email protected] <[email protected]> > *Sent:* Saturday, July 27, 2024 12:36 PM > *To:* 'Santosh' <[email protected]>; [email protected] > *Subject:* RE: [NMusers] Obtaining RSE% > > Dear Santosh, > > There is a good reason for this. Wald (1943) has shown that the inverse of the Hessian (R matrix) evaluated at the maximum likelihood estimates is a consistent estimator of the covariance matrix. It is based on Wald’s approximation that the likelihood surface locally near the maximum likelihood estimates can be approximated by a quadratic function in the parameters. This theory does not hold for any set of parameter estimates along the algorithm’s search path prior to convergence to the maximum likelihood estimates. Moreover, inverting the Hessian evaluated at an interim step prior to convergence would likely be a poor approximation especially early in the search path where the gradients are large (i.e., large changes in OFV for a given change in the parameters would probably have substantial curvature and not be well approximated by a quadratic model in the parameters). > > Thus, the COV step in NONMEM is only applied once convergence is obtained during the EST step. > > Wald, A. “Tests of statistical hypotheses concerning several parameters when the number of observations is large.” /Trans. Amer. Math. Soc./ 1943;54:426. > > Best, > > Ken > > Kenneth G. Kowalski > > President > > Kowalski PMetrics Consulting, LLC > > Email: [email protected] <mailto:[email protected]> > > Cell: 248-207-5082 > > *From:* [email protected] < mailto: [email protected] >< [email protected] < mailto: [email protected] >> *On Behalf Of *Santosh > > *Sent:* Friday, July 26, 2024 3:38 AM > *To:* [email protected] <mailto:[email protected]> > *Subject:* [NMusers] Obtaining RSE% > > Dear esteemed experts! > > When using one or more estimation methods & covariance step in a NONMEM control stream, the resulting ext file contains final estimate (for all estimation steps) & standard error (only for the last estimation step). > > Is there a way that standard error is generated for every estimation step? > > TIA > > Santosh

-----Original Message----- From: [email protected] <[email protected]> On Behalf Of Jeroen Elassaiss-Schaap (PD-value B.V.) Sent: Monday, July 29, 2024 3:37 PM To: [email protected]; 'Santosh' <[email protected]>; [email protected] Cc: 'Alan Maloney' <[email protected]>; Pyry Välitalo <[email protected]> Subject: Re: [NMusers] Obtaining RSE% [Some people who received this message don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] Dear NMusers, This is a great reminder for us to consider the reliability of standard errors in our models, thanks Ken & Alan. The more non-linear the models become, the less reliable and the more important other perspectives on parameter values such as sensitivity analysis and prior knowledge. The nmusers archive has many great threads on the topic that are available to review such as https://www.mail-archive.com/[email protected]/msg05423.html and related https://www.mail-archive.com/[email protected]/msg05419.html . In summary, log-transformation only can get you so far but can perhaps be seen as a sort of minimal effort. To add to the Lewis's quote about SEs - "they are not worth the electrons used to compute them" (see the links), Pyry had some very interesting observations he shared with me about the SE of the CV of a log-normal omega: it inflates with higher values of omega compared to the SE of omega itself. Best regards, Jeroen http://pd-value.com [email protected] @PD_value +31 6 23118438 -- More value out of your data! On 29-07-2024 14:41, [email protected] wrote: > > Dear NMusers, > > It was recently pointed out to me by a statistical colleague that my > recent NMusers post about the inverse Hessian (R matrix) evaluated at > the maximum likelihood estimates is a consistent estimator of the > covariance matrix (i.e., converges to the true value with large N) is > only true for linear models. For nonlinear models, the standard > errors produced by NONMEM and other nonlinear estimation software are > not only asymptotic but also approximate. Moreover, how well that > approximation works will also depend on the parameterization. This I > believe is one of the motivations behind “mu referencing” in NONMEM > and the use of log transformations of the parameters to help improve > Wald-based approximations. I thank Alan Maloney for pointing this out > to me. > > Kind regards, > > Ken > > *From:*[email protected] <[email protected]> > *Sent:* Saturday, July 27, 2024 12:36 PM > *To:* 'Santosh' <[email protected]>; [email protected] > *Subject:* RE: [NMusers] Obtaining RSE% > > Dear Santosh, > > There is a good reason for this. Wald (1943) has shown that the > inverse of the Hessian (R matrix) evaluated at the maximum likelihood > estimates is a consistent estimator of the covariance matrix. It is > based on Wald’s approximation that the likelihood surface locally near > the maximum likelihood estimates can be approximated by a quadratic > function in the parameters. This theory does not hold for any set of > parameter estimates along the algorithm’s search path prior to > convergence to the maximum likelihood estimates. Moreover, inverting > the Hessian evaluated at an interim step prior to convergence would > likely be a poor approximation especially early in the search path > where the gradients are large (i.e., large changes in OFV for a given > change in the parameters would probably have substantial curvature and > not be well approximated by a quadratic model in the parameters). > > Thus, the COV step in NONMEM is only applied once convergence is > obtained during the EST step. > > Wald, A. “Tests of statistical hypotheses concerning several > parameters when the number of observations is large.” /Trans. Amer. > Math. Soc./ 1943;54:426. > > Best, > > Ken > > Kenneth G. Kowalski > > President > > Kowalski PMetrics Consulting, LLC > > Email: [email protected] <mailto:[email protected]> > > Cell: 248-207-5082 > > *From:*[email protected] > <mailto:[email protected]><[email protected] > <mailto:[email protected]>> *On Behalf Of *Santosh > *Sent:* Friday, July 26, 2024 3:38 AM > *To:* [email protected] <mailto:[email protected]> > *Subject:* [NMusers] Obtaining RSE% > > Dear esteemed experts! > > When using one or more estimation methods & covariance step in a > NONMEM control stream, the resulting ext file contains final estimate > (for all estimation steps) & standard error (only for the last > estimation step). > > Is there a way that standard error is generated for every estimation step? > > TIA > > Santosh >

On Mon, Jul 29, 2024 at 8:52 AM Nick Holford <[email protected]> wrote: > Hi Jeroen, > > A small correction. Please re-read my email to nmusers on 12 Feb 2015 > which I quote here. Sorry I cannot show the original but the 1999 URL is > not available to me anymore. > > ================= start quote =================== > Nick Holford Thu, 12 Feb 2015 11:54:59 -0800 > Hi, > The original quote about electrons comes from a remark I made in 1999 on > nmusers. > http://www.cognigencorp.com/nonmem/nm/99nov121999.html > Lewis Sheiner agreed in the same thread. Thanks to the wonders of living > on a sphere Lewis appears to agree with me the day before I made the > comment :-) > ================= end quote =================== > > I had been meaning to add to Ken's great email which confirms my original > assertion about electrons. > > If Santosh really wanted to calculate SE's after every "iteration" (which > I think was Ken's interpretation of every "estimation") then this can be > done by running a non-parametric bootstrap with the parameter estimates > produced after every iteration. > > I wonder if Santosh would like to spend a few hours doing that and adding > to the nmusers collection about standard errors by reporting the results to > us? > > > Best wishes, > Nick > > > -- > Nick Holford, Professor Emeritus Clinical Pharmacology, MBChB, FRACP > mobile: NZ+64(21) 46 23 53 ; FR+33(6) 62 32 46 72 > email: [email protected] > web: http://holford.fmhs.auckland.ac.nz/ > > -----Original Message----- > From: [email protected] <[email protected]> On > Behalf Of Jeroen Elassaiss-Schaap (PD-value B.V.) > Sent: Monday, July 29, 2024 3:37 PM > To: [email protected]; 'Santosh' <[email protected]>; > [email protected] > Cc: 'Alan Maloney' <[email protected]>; Pyry Välitalo < > [email protected]> > Subject: Re: [NMusers] Obtaining RSE% > > [Some people who received this message don't often get email from > [email protected]. Learn why this is important at > https://aka.ms/LearnAboutSenderIdentification ] > > Dear NMusers, > > This is a great reminder for us to consider the reliability of standard > errors in our models, thanks Ken & Alan. The more non-linear the models > become, the less reliable and the more important other perspectives on > parameter values such as sensitivity analysis and prior knowledge. > > The nmusers archive has many great threads on the topic that are available > to review such as > https://www.mail-archive.com/[email protected]/msg05423.html and > related https://www.mail-archive.com/[email protected]/msg05419.html > . In summary, log-transformation only can get you so far but can perhaps be > seen as a sort of minimal effort. > > To add to the Lewis's quote about SEs - "they are not worth the electrons > used to compute them" (see the links), Pyry had some very interesting > observations he shared with me about the SE of the CV of a log-normal > omega: it inflates with higher values of omega compared to the SE of omega > itself. > > Best regards, > > Jeroen > > http://pd-value.com > [email protected] > @PD_value > +31 6 23118438 > -- More value out of your data! > > On 29-07-2024 14:41, [email protected] wrote: > > > > Dear NMusers, > > > > It was recently pointed out to me by a statistical colleague that my > > recent NMusers post about the inverse Hessian (R matrix) evaluated at > > the maximum likelihood estimates is a consistent estimator of the > > covariance matrix (i.e., converges to the true value with large N) is > > only true for linear models. For nonlinear models, the standard > > errors produced by NONMEM and other nonlinear estimation software are > > not only asymptotic but also approximate. Moreover, how well that > > approximation works will also depend on the parameterization. This I > > believe is one of the motivations behind “mu referencing” in NONMEM > > and the use of log transformations of the parameters to help improve > > Wald-based approximations. I thank Alan Maloney for pointing this out > > to me. > > > > Kind regards, > > > > Ken > > > > *From:*[email protected] <[email protected]> > > *Sent:* Saturday, July 27, 2024 12:36 PM > > *To:* 'Santosh' <[email protected]>; [email protected] > > *Subject:* RE: [NMusers] Obtaining RSE% > > > > Dear Santosh, > > > > There is a good reason for this. Wald (1943) has shown that the > > inverse of the Hessian (R matrix) evaluated at the maximum likelihood > > estimates is a consistent estimator of the covariance matrix. It is > > based on Wald’s approximation that the likelihood surface locally near > > the maximum likelihood estimates can be approximated by a quadratic > > function in the parameters. This theory does not hold for any set of > > parameter estimates along the algorithm’s search path prior to > > convergence to the maximum likelihood estimates. Moreover, inverting > > the Hessian evaluated at an interim step prior to convergence would > > likely be a poor approximation especially early in the search path > > where the gradients are large (i.e., large changes in OFV for a given > > change in the parameters would probably have substantial curvature and > > not be well approximated by a quadratic model in the parameters). > > > > Thus, the COV step in NONMEM is only applied once convergence is > > obtained during the EST step. > > > > Wald, A. “Tests of statistical hypotheses concerning several > > parameters when the number of observations is large.” /Trans. Amer. > > Math. Soc./ 1943;54:426. > > > > Best, > > > > Ken > > > > Kenneth G. Kowalski > > > > President > > > > Kowalski PMetrics Consulting, LLC > > > > Email: [email protected] <mailto:[email protected]> > > > > Cell: 248-207-5082 > > > > *From:*[email protected] > > <mailto:[email protected]><[email protected] > > <mailto:[email protected]>> *On Behalf Of *Santosh > > *Sent:* Friday, July 26, 2024 3:38 AM > > *To:* [email protected] <mailto:[email protected]> > > *Subject:* [NMusers] Obtaining RSE% > > > > Dear esteemed experts! > > > > When using one or more estimation methods & covariance step in a > > NONMEM control stream, the resulting ext file contains final estimate > > (for all estimation steps) & standard error (only for the last > > estimation step). > > > > Is there a way that standard error is generated for every estimation > step? > > > > TIA > > > > Santosh > > > >

> On Jul 29, 2024, at 11:12 AM, <[email protected]> > <[email protected]> wrote: > > Hi Nick, > > I don't want to rehash old debates with you about the diagnostic value of the > COV step. However, your statement about SEs "they are not worth the > electrons expended to compute them" seems hyperbolic to me. I suspect that > what Lewis agreed to was the general sentiment that we need to be cautious in > how we use and interpret the SEs generated by NONMEM. I doubt that he felt > that they have absolutely no value. Indeed, in many of Lewis' papers where > he published modeling results, he reports the standard errors of these > estimates from NONMEM. > > It certainly was not my intent to assert that the SEs and the COV step in > general, have no value. I believe they still do, even if we may not be able > to use them say to construct confidence intervals and expect them to have the > proper coverage probabilities for purposes of statistical inference. > > I do not think a non-parametric bootstrap with the parameter estimates > produced after every iteration is going to tell us anything. If for no other > reason that the iteration search path itself is dependent on the starting > values used. That is, the parameter estimates after each iteration will > depend on where you start. Whereas the maximum likelihood estimates obtained > at convergence to the global minimum OFV, should be somewhat invariant to the > starting values provided the starting values are reasonable. The theory > behind the non-parametric bootstrap standard errors still requires that you > obtain the maximum likelihood estimates for each bootstrap dataset. > > Best, > > Ken > > -----Original Message----- > From: Nick Holford <[email protected]> > Sent: Monday, July 29, 2024 11:52 AM > To: Jeroen Elassaiss-Schaap (PD-value B.V.) <[email protected]>; > [email protected]; 'Santosh' <[email protected]>; > [email protected] > Cc: 'Alan Maloney' <[email protected]>; Pyry Välitalo > <[email protected]> > Subject: RE: [NMusers] Obtaining RSE% > > Hi Jeroen, > > A small correction. Please re-read my email to nmusers on 12 Feb 2015 which I > quote here. Sorry I cannot show the original but the 1999 URL is not > available to me anymore. > > ================= start quote =================== Nick Holford Thu, 12 Feb > 2015 11:54:59 -0800 Hi, The original quote about electrons comes from a > remark I made in 1999 on nmusers. > http://www.cognigencorp.com/nonmem/nm/99nov121999.html > Lewis Sheiner agreed in the same thread. Thanks to the wonders of living on a > sphere Lewis appears to agree with me the day before I made the comment :-) > ================= end quote =================== > > I had been meaning to add to Ken's great email which confirms my original > assertion about electrons. > > If Santosh really wanted to calculate SE's after every "iteration" (which I > think was Ken's interpretation of every "estimation") then this can be done > by running a non-parametric bootstrap with the parameter estimates produced > after every iteration. > > I wonder if Santosh would like to spend a few hours doing that and adding to > the nmusers collection about standard errors by reporting the results to us? > > > Best wishes, > Nick > > > -- > Nick Holford, Professor Emeritus Clinical Pharmacology, MBChB, FRACP > mobile: NZ+64(21) 46 23 53 ; FR+33(6) 62 32 46 72 > email: [email protected] > web: http://holford.fmhs.auckland.ac.nz/ > > -----Original Message----- > From: [email protected] <[email protected]> On Behalf > Of Jeroen Elassaiss-Schaap (PD-value B.V.) > Sent: Monday, July 29, 2024 3:37 PM > To: [email protected]; 'Santosh' <[email protected]>; > [email protected] > Cc: 'Alan Maloney' <[email protected]>; Pyry Välitalo > <[email protected]> > Subject: Re: [NMusers] Obtaining RSE% > > [Some people who received this message don't often get email from > [email protected]. Learn why this is important at > https://aka.ms/LearnAboutSenderIdentification ] > > Dear NMusers, > > This is a great reminder for us to consider the reliability of standard > errors in our models, thanks Ken & Alan. The more non-linear the models > become, the less reliable and the more important other perspectives on > parameter values such as sensitivity analysis and prior knowledge. > > The nmusers archive has many great threads on the topic that are available to > review such as > https://www.mail-archive.com/[email protected]/msg05423.html and related > https://www.mail-archive.com/[email protected]/msg05419.html . In > summary, log-transformation only can get you so far but can perhaps be seen > as a sort of minimal effort. > > To add to the Lewis's quote about SEs - "they are not worth the electrons > used to compute them" (see the links), Pyry had some very interesting > observations he shared with me about the SE of the CV of a log-normal omega: > it inflates with higher values of omega compared to the SE of omega itself. > > Best regards, > > Jeroen > > http://pd-value.com > [email protected] > @PD_value > +31 6 23118438 > -- More value out of your data! > > On 29-07-2024 14:41, [email protected] wrote: >> >> Dear NMusers, >> >> It was recently pointed out to me by a statistical colleague that my >> recent NMusers post about the inverse Hessian (R matrix) evaluated at >> the maximum likelihood estimates is a consistent estimator of the >> covariance matrix (i.e., converges to the true value with large N) is >> only true for linear models. For nonlinear models, the standard >> errors produced by NONMEM and other nonlinear estimation software are >> not only asymptotic but also approximate. Moreover, how well that >> approximation works will also depend on the parameterization. This I >> believe is one of the motivations behind “mu referencing” in NONMEM >> and the use of log transformations of the parameters to help improve >> Wald-based approximations. I thank Alan Maloney for pointing this out >> to me. >> >> Kind regards, >> >> Ken >> >> *From:*[email protected] <[email protected]> >> *Sent:* Saturday, July 27, 2024 12:36 PM >> *To:* 'Santosh' <[email protected]>; [email protected] >> *Subject:* RE: [NMusers] Obtaining RSE% >> >> Dear Santosh, >> >> There is a good reason for this. Wald (1943) has shown that the >> inverse of the Hessian (R matrix) evaluated at the maximum likelihood >> estimates is a consistent estimator of the covariance matrix. It is >> based on Wald’s approximation that the likelihood surface locally near >> the maximum likelihood estimates can be approximated by a quadratic >> function in the parameters. This theory does not hold for any set of >> parameter estimates along the algorithm’s search path prior to >> convergence to the maximum likelihood estimates. Moreover, inverting >> the Hessian evaluated at an interim step prior to convergence would >> likely be a poor approximation especially early in the search path >> where the gradients are large (i.e., large changes in OFV for a given >> change in the parameters would probably have substantial curvature and >> not be well approximated by a quadratic model in the parameters). >> >> Thus, the COV step in NONMEM is only applied once convergence is >> obtained during the EST step. >> >> Wald, A. “Tests of statistical hypotheses concerning several >> parameters when the number of observations is large.” /Trans. Amer. >> Math. Soc./ 1943;54:426. >> >> Best, >> >> Ken >> >> Kenneth G. Kowalski >> >> President >> >> Kowalski PMetrics Consulting, LLC >> >> Email: [email protected] <mailto:[email protected]> >> >> Cell: 248-207-5082 >> >> *From:*[email protected] >> <mailto:[email protected]><[email protected] >> <mailto:[email protected]>> *On Behalf Of *Santosh >> *Sent:* Friday, July 26, 2024 3:38 AM >> *To:* [email protected] <mailto:[email protected]> >> *Subject:* [NMusers] Obtaining RSE% >> >> Dear esteemed experts! >> >> When using one or more estimation methods & covariance step in a >> NONMEM control stream, the resulting ext file contains final estimate >> (for all estimation steps) & standard error (only for the last >> estimation step). >> >> Is there a way that standard error is generated for every estimation step? >> >> TIA >> >> Santosh >> > > >

On Mon, Jul 29, 2024 at 4:43 PM <[email protected]> wrote: > Hi Santosh, > > > > It’s important to note the distinction between transformations of the > parameters and transformations of the data such as the > log-transform-both-sides approach for a PK model to assume the residual > errors are log-normally distributed. Here we are specifically focusing on > transformations of the parameters not the data. Note that the likelihood > is invariant to transformations of the parameters, so you will get the same > fit and OFV whether you estimate log(CL) or CL as your theta. However, > Wald-based standard errors are very much dependent on the parameter > transformation. > > > > For example, suppose we estimate the typical value of CL as THETA(1). > Assuming the maximum likelihood estimate of THETA(1) is asymptotically > normal then we could construct a confidence interval to reflect the > uncertainty in that parameter estimate as > > > > theta1 +/- Zalpha(SE_theta1) > > > > where SE_theta1 is the Wald-based SE for theta1 and Zalpha is the > two-sided critical value of the standard normal distribution to obtain a > 100x(1 – alpha)% confidence interval. Note that this confidence interval > is symmetric about the estimate theta1. > > > > Now consider a log-transformation of CL such that THETA(1) corresponds to > log(CL). The Wald-based confidence interval for log(CL) would now be > > > > theta1 +/- Zalpha(SE_theta1) > > > > which is symmetric in the log(CL) scale. However, the corresponding > confidence interval for CL requires exponentiating the endpoints of the > log(CL) confidence interval to obtain the confidence interval in the > original CL scale. That is, > > > > exp(theta1 +/- Zalpha(SE_theta1) ) > > > > which will be asymmetric about the estimate of the typical value of CL, > exp(theta1). > > > > When the parameter estimate space is highly asymmetric, transformations > can help with this asymmetry so that the transformed estimates are more > likely to be symmetric and normally distributed. So, to answer your > question, the precision of the estimates may still be valid, but we need to > recognize that the uncertainty in the estimates may be asymmetric in the > untransformed (original) space. > > > > Best, > > > > Ken > > > > *From:* [email protected] <[email protected]> *On > Behalf Of *Santosh > *Sent:* Monday, July 29, 2024 1:11 PM > *To:* [email protected] > *Subject:* Re: [NMusers] Obtaining RSE% > > > > Dear Prof Holford, Ken, Alan, Jeroen & others, > > > > Thanks for the engaging discussions. > > > > In context of monitoring at the iteration level, I vaguely recall that in > NMUSERS or in one of ACOP conferences , there was a presentation & > demonstration with R scripts on looking at the convergence and other > parameters in real time. > > > > The interpretations of SEs is interesting based on linear or non-linear > models, and also based on size of variance of parameters. > > > > On a different note, I am also interested in hearing from you about SEs > when estimated based on transformed distribution space and their values & > interpretations in back-transformed space. Would the notion of precision > still be valid when viewing both transformed and untransformed space? > This is in context of dealing with untransformed space of non-normal or > non-lognormal distributions. > > > > Best regards > > Santosh > > > > > > On Mon, Jul 29, 2024 at 8:52 AM Nick Holford <[email protected]> > wrote: > > Hi Jeroen, > > A small correction. Please re-read my email to nmusers on 12 Feb 2015 > which I quote here. Sorry I cannot show the original but the 1999 URL is > not available to me anymore. > > ================= start quote =================== > Nick Holford Thu, 12 Feb 2015 11:54:59 -0800 > Hi, > The original quote about electrons comes from a remark I made in 1999 on > nmusers. > http://www.cognigencorp.com/nonmem/nm/99nov121999.html > Lewis Sheiner agreed in the same thread. Thanks to the wonders of living > on a sphere Lewis appears to agree with me the day before I made the > comment :-) > ================= end quote =================== > > I had been meaning to add to Ken's great email which confirms my original > assertion about electrons. > > If Santosh really wanted to calculate SE's after every "iteration" (which > I think was Ken's interpretation of every "estimation") then this can be > done by running a non-parametric bootstrap with the parameter estimates > produced after every iteration. > > I wonder if Santosh would like to spend a few hours doing that and adding > to the nmusers collection about standard errors by reporting the results to > us? > > > Best wishes, > Nick > > > -- > Nick Holford, Professor Emeritus Clinical Pharmacology, MBChB, FRACP > mobile: NZ+64(21) 46 23 53 ; FR+33(6) 62 32 46 72 > email: [email protected] > web: http://holford.fmhs.auckland.ac.nz/ > > -----Original Message----- > From: [email protected] <[email protected]> On > Behalf Of Jeroen Elassaiss-Schaap (PD-value B.V.) > Sent: Monday, July 29, 2024 3:37 PM > To: [email protected]; 'Santosh' <[email protected]>; > [email protected] > Cc: 'Alan Maloney' <[email protected]>; Pyry Välitalo < > [email protected]> > Subject: Re: [NMusers] Obtaining RSE% > > [Some people who received this message don't often get email from > [email protected]. Learn why this is important at > https://aka.ms/LearnAboutSenderIdentification ] > > Dear NMusers, > > This is a great reminder for us to consider the reliability of standard > errors in our models, thanks Ken & Alan. The more non-linear the models > become, the less reliable and the more important other perspectives on > parameter values such as sensitivity analysis and prior knowledge. > > The nmusers archive has many great threads on the topic that are available > to review such as > https://www.mail-archive.com/[email protected]/msg05423.html and > related https://www.mail-archive.com/[email protected]/msg05419.html > . In summary, log-transformation only can get you so far but can perhaps be > seen as a sort of minimal effort. > > To add to the Lewis's quote about SEs - "they are not worth the electrons > used to compute them" (see the links), Pyry had some very interesting > observations he shared with me about the SE of the CV of a log-normal > omega: it inflates with higher values of omega compared to the SE of omega > itself. > > Best regards, > > Jeroen > > http://pd-value.com > [email protected] > @PD_value > +31 6 23118438 > -- More value out of your data! > > On 29-07-2024 14:41, [email protected] wrote: > > > > Dear NMusers, > > > > It was recently pointed out to me by a statistical colleague that my > > recent NMusers post about the inverse Hessian (R matrix) evaluated at > > the maximum likelihood estimates is a consistent estimator of the > > covariance matrix (i.e., converges to the true value with large N) is > > only true for linear models. For nonlinear models, the standard > > errors produced by NONMEM and other nonlinear estimation software are > > not only asymptotic but also approximate. Moreover, how well that > > approximation works will also depend on the parameterization. This I > > believe is one of the motivations behind “mu referencing” in NONMEM > > and the use of log transformations of the parameters to help improve > > Wald-based approximations. I thank Alan Maloney for pointing this out > > to me. > > > > Kind regards, > > > > Ken > > > > *From:*[email protected] <[email protected]> > > *Sent:* Saturday, July 27, 2024 12:36 PM > > *To:* 'Santosh' <[email protected]>; [email protected] > > *Subject:* RE: [NMusers] Obtaining RSE% > > > > Dear Santosh, > > > > There is a good reason for this. Wald (1943) has shown that the > > inverse of the Hessian (R matrix) evaluated at the maximum likelihood > > estimates is a consistent estimator of the covariance matrix. It is > > based on Wald’s approximation that the likelihood surface locally near > > the maximum likelihood estimates can be approximated by a quadratic > > function in the parameters. This theory does not hold for any set of > > parameter estimates along the algorithm’s search path prior to > > convergence to the maximum likelihood estimates. Moreover, inverting > > the Hessian evaluated at an interim step prior to convergence would > > likely be a poor approximation especially early in the search path > > where the gradients are large (i.e., large changes in OFV for a given > > change in the parameters would probably have substantial curvature and > > not be well approximated by a quadratic model in the parameters). > > > > Thus, the COV step in NONMEM is only applied once convergence is > > obtained during the EST step. > > > > Wald, A. “Tests of statistical hypotheses concerning several > > parameters when the number of observations is large.” /Trans. Amer. > > Math. Soc./ 1943;54:426. > > > > Best, > > > > Ken > > > > Kenneth G. Kowalski > > > > President > > > > Kowalski PMetrics Consulting, LLC > > > > Email: [email protected] <mailto:[email protected]> > > > > Cell: 248-207-5082 > > > > *From:*[email protected] > > <mailto:[email protected]><[email protected] > > <mailto:[email protected]>> *On Behalf Of *Santosh > > *Sent:* Friday, July 26, 2024 3:38 AM > > *To:* [email protected] <mailto:[email protected]> > > *Subject:* [NMusers] Obtaining RSE% > > > > Dear esteemed experts! > > > > When using one or more estimation methods & covariance step in a > > NONMEM control stream, the resulting ext file contains final estimate > > (for all estimation steps) & standard error (only for the last > > estimation step). > > > > Is there a way that standard error is generated for every estimation > step? > > > > TIA > > > > Santosh > > > >