Use AND to ignore rows for input dataset

________________________________ From: [email protected] <[email protected]> on behalf of Mark Tepeck <[email protected]> Sent: Monday, December 16, 2019 15:16 To: Nmusers <[email protected]> Subject: [NMusers] Use AND to ignore rows for input dataset Hi Colleagues, Although there are many workarounds to manage the input datasets for NONMEM, use of IGNORE would be a very handy solution. However, I am surprised to see that AND is not supported by NONMEM. e.g. IGNORE=(CMT.EQ.1, AND CMT.EQ.2 ) would throw an error. In contrast, NONMEM use OR for IGNORE=(CMT.EQ.1, CMT.EQ.2 ) by default, which is quite different for the common practice of software. I am wondering if the AND feature has been or will be implemented in the last/future NONMEM version. Thank you, Mark

From: [email protected] <[email protected]> On Behalf Of Sebastien Bihorel Sent: Monday, December 16, 2019 12:39 PM To: Mark Tepeck <[email protected]>; Nmusers <[email protected]> Subject: Re: [NMusers] Use AND to ignore rows for input dataset I would second that feature request! ________________________________ From: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> on behalf of Mark Tepeck <[email protected]<mailto:[email protected]>> Sent: Monday, December 16, 2019 15:16 To: Nmusers <[email protected]<mailto:[email protected]>> Subject: [NMusers] Use AND to ignore rows for input dataset Hi Colleagues, Although there are many workarounds to manage the input datasets for NONMEM, use of IGNORE would be a very handy solution. However, I am surprised to see that AND is not supported by NONMEM. e.g. IGNORE=(CMT.EQ.1, AND CMT.EQ.2 ) would throw an error. In contrast, NONMEM use OR for IGNORE=(CMT.EQ.1, CMT.EQ.2 ) by default, which is quite different for the common practice of software. I am wondering if the AND feature has been or will be implemented in the last/future NONMEM version. Thank you, Mark

________________________________ From: [email protected] <[email protected]> on behalf of Bauer, Robert <[email protected]> Sent: Monday, December 16, 2019 17:39 To: Nmusers <[email protected]> Subject: RE: [NMusers] Use AND to ignore rows for input dataset Guide VIII suggests the following: Suppose it is desired that records be dropped that satisfy the logical ".AND." of several conditions. This can be implemented by using an ACCEPT list with the negations of the conditions. For example, suppose that records to be ignored are those having GEN=1 .AND. AGE > 60. This may be done as follows: ACCEPT=(GEN.NE.1,AGE.LE.60) Robert J. Bauer, Ph.D. Senior Director Pharmacometrics R&D ICON Early Phase 820 W. Diamond Avenue Suite 100 Gaithersburg, MD 20878 Office: (215) 616-6428 Mobile: (925) 286-0769 [email protected]<mailto:[email protected]> http://www.iconplc.com/ From: [email protected] <[email protected]> On Behalf Of Sebastien Bihorel Sent: Monday, December 16, 2019 12:39 PM To: Mark Tepeck <[email protected]>; Nmusers <[email protected]> Subject: Re: [NMusers] Use AND to ignore rows for input dataset I would second that feature request! ________________________________ From: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> on behalf of Mark Tepeck <[email protected]<mailto:[email protected]>> Sent: Monday, December 16, 2019 15:16 To: Nmusers <[email protected]<mailto:[email protected]>> Subject: [NMusers] Use AND to ignore rows for input dataset Hi Colleagues, Although there are many workarounds to manage the input datasets for NONMEM, use of IGNORE would be a very handy solution. However, I am surprised to see that AND is not supported by NONMEM. e.g. IGNORE=(CMT.EQ.1, AND CMT.EQ.2 ) would throw an error. In contrast, NONMEM use OR for IGNORE=(CMT.EQ.1, CMT.EQ.2 ) by default, which is quite different for the common practice of software. I am wondering if the AND feature has been or will be implemented in the last/future NONMEM version. Thank you, Mark

From: Sebastien Bihorel <[email protected]> Sent: Monday, December 16, 2019 5:02 PM To: Bauer, Robert <[email protected]>; Nmusers <[email protected]> Subject: Re: [NMusers] Use AND to ignore rows for input dataset Since nowadays most datasets include a header line, IGNORE=@ (or something along those lines) is almost always used in control streams. Since ACCEPT and IGNORE cannot be used in the same $DATA record, the suggested technique is seldom usable in my experience. ________________________________ From: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> on behalf of Bauer, Robert <[email protected]<mailto:[email protected]>> Sent: Monday, December 16, 2019 17:39 To: Nmusers <[email protected]<mailto:[email protected]>> Subject: RE: [NMusers] Use AND to ignore rows for input dataset Guide VIII suggests the following: Suppose it is desired that records be dropped that satisfy the logical ".AND." of several conditions. This can be implemented by using an ACCEPT list with the negations of the conditions. For example, suppose that records to be ignored are those having GEN=1 .AND. AGE > 60. This may be done as follows: ACCEPT=(GEN.NE.1,AGE.LE.60) Robert J. Bauer, Ph.D. Senior Director Pharmacometrics R&D ICON Early Phase 820 W. Diamond Avenue Suite 100 Gaithersburg, MD 20878 Office: (215) 616-6428 Mobile: (925) 286-0769 [email protected]<mailto:[email protected]> http://www.iconplc.com/ From: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> On Behalf Of Sebastien Bihorel Sent: Monday, December 16, 2019 12:39 PM To: Mark Tepeck <[email protected]<mailto:[email protected]>>; Nmusers <[email protected]<mailto:[email protected]>> Subject: Re: [NMusers] Use AND to ignore rows for input dataset I would second that feature request! ________________________________ From: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> on behalf of Mark Tepeck <[email protected]<mailto:[email protected]>> Sent: Monday, December 16, 2019 15:16 To: Nmusers <[email protected]<mailto:[email protected]>> Subject: [NMusers] Use AND to ignore rows for input dataset Hi Colleagues, Although there are many workarounds to manage the input datasets for NONMEM, use of IGNORE would be a very handy solution. However, I am surprised to see that AND is not supported by NONMEM. e.g. IGNORE=(CMT.EQ.1, AND CMT.EQ.2 ) would throw an error. In contrast, NONMEM use OR for IGNORE=(CMT.EQ.1, CMT.EQ.2 ) by default, which is quite different for the common practice of software. I am wondering if the AND feature has been or will be implemented in the last/future NONMEM version. Thank you, Mark

On Mon, Dec 16, 2019 at 9:41 PM Bauer, Robert <[email protected]> wrote: > > Sebastien: > > It is not clear in the guide, but only parenthesized IGNORE and ACCEPT lists, > such as IGNORE=(list) and ACCEPT=(list), are incompatible. IGNORE=c, where > c could be any character except space, should work with IGNORE=(list) or > ACCEPT=(list). For example: > > > > $DATA example1.csv ACCEPT=(CMT.EQN.1) IGNORE=@ > > > > Also, any data record starting with first non-blank character of # will > always be ignored by NMTRAN. > > Robert J. Bauer, Ph.D. > > Senior Director > > Pharmacometrics R&D > > ICON Early Phase > > 820 W. Diamond Avenue > > Suite 100 > > Gaithersburg, MD 20878 > > Office: (215) 616-6428 > > Mobile: (925) 286-0769 > > [email protected] > > www.iconplc.com > > > > From: Sebastien Bihorel <[email protected]> > Sent: Monday, December 16, 2019 5:02 PM > To: Bauer, Robert <[email protected]>; Nmusers <[email protected]> > Subject: Re: [NMusers] Use AND to ignore rows for input dataset > > > > Since nowadays most datasets include a header line, IGNORE=@ (or something > along those lines) is almost always used in control streams. Since ACCEPT and > IGNORE cannot be used in the same $DATA record, the suggested technique is > seldom usable in my experience. > > ________________________________ > > From: [email protected] <[email protected]> on behalf > of Bauer, Robert <[email protected]> > Sent: Monday, December 16, 2019 17:39 > To: Nmusers <[email protected]> > Subject: RE: [NMusers] Use AND to ignore rows for input dataset > > > > Guide VIII suggests the following: > > > > Suppose it is desired that records be dropped that satisfy the logical > ".AND." of several conditions. This can be implemented by using an ACCEPT > list with the negations of the conditions. For example, suppose that records > to be ignored are those having GEN=1 .AND. AGE > 60. This may be done as > follows: > > ACCEPT=(GEN.NE.1,AGE.LE.60) > > > > Robert J. Bauer, Ph.D. > > Senior Director > > Pharmacometrics R&D > > ICON Early Phase > > 820 W. Diamond Avenue > > Suite 100 > > Gaithersburg, MD 20878 > > Office: (215) 616-6428 > > Mobile: (925) 286-0769 > > [email protected] > > www.iconplc.com > > > > From: [email protected] <[email protected]> On Behalf > Of Sebastien Bihorel > Sent: Monday, December 16, 2019 12:39 PM > To: Mark Tepeck <[email protected]>; Nmusers <[email protected]> > Subject: Re: [NMusers] Use AND to ignore rows for input dataset > > > > I would second that feature request! > > ________________________________ > > From: [email protected] <[email protected]> on behalf > of Mark Tepeck <[email protected]> > Sent: Monday, December 16, 2019 15:16 > To: Nmusers <[email protected]> > Subject: [NMusers] Use AND to ignore rows for input dataset > > > > Hi Colleagues, > > > > Although there are many workarounds to manage the input datasets for NONMEM, > use of IGNORE would be a very handy solution. However, I am surprised to see > that AND is not supported by NONMEM. e.g. IGNORE=(CMT.EQ.1, AND CMT.EQ.2 ) > would throw an error. In contrast, NONMEM use OR for IGNORE=(CMT.EQ.1, > CMT.EQ.2 ) by default, which is quite different for the common practice of > software. > > > > I am wondering if the AND feature has been or will be implemented in the > last/future NONMEM version. > > > > Thank you, > > > > Mark > > > >

From: Mark Tepeck <[email protected]> Sent: Tuesday, December 17, 2019 8:21 PM To: Nmusers <[email protected]> Cc: Bauer, Robert <[email protected]> Subject: Re: [NMusers] Use AND to ignore rows for input dataset Hi Bob, Thanks for providing the solution applicable to some cases. Meanwhile, I would like to share my thoughts on this issue. My sole purpose is to make NONMEM even better. 1. Use of OR alone may be subject to a very limited utility. Actually, my scenario is way more complicated. E.g. I want to exclude the year-span between (10,18) and (60,70) from the pool of (0,90). I cannot find a solution with ACCEPT/OR for this case. 2. Implicit/default use of OR is quite counter-intuitive and could raise the NONMEM learning curve. As we all know, AND (instead of OR) is the widely implemented default operator, E.g., Google search and programming languages (like R package dplyr/filter). NONMEM developers may consider following this general practice. 3. A lot of solutions provided on this platform come from the Guide -VIII. They are still useful but seems cumbersome from the current perspective. While we stick to a six-year-old guide, we entered a new era of machine learning, artificial intelligence, self-driving car, smart phones, etc, technologies are evolving quickly to become much easier than ever to use. Maybe, we can make NONMEM more user-friendly as well by solving this long-standing issue. 😊 With this fixed, IGNORE can serve as a universal filtering command. I really appreciate your input and feel free to correct me if I am wrong. Mark

On Fri, Dec 20, 2019 at 10:42 AM Bauer, Robert <[email protected]> wrote: > Mark: > > The best solution would be if an additional ignore filtering can be done > within a code record, so any degree of elaborate filtering can be done. > For example: > > > > $INFN > > DATA_IGNORE=0 > > IF(SEX==1 .AND. BW<30) DATA_IGNORE=1 > > IF(AGE>10 .AND. AGE<70) DATA_IGNORE=1 > > IF(GEN1==3 .OR. GEN2==10 .AND. GEN3>20) DATA_IGNORE=1 > > > > This can be available in the next version NONMEM 7.5 > > > > Guide VIII is kept up to date with the version. The date of guide VIII is > January 2018. > > > > > > Robert J. Bauer, Ph.D. > > Senior Director > > Pharmacometrics R&D > > ICON Early Phase > > 820 W. Diamond Avenue > > Suite 100 > > Gaithersburg, MD 20878 > > Office: (215) 616-6428 > > Mobile: (925) 286-0769 > > [email protected] > > www.iconplc.com > > > > *From:* Mark Tepeck <[email protected]> > *Sent:* Tuesday, December 17, 2019 8:21 PM > *To:* Nmusers <[email protected]> > *Cc:* Bauer, Robert <[email protected]> > *Subject:* Re: [NMusers] Use AND to ignore rows for input dataset > > > > Hi Bob, > > Thanks for providing the solution applicable to some cases. Meanwhile, > I would like to share my thoughts on this issue. My sole purpose is to > make NONMEM even better. > > 1. Use of OR alone may be subject to a very limited utility. Actually, > my scenario is way more complicated. E.g. I want to exclude the > year-span between (10,18) and (60,70) from the pool of (0,90). I > cannot find a solution with ACCEPT/OR for this case. > > 2. Implicit/default use of OR is quite counter-intuitive and could > raise the NONMEM learning curve. As we all know, AND (instead of OR) > is the widely implemented default operator, E.g., Google search and > programming languages (like R package dplyr/filter). NONMEM developers > may consider following this general practice. > > 3. A lot of solutions provided on this platform come from the Guide > -VIII. They are still useful but seems cumbersome from the current > perspective. While we stick to a six-year-old guide, we entered a new > era of machine learning, artificial intelligence, self-driving car, > smart phones, etc, technologies are evolving quickly to become much > easier than ever to use. Maybe, we can make NONMEM more user-friendly > as well by solving this long-standing issue. 😊 With this fixed, > IGNORE can serve as a universal filtering command. > > I really appreciate your input and feel free to correct me if I am wrong. > > Mark > > > > > >

From: Mark Tepeck <[email protected]> Sent: Monday, December 23, 2019 10:50 AM To: Bauer, Robert <[email protected]> Cc: Nmusers <[email protected]> Subject: Re: [NMusers] Use AND to ignore rows for input dataset Hi Bob, Thank you so much for your feedback. I am so glad to see that NONMEM 7.5 offers a full treatment to this issue, which would definitely improve end-user experience. I can find other guides but VIII at the root folder of https://nonmem.iconplc.com/?token=8D25A977-25B2-11ea-A324-005056911489. Would you please direct me to the right link to the latest version of guide VIII? Have a good holiday, Mark On Fri, Dec 20, 2019 at 10:42 AM Bauer, Robert <[email protected]<mailto:[email protected]>> wrote: Mark: The best solution would be if an additional ignore filtering can be done within a code record, so any degree of elaborate filtering can be done. For example: $INFN DATA_IGNORE=0 IF(SEX==1 .AND. BW<30) DATA_IGNORE=1 IF(AGE>10 .AND. AGE<70) DATA_IGNORE=1 IF(GEN1==3 .OR. GEN2==10 .AND. GEN3>20) DATA_IGNORE=1 This can be available in the next version NONMEM 7.5 Guide VIII is kept up to date with the version. The date of guide VIII is January 2018. Robert J. Bauer, Ph.D. Senior Director Pharmacometrics R&D ICON Early Phase 820 W. Diamond Avenue Suite 100 Gaithersburg, MD 20878 Office: (215) 616-6428 Mobile: (925) 286-0769 [email protected]<mailto:[email protected]> http://www.iconplc.com/ From: Mark Tepeck <[email protected]<mailto:[email protected]>> Sent: Tuesday, December 17, 2019 8:21 PM To: Nmusers <[email protected]<mailto:[email protected]>> Cc: Bauer, Robert <[email protected]<mailto:[email protected]>> Subject: Re: [NMusers] Use AND to ignore rows for input dataset Hi Bob, Thanks for providing the solution applicable to some cases. Meanwhile, I would like to share my thoughts on this issue. My sole purpose is to make NONMEM even better. 1. Use of OR alone may be subject to a very limited utility. Actually, my scenario is way more complicated. E.g. I want to exclude the year-span between (10,18) and (60,70) from the pool of (0,90). I cannot find a solution with ACCEPT/OR for this case. 2. Implicit/default use of OR is quite counter-intuitive and could raise the NONMEM learning curve. As we all know, AND (instead of OR) is the widely implemented default operator, E.g., Google search and programming languages (like R package dplyr/filter). NONMEM developers may consider following this general practice. 3. A lot of solutions provided on this platform come from the Guide -VIII. They are still useful but seems cumbersome from the current perspective. While we stick to a six-year-old guide, we entered a new era of machine learning, artificial intelligence, self-driving car, smart phones, etc, technologies are evolving quickly to become much easier than ever to use. Maybe, we can make NONMEM more user-friendly as well by solving this long-standing issue. 😊 With this fixed, IGNORE can serve as a universal filtering command. I really appreciate your input and feel free to correct me if I am wrong. Mark