Validation Strategy for NONMEM

On Tue, Oct 21, 2008 at 9:18 AM, <[EMAIL PROTECTED]> wrote: > > Dear Mark, > > I am engaged in this question since the beginning of this year (not finished > yet), and I am happy to share some basics of my experiences: > > 1. It is important to specify where the data for NONMEM analysis come > from. If they come from a GCP source and are already QA-ed when they arrive > at the doorstep of NONMEM then your system will only subject to GCP > regulation. Otherwise, you also have to comply with GLP. > > 2. There is no way around what is called here a 'Validation Plan' and > a 'Risk Analysis'. These documents will trigger a slate of other documents > (in our case here about 15) which describe Installation, Installation > Validation, Qualification of Users, Modeling Strategy, Review Processes, > System Life Cycle Management etc. > > 3. We found it useful to differentiate between 'Exploratory Work' and > 'Submission Work'. > > 4. Before you worry about passing inspection by the FDA, you need to > worry about passing inspection of your own company QA officers. > > 5. Just installing NONMEM with NMQual does not render you new system > 'validated' or 'qualified'. Here my apologies to the excellent folks at > Metrum, but for various reasons, we ended up not using NMQual. > > 6. You have to know what you are trying to build before you concern > yourself about QA processes. A number of separate installations on PCs > linked to a file server is a different animal from a server-based > installation with a grid engine. > > 7. It all takes more time than you think: make generous budgets and > time lines. > > I hope I helped more than I confused, > > Joachim > > __________________________________________ > Joachim GREVEL, Ph.D. > MERCK SERONO International S.A. > Exploratory Medicine > 1202 Geneva > Tel: +41.22.414.4751 > Fax: +41.22.414.3059 > Email: [EMAIL PROTECTED] > > > > > "Vilicich, Mark" <[EMAIL PROTECTED]> > Sent by: [EMAIL PROTECTED] > > 10/17/2008 10:08 PM > > To > <[email protected]> > cc > Subject > [NMusers] Validation Strategy for NONMEM > > > > > Dear All, > > I am interested in perspectives on strategies for "validating" NONMEM. Also, > experiences from or with the FDA since the FDA is: a key user, customer of > analysis and auditor of NONMEM use in the industry. Without a large nonmem > staff here, the challenge I see is in scaling the validation strategy to > provide the most efficient environment for doing analysis that is defensible > to both internal and external audits based on the associated GxP risk level. > > Below are the concepts I've cobbled together, though instead of my > reinventing the wheel I appreciate anything you could share. Any and all > gems of insight you can share whether it regard the big picture or some > detailed specifics, IT centric or business process related. You may send > them back to the listserver or me directly as you feel appropriate. > > Details: > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > From searching the archives and other random bits of knowledge on NONMEM, > part of the validation strategy is to recognize that NONMEM is not to be > literally validated. NONMEM may be considered more of a development > environment, optimized for developing specialized forms of complex analysis > and modeling. As a development platform, an approach could be that NONMEM > itself is qualified and each specific analysis is validated individually. > > To support establishing a defensible NONMEM environment, I've also read > discussions on integrating common software development best practices such > as version control of the "programming" of nonmem, NMQual and other > commercial and custom tools for capturing all the metadata related to > running a specific NONMEM job. These themes support defining the state of > the NONMEM environment and ability to reproduce the outcomes. > > Also, reading in the archives about the differences in the numeric outcomes > of NONMEM analyses based on the hardware platform, etc. are helpful to know > up front and to consider in the validation strategy so it is not destined to > failure if the target environment is multiplatform or otherwise complex. > > Gaps noticed/topics not discussed: > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Is there opportunity in looking at the risk based approached sanctioned by > the FDA a few years ago that would make the total validation deliverable, > including both the application and the model development process, more lean > and targeted at the primary risk targets? > > Does this scientific software environment lend itself to use of modern agile > software development methodologies that go far beyond basic iterative > approaches. These methodologies are being used in software development for > the regulated/GxP industry. > > I've seen the excellent presentation from 2004 that Joga Gobburu from the > FDA gave, seems like there has been some progression of thought or actions > on the proposals included there. Any references to follow-up information on > it would be helpful? > > Regards , > > Mark Vilicich > Early Development > [EMAIL PROTECTED] > > ________________________________ > > This message and any attachment are confidential, may be privileged or > otherwise protected from disclosure and are intended only for use by the > addressee(s) named herein. If you are not the intended recipient, you must > not copy this message or attachment or disclose the contents to any other > person. If you have received this transmission in error, please notify the > sender immediately and delete the message and any attachment from your > system.

> On Tue, Oct 21, 2008 at 7:37 PM, <[EMAIL PROTECTED]> wrote: > > > After reading this, it is no wonder scientific productivity is at an > > all-time low. Imagine if Marie Curie had to qualify her radium-purifying > > equipment, or if Alexander Fleming had to validate his petri dishes before > > culturing Penecillium. One day scientists are going to push back against > > these IT people, who just make busy work for everyone. > > > > [EMAIL PROTECTED] > > Sent by: [EMAIL PROTECTED] > > > > 21-Oct-2008 03:18 > > > > To > > [email protected] > > cc > > Subject > > Re: [NMusers] Validation Strategy for NONMEM > > > > Dear Mark, > > > > I am engaged in this question since the beginning of this year (not finished > > yet), and I am happy to share some basics of my experiences: > > > > 1. It is important to specify where the data for NONMEM analysis come > > from. If they come from a GCP source and are already QA-ed when they arrive > > at the doorstep of NONMEM then your system will only subject to GCP > > regulation. Otherwise, you also have to comply with GLP. > > > > 2. There is no way around what is called here a 'Validation Plan' and > > a 'Risk Analysis'. These documents will trigger a slate of other documents > > (in our case here about 15) which describe Installation, Installation > > Validation, Qualification of Users, Modeling Strategy, Review Processes, > > System Life Cycle Management etc. > > > > 3. We found it useful to differentiate between 'Exploratory Work' and > > 'Submission Work'. > > > > 4. Before you worry about passing inspection by the FDA, you need to > > worry about passing inspection of your own company QA officers. > > > > 5. Just installing NONMEM with NMQual does not render you new system > > 'validated' or 'qualified'. Here my apologies to the excellent folks at > > Metrum, but for various reasons, we ended up not using NMQual. > > > > 6. You have to know what you are trying to build before you concern > > yourself about QA processes. A number of separate installations on PCs > > linked to a file server is a different animal from a server-based > > installation with a grid engine. > > > > 7. It all takes more time than you think: make generous budgets and > > time lines. > > > > I hope I helped more than I confused, > > > > Joachim > > > > __________________________________________ > > Joachim GREVEL, Ph.D. > > MERCK SERONO International S.A. > > Exploratory Medicine > > 1202 Geneva > > Tel: +41.22.414.4751 > > Fax: +41.22.414.3059 > > Email: [EMAIL PROTECTED] > > > > "Vilicich, Mark" <[EMAIL PROTECTED]> > > Sent by: [EMAIL PROTECTED] > > > > 10/17/2008 10:08 PM > > > > To > > <[email protected]> > > cc > > Subject > > [NMusers] Validation Strategy for NONMEM > > > > Dear All, > > > > I am interested in perspectives on strategies for "validating" NONMEM. Also, > > experiences from or with the FDA since the FDA is: a key user, customer of > > analysis and auditor of NONMEM use in the industry. Without a large nonmem > > staff here, the challenge I see is in scaling the validation strategy to > > provide the most efficient environment for doing analysis that is defensible > > to both internal and external audits based on the associated GxP risk level. > > > > Below are the concepts I've cobbled together, though instead of my > > reinventing the wheel I appreciate anything you could share. Any and all > > gems of insight you can share whether it regard the big picture or some > > detailed specifics, IT centric or business process related. You may send > > them back to the listserver or me directly as you feel appropriate. > > > > Details: > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > From searching the archives and other random bits of knowledge on NONMEM, > > part of the validation strategy is to recognize that NONMEM is not to be > > literally validated. NONMEM may be considered more of a development > > environment, optimized for developing specialized forms of complex analysis > > and modeling. As a development platform, an approach could be that NONMEM > > itself is qualified and each specific analysis is validated individually. > > > > To support establishing a defensible NONMEM environment, I've also read > > discussions on integrating common software development best practices such > > as version control of the "programming" of nonmem, NMQual and other > > commercial and custom tools for capturing all the metadata related to > > running a specific NONMEM job. These themes support defining the state of > > the NONMEM environment and ability to reproduce the outcomes. > > > > Also, reading in the archives about the differences in the numeric outcomes > > of NONMEM analyses based on the hardware platform, etc. are helpful to know > > up front and to consider in the validation strategy so it is not destined to > > failure if the target environment is multiplatform or otherwise complex. > > > > Gaps noticed/topics not discussed: > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Is there opportunity in looking at the risk based approached sanctioned by > > the FDA a few years ago that would make the total validation deliverable, > > including both the application and the model development process, more lean > > and targeted at the primary risk targets? > > > > Does this scientific software environment lend itself to use of modern agile > > software development methodologies that go far beyond basic iterative > > approaches. These methodologies are being used in software development for > > the regulated/GxP industry. > > > > I've seen the excellent presentation from 2004 that Joga Gobburu from the > > FDA gave, seems like there has been some progression of thought or actions > > on the proposals included there. Any references to follow-up information on > > it would be helpful? > > > > Regards , > > > > Mark Vilicich > > Early Development > > [EMAIL PROTECTED] > > > > ________________________________ > > > > This message and any attachment are confidential, may be privileged or > > otherwise protected from disclosure and are intended only for use by the > > addressee(s) named herein. If you are not the intended recipient, you must > > not copy this message or attachment or disclose the contents to any other > > person. If you have received this transmission in error, please notify the > > sender immediately and delete the message and any attachment from your > > system.

On Tue, Oct 21, 2008 at 4:11 PM, Nick Holford <[EMAIL PROTECTED]>wrote: > Tony, > > To expand a bit on your useful validation definition. > > Making sure computational results are reproducible does not mean they are > correct. I believe that most NONMEM analyses cannot be proven to be correct > therefore the validation efforts only assure reproducibility. Whether this > makes the validation useful or useless is debatable. > > Nick > > > > > A.J. Rossini wrote: > >> There is useless validation, and then there is useful validation. The >> latter is about making sure your computational results are >> reproducible, the former is about making sure that your documentation >> can be photocopied. It's sort of the same thing, if you aren't a >> modeler. >> >> Unfortunately, most on this list tend to be modelers. >> >> On Tue, Oct 21, 2008 at 7:37 PM, <[EMAIL PROTECTED]> wrote: >> >> >>> After reading this, it is no wonder scientific productivity is at an >>> all-time low. Imagine if Marie Curie had to qualify her radium-purifying >>> equipment, or if Alexander Fleming had to validate his petri dishes >>> before >>> culturing Penecillium. One day scientists are going to push back against >>> these IT people, who just make busy work for everyone. >>> >>> >>> >>> [EMAIL PROTECTED] >>> Sent by: [EMAIL PROTECTED] >>> >>> 21-Oct-2008 03:18 >>> >>> >>> To >>> [email protected] >>> cc >>> Subject >>> Re: [NMusers] Validation Strategy for NONMEM >>> >>> >>> >>> >>> >>> Dear Mark, >>> >>> I am engaged in this question since the beginning of this year (not >>> finished >>> yet), and I am happy to share some basics of my experiences: >>> >>> 1. It is important to specify where the data for NONMEM analysis >>> come >>> from. If they come from a GCP source and are already QA-ed when they >>> arrive >>> at the doorstep of NONMEM then your system will only subject to GCP >>> regulation. Otherwise, you also have to comply with GLP. >>> >>> 2. There is no way around what is called here a 'Validation Plan' >>> and >>> a 'Risk Analysis'. These documents will trigger a slate of other >>> documents >>> (in our case here about 15) which describe Installation, Installation >>> Validation, Qualification of Users, Modeling Strategy, Review Processes, >>> System Life Cycle Management etc. >>> >>> 3. We found it useful to differentiate between 'Exploratory Work' >>> and >>> 'Submission Work'. >>> >>> 4. Before you worry about passing inspection by the FDA, you need >>> to >>> worry about passing inspection of your own company QA officers. >>> >>> 5. Just installing NONMEM with NMQual does not render you new >>> system >>> 'validated' or 'qualified'. Here my apologies to the excellent folks at >>> Metrum, but for various reasons, we ended up not using NMQual. >>> >>> 6. You have to know what you are trying to build before you >>> concern >>> yourself about QA processes. A number of separate installations on PCs >>> linked to a file server is a different animal from a server-based >>> installation with a grid engine. >>> >>> 7. It all takes more time than you think: make generous budgets >>> and >>> time lines. >>> >>> I hope I helped more than I confused, >>> >>> Joachim >>> >>> __________________________________________ >>> Joachim GREVEL, Ph.D. >>> MERCK SERONO International S.A. >>> Exploratory Medicine >>> 1202 Geneva >>> Tel: +41.22.414.4751 >>> Fax: +41.22.414.3059 >>> Email: [EMAIL PROTECTED] >>> >>> >>> >>> "Vilicich, Mark" <[EMAIL PROTECTED]> >>> Sent by: [EMAIL PROTECTED] >>> >>> 10/17/2008 10:08 PM >>> >>> To >>> <[email protected]> >>> cc >>> Subject >>> [NMusers] Validation Strategy for NONMEM >>> >>> >>> >>> >>> >>> >>> Dear All, >>> >>> I am interested in perspectives on strategies for "validating" NONMEM. >>> Also, >>> experiences from or with the FDA since the FDA is: a key user, customer >>> of >>> analysis and auditor of NONMEM use in the industry. Without a large >>> nonmem >>> staff here, the challenge I see is in scaling the validation strategy to >>> provide the most efficient environment for doing analysis that is >>> defensible >>> to both internal and external audits based on the associated GxP risk >>> level. >>> >>> Below are the concepts I've cobbled together, though instead of my >>> reinventing the wheel I appreciate anything you could share. Any and all >>> gems of insight you can share whether it regard the big picture or some >>> detailed specifics, IT centric or business process related. You may send >>> them back to the listserver or me directly as you feel appropriate. >>> >>> Details: >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> From searching the archives and other random bits of knowledge on NONMEM, >>> part of the validation strategy is to recognize that NONMEM is not to be >>> literally validated. NONMEM may be considered more of a development >>> environment, optimized for developing specialized forms of complex >>> analysis >>> and modeling. As a development platform, an approach could be that NONMEM >>> itself is qualified and each specific analysis is validated individually. >>> >>> To support establishing a defensible NONMEM environment, I've also read >>> discussions on integrating common software development best practices >>> such >>> as version control of the "programming" of nonmem, NMQual and other >>> commercial and custom tools for capturing all the metadata related to >>> running a specific NONMEM job. These themes support defining the state of >>> the NONMEM environment and ability to reproduce the outcomes. >>> >>> Also, reading in the archives about the differences in the numeric >>> outcomes >>> of NONMEM analyses based on the hardware platform, etc. are helpful to >>> know >>> up front and to consider in the validation strategy so it is not destined >>> to >>> failure if the target environment is multiplatform or otherwise complex. >>> >>> Gaps noticed/topics not discussed: >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> Is there opportunity in looking at the risk based approached sanctioned >>> by >>> the FDA a few years ago that would make the total validation deliverable, >>> including both the application and the model development process, more >>> lean >>> and targeted at the primary risk targets? >>> >>> Does this scientific software environment lend itself to use of modern >>> agile >>> software development methodologies that go far beyond basic iterative >>> approaches. These methodologies are being used in software development >>> for >>> the regulated/GxP industry. >>> >>> I've seen the excellent presentation from 2004 that Joga Gobburu from the >>> FDA gave, seems like there has been some progression of thought or >>> actions >>> on the proposals included there. Any references to follow-up information >>> on >>> it would be helpful? >>> >>> Regards , >>> >>> Mark Vilicich >>> Early Development >>> [EMAIL PROTECTED] >>> >>> ________________________________ >>> >>> This message and any attachment are confidential, may be privileged or >>> otherwise protected from disclosure and are intended only for use by the >>> addressee(s) named herein. If you are not the intended recipient, you >>> must >>> not copy this message or attachment or disclose the contents to any other >>> person. If you have received this transmission in error, please notify >>> the >>> sender immediately and delete the message and any attachment from your >>> system.

On Tue, Oct 21, 2008 at 7:37 PM, <[EMAIL PROTECTED]> wrote: > > After reading this, it is no wonder scientific productivity is at an > all-time low. Imagine if Marie Curie had to qualify her radium-purifying > equipment, or if Alexander Fleming had to validate his petri dishes before > culturing Penecillium. One day scientists are going to push back against > these IT people, who just make busy work for everyone. > > > > [EMAIL PROTECTED] > Sent by: [EMAIL PROTECTED] > > 21-Oct-2008 03:18 > > > To > [email protected] > cc > Subject > Re: [NMusers] Validation Strategy for NONMEM > > > > > > Dear Mark, > > I am engaged in this question since the beginning of this year (not finished > yet), and I am happy to share some basics of my experiences: > > 1. It is important to specify where the data for NONMEM analysis come > from. If they come from a GCP source and are already QA-ed when they arrive > at the doorstep of NONMEM then your system will only subject to GCP > regulation. Otherwise, you also have to comply with GLP. > > 2. There is no way around what is called here a 'Validation Plan' and > a 'Risk Analysis'. These documents will trigger a slate of other documents > (in our case here about 15) which describe Installation, Installation > Validation, Qualification of Users, Modeling Strategy, Review Processes, > System Life Cycle Management etc. > > 3. We found it useful to differentiate between 'Exploratory Work' and > 'Submission Work'. > > 4. Before you worry about passing inspection by the FDA, you need to > worry about passing inspection of your own company QA officers. > > 5. Just installing NONMEM with NMQual does not render you new system > 'validated' or 'qualified'. Here my apologies to the excellent folks at > Metrum, but for various reasons, we ended up not using NMQual. > > 6. You have to know what you are trying to build before you concern > yourself about QA processes. A number of separate installations on PCs > linked to a file server is a different animal from a server-based > installation with a grid engine. > > 7. It all takes more time than you think: make generous budgets and > time lines. > > I hope I helped more than I confused, > > Joachim > > __________________________________________ > Joachim GREVEL, Ph.D. > MERCK SERONO International S.A. > Exploratory Medicine > 1202 Geneva > Tel: +41.22.414.4751 > Fax: +41.22.414.3059 > Email: [EMAIL PROTECTED] > > > > "Vilicich, Mark" <[EMAIL PROTECTED]> > Sent by: [EMAIL PROTECTED] > > 10/17/2008 10:08 PM > > To > <[email protected]> > cc > Subject > [NMusers] Validation Strategy for NONMEM > > > > > > > Dear All, > > I am interested in perspectives on strategies for "validating" NONMEM. Also, > experiences from or with the FDA since the FDA is: a key user, customer of > analysis and auditor of NONMEM use in the industry. Without a large nonmem > staff here, the challenge I see is in scaling the validation strategy to > provide the most efficient environment for doing analysis that is defensible > to both internal and external audits based on the associated GxP risk level. > > Below are the concepts I've cobbled together, though instead of my > reinventing the wheel I appreciate anything you could share. Any and all > gems of insight you can share whether it regard the big picture or some > detailed specifics, IT centric or business process related. You may send > them back to the listserver or me directly as you feel appropriate. > > Details: > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > From searching the archives and other random bits of knowledge on NONMEM, > part of the validation strategy is to recognize that NONMEM is not to be > literally validated. NONMEM may be considered more of a development > environment, optimized for developing specialized forms of complex analysis > and modeling. As a development platform, an approach could be that NONMEM > itself is qualified and each specific analysis is validated individually. > > To support establishing a defensible NONMEM environment, I've also read > discussions on integrating common software development best practices such > as version control of the "programming" of nonmem, NMQual and other > commercial and custom tools for capturing all the metadata related to > running a specific NONMEM job. These themes support defining the state of > the NONMEM environment and ability to reproduce the outcomes. > > Also, reading in the archives about the differences in the numeric outcomes > of NONMEM analyses based on the hardware platform, etc. are helpful to know > up front and to consider in the validation strategy so it is not destined to > failure if the target environment is multiplatform or otherwise complex. > > Gaps noticed/topics not discussed: > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Is there opportunity in looking at the risk based approached sanctioned by > the FDA a few years ago that would make the total validation deliverable, > including both the application and the model development process, more lean > and targeted at the primary risk targets? > > Does this scientific software environment lend itself to use of modern agile > software development methodologies that go far beyond basic iterative > approaches. These methodologies are being used in software development for > the regulated/GxP industry. > > I've seen the excellent presentation from 2004 that Joga Gobburu from the > FDA gave, seems like there has been some progression of thought or actions > on the proposals included there. Any references to follow-up information on > it would be helpful? > > Regards , > > Mark Vilicich > Early Development > [EMAIL PROTECTED] > > ________________________________ > > This message and any attachment are confidential, may be privileged or > otherwise protected from disclosure and are intended only for use by the > addressee(s) named herein. If you are not the intended recipient, you must > not copy this message or attachment or disclose the contents to any other > person. If you have received this transmission in error, please notify the > sender immediately and delete the message and any attachment from your > system.

On Wed, Oct 22, 2008 at 1:21 AM, <[EMAIL PROTECTED]> wrote: > > Dear all, > > one of the more useful things we did was an exchange of data files and > models with a prominent member of our modeling community. This way we > challenged various aspects of our installation, found some soft spot, > corrected it, and wrote a document that is worth more than the ink and the > paper. Yet again, all we did was assuring that our results match those of a > number of other installations (different compilers, operating systems, > processors). Furthermore, we got an idea of the comparative speed of our > installation. These are useful things to have documented when you are a > system administrator and modeler at the same time. > > Joachim > > __________________________________________ > Joachim GREVEL, Ph.D. > MERCK SERONO International S.A. > Exploratory Medicine > 1202 Geneva > Tel: +41.22.414.4751 > Fax: +41.22.414.3059 > Email: [EMAIL PROTECTED] > > > > > *"A.J. Rossini" <[EMAIL PROTECTED]>* > > 10/21/2008 10:41 PM > To > [EMAIL PROTECTED] > cc > [EMAIL PROTECTED], [email protected] > Subject > Re: [NMusers] Validation Strategy for NONMEM > > > > > There is useless validation, and then there is useful validation. The > latter is about making sure your computational results are > reproducible, the former is about making sure that your documentation > can be photocopied. It's sort of the same thing, if you aren't a > modeler. > > Unfortunately, most on this list tend to be modelers. > > On Tue, Oct 21, 2008 at 7:37 PM, <[EMAIL PROTECTED]> wrote: > > > > After reading this, it is no wonder scientific productivity is at an > > all-time low. Imagine if Marie Curie had to qualify her radium-purifying > > equipment, or if Alexander Fleming had to validate his petri dishes > before > > culturing Penecillium. One day scientists are going to push back against > > these IT people, who just make busy work for everyone. > > > > > > > > [EMAIL PROTECTED] > > Sent by: [EMAIL PROTECTED] > > > > 21-Oct-2008 03:18 > > > > > > To > > [email protected] > > cc > > Subject > > Re: [NMusers] Validation Strategy for NONMEM > > > > > > > > > > > > Dear Mark, > > > > I am engaged in this question since the beginning of this year (not > finished > > yet), and I am happy to share some basics of my experiences: > > > > 1. It is important to specify where the data for NONMEM analysis > come > > from. If they come from a GCP source and are already QA-ed when they > arrive > > at the doorstep of NONMEM then your system will only subject to GCP > > regulation. Otherwise, you also have to comply with GLP. > > > > 2. There is no way around what is called here a 'Validation Plan' > and > > a 'Risk Analysis'. These documents will trigger a slate of other > documents > > (in our case here about 15) which describe Installation, Installation > > Validation, Qualification of Users, Modeling Strategy, Review Processes, > > System Life Cycle Management etc. > > > > 3. We found it useful to differentiate between 'Exploratory Work' > and > > 'Submission Work'. > > > > 4. Before you worry about passing inspection by the FDA, you need > to > > worry about passing inspection of your own company QA officers. > > > > 5. Just installing NONMEM with NMQual does not render you new > system > > 'validated' or 'qualified'. Here my apologies to the excellent folks at > > Metrum, but for various reasons, we ended up not using NMQual. > > > > 6. You have to know what you are trying to build before you > concern > > yourself about QA processes. A number of separate installations on PCs > > linked to a file server is a different animal from a server-based > > installation with a grid engine. > > > > 7. It all takes more time than you think: make generous budgets > and > > time lines. > > > > I hope I helped more than I confused, > > > > Joachim > > > > __________________________________________ > > Joachim GREVEL, Ph.D. > > MERCK SERONO International S.A. > > Exploratory Medicine > > 1202 Geneva > > Tel: +41.22.414.4751 > > Fax: +41.22.414.3059 > > Email: [EMAIL PROTECTED] > > > > > > > > "Vilicich, Mark" <[EMAIL PROTECTED]> > > Sent by: [EMAIL PROTECTED] > > > > 10/17/2008 10:08 PM > > > > To > > <[email protected]> > > cc > > Subject > > [NMusers] Validation Strategy for NONMEM > > > > > > > > > > > > > > Dear All, > > > > I am interested in perspectives on strategies for "validating" NONMEM. > Also, > > experiences from or with the FDA since the FDA is: a key user, customer > of > > analysis and auditor of NONMEM use in the industry. Without a large > nonmem > > staff here, the challenge I see is in scaling the validation strategy to > > provide the most efficient environment for doing analysis that is > defensible > > to both internal and external audits based on the associated GxP risk > level. > > > > Below are the concepts I've cobbled together, though instead of my > > reinventing the wheel I appreciate anything you could share. Any and all > > gems of insight you can share whether it regard the big picture or some > > detailed specifics, IT centric or business process related. You may send > > them back to the listserver or me directly as you feel appropriate. > > > > Details: > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > From searching the archives and other random bits of knowledge on NONMEM, > > part of the validation strategy is to recognize that NONMEM is not to be > > literally validated. NONMEM may be considered more of a development > > environment, optimized for developing specialized forms of complex > analysis > > and modeling. As a development platform, an approach could be that NONMEM > > itself is qualified and each specific analysis is validated individually. > > > > To support establishing a defensible NONMEM environment, I've also read > > discussions on integrating common software development best practices > such > > as version control of the "programming" of nonmem, NMQual and other > > commercial and custom tools for capturing all the metadata related to > > running a specific NONMEM job. These themes support defining the state of > > the NONMEM environment and ability to reproduce the outcomes. > > > > Also, reading in the archives about the differences in the numeric > outcomes > > of NONMEM analyses based on the hardware platform, etc. are helpful to > know > > up front and to consider in the validation strategy so it is not destined > to > > failure if the target environment is multiplatform or otherwise complex. > > > > Gaps noticed/topics not discussed: > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Is there opportunity in looking at the risk based approached sanctioned > by > > the FDA a few years ago that would make the total validation deliverable, > > including both the application and the model development process, more > lean > > and targeted at the primary risk targets? > > > > Does this scientific software environment lend itself to use of modern > agile > > software development methodologies that go far beyond basic iterative > > approaches. These methodologies are being used in software development > for > > the regulated/GxP industry. > > > > I've seen the excellent presentation from 2004 that Joga Gobburu from the > > FDA gave, seems like there has been some progression of thought or > actions > > on the proposals included there. Any references to follow-up information > on > > it would be helpful? > > > > Regards , > > > > Mark Vilicich > > Early Development > > [EMAIL PROTECTED] > > > > ________________________________ > > > > This message and any attachment are confidential, may be privileged or > > otherwise protected from disclosure and are intended only for use by the > > addressee(s) named herein. If you are not the intended recipient, you > must > > not copy this message or attachment or disclose the contents to any other > > person. If you have received this transmission in error, please notify > the > > sender immediately and delete the message and any attachment from your > > system.

On Oct 17, 2008, at 4:08 PM, Vilicich, Mark wrote: > Dear All, > > I am interested in perspectives on strategies for "validating" NONMEM. Also, experiences from or with the FDA since the FDA is: a key user, customer of analysis and auditor of NONMEM use in the industry. Without a large nonmem staff here, the challenge I see is in scaling the validation strategy to provide the most efficient environment for doing analysis that is defensible to both internal and external audits based on the associated GxP risk level. > > Below are the concepts I've cobbled together, though instead of my reinventing the wheel I appreciate anything you could share. Any and all gems of insight you can share whether it regard the big picture or some detailed specifics, IT centric or business process related. You may send them back to the listserver or me directly as you feel appropriate. > > Details: > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > From searching the archives and other random bits of knowledge on NONMEM, part of the validation strategy is to recognize that NONMEM is not to be literally validated. NONMEM may be considered more of a development environment, optimized for developing specialized forms of complex analysis and modeling. As a development platform, an approach could be that NONMEM itself is qualified and each specific analysis is validated individually. > > To support establishing a defensible NONMEM environment, I've also read discussions on integrating common software development best practices such as version control of the "programming" of nonmem, NMQual and other commercial and custom tools for capturing all the metadata related to running a specific NONMEM job. These themes support defining the state of the NONMEM environment and ability to reproduce the outcomes. > > Also, reading in the archives about the differences in the numeric outcomes of NONMEM analyses based on the hardware platform, etc. are helpful to know up front and to consider in the validation strategy so it is not destined to failure if the target environment is multiplatform or otherwise complex. > > Gaps noticed/topics not discussed: > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > Is there opportunity in looking at the risk based approached sanctioned by the FDA a few years ago that would make the total validation deliverable, including both the application and the model development process, more lean and targeted at the primary risk targets? > > Does this scientific software environment lend itself to use of modern agile software development methodologies that go far beyond basic iterative approaches. These methodologies are being used in software development for the regulated/GxP industry. > > I've seen the excellent presentation from 2004 that Joga Gobburu from the FDA gave, seems like there has been some progression of thought or actions on the proposals included there. Any references to follow-up information on it would be helpful? > > Regards , > > Mark Vilicich > Early Development > [EMAIL PROTECTED]

________________________________ From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of [EMAIL PROTECTED] Sent: Wednesday, October 22, 2008 7:44 AM To: [email protected] Subject: Re: [NMusers] Validation Strategy for NONMEM Dear Jun, the outside reference used: NONMEM (Version VI) with Wings for NONMEM; installed using NMQUAL version 6.3.2 (Metrum Institute); Intel Visual Fortran compiler (Version 10.1 ) or GNU Fortran g77 (Version 3.1 release 20020514); compiler options for Intel Fortran were /nologo /nbs /w /4Yportlib /Gs /Ob1gyti /Qprec_div. ; compiler options for g77 were -fno-backslash -O; hardware: Intel Core Duo 7500 2.19GHz processor on an IBM Thinkpad Model T61 with 1 Gb RAM. The operating system was Windows XP Professional 2002 Service Pack 3. We internally used: NONMEM (Version VI,2) PsN (Version 2.2.5) installed without NMQual; gfortran compiler with option -O ; hardware: HP PROLIANT DL585 G2 Servers with many cores (speed 2,8 GHz) with 16 Gb RAM and the LINUX (SUSE Enterprise Edition 10 SP1 64 Bit) operating system with SUN Grid engine (N1). We reproduced the reference models well. Speed was equal with ANVAN6 (long runtimes in the hours). But with precompiled models ADVAN4 and with naked PRED the Intel compiler was sometimes twice as fast (runtimes < 1 min). Hope this is helpful for some of you about to make decisions about hard- and software. Joachim __________________________________________ Joachim GREVEL, Ph.D. MERCK SERONO International S.A. Exploratory Medicine 1202 Geneva Tel: +41.22.414.4751 Fax: +41.22.414.3059 Email: [EMAIL PROTECTED] "Jun Shen" <[EMAIL PROTECTED]> 10/22/2008 04:08 PM To [EMAIL PROTECTED] cc [email protected] Subject Re: [NMusers] Validation Strategy for NONMEM Dear Joachim, Can you share some of your findings? What system configuration seems to be optimal (hardware/OS system, compiler, any aiding software)? Thanks. Jun On Wed, Oct 22, 2008 at 1:21 AM, <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> > wrote: Dear all, one of the more useful things we did was an exchange of data files and models with a prominent member of our modeling community. This way we challenged various aspects of our installation, found some soft spot, corrected it, and wrote a document that is worth more than the ink and the paper. Yet again, all we did was assuring that our results match those of a number of other installations (different compilers, operating systems, processors). Furthermore, we got an idea of the comparative speed of our installation. These are useful things to have documented when you are a system administrator and modeler at the same time. Joachim __________________________________________ Joachim GREVEL, Ph.D. MERCK SERONO International S.A. Exploratory Medicine 1202 Geneva Tel: +41.22.414.4751 Fax: +41.22.414.3059 Email: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> "A.J. Rossini" <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> > 10/21/2008 10:41 PM To [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> cc [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> , [email protected] <mailto:[email protected]> Subject Re: [NMusers] Validation Strategy for NONMEM There is useless validation, and then there is useful validation. The latter is about making sure your computational results are reproducible, the former is about making sure that your documentation can be photocopied. It's sort of the same thing, if you aren't a modeler. Unfortunately, most on this list tend to be modelers. On Tue, Oct 21, 2008 at 7:37 PM, <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> > wrote: > > After reading this, it is no wonder scientific productivity is at an > all-time low. Imagine if Marie Curie had to qualify her radium-purifying > equipment, or if Alexander Fleming had to validate his petri dishes before > culturing Penecillium. One day scientists are going to push back against > these IT people, who just make busy work for everyone. > > > > [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> > Sent by: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> > > 21-Oct-2008 03:18 > > > To > [email protected] <mailto:[email protected]> > cc > Subject > Re: [NMusers] Validation Strategy for NONMEM > > > > > > Dear Mark, > > I am engaged in this question since the beginning of this year (not finished > yet), and I am happy to share some basics of my experiences: > > 1. It is important to specify where the data for NONMEM analysis come > from. If they come from a GCP source and are already QA-ed when they arrive > at the doorstep of NONMEM then your system will only subject to GCP > regulation. Otherwise, you also have to comply with GLP. > > 2. There is no way around what is called here a 'Validation Plan' and > a 'Risk Analysis'. These documents will trigger a slate of other documents > (in our case here about 15) which describe Installation, Installation > Validation, Qualification of Users, Modeling Strategy, Review Processes, > System Life Cycle Management etc. > > 3. We found it useful to differentiate between 'Exploratory Work' and > 'Submission Work'. > > 4. Before you worry about passing inspection by the FDA, you need to > worry about passing inspection of your own company QA officers. > > 5. Just installing NONMEM with NMQual does not render you new system > 'validated' or 'qualified'. Here my apologies to the excellent folks at > Metrum, but for various reasons, we ended up not using NMQual. > > 6. You have to know what you are trying to build before you concern > yourself about QA processes. A number of separate installations on PCs > linked to a file server is a different animal from a server-based > installation with a grid engine. > > 7. It all takes more time than you think: make generous budgets and > time lines. > > I hope I helped more than I confused, > > Joachim > > __________________________________________ > Joachim GREVEL, Ph.D. > MERCK SERONO International S.A. > Exploratory Medicine > 1202 Geneva > Tel: +41.22.414.4751 > Fax: +41.22.414.3059 > Email: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> > > > > "Vilicich, Mark" <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> > > Sent by: [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> > > 10/17/2008 10:08 PM > > To > <[email protected] <mailto:[email protected]> > > cc > Subject > [NMusers] Validation Strategy for NONMEM > > > > > > > Dear All, > > I am interested in perspectives on strategies for "validating" NONMEM. Also, > experiences from or with the FDA since the FDA is: a key user, customer of > analysis and auditor of NONMEM use in the industry. Without a large nonmem > staff here, the challenge I see is in scaling the validation strategy to > provide the most efficient environment for doing analysis that is defensible > to both internal and external audits based on the associated GxP risk level. > > Below are the concepts I've cobbled together, though instead of my > reinventing the wheel I appreciate anything you could share. Any and all > gems of insight you can share whether it regard the big picture or some > detailed specifics, IT centric or business process related. You may send > them back to the listserver or me directly as you feel appropriate. > > Details: > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > From searching the archives and other random bits of knowledge on NONMEM, > part of the validation strategy is to recognize that NONMEM is not to be > literally validated. NONMEM may be considered more of a development > environment, optimized for developing specialized forms of complex analysis > and modeling. As a development platform, an approach could be that NONMEM > itself is qualified and each specific analysis is validated individually. > > To support establishing a defensible NONMEM environment, I've also read > discussions on integrating common software development best practices such > as version control of the "programming" of nonmem, NMQual and other > commercial and custom tools for capturing all the metadata related to > running a specific NONMEM job. These themes support defining the state of > the NONMEM environment and ability to reproduce the outcomes. > > Also, reading in the archives about the differences in the numeric outcomes > of NONMEM analyses based on the hardware platform, etc. are helpful to know > up front and to consider in the validation strategy so it is not destined to > failure if the target environment is multiplatform or otherwise complex. > > Gaps noticed/topics not discussed: > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Is there opportunity in looking at the risk based approached sanctioned by > the FDA a few years ago that would make the total validation deliverable, > including both the application and the model development process, more lean > and targeted at the primary risk targets? > > Does this scientific software environment lend itself to use of modern agile > software development methodologies that go far beyond basic iterative > approaches. These methodologies are being used in software development for > the regulated/GxP industry. > > I've seen the excellent presentation from 2004 that Joga Gobburu from the > FDA gave, seems like there has been some progression of thought or actions > on the proposals included there. Any references to follow-up information on > it would be helpful? > > Regards , > > Mark Vilicich > Early Development > [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]> > > ________________________________ > > This message and any attachment are confidential, may be privileged or > otherwise protected from disclosure and are intended only for use by the > addressee(s) named herein. If you are not the intended recipient, you must > not copy this message or attachment or disclose the contents to any other > person. If you have received this transmission in error, please notify the > sender immediately and delete the message and any attachment from your > system.

From: owner-nmusers On Behalf Of Joachim.Grevel Sent: Wednesday, October 22, 2008 7:44 AM To: nmusers Subject: Re: [NMusers] Validation Strategy for NONMEM Dear Jun, the outside reference used: NONMEM (Version VI) with Wings for NONMEM; installed using NMQUAL version 6.3.2 (Metrum Institute); Intel Visual Fortran compiler (Version 10.1 ) or GNU Fortran g77 (Version 3.1 release 20020514); compiler options for Intel Fortran were /nologo /nbs /w /4Yportlib /Gs /Ob1gyti /Qprec_div. ; compiler options for g77 were -fno-backslash ?O; hardware: Intel Core Duo 7500 2.19GHz processor on an IBM Thinkpad Model T61 with 1 Gb RAM. The operating system was Windows XP Professional 2002 Service Pack 3. We internally used: NONMEM (Version VI,2) PsN (Version 2.2.5) installed without NMQual; gfortran compiler with option -O ; hardware: HP PROLIANT DL585 G2 Servers with many cores (speed 2,8 GHz) with 16 Gb RAM and the LINUX (SUSE Enterprise Edition 10 SP1 64 Bit) operating system with SUN Grid engine (N1). We reproduced the reference models well. Speed was equal with ANVAN6 (long runtimes in the hours). But with precompiled models ADVAN4 and with naked PRED the Intel compiler was sometimes twice as fast (runtimes < 1 min). Hope this is helpful for some of you about to make decisions about hard- and software. Joachim __________________________________________ Joachim GREVEL, Ph.D. MERCK SERONO International S.A. Exploratory Medicine 1202 Geneva Tel: +41.22.414.4751 Fax: +41.22.414.3059 Email: joachim.grevel "Jun Shen" <jun.shen.ut 10/22/2008 04:08 PM To Joachim.Grevel cc nmusers Subject Re: [NMusers] Validation Strategy for NONMEM Dear Joachim, Can you share some of your findings? What system configuration seems to be optimal (hardware/OS system, compiler, any aiding software)? Thanks. Jun On Wed, Oct 22, 2008 at 1:21 AM, <Joachim.Grevel Dear all, one of the more useful things we did was an exchange of data files and models with a prominent member of our modeling community. This way we challenged various aspects of our installation, found some soft spot, corrected it, and wrote a document that is worth more than the ink and the paper. Yet again, all we did was assuring that our results match those of a number of other installations (different compilers, operating systems, processors). Furthermore, we got an idea of the comparative speed of our installation. These are useful things to have documented when you are a system administrator and modeler at the same time. Joachim __________________________________________ Joachim GREVEL, Ph.D. MERCK SERONO International S.A. Exploratory Medicine 1202 Geneva Tel: +41.22.414.4751 Fax: +41.22.414.3059 Email: joachim.grevel "A.J. Rossini" <blindglobe 10/21/2008 10:41 PM To Michael.J.Fossler cc Joachim.Grevel Subject Re: [NMusers] Validation Strategy for NONMEM There is useless validation, and then there is useful validation. The latter is about making sure your computational results are reproducible, the former is about making sure that your documentation can be photocopied. It's sort of the same thing, if you aren't a modeler. Unfortunately, most on this list tend to be modelers. On Tue, Oct 21, 2008 at 7:37 PM, <Michael.J.Fossler > > After reading this, it is no wonder scientific productivity is at an > all-time low. Imagine if Marie Curie had to qualify her radium-purifying > equipment, or if Alexander Fleming had to validate his petri dishes before > culturing Penecillium. One day scientists are going to push back against > these IT people, who just make busy work for everyone. > > > > Joachim.Grevel > Sent by: owner-nmusers > > 21-Oct-2008 03:18 > > > To > nmusers > cc > Subject > Re: [NMusers] Validation Strategy for NONMEM > > > > > > Dear Mark, > > I am engaged in this question since the beginning of this year (not finished > yet), and I am happy to share some basics of my experiences: > > 1. It is important to specify where the data for NONMEM analysis come > from. If they come from a GCP source and are already QA-ed when they arrive > at the doorstep of NONMEM then your system will only subject to GCP > regulation. Otherwise, you also have to comply with GLP. > > 2. There is no way around what is called here a 'Validation Plan' and > a 'Risk Analysis'. These documents will trigger a slate of other documents > (in our case here about 15) which describe Installation, Installation > Validation, Qualification of Users, Modeling Strategy, Review Processes, > System Life Cycle Management etc. > > 3. We found it useful to differentiate between 'Exploratory Work' and > 'Submission Work'. > > 4. Before you worry about passing inspection by the FDA, you need to > worry about passing inspection of your own company QA officers. > > 5. Just installing NONMEM with NMQual does not render you new system > 'validated' or 'qualified'. Here my apologies to the excellent folks at > Metrum, but for various reasons, we ended up not using NMQual. > > 6. You have to know what you are trying to build before you concern > yourself about QA processes. A number of separate installations on PCs > linked to a file server is a different animal from a server-based > installation with a grid engine. > > 7. It all takes more time than you think: make generous budgets and > time lines. > > I hope I helped more than I confused, > > Joachim > > ________________________ __________________ > Joachim GREVEL, Ph.D. > MERCK SERONO International S.A. > Exploratory Medicine > 1202 Geneva > Tel: +41.22.414.4751 > Fax: +41.22.414.3059 > Email: joachim.grevel > > > > "Vilicich, Mark" <Mark.Vilicich > Sent by: owner-nmusers > > 10/17/2008 10:08 PM > > To > <nmusers > cc > Subject > [NMusers] Validation Strategy for NONMEM > > > > > > > Dear All, > > I am interested in perspectives on strategies for "validating" NONMEM. Also, > experiences from or with the FDA since the FDA is: a key user, customer of > analysis and auditor of NONMEM use in the industry. Without a large nonmem > staff here, the challenge I see is in scaling the validation strategy to > provide the most efficient environment for doing analysis that is defensible > to both internal and external audits based on the associated GxP risk level. > > Below are the concepts I've cobbled together, though instead of my > reinventing the wheel I appreciate anything you could share. Any and all > gems of insight you can share whether it regard the big picture or some > detailed specifics, IT centric or business process related. You may send > them back to the listserver or me directly as you feel appropriate. > > Details: > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > From searching the archives and other random bits of knowledge on NONMEM, > part of the validation strategy is to recognize that NONMEM is not to be > literally validated. NONMEM may be considered more of a development > environment, optimized for developing specialized forms of complex analysis > and modeling. As a development platform, an approach could be that NONMEM > itself is qualified and each specific analysis is validated individually. > > To support establishing a defensible NONMEM environment, I've also read > discussions on integrating common software development best practices such > as version control of the "programming" of nonmem, NMQual and other > commercial and custom tools for capturing all the metadata related to > running a specific NONMEM job. These themes support defining the state of > the NONMEM environment and ability to reproduce the outcomes. > > Also, reading in the archives about the differences in the numeric outcomes > of NONMEM analyses based on the hardware platform, etc. are helpful to know > up front and to consider in the validation strategy so it is not destined to > failure if the target environment is multiplatform or otherwise complex. > > Gaps noticed/topics not discussed: > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Is there opportunity in looking at the risk based approached sanctioned by > the FDA a few years ago that would make the total validation deliverable, > including both the application and the model development process, more lean > and targeted at the primary risk targets? > > Does this scientific software environment lend itself to use of modern agile > software development methodologies that go far beyond basic iterative > approaches. These methodologies are being used in software development for > the regulated/GxP industry. > > I've seen the excellent presentation from 2004 that Joga Gobburu from the > FDA gave, seems like there has been some progression of thought or actions > on the proposals included there. Any references to follow-up information on > it would be helpful? > > Regards , > > Mark Vilicich > Early Development > mark.vilicich > > ________________________ ________ > > This message and any attachment are confidential, may be privileged or > otherwise protected from disclosure and are intended only for use by the > addressee(s) named herein. If you are not the intended recipient, you must > not copy this message or attachment or disclose the contents to any other > person. If you have received this transmission in error, please notify the > sender immediately and delete the message and any attachment from your > system.

From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of [EMAIL PROTECTED] Sent: Wednesday, October 22, 2008 7:44 AM To: [email protected] Subject: Re: [NMusers] Validation Strategy for NONMEM Dear Jun, the outside reference used: NONMEM (Version VI) with Wings for NONMEM; installed using NMQUAL version 6.3.2 (Metrum Institute); Intel Visual Fortran compiler (Version 10.1 ) or GNU Fortran g77 (Version 3.1 release 20020514); compiler options for Intel Fortran were /nologo /nbs /w /4Yportlib /Gs /Ob1gyti /Qprec_div. ; compiler options for g77 were -fno-backslash ?O; hardware: Intel Core Duo 7500 2.19GHz processor on an IBM Thinkpad Model T61 with 1 Gb RAM. The operating system was Windows XP Professional 2002 Service Pack 3. We internally used: NONMEM (Version VI,2) PsN (Version 2.2.5) installed without NMQual; gfortran compiler with option -O ; hardware: HP PROLIANT DL585 G2 Servers with many cores (speed 2,8 GHz) with 16 Gb RAM and the LINUX (SUSE Enterprise Edition 10 SP1 64 Bit) operating system with SUN Grid engine (N1). We reproduced the reference models well. Speed was equal with ANVAN6 (long runtimes in the hours). But with precompiled models ADVAN4 and with naked PRED the Intel compiler was sometimes twice as fast (runtimes < 1 min). Hope this is helpful for some of you about to make decisions about hard- and software. Joachim __________________________________________ Joachim GREVEL, Ph.D. MERCK SERONO International S.A. Exploratory Medicine 1202 Geneva Tel: +41.22.414.4751 Fax: +41.22.414.3059 Email: [EMAIL PROTECTED] "Jun Shen" <[EMAIL PROTECTED]> 10/22/2008 04:08 PM To [EMAIL PROTECTED] cc [email protected] Subject Re: [NMusers] Validation Strategy for NONMEM Dear Joachim, Can you share some of your findings? What system configuration seems to be optimal (hardware/OS system, compiler, any aiding software)? Thanks. Jun On Wed, Oct 22, 2008 at 1:21 AM, <[EMAIL PROTECTED]> wrote: Dear all, one of the more useful things we did was an exchange of data files and models with a prominent member of our modeling community. This way we challenged various aspects of our installation, found some soft spot, corrected it, and wrote a document that is worth more than the ink and the paper. Yet again, all we did was assuring that our results match those of a number of other installations (different compilers, operating systems, processors). Furthermore, we got an idea of the comparative speed of our installation. These are useful things to have documented when you are a system administrator and modeler at the same time. Joachim __________________________________________ Joachim GREVEL, Ph.D. MERCK SERONO International S.A. Exploratory Medicine 1202 Geneva Tel: +41.22.414.4751 Fax: +41.22.414.3059 Email: [EMAIL PROTECTED] "A.J. Rossini" <[EMAIL PROTECTED]> 10/21/2008 10:41 PM To [EMAIL PROTECTED] cc [EMAIL PROTECTED], [email protected] Subject Re: [NMusers] Validation Strategy for NONMEM There is useless validation, and then there is useful validation. The latter is about making sure your computational results are reproducible, the former is about making sure that your documentation can be photocopied. It's sort of the same thing, if you aren't a modeler. Unfortunately, most on this list tend to be modelers. On Tue, Oct 21, 2008 at 7:37 PM, <[EMAIL PROTECTED]> wrote: > > After reading this, it is no wonder scientific productivity is at an > all-time low. Imagine if Marie Curie had to qualify her radium-purifying > equipment, or if Alexander Fleming had to validate his petri dishes before > culturing Penecillium. One day scientists are going to push back against > these IT people, who just make busy work for everyone. > > > > [EMAIL PROTECTED] > Sent by: [EMAIL PROTECTED] > > 21-Oct-2008 03:18 > > > To > [email protected] > cc > Subject > Re: [NMusers] Validation Strategy for NONMEM > > > > > > Dear Mark, > > I am engaged in this question since the beginning of this year (not finished > yet), and I am happy to share some basics of my experiences: > > 1. It is important to specify where the data for NONMEM analysis come > from. If they come from a GCP source and are already QA-ed when they arrive > at the doorstep of NONMEM then your system will only subject to GCP > regulation. Otherwise, you also have to comply with GLP. > > 2. There is no way around what is called here a 'Validation Plan' and > a 'Risk Analysis'. These documents will trigger a slate of other documents > (in our case here about 15) which describe Installation, Installation > Validation, Qualification of Users, Modeling Strategy, Review Processes, > System Life Cycle Management etc. > > 3. We found it useful to differentiate between 'Exploratory Work' and > 'Submission Work'. > > 4. Before you worry about passing inspection by the FDA, you need to > worry about passing inspection of your own company QA officers. > > 5. Just installing NONMEM with NMQual does not render you new system > 'validated' or 'qualified'. Here my apologies to the excellent folks at > Metrum, but for various reasons, we ended up not using NMQual. > > 6. You have to know what you are trying to build before you concern > yourself about QA processes. A number of separate installations on PCs > linked to a file server is a different animal from a server-based > installation with a grid engine. > > 7. It all takes more time than you think: make generous budgets and > time lines. > > I hope I helped more than I confused, > > Joachim > > __________________________________________ > Joachim GREVEL, Ph.D. > MERCK SERONO International S.A. > Exploratory Medicine > 1202 Geneva > Tel: +41.22.414.4751 > Fax: +41.22.414.3059 > Email: [EMAIL PROTECTED] > > > > "Vilicich, Mark" <[EMAIL PROTECTED]> > Sent by: [EMAIL PROTECTED] > > 10/17/2008 10:08 PM > > To > <[email protected]> > cc > Subject > [NMusers] Validation Strategy for NONMEM > > > > > > > Dear All, > > I am interested in perspectives on strategies for "validating" NONMEM. Also, > experiences from or with the FDA since the FDA is: a key user, customer of > analysis and auditor of NONMEM use in the industry. Without a large nonmem > staff here, the challenge I see is in scaling the validation strategy to > provide the most efficient environment for doing analysis that is defensible > to both internal and external audits based on the associated GxP risk level. > > Below are the concepts I've cobbled together, though instead of my > reinventing the wheel I appreciate anything you could share. Any and all > gems of insight you can share whether it regard the big picture or some > detailed specifics, IT centric or business process related. You may send > them back to the listserver or me directly as you feel appropriate. > > Details: > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > From searching the archives and other random bits of knowledge on NONMEM, > part of the validation strategy is to recognize that NONMEM is not to be > literally validated. NONMEM may be considered more of a development > environment, optimized for developing specialized forms of complex analysis > and modeling. As a development platform, an approach could be that NONMEM > itself is qualified and each specific analysis is validated individually. > > To support establishing a defensible NONMEM environment, I've also read > discussions on integrating common software development best practices such > as version control of the "programming" of nonmem, NMQual and other > commercial and custom tools for capturing all the metadata related to > running a specific NONMEM job. These themes support defining the state of > the NONMEM environment and ability to reproduce the outcomes. > > Also, reading in the archives about the differences in the numeric outcomes > of NONMEM analyses based on the hardware platform, etc. are helpful to know > up front and to consider in the validation strategy so it is not destined to > failure if the target environment is multiplatform or otherwise complex. > > Gaps noticed/topics not discussed: > ~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Is there opportunity in looking at the risk based approached sanctioned by > the FDA a few years ago that would make the total validation deliverable, > including both the application and the model development process, more lean > and targeted at the primary risk targets? > > Does this scientific software environment lend itself to use of modern agile > software development methodologies that go far beyond basic iterative > approaches. These methodologies are being used in software development for > the regulated/GxP industry. > > I've seen the excellent presentation from 2004 that Joga Gobburu from the > FDA gave, seems like there has been some progression of thought or actions > on the proposals included there. Any references to follow-up information on > it would be helpful? > > Regards , > > Mark Vilicich > Early Development > [EMAIL PROTECTED] > > ________________________________ > > This message and any attachment are confidential, may be privileged or > otherwise protected from disclosure and are intended only for use by the > addressee(s) named herein. If you are not the intended recipient, you must > not copy this message or attachment or disclose the contents to any other > person. If you have received this transmission in error, please notify the > sender immediately and delete the message and any attachment from your > system.

On Tue, Oct 21, 2008 at 11:11 PM, Nick Holford <[EMAIL PROTECTED]> wrote: > Tony, > > To expand a bit on your useful validation definition. > > Making sure computational results are reproducible does not mean they are > correct. I believe that most NONMEM analyses cannot be proven to be correct > therefore the validation efforts only assure reproducibility. Whether this > makes the validation useful or useless is debatable. > > Nick > > > > A.J. Rossini wrote: >> >> There is useless validation, and then there is useful validation. The >> latter is about making sure your computational results are >> reproducible, the former is about making sure that your documentation >> can be photocopied. It's sort of the same thing, if you aren't a >> modeler. >> >> Unfortunately, most on this list tend to be modelers. >> >> On Tue, Oct 21, 2008 at 7:37 PM, <[EMAIL PROTECTED]> wrote: >> >>> >>> After reading this, it is no wonder scientific productivity is at an >>> all-time low. Imagine if Marie Curie had to qualify her radium-purifying >>> equipment, or if Alexander Fleming had to validate his petri dishes >>> before >>> culturing Penecillium. One day scientists are going to push back against >>> these IT people, who just make busy work for everyone. >>> >>> >>> >>> [EMAIL PROTECTED] >>> Sent by: [EMAIL PROTECTED] >>> >>> 21-Oct-2008 03:18 >>> >>> >>> To >>> [email protected] >>> cc >>> Subject >>> Re: [NMusers] Validation Strategy for NONMEM >>> >>> >>> >>> >>> >>> Dear Mark, >>> >>> I am engaged in this question since the beginning of this year (not >>> finished >>> yet), and I am happy to share some basics of my experiences: >>> >>> 1. It is important to specify where the data for NONMEM analysis >>> come >>> from. If they come from a GCP source and are already QA-ed when they >>> arrive >>> at the doorstep of NONMEM then your system will only subject to GCP >>> regulation. Otherwise, you also have to comply with GLP. >>> >>> 2. There is no way around what is called here a 'Validation Plan' >>> and >>> a 'Risk Analysis'. These documents will trigger a slate of other >>> documents >>> (in our case here about 15) which describe Installation, Installation >>> Validation, Qualification of Users, Modeling Strategy, Review Processes, >>> System Life Cycle Management etc. >>> >>> 3. We found it useful to differentiate between 'Exploratory Work' >>> and >>> 'Submission Work'. >>> >>> 4. Before you worry about passing inspection by the FDA, you need >>> to >>> worry about passing inspection of your own company QA officers. >>> >>> 5. Just installing NONMEM with NMQual does not render you new >>> system >>> 'validated' or 'qualified'. Here my apologies to the excellent folks at >>> Metrum, but for various reasons, we ended up not using NMQual. >>> >>> 6. You have to know what you are trying to build before you >>> concern >>> yourself about QA processes. A number of separate installations on PCs >>> linked to a file server is a different animal from a server-based >>> installation with a grid engine. >>> >>> 7. It all takes more time than you think: make generous budgets >>> and >>> time lines. >>> >>> I hope I helped more than I confused, >>> >>> Joachim >>> >>> __________________________________________ >>> Joachim GREVEL, Ph.D. >>> MERCK SERONO International S.A. >>> Exploratory Medicine >>> 1202 Geneva >>> Tel: +41.22.414.4751 >>> Fax: +41.22.414.3059 >>> Email: [EMAIL PROTECTED] >>> >>> >>> >>> "Vilicich, Mark" <[EMAIL PROTECTED]> >>> Sent by: [EMAIL PROTECTED] >>> >>> 10/17/2008 10:08 PM >>> >>> To >>> <[email protected]> >>> cc >>> Subject >>> [NMusers] Validation Strategy for NONMEM >>> >>> >>> >>> >>> >>> >>> Dear All, >>> >>> I am interested in perspectives on strategies for "validating" NONMEM. >>> Also, >>> experiences from or with the FDA since the FDA is: a key user, customer >>> of >>> analysis and auditor of NONMEM use in the industry. Without a large >>> nonmem >>> staff here, the challenge I see is in scaling the validation strategy to >>> provide the most efficient environment for doing analysis that is >>> defensible >>> to both internal and external audits based on the associated GxP risk >>> level. >>> >>> Below are the concepts I've cobbled together, though instead of my >>> reinventing the wheel I appreciate anything you could share. Any and all >>> gems of insight you can share whether it regard the big picture or some >>> detailed specifics, IT centric or business process related. You may send >>> them back to the listserver or me directly as you feel appropriate. >>> >>> Details: >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> From searching the archives and other random bits of knowledge on NONMEM, >>> part of the validation strategy is to recognize that NONMEM is not to be >>> literally validated. NONMEM may be considered more of a development >>> environment, optimized for developing specialized forms of complex >>> analysis >>> and modeling. As a development platform, an approach could be that NONMEM >>> itself is qualified and each specific analysis is validated individually. >>> >>> To support establishing a defensible NONMEM environment, I've also read >>> discussions on integrating common software development best practices >>> such >>> as version control of the "programming" of nonmem, NMQual and other >>> commercial and custom tools for capturing all the metadata related to >>> running a specific NONMEM job. These themes support defining the state of >>> the NONMEM environment and ability to reproduce the outcomes. >>> >>> Also, reading in the archives about the differences in the numeric >>> outcomes >>> of NONMEM analyses based on the hardware platform, etc. are helpful to >>> know >>> up front and to consider in the validation strategy so it is not destined >>> to >>> failure if the target environment is multiplatform or otherwise complex. >>> >>> Gaps noticed/topics not discussed: >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> Is there opportunity in looking at the risk based approached sanctioned >>> by >>> the FDA a few years ago that would make the total validation deliverable, >>> including both the application and the model development process, more >>> lean >>> and targeted at the primary risk targets? >>> >>> Does this scientific software environment lend itself to use of modern >>> agile >>> software development methodologies that go far beyond basic iterative >>> approaches. These methodologies are being used in software development >>> for >>> the regulated/GxP industry. >>> >>> I've seen the excellent presentation from 2004 that Joga Gobburu from the >>> FDA gave, seems like there has been some progression of thought or >>> actions >>> on the proposals included there. Any references to follow-up information >>> on >>> it would be helpful? >>> >>> Regards , >>> >>> Mark Vilicich >>> Early Development >>> [EMAIL PROTECTED] >>> >>> ________________________________ >>> >>> This message and any attachment are confidential, may be privileged or >>> otherwise protected from disclosure and are intended only for use by the >>> addressee(s) named herein. If you are not the intended recipient, you >>> must >>> not copy this message or attachment or disclose the contents to any other >>> person. If you have received this transmission in error, please notify >>> the >>> sender immediately and delete the message and any attachment from your >>> system.