Open Test Method Development Pilot Program

ONC’s Open Test Method Development Pilot Program builds on our continued commitment to collaborate with health IT stakeholders and to enhance stakeholder engagement in ONC’s Health IT Certification Program. This pilot program is open to all stakeholders willing to contribute their expertise towards the development of the test methods that could ultimately be used by accredited testing laboratories (ATLs) for health IT testing. This pilot program is designed to provide stakeholders with an expanded opportunity to apply their in-the-field experience to test method development (including test procedures, test data, and test tools) to support improvements to the nation’s health care system.

The pilot program will be limited to two 2014 Edition EHR certification criteria and, at its conclusion, will be evaluated for feasibility, efficiency, and scalability relative to future certification criteria editions.

The pilot program will follow the process below and aims to complete the development of two test methods by October 31, 2014

If you have questions about the Open Test Development Pilot Program Email Us

Guidelines
  1. It is acceptable to contribute to only a portion of this test method, e.g. only the test data.
  2. The test method should meet only the requirements of the Final Rule (45 CFR Part 170, September 4, 2012) for the certification criteria.   Conversely, no requirements should be added.
  3. The test procedure should align with the test data and vice versa.
  4. There may or may not be a need for a conformance tool for the test method.
  5. Referenced standards are called out by the Final Rule and cannot be ignored or changed.
  6. Concentrate on the “what” and not the “how.”   That is, do not try to lead the tester through the process in a step-by-step or prescriptive manner.
  7. Do not tie the test method to a particular clinical practice workflow.  (Note:  If you are proposing a test scenario involving multiple unit tests, then clinical practice workflow must be described.  However, do not make the workflow overly specific, thereby making it plausible in only a very restricted and prescribed environment.)
  8. Provide enough detail so that a determination of pass/fail can be made.  That is, make a clear statement of the expected outcome for each step (or group of steps).
Definitions

Derived Test Requirements - describes a specific portion of the certification criterion which will be addressed in the test procedure. To provide traceability, each is denoted using the following form: DTR [FR certification criterion number] – [Sequence number]

Required Vendor Information - describes the information needed from the Vendor in order to perform the test procedure. To provide traceability, each is denoted using the following form: VE [FR certification criterion number] – [Sequence number] 

Required Test Procedure – describes the test activities required to be performed by the Tester. To provide traceability, each is denoted using the following form: TE [FR certification criterion number] – [Sequence number]

Inspection Guide – provides additional guidance to the Tester on how to evaluate conformance to the certification criterion. To provide traceability, each is denoted using the following form: IN [FR certification criterion number] – [Sequence number]

Test Story – an English language description of a clinical situation that gives context to the test data.  It can describe the patient-physician interactions or the physician-lab interactions, for instance.  They may be very simple and short or long and complex.

Suggest a Template

If you would like to suggest a template, please keep in mind that a template could be used for a large number of criteria and unit or scenario based testing. While ONC recognizes that this program may result in two different templates for two different criteria, it encourages the industry to suggest one template that could address both. The existing template above is meant to serve as a starting point. Suggested templates will be posted and made available for feedback as they are received and reviewed.

 

Could not access the content at the URL because it is not from an allowed source.

http://confluence.oncprojectracking.org/OTMDPDocuments/Template.html

You may contact your site administrator and request that this URL be added to the list of allowed sources.

 

 

 

 

 

Existing Test Method Template (Unit-based)

This template is provided as a starter template. If you have feedback for this template, please provide it below.

Loading

 

Could not access the content at the URL because it is not from an allowed source.

http://confluence.oncprojectracking.org/OTMDPDocuments/Feedback.html

You may contact your site administrator and request that this URL be added to the list of allowed sources.

*Thank you for your feedback. It will be reviewed and posted.

 

Feedback

 

8/12/14 Work Group

Overall goal:  Aim for one template and add sections or mark sections as “non-applicable” as appropriate for specific criteria, with the understanding that this template may adapt during full content build.

 

Work Group Feedback:

  • Currently, there are multiple sections with the same information that may make the procedures confusing.  Should consolidate the info and move the test narrative to the top, so that people can get right to what they need to do.
  • Build a flexible template model that fits with all of the criteria and potential test methods, but isn’t too complex.
  • Don’t remove content from the existing template, just think about restructuring it so that:
    • More of what needs to be DONE is at the beginning
    • Descriptions/additional information is at the end
    • Update the Standards section so that as standards get revised, there is clear documentation of versioning (i.e. list the 2011 standards and effective dates, the 2014, etc.)
    • Consider adding the following sections to the existing template:
      • Special considerations/potential “curveballs”
      • Technical requirements
      • Dependencies on other test criteria
      • Attestation (as a test method)
      • Consider clarifying how the system is meant to be used vs. how it is used within the test method or in the reporting of test results.
8/15/2014 FeedbackFirst off, we wanted to mention that the test method template itself hasn’t been problematic for us in the past; it’s the content within the template that’s been the most problematic. We look forward to giving feedback as the group adapts the ePrescribing and clinical decision support measures to the new template. 

Similar to the feedback on the template given by the work group on 8/12, we agree that the template could be consolidated. Certain sections of the template are duplicative. For example, the testing tools being used for a given test are described in the informative test description and again in the testing tools section. We’d like to see the testing tools be included as a sub-point in the informative test description section to consolidate the document. The template also contains extraneous info that applies to all templates such as “This introductory section is identical in all 2014 Edition Test Procedures. It describes the purpose of the document and cites relevant regulatory policy and stakeholders”. Since this is on every template, we get used to seeing it and skip over it. Could these sections be moved to a central document that applies to all test procedures to make the test procedure documents themselves more concise? 

We’ve found the certification criteria, preamble language and informative test description sections to be the most helpful. The certification criteria and preamble language minimize the cross-referencing we need to do between documents and the informative test procedure gives us an overview of the test before getting into the details. The referenced standards section is also helpful and necessary to keep everyone on the same page, but we would suggest moving it to the end of the document so that it doesn’t break up the flow between “general description of test” and the detailed “test procedure”. 
8/15/2014 FeedbackGeneral Feedback for Test Procedure Template: 
• Set the expectation at the beginning of the document that readers should familiarize themselves with the final certification criteria edition rule as prerequisite knowledge to content contained in the test procedure. 
• Content within the test procedure template should be reflective of true test procedures rather than replication of content contained in the final rule. For reference to final rule citations, we suggest including links to the regulatory section of the rule possibly in a parsed out version of the final rule rather than re-inclusion of only sections of the regulatory text that may risk being out of context to the whole of the preamble and regulation found in the final rule. 
• Consider use of a tabular format rather than a narrative one for presenting the test procedure similar to what ATLs have provided vendors. Each section of the test procedure as appropriate for each step should contain interpretive guidance, regulatory citations, and links to conformance testing tools so that content is presented in context to the testing step. The following link provides an example from ICSA:https://icsalabs.s3.amazonaws.com/downloads/2014%20Edition%20Test%20Script.docx 
• The information within each section of the template should contain only what is consistent with the description of the section as stated in order to be consumed effectively. 
• Content within the respective ONC test procedures should be evaluated to consider whether given information could be cross-referenced in the respective CMS Technical Specification Sheets for the related meaningful use objectives. For example, content within the Informative Test Description contains information that would be helpful in the Certification and Standards Criteria section of respective Technical Specification Sheet. Testing information can always provide insight into requirements for the design of software. 
  
Preamble Language Section Feedback: 
• Replication of content from the final rule is not necessary. We propose either summarizing key points in a bulleted format or referencing the final rule via hyperlink. 
  
  
Informative Test Procedure Section Feedback: 
• The audience of this document should be mentioned as both vendors and testing bodies. 
o If this section is intended to provide guidance and clarity to ATLs regarding testing latitude, we suggest either relabeling this section or creating a distinct section summarizing testing expectations. Consideration should be taken to clarify the audience of content should it be distinct to either tester or vendor expectations. 
• Content should be organized in sequence of major context; thus, organized in sections of how a test script would be organized, yet not to the detailed level of test steps. 
o For all sections, links should be provided to reference relevant content within the final rule. Requirements should also be anchored to the respective sections of all test procedures. 
• It would be helpful to understand how content in this section is fully vetted to assure it fulfills the regulatory intent without exceeding it. 
o For example, the 2014 test procedure for 170.314(c)(1)-(3) originally indicated only a complete EHR model of certification and testing could include testing to all three of the clinical quality measure criteria which was incorrect and not substantiated by the 2014 criteria edition final rule. 
• This section would be more user-friendly if formatted in a tabular fashion similar to test scripts formatted by ATLs. 
• Testing expectations should be written explicitly to the degree possible in order to leave little room for interpretation or negotiation between vendors and ATLs to debate regulatory intent or acceptable testing approaches. We encourage further guidance in this section so there is less debate between vendors and ATLs to address fewer matters of interpretation. 
o For example, in the test procedure for 170.314(a)(8), it was not entirely clear from the informative test procedure section if it would be permissible to show one user undertaking an action that would cause a CDS intervention to occur that may be directed to a second user or if both triggering action and response had to be interactive to the same user 
• Maintain a section including the changes from the previous rule to the current rule, but be more detailed in the description. We would request a more detailed section, perhaps including a table that includes one column with text from the previous rule with the second column containing content from the current rule. 
8/15/2014 FeedbackOne of the most useful tools I have found in developing and building testing scenarios is the use of flow diagrams - technical and, this case, clinical. It is extremely helpful for everyone to have a visualization of the different people, technology, and processes involved. It helps determine at what point the testing failed and how the workflows were modified, through refinements visualization and iterative illustrations, to determine the proper path or to exhibit where potential may, or actual failure did, occur.
8/15/2014 FeedbackLimit the amount of historical MU criterion information. Focus on the specific goal of the MU criterion as it appears in the respective cycle/stage (MU 2014) being tested towards. It clutters the space and confuses the reader.

 

Suggested Templates:

 

Suggested Test Method Template 8-13-2014

 

  

Loading

Could not access the content at the URL because it is not from an allowed source.

http://confluence.oncprojectracking.org/OTMDPDocuments/Feedback.html

You may contact your site administrator and request that this URL be added to the list of allowed sources.

*Thank you for your feedback. It will be reviewed and posted.

 

Feedback

8/14/2014 FeedbackI believe this template for a scenario based testing method is more desirable than the template for unit based testing method. It should permit EHR technology vendors to demonstrate compliance with the certification criteria in a way that more closely resembles a typical workflow in a clinical setting. 

One of the more significant challenges we have faced in delivering 2014 CEHRT to our clients has been to adapt the methodology used to achieve certification into workflows that meet the needs of our clients. It is one thing to demonstrate an EHR technology's capability to meet a specific certification criterion as the unit based testing method requires. It is quite another to do so with comprehensive clinical workflows appropriate to the setting and applicable to the certification criteria. 

Our clients routinely seek our guidance on how to best implement our CEHRT to meet their meaningful use objectives and often want to follow the exact methods we followed to achieve certification, only to find that while the unit based testing method verified our capability, it did not always do so in a context meaningful to clinical workflows they could implement. Unit-based testing, by design, does not foster the holistic approach to developing meaningful clinical workflow design for use in certification testing. Scenario based testing should permit this and should help speed delivery of workable certified solutions to our clients.
8/15/2014 FeedbackWe have a few concerns with the newly proposed scenario-based template. While scenario-based testing is capable of certifying you in multiple criteria at once, we anticipate difficulties surrounding vendors who aren't certifying on all criteria. The number of potential combinations of certification criteria and how they could interact with each scenario could be difficult to account for. We also believe that test data is more straightforward when it’s linked to a specific criterion and would therefore suggest sticking with the criteria-based template to allow vendors greater flexibly in the criteria they certify on and to avoid introducing additional complexity.

 

 

  • No labels