Altarum

Testing Session for the (f)(4) criteria

On August 29, ONC and Altarum conducted a testing session via WebEx. ONC provided four files that pass the CDC validator IG validation. Altarum loaded each of the four respectively, and received many errors (see the report in attachments.) We don't know if the errors are due to the fact that Altarum has updated their validator for subsequently published IG ERRATA–or other reasons. Altarum is to perform an analysis of the errors and send us the details.

September 4 update: a link to the analysis: https://app.smartsheet.com/b/publish?EQBCT=2b2fad5c6c2d4f4993f6c7d63e209d55

It appears the majority of the errors were triggered because the schematron included in the HL7 IG package, used for NIST validation, was incorrect. Altarum has made the corrections in our validation. Altarum implemented the schematron provided within the HL7 download, which comes along with the standard. They included errata modifications through errata 1 and 2, but none of these errors were determined to be related to errata. The majority of the errors were stemming from test logic present in the original schematron, as the tests were written (7 of the 9 unique errors).

For the remaining 2 unique errors, we had implemented the vocabulary as-provided (from the same download), which was either incomplete, or difficult to verify due the obscurity of the value sets. The IG names the TNM value sets, which are presented incompletely in the guide. The PHINVADS links are not valid. A comprehensive list of codes (or iterations) is not readily accessible.

September 6 from Altarum:

We are in the habit of releasing changes/updates after we’ve completed development sprints. This happens through a scheduled maintenance window, during which we apply the updates to the tool hosted by the MiHIN organization. When we’ve posted the updates (perhaps in a few weeks?), I should be able to demonstrate clean validations.

September 25:

Met with Altarum on September 25 and re-ran all of the test data that had failed validation in their tool previously: the demonstration showed the four test files to pass. Then, I sent a file that they did not have beforehand (that I purposely saved for this occasion) and that file failed basic validation; of note is that this same file passes the CDC validator. Altarum is to investigate and get back to us again--my subsequent analysis determined that the underlying schema for the Cancer Report format is "CDA R2.0", which is different than "C-CDA R2.1", and it appears that the validator used by Altarum is checking to the C-CDA schema; I asked them to get additional test data to run before the next demonstration. I also asked them to get additional test data to run before the next demonstration.

October 15:

From: Laura Rappleye
Sent: Monday, October 15, 2018 11:51 AM
To: 'Chaput, Daniel (OS/ONC)' <Daniel.Chaput@hhs.gov>; Snyder, John (OS/ONC) <John.Snyder2@hhs.gov>; Timothy DeMint <Timothy.DeMint@altarum.org>
Cc: Davis, Seon (OS/ONC) <Seon.Davis@hhs.gov>; Rick Duncan <Rick.Duncan@altarum.org>; Wu, Lauren (OS/ONC) <Lauren.Wu@hhs.gov>; Marshall, Carmelita (OS/ONC) <Carmelita.Marshall@hhs.gov>; Janzen, Glenn (OS/ONC) <Glenn.Janzen@hhs.gov>; Troy Abrahams <Troy.Abrahams@altarum.org>
Subject: RE: Altarum Alternate Test Method - (f)(4) Cancer Testing Demo

We have separated out the Michigan specific requirements. The validation tool we are proposing to use for an alternative testing method follows the implementation guides stated in the final rule.

We also are proposing to have the alternative testing method in Altarum’s secure network or Amazon cloud. We would not use the Validator at the MiHIN.

Dec 7, 2018:

Reviewed additional documentation from Altarum; requested Altarum 1) submit a draft Test Procedure, 2) provide the location of the Altarum validator and an outline of the plans to operationalize if this differs from the current MIHIN approach.