What a testing claim should tell us, and what it often leaves unanswered
At the recent WEDI Conference, testing came up again and again, from the podium and in the hallways, most often in discussions of the CMS Interoperability and Prior Authorization Final Rule (CMS-0057-F). I was glad to hear it. I also left with a question: were we all talking about the same thing?
“We tested it” can mean that two developers exchanged a resource successfully at a Connectathon. It can mean that an implementation passed a set of validation checks. Or it can mean that a team verified requirements across a full workflow, documented the results, and examined how the system behaves when something goes wrong. These are different accomplishments, and our industry routinely reports all of them with the same word.
The cost of blurring them shows up later, in integrations that fail after go-live, in rework, and in patient data that does not arrive where and when a clinician needs it.
When someone tells me an implementation has been tested, I want to know four things: against which requirements, under what conditions, with what evidence, and what was left untested. This article opens a series on those questions. It lays out how AEGIS defines testing, where we think the industry is under-investing, and what we are asking the implementation community to help build.
What Counts as Testing
A Connectathon, a demonstration, or a certification exercise can each be an occasion for testing. Attending one does not, by itself, tell anyone what an implementation can reliably do.
At AEGIS, we treat testing as a repeatable process with a defined scope, defined inputs, expected results, and evidence that someone outside the team can examine. Done well, it shows what worked today, how confident you can be that it will keep working, and where the gaps are.
Running a few transactions and seeing a 200 OK is an exercise. Testing is knowing which requirements those transactions verified, which they did not, and what remains untested.
Take a request that returns 200 OK. That response alone does not tell you whether the payload contains the required elements, whether access was correctly restricted, or whether the next step in the workflow will succeed. Each of those is a separate question with its own expected outcome and its own evidence. A successful exchange is useful information, as long as we are precise about what it demonstrated.
Start With the Requirements
Our approach is requirements-based. Every test traces to something the specification or implementation guide says an implementation must or should do: a SHALL statement, an applicable SHOULD, a constraint, a business rule, a workflow expectation. Those distinctions matter. A mandatory requirement and a recommendation should not quietly become the same pass/fail criterion.
For each requirement, we should be able to answer:
- Where does it come from?
- What conditions apply?
- Which tests verify it?
- What result do we expect?
- What evidence did the test produce?
A test that cannot be tied to a requirement or a stated objective needs a reason to exist. A requirement with no test is unverified, and we should say so.
Systems engineering addressed this problem decades ago with the Requirements Traceability Matrix (RTM). It was never meant as paperwork. Complex systems fail in the gaps between what was specified, what was built, and what was verified, and the RTM closes those gaps by linking each requirement to its source, its implementation, and the tests that prove it.
FHIR implementation guides adopted across an entire industry need the same treatment. That means decomposing each IG into discrete, testable statements instead of reporting that “we tested the profile” or “we tested the operation.” It means designing test approaches around use cases and workflows, because real exchanges happen as sequences of interactions. And once the RTM exists, it answers the question sponsors and adopters care about most: what share of an IG’s requirements has been tested, and by how many implementers?
Without traceability, “we tested it” is an assertion. With traceability, it becomes evidence.
The Uncomfortable Cases Belong in the Test Plan
Most interoperability testing today concentrates on the happy path: a well-formed request, a valid resource, the expected response. That path has to work. Production systems also have to cope with missing data, malformed payloads, expired tokens, unsupported versions, and callers who should not have access.
Negative and exception testing asks how a system behaves in those conditions. Does it reject what it should? Does it return the expected error, in a form the other system can act on? Does it fail safely?
These scenarios deserve planned test cases with expected outcomes, written in advance, rather than a few improvised attempts after the main demonstration succeeds. Workflows need the same attention. One API call can behave correctly while the larger sequence fails.
Coverage reports should make these boundaries visible. “The successful request was verified; the expired-token scenario was not” is a useful result, because it tells the next implementer where work remains. An ecosystem that tests only for success discovers its failure modes in production, and clinicians and patients absorb the cost.
Standards Change on Different Schedules
No implementation guide runs alone. A prior authorization workflow built for CMS-0057-F can depend on FHIR R4, US Core, the Da Vinci CRD, DTR, and PAS guides, SMART App Launch, terminology services, and security requirements, each on its own release cycle.
Knowing that two systems interoperated once, on a given date, against a given set of versions says little about next quarter. Test results should record the versions and dependencies in play, and test suites should re-run as those dependencies change, so breakage surfaces before it reaches production. We call this continuous interoperability, and we think it should be an explicit community goal.
The same logic applies to the specifications themselves. Testing usually begins after an IG is balloted and published, when ambiguities are most expensive to fix. We would rather see testing travel with the IG through its lifecycle: checking requirements for testability while the guide is being written, re-running suites with each ballot cycle and technical correction, and sending findings back to authors so the specification and its tests mature together.
Testers Belong in the Room When Testing Is Defined
Standards authors, architects, clinicians, policy leaders, and program sponsors all bring expertise that testing depends on. Testing is also a profession, with its own methods and body of knowledge. Experienced systems testers, business testers, and QA engineers spend their careers asking what could fail, finding missing conditions, setting expected results, and deciding what evidence is sufficient. We would not ask a test engineer to write clinical guidance. By the same logic, test strategy and conformance criteria should not be set without the people who practice testing.
I want those practitioners involved while testing approaches and conformance criteria are being developed, before the important decisions are made. Implementers belong there too. They see what happens when a specification meets a real product, workflow, and dependency stack, and their findings need a clear route back to the people who maintain the standard.
Where AI Fits
AEGIS is building AI into our own platforms, and we see real value in it: drafting candidate test cases from specifications, generating test data, flagging ambiguous IG language, and speeding up analysis of results. It can lower the cost that has kept rigorous testing out of reach for many organizations.
Alongside that enthusiasm, I hear a more troubling idea: that an AI model can examine an implementation and confirm it works, with no test cases, no expected results, and no evidence anyone else can review. That idea is wrong, and in healthcare it is dangerous.
A test compares an actual result with an expected result defined in advance. The outcome is pass or fail. Run the same test against the same system and you get the same answer, and anyone can rerun it to confirm. That repeatability is what allows a test result to count as evidence.
AI works on a different principle. A model produces a statistical inference: the answer its training makes most probable. Even when it is configured to return the same output every time, that output is still a prediction rather than a check against a defined requirement. An AI that reports “this implementation is conformant” has made an educated guess, and a guess cannot be audited the way a test can.
Testing is deterministic by definition. AI is not. That difference decides the role each one can play.
The same distinction explains AI’s other limits in this work. An AI-generated test case still needs an expected result, and a qualified person has to decide why that result is correct and which requirement it verifies. Generating more tests does not add up to coverage, because verifying requirements across negative, workflow, version, and security scenarios takes systematic coverage that a probabilistic tool cannot guarantee. And AI cannot settle an ambiguous specification. It can flag unclear language or propose a reading, but when a guide will be implemented across an industry, the community has to resolve the ambiguity. Otherwise the AI’s interpretation becomes one more divergent implementation.
AI belongs inside a disciplined testing practice, where it helps skilled testers work faster and cover more ground. The verdict on whether an implementation meets a requirement has to come from a deterministic test with a documented expected result.
Measuring Maturity and Coverage
Today it is hard to answer basic questions about our ecosystem. How many implementers support a given IG? Which requirements have been independently verified? Which optional capabilities are in use? Where do failures keep recurring?
We think implementation maturity and implementation coverage should be measured openly and constructively. The purpose is to give authors, sponsors, and adopters an evidence-based picture of where the ecosystem stands and where investment would do the most good. Ranking vendors is not the goal.
What This Means for Connectathons
We have supported Connectathons for many years, and they have been central to FHIR’s growth. They can produce stronger evidence if expectations are set before the event starts:
- Identify the requirements and workflows each track will examine.
- Include negative and exception scenarios in the planned work.
- Record the versions and dependencies under test.
- Agree on a common vocabulary for reporting results and gaps, so findings can be compared across events and sent back to IG authors and sponsors.
- Offer practice beforehand, and carry unresolved findings into follow-up work afterward.
I would also like participants to leave with test assets and testing skills they can use at home. An education-first event keeps the work going between Connectathons.
The Work We Are Taking Forward
We are pursuing these ideas through HL7, working with the FHIR Infrastructure (FHIR-I) Work Group.
PSS-2770, sponsored by FHIR-I with the EHR Work Group as co-sponsor, focuses on bringing structure, traceability, and shared methods to how FHIR implementation guides are verified. It builds on our companion work under PSS-2717, which includes a FHIR Reference Implementation Maturity Framework and our paper “Reimagining the Connectathon.”
PSS-2834 introduces a Dual Rail Architecture, in which specifications and testing evidence are developed in parallel and each informs the other.
The question behind both projects is how to make testing useful while a specification is still being written, instead of waiting until publication to discover what is hard to implement or verify. Answering it takes contributions from the people doing the implementation and testing work.
Help Us Define the Evidence
We believe testing should be owned by the implementation community. Standards authors define what a specification requires. Implementers, testers, and QA professionals are best placed to determine how those requirements are verified in practice, and to report what they learn to the standards community and to the sponsors who fund adoption.
We are inviting implementers, testers, QA engineers, vendors, payers, providers, public health agencies, and program sponsors to take part. Bring a test approach you use. Bring a requirement that is hard to verify. Bring a finding the authors of an implementation guide need to hear. Help us build shared test assets, examine coverage, train the next generation of interoperability testers, and report results in a form other teams can use.
We also need to be candid about gaps. An untested requirement is useful information once it is identified and reported. It becomes a problem when a broad claim of “tested” hides it.
If you have ever found a defect in production that a better test would have caught, you already know why this matters. The next time any of us says “we tested it,” we should be ready to explain exactly what we mean.
Coming Next
Part 2 covers the RTM: how to break a FHIR implementation guide into testable requirements and connect each one to evidence. Later articles will take up negative and exception test design, the practical role of AI in testing, version-aware continuous interoperability, and how to measure implementation maturity across the ecosystem.
To take part in this work, contact mario.hyland@aegis.net or connect with me on LinkedIn at linkedin.com/in/interopguy.