Decrypting the Jedi Scrolls – Qualification is different from validation

In Finland, it is a regular joke that to license technology for nuclear power plant use is actually an art of interpretation of the will of the safety authority. But sometimes it is not about the interpretation why the nuclear regulatory text is so hard to understand. Making mistakes is something that happens to everybody, even to Safety Authorities. Within this post, I present one example of such a case in public, because the issue at hand is bit broader than just correcting wording, and good to understand by domain experts.

During the updates of YVL guides for nuclear safety in 2019, STUK changed the definition of "qualification". The March 2019 version of YVL guide E.7 defines "Qualification is normally used as a synonym for ’validation’ in YVL-guides.", and later in June, the content of YVL guide B.1 with content related to system level qualification followed this approach and finally abolished the difference between concepts of qualification and validation. That made the understanding of different YVL guide requirements regarding qualification totally hard.

Verification and validation

In order to understand qualification, it is necessary to begin from verification and validation. Practically qualification as well as verification and validation are all about assessing requirement conformance. The fundamental difference comes from criteria used, as well as from whose aspect the decision about the conformance is made.

ISO 9001 definitions for verification and validation are quite similar at first sight. ”Confirmation” of ”objective evidence”. The beef is written in purpose of the activity: Verification is defined as confirmation that "specified requirements are fulfilled" when the validation is defined as confirmation that "requirements for a specific intended use or application have been fulfilled". The criteria for verification is documented specification, and the criteria for validation more or less documented ”intended use” or customer ”real needs”. Basically it’s ok to pass verification succesfully even the product is not adequate for intented use if the specification is fulfilled. But passing the validation in such a case would be a critical mistake. Sometimes the difference is expressed so that verification is to ensure the product has been done correctly – according the specifications, where the validation is ensuring that the product is correct – ok for intended use.

The above makes also the role of decision making visible. Verification and related decision of conformance against specification is usually performed by the supplying organization, which is also necessary when there is multiple levels of development specifications. In turn, validation is usually witnessed, approved and sometimes even performed by the customer in large projects.

So, from the theoretical, ISO 9000 framework point of view, the difference is clear. Practically, the situation is a bit more complicated. Compared to stereotypical single customer-supplier situation of ISO 9001, in larger projects there is found a stack of organizations where organizations are suppliers for their customers, but on the same time customers for their suppliers. Because the situation is relative to the organization position in the supply chain, the easy-to-use integrator approach is simply to talk about V&V.

Unfortunately, within the nuclear domain the IAEA safety glossary definitions are not as good as in the ISO framework. Especially an idea of verification against ”intended quality” is unclear. However, with closer examination it can be seen, that the very idea behind definitions is similar to ISO one.

What about qualification?

Concept of ”qualification” is more domain-dependent than ”verification” or ”validation”. There is various types of product qualification, process qualification, method/activity qualification, personnel qualification etc. In the 2005 revision of ISO 9000 vocabulary, qualification process was defined as a demonstration process: ”process to demonstrate the ability to fulfil specified requirements”. It’s obvious, that for the demonstration process, the very same questions of the conformance criteria and decision making are valid. IAEA safety glossary 2018 notes that specific ”qualification requirements” are existing, as well as qualification decision is not necessarily bound to the application/project time (concept of component prequalification).

Unfortunately, the 2015 revision of ISO 9000 dropped the definition of ”qualification process” and complicated the issue, but lets come to that a bit later.

Regarding the nuclear technology domain, it is important to recognize that there are two levels of qualification: equipment (or component) qualification and systems qualification. The first one is traditional and well understood, the later one is challenging.

The nuclear domain works by using systems, structures and components. This is a classical division: Structures are passive elements like parts of buildings. Components are combined together to form systems performing the functionality needed. From the beginning of nuclear industry, the system design has been considered as an ”art of combining proper components” performed by ”guru designers”. The mindset is still visible in the nuclear standardization of today’s. Usually there are strict regulatory rules utilizing international standards for components, and more variant principles for the system level design and quality management.

The above is also reason why the nuclear qualification is most clearly defined for the component level. The massive amount of inspections and testing of components is outsourced to specialized bodies doing the job. The difference to verification and validation is significant: The qualification criteria are publicly available, open for discussion like harmonized standards or regulatory guidelines, where the V&V criteria is project specific at least on highest levels of supply chain and definitely not open for public discussion. Also the decision making structure is different: V&V are dealing mainly with supplier activities possibly extended by customer, where the qualification attestation is done by 3rd party assessment body or safety authority body.

”System level qualification” is new (and challenging) concept

Until the latest version of IAEA safety glossary (2018), IAEA made qualification as an synonym for equipment qualification. The latest version includes also the systems qualification. Clearly, there is a process of ”facility specific system qualification” but what does it mean? Unfortunately, the standardization seems to end here, and the ”facility specific system qualification” is about national regulatory pecularities, more or less bound to overall licensing process.

In Finnish YVL guides valid until the 1990s, there was not a lot of actual requirements on the system level design. The approach was more or less ”assess-and-analyze” for all design disciplines. The rise of software intensive technology however concretized the need of requirement oriented, systems engineering based approach, and also the need of the system level qualification process. In Finnish practice, the focus was raised first time clearly to the system level in guide YVL 5.5 for I&C (2002), and later in YVL 5.2 for electrical (2004). These guides introduced also the concept of system qualification for I&C and electrical systems. Unfortunately, when YVL guides mentioned defined that ”Qualification verifies the conformity of the systems and their components with the requirements”, there was never written advice, what was the exact relation to verification and validation, and what the required qualification exactly meant in practice.

For that reason, the definition of the ”qualification process” in ISO 9000 (2005) was useful. Taking into account the criteria and decision making, the triplet verification-validation-qualification made sense. The system qualification was a demonstration for the safety authority that the system is ok to use, and the safety authority made decision based on his public YVL guidelines either explicitly (by approval) or implicitly (not rejecting). The usage of the ”system qualification” in the next major update of regulatory guides in 2013 was analogous to the interpretation, as well as I&C link to the nuclear IEC-EN standardization.

In 2014 STUK released regulatory guide YVL A.5 considering construction and commissioning of the nuclear facility. By YVL A.5 STUK made an implicit division between licensing and qualification activities. This happened by requiring licensing plan ”describing how the fulfilment of nuclear and radiation safety requirements is ensured and demonstrated in the different phases of the construction or plant modification project”. With simultanious existence of requirements relating system and component level qualification processes, there was small philosophical overlap, but when reading the actual text, the division was quite clear. Licensing was plant level process, dealing with the information defined in the laws and decrees. Qualification was system and component level process, dealing with the information defined in the application specific YVL guides.

Something went wrong

STUK changed the definition of qualification in 2019, and after that it has been very hard to guess, what the regulator means by requirement texts related to qualification. Of course there is room for interpretation in wording "Qualification is normally used as a synonym for ’validation’”. However, without any documented idea of abnormal (or paranormal) cases compared to normal, the definition of ”normal” becomes a rule.

Based on the information available, it cannot be traced why STUK made qualification synonym to validation. One reason may be the (philosophical) overlapping between qualification processes and licensing process presented by YVL A.5, combined with the fact that prior to the vocabulary change, there was actually no explicitly required validation planning for process system design. On the other hand, the change may be result of cumulative defects: In 2015 ISO 9000 standard revision, ISO released ”qualification” free for interpretation and removed the definition of ”qualification process” from the base vocabulary. The funny part is, that also ISO made a mistake: Note 2 of 3.8.6 defines ”activities carried out for verification are sometimes called a qualification process”. Even this is theoretically justifiable for certain cases, without specific definition of qualification process the footnote becomes a rule.

As a result, YVL guides are defining qualification is same as validation, ISO is saying – by footnote - that qualification is same as verification, and combining these in the same project it means that when deriving from the definitions, logically all three are the same. Clearly something went wrong.

Need of harmonization

From practical point of view, I&C is quite safe due to relatively well defined terminology and concepts in the framework of IEC SC45 standardization. However, the Finnish national practice has been that the YVL terminology is preferred to international one, and in this case the exception is inevitable due to the above described defect in YVL terminology. For other disciplines, the situation is more unclear.

The simpliest way to fix the YVL guidance defect is to restore pre-2019 terminology of regulatory guides for qualification. From engineering and life cycle processes point of view, qualification is different from validation and shoehorning them into the same means only problems for everybody.

Additionally, there is need for clarification for application of qualification processes in system level, and what is the relationship between system qualification process and licensing process. How to divide the processes is a regulatory decision, because both are in close connection to regulatory interface of the licensees/license holders. Practically, there is probably no major difference, because the safety demonstration has to be done in both plant and system level, and the question is more about documentation structure and document titles.

Conclusions

Definitions are too many times considered as a sidestep to actual requirement/content writing, but the fact is that the requirement set or document content is at most as valid as the definitions behind.

The above was story of the one small defect in YVL definitions. However, counting how many requirement, design, planning and licensing documents are dependent of definitions of verification, validation and qualification, the financial effects of defective definition may become surprisingly large.


Mika Koskela

CEO, partner and principal expert in IntoWorks Oy, consultant company specialized for safety critical applications.

Disclaimer: The author acknowledges, that the only valid organization to interpret STUK YVL guides is STUK itself, and faulty interpretation by the author is possible. The author has presented the information as in the role of ”concerned citizen”, and the author, or IntoWorks Oy do not bear any responsibility of any use of information presented.

Posted in Uncategorized.