UTP11 — ValidatonAction should provide a reason why the verdict has been assigned

Most test execution systems allow the definition of a reason why a particular test cases has failed or passed.

In TTCN-3, for example, the user can attach a string message to the setverdict() operation indicating the reason that describes verdict from a logical and human-readable perspective.

In JUnit the same can be done within the Assert.XXX methods.

Such a human-readable message guides the analysis commonly performed by testers and developers, by providing a logical explanation (logical in the sense of what does the verdict state with regard to the intention of the test case) to the technical verdict.

A message "The system was expected to respond y and when X was submitted" is more obvious and can be understood even by people outside of the testing/development process than a simple FAIL, where the logical reason behind that fail has to be derived from the stimuli and corresponding reponse within the test case execution trace.

To support this, ValidationAction should have an additional attribute 'reason : String [0..1]'

This would just be consequent, since a TestLog in (the extended version) UTP is able to expose the reason why the final verdict has beedn assigned to the test log.

The RTF decided to extend the ValidationAction with a new tag definition reason. That tag definition allows a user to directly log arbitrary information directly while setting a verdict. This is required by relevant industry standards like ISO 29119, but is part of almost any test execution tool. UTP should cope with that concepts, whereas the RTF decided to not predefine how such a reason has to be specified, nor how many reasons might be logged for a ValidationAction, nor what reasons are actually logged as the final verdictReason in TestLog. Having this premise, the RTF agreed on having an optional tag definition in ValidationAction with an unbound upper multiplicity. The type of reason is ValueSpecification, since this allows the highest flexibility to the user. A user can then decide to either log plain messages (LiteralString) or even complex InstanceSpecifications (InstanceValue).