UMLTP2 — Clarification of the sets expectedElement, forbiddenElement and ignoredElement in ExpectResponseAction

When designing test cases, the formalization of expected responses of the SUT in accordance to previously sent stimuli are of utmost importance for eventual verdict calculation. The expectations of the SUT's behavior can be expressed by at least two sets, the set of expected responses and the set of forbidden elements. In some situations it accelerates the test design if the expected responses are specified (when X is submitted Y is expected), in some situations it is easier to simply specify what is not expected (when X is submitted everything else but not Y is expected). Which set shall rather be used cannot be answered and depends heavily on the context of the test case. These two sets are complemented by the set of responses that can be ignored as a response of a stimulus. Ignoring elements (i.e., excluding them from verdict arbitration) is an important capability in automated testing.

Currently, ExpectResponseAction explicitly defines the set of expected and forbidden elements. The set of ignored elements is implicitly represented by the complement set of the union of expected and forbidden elements.

The current semantics of forbiddenElements says that elements that are forbidden (i.e., that expected to not being sent by the SUT at all) need to be explicitly declared, and if no elements are specified, no elements are forbidden. This definition has a big disadvantage for testing, for it is prone to result in 'false negatives' during test execution. Let's assume that a tester makes a mistake while specifying the set of forbidden elements. Response Y should actually be forbidden, but the tester forgot to explicitly assign Y to the set of forbidden elements. Let's further assume that during test case execution Y occurs in a situation where it ought to not occur. Since it is implicitly (but erroneously) accepted as at least ignored element, the test case would conclude with a 'false negative' verdict. That means that an existing failure of the SUT was not detected as such. 'False negatives' are to be avoided under all circumstances for they provide a wrong assumption about the quality of the SUT.

Therefore, UTP should remove the set of forbidden elements from ExpectResponseAction and replace it by the set of ignored elements. By doing so, a tester would have to explicitly specify what is expected and what shall be ignored. All other elements, i.e., the complement set of the union of expected and ignored elements, are implicitly treated as forbidden elements. This would prevent the creation of implicit 'false negatives', for everything that the tester does not explicitly allow or ignore is considered to be forbidden.

A second alternative could be to enable the specification of all three sets for ExpectResponseAction and to introduce an enumeration ImplicitExpectationKind with literals ImplicitIgnore, ImplicitForbid (default) and ImplicitExpect. This enables the tester to define exactly those sets that are most appropriate for the respective ExpectResponseAction.

The three sets shall be made explicit

Flexibility is of utmost importance for a standard like UTP 2. Since it is not possible to foresee which way is the ideal way to deal with the three different sets, UTP 2 will make the three sets explicit.

In addition, to these three sets, a mechanism will be introduced that determines which set will be implicitly merged (union) with all other received responses. By default, the set of forbidden element will be implicitly extended. This prevents the occurrence of false negative results in case an actually forbidden element was forgotten to be specified as such.