UTP offers a concept to bind arbitration specifications (henceforth, the acronym AS is used) to test sets, test cases and procedural elements. The binding comes in three flavors (see page 136):

• default arbitration specification realized by an implicit binding
• static arbitration specification realized by a static binding directly by the respective test set, test case or procedural element
• dynamic arbitration specification (dynamic binding) overrides any previously existing binding in the context of a procedure invocation

Albeit these three binding kinds makes definitely sense (and should not be removed from the spec) the problem is the technical representation of how bindings shall be expressed, i.e., directly by the test set, test case, procedural element or procedure invocation for dynamic binding. The main issue is that the technical binding mechanism does not scale for large test sets that exhibits exhaustive modularization and reuse of test procedures. In theory, there can be an unlimited number of arbitration specifications set for a test set, test case or procedural element. The classification into static and dynamic binding is a mere synthetic (and unnecessarily complex) concept. In fact, there can only be one static binding being present at any point in time and any further binding must be made via dynamic binding, which in turn requires the use of a <<ProcedureInvocation>> and, thus, a TestExecutionSchedule to actually invoke a test set/test case to dynamically bind the respective AS. This imposes a certain methodology on the user, which is too prescriptive.

The entire mechanism becomes more complicated when procedural element AS shall be dynamically bound. This can only be achieved by simply listing procedural element AS in the set of 'ProcedureInvocation.dynamicArbitrationSpecification' and the use of the boolean field 'transitiveOverride'. It is not possible to explicitly bind a certaint procedural element AS to just a single procedural element (or a subset thereof). Furthermore, it is not easily possible to specify whether a transitiveOverride shall be applied to all deeply nested ProcedureInvocations or not.

Eventually, it must be stated that the approach to bind different AS to the very same test set/test case/procedural element is a good idea (that should be preserved), but as it is realized now, not usable and excessively complex.

A possible solution to that inflexible and overly complex binding mechanism could be to extract the binding from test set/test case/procedural element into a dedicated concept that allow for expressing these bindings external to the test set/test case/procedural element. This would have two advantages if done properly:

1. The test set/test case/procedural element would remain clean of any AS information and would only represent the information required to eventually execute a test case. Test sets/test cases/procedural elements would be easier to maintain after all.

2. An external binding mechanism would be able to explicitly specify which AS shall be set for which element in a certain context. This holds true even for deeply nested test procedures and procedural elements, which is not the case with the current binding mechanism. A kind of ArbitrationDirective would enable a flexible and unrestricted mechanism to define how the verdict for a given test set/test case/procedural element shall be calculated without imposing certain design decision (like the use of TestExecutionSchedules and ProcedureInvocations) on the test designer. This is in turn another fact that positively impacts maintainability of existing test specifications. Practically, an unlimited number of ArbitrationDirectives could be defined for any test set/test case, which would make the AS mechanism most flexible.

If extracted, there would only be two kinds of binding remain:

• implicit binding: if no additional AS information is specified for a certain test set/test case/procedural element, the default AS would take over (like it is right now)
• explicit binding: (Parts of) the AS bindings for test sets/test cases and procedural elements are explicitly bound and can be put into effect by using a certain ArbitrationDirectve. Additionally, other ArbitrationDirectives can be defined to declare further AS bindings, without the need to change (or even touch), the existing test set/test case/procedural elements.

Arbitration specification bindings will be cleaned up from technical debts

The FTF agree that the mechanism to bind multiple Arbitration Specifications to test cases, test sets and procedural elements is incomplete and blocking implementations. Parts of the specification needs to be cleaned up from techhnical debts.

Furthermore, the FTF agreed that UTP 2 would benefit from a mechanism that offers consistent and scalable way to bind different arbitration specifications for different analysis purposes. Idealy, the elements that shall be arbitrated and the corresponding arbitration specifications shall be loosely coupled by a dedicated and reliable UTP 2 capability for arbitration specification binding.

Adding such a new capability is outside the scope of the FTF. Therefore, the FTF will submit another issue for the upcoming RTF to incorporate that specific functionality.

Since dynamic binding of arbitration specifications is not anymore part of the specification, for it was never finished properly, editorial parts of the specification that deals with the different binding mechanisms must be revised to remove any description about the dynamic arbitration specification binding.

A few mistakes were made during the elaboration of the issue resolutions that led to a situation where editing instructions were fogotten to be stated, however, the changes have been applied to the language. The FTF had created an errata sheet to keep track of these supplementing editing instructions. Due to OMG policies, the changes described by the errata sheet need to be voted on as well. This was requested by the AB reviewer.
Therefore, the following list summarizes incomplete editing instructions of already resolved FTF issues (the errata sheet that was submitted with the first FTF submission is attached to the issues):

• Issue UMLTP2-14
  Summary of the correction: The predefined verdict 'fault' was removed from the UTP 2 Types Library. The corresponding element 'Fault' of the conceptual model (Clause 7.6 Arbitration Specification) was not removed by the resolution. This concept must be removed to keep the conceptual model and the profile specification in synch.

• Issue UMLTP2-24
  Summary of the correction: An obsolete entry 'Detailed Semantics' of 8.7.1.1.2.2 ArbitrationSpecification needs to be removed from the specification. Additionally, the Figure 8.1 Language Architecture need to reflect the removal of the UTP Arbitration library (old Clause 9.3, which is removed by the resolution of Issue UMLTP2-24)

• Issue UMLTP2-27
  Summary of the correction: The newly introduced stereotype "ArbitrationResult" needs to be listed in the profile summary table in Clause 8.2 Because of the introduction of a new stereotype, the numbering of the sub-clauses in clause 8.7.1.1.2 Stereotype Specifications needed to be updated. The new numbers that
  mentioned by the original issue resolution had a typo and were not correct. This is corrected by this errata document.

• Issue UMLTP2-29
  Summary of the correction: The newly introduced attribute 'expectationKind' of Clause 8.5.3.4.4 ExpectResponseAction was omitted by the issue resolution.

Corrections of incomplete editing instructions listed by the FTF errata sheet

Apply corrections for incomplete editing instructions of the following issues:

• Issue UMLTP2-14
  Summary of the correction: The predefined verdict 'fault' was removed from the UTP 2 Types Library. The corresponding element 'Fault' of the conceptual model (Clause 7.6 Arbitration Specification) was not removed by the resolution. This concept must be removed to keep the conceptual model and the profile specification in synch.

• Issue UMLTP2-24
  Summary of the correction: An obsolete entry 'Detailed Semantics' of 8.7.1.1.2.2 ArbitrationSpecification needs to be removed from the specification. Additionally, the Figure 8.1 Language Architecture need to reflect the removal of the UTP Arbitration library (old Clause 9.3, which is removed by the resolution of Issue UMLTP2-24)

• Issue UMLTP2-27
  Summary of the correction: The newly introduced stereotype "ArbitrationResult" needsto be listed in the profile summary table in Clause 8.2. Because of the introduction of a new stereotype, the numbering of the sub-clauses in clause 8.7.1.1.2 Stereotype Specifications needed to be updated. The new numbers that
  mentioned by the original issue resolution had a typo and were not correct. This is corrected by this errata document.

• Issue UMLTP2-29
  Summary of the correction: The newly introduced attribute 'expectationKind' of Clause 8.5.3.4.4 ExpectResponseAction was omitted by the issue resolution.

The stereotype ProcedureInvocation allows fro invoking test-specific procedures such as TestExecutionSchedules, TestCases and TestProcedures, but also simple libraries.

It is currently specializes CompoundProceduralElement, which seems to be erroneous. The invokation of a test-specific procedure is an atomic instruction, and as such, it should specialize AtomicProceduraleElement instead.

The corresponding ProcedureInvocationArbitrationSpecification specializes AtomicProceduralElementArbitrationSpecification, though. So, what ever decision will be made with respect to ProcedureInvocation, the corresponding arbitration specification might require some re-work, too.

Change ProcedureInvocation to specialize AtomicProceduralElement instead

The FTF decided that ProcedureInvocation is not a CompoundProceduralElement, but an AtomicProceduralElement. This is already reflected in the editorial parts of the specification (i.e., the conceptual model and the arbitration specification section of clause 8), but was obviously incorrectly stated in the diagrams of both the conceptual model and the profile specification (maybe as a relic from the initial submission).

This issues will be resolved together with proposal #54 for issue #24

Rules DRTC01 (should be "DRTR01") and DRTC01 have the same identifier.

Rename rule DRTC01 to DRTR01

The rule in clause 7.2.1 has accidently named the same way as a rule in clause 7.3.1.1. This must be fixed simply by renaming the rule.

UTP should better distinguish the origin of the terms in the vocabulary. E.g., there are concepts and definitions, which are solely listed in the conceptual model or the profile implementation vice versa. The UTP vocabulary should indicate where this term originated from.

A potential solution is to add a introductory sentence to section 4 to explain that the terms in section 4 are a summary of the conceptual model of UTP 2 (section 7). Furthermore, improve section 6.2 (Typographical Conventions) with respect to the kids of cross references in the spec (to the definition, to the conceptual model to stereotypes)

Clarify the nature of Clause 4

This issue will be merged with the issue #6 and resolved in the course of the resolution of issue #6 with is done with the proposal #47.

This issue was raised by Manfred Koethe as UTP 2 revised submission reviewer

All OCL constraints were removed from the specification

During the resolution of UMLTP2-16, it was decided to firstly removed the entire OCL code from the specification for it was highly erroneous and written in a rush. This means that also the incorrect references to UML 2.5 will removed in the course of the resolution of issue #16, thus, this issue will be resolved by proposal #UMLTP2-39.

This issue has been raised by Manfred Koethe as UTP 2 revised submission reviewer.

Clarify the nature of Clause 4

The terms and definitions found in clause 4 are a summary of the conceptual model that is defined in clause 7. Examples and further details can be found in the respective sub-section of clause 7.

Therfore, the FTF agreed to add an introduction to clause 4 that helps the reader to understand the nature of clause 4 and refer to clause 7 for further examples and details.

Furhter, the FTF decided that the definitions of test type and test level suffice as a summary, for further examples and details for both definitions are given in clause 7. Clause 6.1 "How to read this document" shall be updated accordingly.

In the course of this discussion, the FTF found out that the definition of test objective can be improved for clarity.

The description of the enumeration ProcedurePhaseKind is not part of the specification.

Incorporate ProcedurePhaseKind into the specification

The FTF decided that is an editorial issue that has to be fixed.

The tag definition specification of a TestObejctive and TestRequirement is currently typed by String. Although this would most likely suffice in many situation, it prevents tools to specify other forms than plain text (e.g., structured text with formal semantics such as ETSI's TPLan). If this tag definition would be of type ValueSpecification, tools/implementers could decide to go for a more capabale representation of a 'specification' such as OpaqueExpressions or Expression. Eventually, plain text is again supported by the use of LiteralString.

Replace String attributes 'specification' with ValueSpecification

As the issue description mentiones, replacing the String-typed attribute 'specification' of TestRequirement and TestObjective with ValueSpecification widens the possibilities of how to express these information.

The ID attribute offered by the TestSet and TestContext should be 0..1 similar to TestConfiguration, TestCase etc.
The reason is that those IDs shall act as traceability element in context of a testing tools landscapes, in particular for synchonization with test management and test reporting tools.
If such a traceability link is not required, models shall not become invalid.

Change multiplicity of attribute ID of TestSet and TestContext to 0..1

As recommended by the issue description, the mutliplicity of the attribute 'ID' should be changed to 0..1.
This proposal incorporates also the resolution of proposal UMLTP2-38. As agreed in that proposal, the tag definitions 'testLevel' and 'testType' of TestContext ought to have an upper bound of *.

The two tag definitions 'testLevel' and 'testType' determine the corresponding test item level and the test item's quality criteria that are targeted by the artifacts of the test context. Currently, this is set to an optional multiplicity of 1, but ought to be 0..*

Change upper bound to 0..*

The issue is correct; it is easily possible that tester want to reuse the test artifacts assembled by the test context for more than just one test level or test type.

In fact, the way it is written in the spec (plural), it seems that it was intended from the very beginning to assign an upper bound of * to both tag definitions.

The resolution of this issue was done in conjunction with proposal UMLTP2-34.

The formal specification of ChoiceOfValues is incomplete. These paragraphs are relics from an approach to provide a precise semantics of data value extensions.

Suggestion: Remove these paragraphs.

Remove incomplete formal specification of ChoiceOfValue

The paragraphs mentioned in the issue descriptions are relics of the attempt to state the semantics of some ValueSpecifications extensions more precise during the revised submission phase. It shall simply be removed from the description.

The last two paragraphs of AnyValue description are incomplete.

Suggest resolution: Remove those two paragraphs

Remove incomplete formal specification of AnyValue

The paragraphs mentioned in the issue descriptions are relics of the attempt to state the semantics of some ValueSpecifications extensions more precise during the revised submission phase. It shall simply be removed from the description.

Most changes specified in the errata to the beta specification (ad/17-06-01) have not yet been included (except for the changes on the cover page). These changes must be included in the final specification.

Resolve errata issues that are still valid

The FTF decided to check each errata issue for validity and to resolve them in case they have not yet been resolved in course of the production of the beta specification.

The following proposal compares the errata (OMG doc# ad/2017-06-01) issues (by number) with the published document with doc# ptc/2017-09-29. In case the errata issue has already been resolved during the production of the Beta specification (ptc/2017-09-29) na action will be taken and the corresponding errata issue will declared as obsolete.

UTP 2 provides the predefined verdict instances none, pass, inconclusive, fail, error. These instances are currenty represented by LiteralStrings of type 'verdict'. Using LiteralStrings is fine for the UTP test logging facility for TestCaseLog and TestSetLog simply refers to ValueSpecifications of type 'verdict'. But in situations where those predefined verdict instances shall be used in a different context e.g. in the context of types of Properties of a StructuredClassifier and InstanceSpecifications thereof, LiteralStrings cannot be really reused for Slot values. Therefore, the LiteralString should be reworked into InstanceSpecifications of type 'verdict' with their specification set to LiteralStrings (of type 'verdict') with name and value set to the 5 predefined verdicts.

Replace LiteralString verdict instances with InstanceSpecification verdict instances

As the issue description indicates the use of LiteralStrings as predefined verdict instances may prevent reuse of these predefined verdict instances in certain situations.
If one of the predefined verdict instances shall be used in a context other than associating test logs with these predefined verdicts, it happens that reuse is not possible because of composite associations between a UML metaclass and ValueSpecification.
For example, if there is a user-defined complex verdict, expressed as StructuredClassifer with Properties and a predefined verdict shall be assigned to one of the Properties in the context of an InstanceSpecification's Slot, it happens that Slot.value is a composite association to ValueSpecification. This means that the Slot physically owns the ValueSpecification, which is not possible then in the context of a predefined (i.e., read-only) model library.
Therefore, the proposal is to use InstanceSpecifications instead to express the predefined verdicts. By doing so, these verdict instances can be reused by means of InstanceValues.
The specification shall not, however, prevent using other ValueSpecifications as verdict instances in case of user-specific test models. It must be clearly stated that the use of InstanceSpecifications is simply the default UTP way to express verdict instances.

The relationship between "8.7.1 Arbitration Specifications" and "9.3 UTP Arbitration" is not clear. Probably, 9.3 is only the (informal?) semantics of 8.7.1?

Natural language description of default arbitration specification suffices for the moment

During the meeting on 19th April, the group agreed to do the following:

• Remove complete clause 9.3 UTP Arbitration from the spec for it represents an informative part of the spec. The classes provided by that library were never meant to be implemented by a vendor; they serve the mere purpose to unambiguously define the semantics of the default arbitration specifications.
• Provide a natrual language decscription of the default arbitration specification as normative part of the stereotype descriptions

Removal of clause 9.3 is the easiest way to solve this issue. Having an executable specification is a nice add-on, especially for tool vendors. But this should be a self-contained clause of mere informative nature. It is therefore heavily recommended and indorsed by the FTF to re-incorporate the precise yet executable semantics of the default arbitration specifications back into the document.

This issues was raised by Manfred Koethe as UTP 2 revised submission reviewer

All OCL constraints were removed from the specification

During the resolution of UMLTP2-16, it was decided to firstly removed the entire OCL code from the specification for it was highly erroneous and written in a rush. This means that also the incorrect references to UTP stereotypes in the OCL constraints were removed, thus, this issue was resolved by proposal UMLTP2-39.

This issue was raised by Manfred Koethe as UTP 2 revised submission reviewer.

Duplicated submission - see UMLTP2-6

This issue is a duplicated submission of UMLTP2-6

Most of the intra-document links (the blue underlined words) do not refer to the correct target. This seems to be caused by the transformation process between the original specification model generated out of the model and the final PDF document published as the beta version of UTP.

Obsolete without manual transformation of generated document

This issue should become obsolete without manual transformation of generated document

The concpetual model (Clause 7) and profile specification (Clause 8) uses the term 'procedure' as parent concept for the test-specific procedures introduced by UTP, which are TestCase, TestExecutionSchedule and TestProcedure.

While the test-specific procedures are reflected by stereotype definitions in Clause 8, the rather abstract concept 'procedure' is not represented as stereotype.

No added value by a stereotype Procedure

Having a stereotype <<Procedure>> would not add any value to the language.
The UML representation of the concept 'Procedure' is given by Behavior. A stereotype <<Procedure>> would not add any additional property. Therefore, the FTF decided to close this issue with no change.

Instead of creating a new stereotype, the FTF decided to introduce an additional derived tag definition into <<ProcedurInvocation>> of type Behavior that retrieves the invoked Behavior from the underlying UML metaclasses.

UTP allows for capturing the most important test execution information for test cases and test sets. Those information are the minimum required information to document and reasoning over coverage of test requirements or test objectives.

In theory and practice, it is possible and required to calculate verdicts by different verdict arbitrations for the very same test case or test set. A test log should only contain the mere information about the exchange of message between the test environment and the test item. Information about comparison or verdict arbitration should not be part of a test log, but of a dedicated ArbitrationResult concept.

The extraction of verdict into a dedicated ArbitrationResult is also necessary to avoid introducing redundant information into the model by copying the test log elements for the mere purpose of verdict association. Currently, it would not introduce much redundancy by copying just a single TestLog element (and the respective stereotype, of courde), but it is expected that UTP will introduce a fine-grained log model in a future version that is capable of recording all relevant information of test executions. If verdict still remained to test log or the test log entries later on, it would be necessary to potentially copy entire (and long) test log information with dozens of actual procedural elements, just for the sake of additional verdict arbitration. This situation would not scale at all for large test projects.

ArbitrationResult is an important concept to avoid redundancy

The concept ArbitrationResult will be integrated into the Profile Specification to avoid redundancy in case the very same test case shall be checked by different ArbitrationSpecifications which may produce different verdicts.
No change of the conceptual model required, for the ArbitrationResult is simply a technical solution to avaoid redundancy, not a conceptual change.
The solution will be based on UML's InstanceSpecification concept, for an ArbitrationResult is actually an instance of the execution of an ArbitrationSpecification.
ArbitrationResult will contain at least the attribute ‘verdict’. Additional information might be added by means of the pre-defined auxiliary library (later on)

Structural aspects other than the predefined ones for the stereotype TestLog can be added by means of a <<TestLogStructure>> Classifier. Currently, UTP restricts a TestLog to be classified by at most one TestLogStructure. But it may easily happen that a TestLog is supposed to capture the information form more than just one <<TestLogStructure>> Classifier. Therefore, the multiplicity of TestLog.instanceOf should be 0..*.

Allow for multiple <<TestLogStructure>> Classifiers

The abstract stereotype <<TestLog>> has a derived association to TestLogStructure. It is derived from the set of <<TestLogStructure>>Classifiers of the underlying InstanceSpecification (see InstanceSpecification.classifier). It should be possible to have more than just one <<TestLogStructure>>Classifier for any concrete <<TestLog>> to remain flexible and extendible for user needs.

This issue will be resolved in the course of resolving issue UMLTP2-27 with proposal UMLTP2-36.

When designing test cases, the formalization of expected responses of the SUT in accordance to previously sent stimuli are of utmost importance for eventual verdict calculation. The expectations of the SUT's behavior can be expressed by at least two sets, the set of expected responses and the set of forbidden elements. In some situations it accelerates the test design if the expected responses are specified (when X is submitted Y is expected), in some situations it is easier to simply specify what is not expected (when X is submitted everything else but not Y is expected). Which set shall rather be used cannot be answered and depends heavily on the context of the test case. These two sets are complemented by the set of responses that can be ignored as a response of a stimulus. Ignoring elements (i.e., excluding them from verdict arbitration) is an important capability in automated testing.

Currently, ExpectResponseAction explicitly defines the set of expected and forbidden elements. The set of ignored elements is implicitly represented by the complement set of the union of expected and forbidden elements.

The current semantics of forbiddenElements says that elements that are forbidden (i.e., that expected to not being sent by the SUT at all) need to be explicitly declared, and if no elements are specified, no elements are forbidden. This definition has a big disadvantage for testing, for it is prone to result in 'false negatives' during test execution. Let's assume that a tester makes a mistake while specifying the set of forbidden elements. Response Y should actually be forbidden, but the tester forgot to explicitly assign Y to the set of forbidden elements. Let's further assume that during test case execution Y occurs in a situation where it ought to not occur. Since it is implicitly (but erroneously) accepted as at least ignored element, the test case would conclude with a 'false negative' verdict. That means that an existing failure of the SUT was not detected as such. 'False negatives' are to be avoided under all circumstances for they provide a wrong assumption about the quality of the SUT.

Therefore, UTP should remove the set of forbidden elements from ExpectResponseAction and replace it by the set of ignored elements. By doing so, a tester would have to explicitly specify what is expected and what shall be ignored. All other elements, i.e., the complement set of the union of expected and ignored elements, are implicitly treated as forbidden elements. This would prevent the creation of implicit 'false negatives', for everything that the tester does not explicitly allow or ignore is considered to be forbidden.

A second alternative could be to enable the specification of all three sets for ExpectResponseAction and to introduce an enumeration ImplicitExpectationKind with literals ImplicitIgnore, ImplicitForbid (default) and ImplicitExpect. This enables the tester to define exactly those sets that are most appropriate for the respective ExpectResponseAction.

The three sets shall be made explicit

Flexibility is of utmost importance for a standard like UTP 2. Since it is not possible to foresee which way is the ideal way to deal with the three different sets, UTP 2 will make the three sets explicit.

In addition, to these three sets, a mechanism will be introduced that determines which set will be implicitly merged (union) with all other received responses. By default, the set of forbidden element will be implicitly extended. This prevents the occurrence of false negative results in case an actually forbidden element was forgotten to be specified as such.

Many Constraints of stereotypes are inconsistent with respect to OCL and natural language.

See section 8.5.1.2.2 TestCase:

• Constraint 'Use of BehavioredClassifier' is inconsistent
• Constraint /* DRTC02: It is necessary that each test execution schedule requires at most one preconditon. */ does not belong to the stereotype test case at all.

This is the case for most of the Constraints in section 8. A thorough check would be required to clean that up.

Remove Constraints and OCL syntax of natural langaue description if necessary

The formalization of natural language constraints into OCL code was rather done in a rush for the revised submission. As a result, many OCL constraints were not correctly written. In these situation, the OCL code shall be removed in the first place.
In some cases, entire Constraints were too restrictive and will be removed.

The predefined verdict fault and error are synonyms. Fault shall be removed. Error is more common in the test automation community to describe situations where some technical exception occured in the test environment.

Remove predefined 'fault' verdict

The predefined verdict 'fault' has no clear meaning to the TF. Therefore, it shall be removed from the specification entirely.

This issue shall be resolved by the resolution of issue UMLTP2-14.

Double-check that the LoginExample copes with the latest version of the spec.

Correct the details Login example in a later RTF

The example is based on an earlier submission (i.e., the initial submission) of the UTP 2. Even though the example is not anymore in synch with the current version of the profile specification, it still helps understanding how to apply the UTP ina dynamic test process.

Due to limited time, it was decided to defer the correction of the Login example to a later RTF.

The last two paragraphs represents an incomplete formal specification of the semantics of ComplementedValue. It should be removed for does not have an added value in the current state.

Remove incomplete formal specification of ComplementedValue

The paragraphs mentioned in the issue descriptions are relics of the attempt to state the semantics of some ValueSpecifications extensions more precise during the revised submission phase. It shall simply be removed from the description.

UTP 2 offers currently rudimentary test log capabilities. It is possible to capture the minimal test execution information for test cases and test sets. These information are sufficient for the sake of coverage calculation or high-level test reporting, however, the details of a test case execution (e.g., the execution of CreateStimulusActions or ExpectedResponseAction) are completely omitted. This wastes a lot of potential UTP could offer for building test automation architectures like post-execution comparison/verdict calculation, incorporation of test execution traces from heterogeneous test tools in order to process the captured information solely on model level (raising the level of abstraction for automated results checking), comprehensibility or high-level failure analysis and test execution comparison (trends etc.).

As high-level test modelling language, UTP ought to provide at least a facility to enable sophisticated activities on fine-grained test execution logs by capturing the execution on procedural element level.

Will be addressed by a later RTF due to timing reasons

During the F2F in Santander, the FTF agreed that this capability is an important aspect for UTP 2. It will be resolved in a subsequent RTF.

The current TestDesignFacility og UTP 2 is based on the stereotypes TestDesignDirective, TestDesignTechnique, TestDesignDirectiveStructure and TestDesignTechniqueStructure. These elements are tightly integrated by means of the Profile's abstract syntax. So far, this is fine for the test design purposes. As such, the offered concepts support the definition of platform-independent instructions how to derive test cases, test data, test sets, test procedures and/or test execution schedules. These instruction can be put into effect by either a manual test designer or translated into the inpurt for a test generator.

UTP 2, however, is capable of describing test automation architectures (ISTQB TAA) or the architecture of automated testware - both terms are considered to be synonyms. In a test automation architecture, further activities need to be specified such as the generation of executable test code, the configuration and automated setup of the test execution environment and tool (setup and teardown of the automated test environment; compilation of test scripts, automated scheduling of test cases in a test set according to a given test execution schedule etc.), the configuration of the logging facility (e.g., depths and structure of logged information), the configuration of the arbitration specification (which arbitration specifications shall be used, linked to test cases, test sets and procedural elements, etc.).

All these configurations are essentials parts of an automatedt testware and can be used in combination or separately from each other (e.g., specifying a specific arbitration specification for verdict calculation on already executed set of test cases in a post-execution manner).

UTP is currently too restrictive and only offers concepts for configuring the test design activities. As a test modelling language designed in particular for test automation, UTP should offer also concepts for the other activities that need to be performed in an test automation solution.

Therfore, UTP should abstract from TestDesignDirective and TestDesignTechnqiue into abstract classes TestArchitectureDirective and TestArchitectureCapability (and the respective structures). That would, one hand, enables user-specific extensions in order to cope with the aforementioned, not yet supported concepts of test script generation, test execution, test logging and verdict arbitration, on the other hand prepare UTP 2 for potential improvements in that area in further revisions of the language.

Wil be addressed by a later RTF due to time reasons

The FTF agreed that this is an important capability of which UTP 2 would benefit from. But it is not a implementation bocking issue. It would also not break backward compatibilitry, therefore, it will be deferred to a later RTF.