The current document generation process used to produce the UTP 2 specs is limited with respect to the illustration of property/association end modifiers (i.e., unique, ordered, redefines, subsets, read-only, union). According to UML, those modifiers shall be illustrated after der name and type and multiplicity of a property. The following grammar rule (taken from UML 2.5, Clause 9.54) specifies the textual notation of properties:
<property> ::= [<visibility>] [‘/’] <name> [‘:’ <prop-type>] [‘[‘ <multiplicity-range> ‘]’] [‘=’ <default>] [‘

’]

In the UTP 2 document generation framework the illutration of modifiers was circumvented in a way that is is close to, but different to the notation prescribed by UML. Although, the aligned notation of UTP 2 is okay from a readers point of view, it appears that the modifiers have not been consistenly illustrated. See two examples from the spec:

In section 8.3.2.7.17 TestDesignDirective , TestDesignDirective modifies are displayed before the name of the propery:

: RoleConfiguration [*]

Even though both ways are still good enough to comprehend, from an editorial point of view, one way should be preferred to other.

Deferred due to lack of time

The RTF agreed that this a valid, but pure editorial issues. As explained in the issue description, the reason for these varying property modifiers lies in the document generation framework. Since all required information is available, though, the RTF agreed to resolve this issue in a future RTF.

In the current spec, the abstract syntax diagrams reside in dedictated sub-sections directly underneath the conceptual context section (e.g., 8.7.2. Test Logging) they belong to. Stereotype definitions of ths conceptual context are grouped in a dedicated sub-section 'stereotype definitions'.

In UTP 2.1 RTF turned out that this structure does not support maintainability and modifiability of the spec. Everytime an additional diagram is added to the conceptual context section (maybe due to enhacements, clean-ups, better seperation and illustration of complex diagrams), the entire stereotype definitions need to be renumbered, because the respective diagram sub-sections and the 'stereotype specification' sub-section reside on the same level.

If the abstract syntax diagrams would be sub-sections of a section called 'Abstract Syntax', there would be no need to renumber stereotypes.

Therefore, I propose to make the spec more maintainable and stable by incoporating the following structure for every concepual context section in the spec.

8.7. Test Evaluation
8.7.1 ArbitrationSpecifications
8.7.2 Test Logging
8.7.2.1 Abstract Syntax
8.7.2.2 Stereotype Specfications

Deferred due to lack of time

The issues requests a pure editorial change for better maintainability of the specifcation. The RTF agreed that the proposed change makes sense, but due to lack of time, the resolution of that issue will be resolved in a future RTF.

In the current spec, non-navigable association ends (i.e., association ends that shall belong to the association) are still listed in the section 'Associations' of the stereotype on the other side.

See for example in section 8.4.2.5 TestConfigurationRole the following entry in the sub-section 'Associations':
: TestConfiguration

The spec should be cleaned up in that regard.

Deferred due to lack of time

The RTF agreed that this is indeed a valid, yet pure editorial issue. The reason for showing non-navigable association ends in the stereotype specifications lies in the document generation framework that is used to produce the specification document. It is, at the moment, not possible to spare non-navigable association ends from the generation process. In the next RTF, it will be resolved by either enhancing the document generation framework in a way so that it can spare non-navigable association ends or by adding an editorial explanation to clause 6.2 Typographical conventions.

Decision table testing is a major and frequently used black-box test design technique. In system and acceptance testing as well as testing of business processes, decision table testing is used to identify the test conditions within business rules by representing these rules in a tabluar format.

The Decision Modeling Notation (DMN) provides a MOF-based metamodel for modeling decision tables. Within the DMN, decition tables are an isolated part, well suited for being extracted as profile as part of UTP.

Incorporating a decision table profile into UTP, inspired by the DMN MOF-based metamodel, would improve UTP in the area of business rule testing and make UTP more complete regarding specification-oriented test design techniques.

Deferred due to lack of time

The RTF agreed that there should be a tighter integration between decision tables and utp2 to perform decision table testing. Due to lack of time, the RTF agreed to defer the resolution of this issue to a subsequence RTF

Curently, UTP2 supports capturing actual results of a Test Set or a Test Case execution via its Test Log capability. A TestLog is an instance of a Test Log Structure that shall specify, what exacty is to be logged. However, the current UTP2 specification does not specify, how such a Test Log Structure is to be expressed. Furthermore, when one or more Test Logs have been instantiated (after executing some Test Sets or Test Cases), the tester may whish to extract only those parts of an existing (potentially huge) Test Log that are relevant to investigate a specific problem. This is not supported by the current UTP2 specification.

Deferred due to lack of time

RTF agreed that this issues is a helpful extension to the language. It will, thus, be addressed in a future revision due to lack of time.

Double-check that the LoginExample copes with the latest version of the spec.

Deferred due to lack of time

This issue is still valid, but will be addressed in a future RTF due to lack of time.

"Figure 8.15 - Test-specific actions Overview" needs to be cleaned up. Stereotype shapes are overlapping.

Replace Figure 8.15 with new figure

RTF agreed to correct this pure editorial issue.

The specification documents (both ptc/23-04-02 and ptc/23-04-03) that were submitted with the UTP 2.2 RTF report (doc#: ptc/23-04-01) contained some issue with editiorial content. Some words that were set in (bold) blue do not represent links, but the typographical convention for these words are not given in the specification. Furthermore, the newly added clauses made not use of those links at all.

Complementary editorial/typographical changes from AB comments

The following resolution and the editing instructions represent complementary editorial corrections of the already submitted specification documents (both ptc/23-04-02 and ptc/23-04-03) because of the Ab review comments recieved after submission of the UTP 2.2 in May 2023 (doc# ptc/23-04-01).
The editing instructions describe only the delta of changes, thus, all the changes reported by the editing instructions of the formert report (doc# ptc/23-04-01) are present in the new specifications documents. In addition, the new submitted specification documents (ptc/24-07-02 and ptc/23-07-03) contain editing instructions as a result of the AB review of the former submission (ptc/23-04-01).

The machine-readable files submitted with UTP 2.2 RTF report (doc#: ptc/23-04-01) contained some syntactical issues that prevented the import of the machine readable files into compliant tools.

Minor editorial changes to the XMI

The machine-readable file for the UTP2 metamodel (utp2.xm, ptc/23-04-06) contained the following issues that prevent import/validity:

Issue in Line 1091
<ownedEnd xmi:type="uml:Property" xmi:id="_1CSfwPAuEeW2VZXfgcC2cA" name="referencedBy" type="_omHdQKJVEeS8gLY2208_PA" isDerived="true"association="_1CR4sPAuEeW2VZXfgcC2cA">

Corrected to:
<ownedEnd xmi:type="uml:Property" xmi:id="_1CSfwPAuEeW2VZXfgcC2cA" name="referencedBy" type="_omHdQKJVEeS8gLY2208_PA" isDerived="true" association="_1CR4sPAuEeW2VZXfgcC2cA">

Issue in line 1231:
<ownedLiteral xmi:type="uml:EnumerationLiteral" xmi:id="_MPcSIFeHEeihnpddrZbjug" name="implicitExpect"i/>

Corrected to:
<ownedLiteral xmi:type="uml:EnumerationLiteral" xmi:id="_MPcSIFeHEeihnpddrZbjug" name="implicitExpect"/>

Issue in Line 1566:
<ownedEnd xmi:type="uml:ExtensionEnd" xmi:id="KAmGUHJKEemyx_XW_ukewg" name="extension_TestTechniqueStructure" type="_90Z0HJJEemyx_XW_ukewg" aggregation="composite"association="_KAc8YHJKEemyx_XW_ukewg"/>

Corrected to:
<ownedEnd xmi:type="uml:ExtensionEnd" xmi:id="KAmGUHJKEemyx_XW_ukewg" name="extension_TestTechniqueStructure" type="_90Z0HJJEemyx_XW_ukewg" aggregation="composite" association="_KAc8YHJKEemyx_XW_ukewg"/>

There is an obvious misplaced change bar marker in the beginning of section "8.6.1 Data Specification". It needs to be removed.

Changebar not present anymore in UTP 2.1 official document

This pure editotial issue has been already fixed with the release of the UTP 2.1 document

UTP 2 defines for each element that may produce a verdict (i.e. test set, test case and all procedural elements), default arbitration specifications that prescribe how to calculate that verdict. The default arbitration specifications are optimized for functional testing, though. In case of testing non-functional aspects such as fault tolerance, performance or security, the rules to determine functional verdicts do usually not suffice anymore. Therfore, UTP 2 enables the definiton of proprietary arbitration specification (mainfested as stereotype <<ArbitrationSpecification>>) to grant the hightest flexibility possible to the user.

However, the current mechanism to associate a test set, test case or procedural element with a dedicated arbitration specification is restricted to only one arbitration specification. Even though this would be sufficient in many situations, the problem with the current binding mechanism is that the test set, test case or procedural element is literally changed by the binding. This may lead to situations where elements are copied only for the sake of associating a different arbitration specification to a that element. This is a situation that should be avoided under all circumstances.

We suggest adding a loosely coupled arbitration specification binding mechanism that keeps the elements and the arbitration specifications that ought to be bound to that element separated from each other. This would bring the benefit that test sets, test cases and procedural elements stay clean of any arbitration specification. This would also foster reusability of these elements.

The loosely coupled mechanism to bind arbitration specifications to the test sets, test cases and procedural elements could be based on a new concept 'ArbitrationDirective' , similar to the 'TestDesignDirective' concept that already exists in the language and proved its applicability. By doing so, users could define and assign an arbitrary number of arbitration specifications to a test set, test case or procedural element without technically modifying any of these elements.

Introducing an arbitration specification binding mechanism

The RTF agreed that the current arbitration facility is not flexile enough and will result in a number of duplicated model elements just for the purpose of assigning different pass/fail criteria. The RTF has developed an arbitration specification binding mechanism that loosely couples arbitration targets with arbitration specifications. Bindings can be overridden in a cascading manner that fosters reuse of existing arbitration specification.

The last paragraph of Clause 8.6.2 Data Values is an unintended left over from the revised submission. In the revised submission, there was the attempt to precisely define the semantics of matching ValueSpecifications. This attempt ought to be entirely removed. Thus, this paragraph should be removed.
For the sake of clarity:

Both stimuli and expected responses yield data values for distinct signature elements. A signature element is defined
as instance of either a Parameter or Property (i.e., this specification introduces a virtual metaclass SignatureElement that is the joint superclass of Property and Parameter and has at least the following attributes: type : UML::Type,
lower : Integer, upper : UnlimitedNatural). Given by UML [UML25], a "... Type specifies a set of allowed values
known as the instances of the Type." This specification denotes this set in the context of a SignatureElement
expressed as type(se), with type(se) as SignatureElement.type, and use T as abbreviation for type(se).
We specify
with se instance of SignatureElement and lower(se) as SignatureElement.lower and denote it by SE type.
A ValueSpecification V as an argument for a SignatureElement is specified as
These basic definitions are further used for the specific ValueSpecification matching mechanism extensions
introduced by UTP.

Remove last paragraph

RTF agreed to remove the last paragraph as it represents a relict from the initial submission.

Figure 9.3 shows a contained package "UTP Types Library" which is a relict from the initial submission. The UTP Types Library is a stand-alone library, technically unrelated to the UTP Auxiliary Library. This is also reflected in the machine readable files. Figure 9.3 needs to be corrected in that regard.

Replace Figure 9.3 with new figure

Task force agreed that this minor, pure editorial issue should be resolved by replacing the erroneous figure with a corrected one

This linebreak can be removed.

Issue not present anymore

The reported issue is not anymore present in the document with number formal/20-08-05.

This is an urgent issue, raised by the AB with respect to the correctness of the XMI files as well as the URI of the XMI files and the correct URIs mentioned on the cover page of the UTP 2.1 specification.

Correct XMI files and URI of machine-readable files for UTP 2.1

As discussed during the AB meeting at the Amsterdam meeting, the XMI file needed some cleanup and also the URI of the machine-readable files need to be corrected on the cover page.

The current TestDesignFacility og UTP 2 is based on the stereotypes TestDesignDirective, TestDesignTechnique, TestDesignDirectiveStructure and TestDesignTechniqueStructure. These elements are tightly integrated by means of the Profile's abstract syntax. So far, this is fine for the test design purposes. As such, the offered concepts support the definition of platform-independent instructions how to derive test cases, test data, test sets, test procedures and/or test execution schedules. These instruction can be put into effect by either a manual test designer or translated into the inpurt for a test generator.

UTP 2, however, is capable of describing test automation architectures (ISTQB TAA) or the architecture of automated testware - both terms are considered to be synonyms. In a test automation architecture, further activities need to be specified such as the generation of executable test code, the configuration and automated setup of the test execution environment and tool (setup and teardown of the automated test environment; compilation of test scripts, automated scheduling of test cases in a test set according to a given test execution schedule etc.), the configuration of the logging facility (e.g., depths and structure of logged information), the configuration of the arbitration specification (which arbitration specifications shall be used, linked to test cases, test sets and procedural elements, etc.).

All these configurations are essentials parts of an automatedt testware and can be used in combination or separately from each other (e.g., specifying a specific arbitration specification for verdict calculation on already executed set of test cases in a post-execution manner).

UTP is currently too restrictive and only offers concepts for configuring the test design activities. As a test modelling language designed in particular for test automation, UTP should offer also concepts for the other activities that need to be performed in an test automation solution.

Therfore, UTP should abstract from TestDesignDirective and TestDesignTechnqiue into abstract classes TestArchitectureDirective and TestArchitectureCapability (and the respective structures). That would, one hand, enables user-specific extensions in order to cope with the aforementioned, not yet supported concepts of test script generation, test execution, test logging and verdict arbitration, on the other hand prepare UTP 2 for potential improvements in that area in further revisions of the language.

Introduce an abstract TestDirective facility

The RTF agreed that an integration of an abstract test directive concepts (similar to the already existing and proved test design directive facility) enables test engineers to capture information relevant for carrying out certain test-related activities such as test design, test execution and/or arbitration. Allowing test engineers to model these kind of information opens the door for automating them.

UTP 2 offers currently rudimentary test log capabilities. It is possible to capture the minimal test execution information for test cases and test sets. These information are sufficient for the sake of coverage calculation or high-level test reporting, however, the details of a test case execution (e.g., the execution of CreateStimulusActions or ExpectedResponseAction) are completely omitted. This wastes a lot of potential UTP could offer for building test automation architectures like post-execution comparison/verdict calculation, incorporation of test execution traces from heterogeneous test tools in order to process the captured information solely on model level (raising the level of abstraction for automated results checking), comprehensibility or high-level failure analysis and test execution comparison (trends etc.).

As high-level test modelling language, UTP ought to provide at least a facility to enable sophisticated activities on fine-grained test execution logs by capturing the execution on procedural element level.

Enable logging on procedural element level

The task force agreed that the ability of abstracting from technical test logs into platform-independent test logs opens the door for UTP 2 to integrate the execution details of different test results potentially created by different test exeuction tools, as well as additional information such as information obtained from dynamic analysis tools by means of a single platform-independent language. UTP 2 may use these information for many purposes such as visualizing the executed test cases in a unique way; post-processing the execution results to calculate an overall verdict; perform further analysis like timing violations etc.
Having a single, integrated view on the results of test executions may become very conventient particulary in situations with heterogeneous testing landscapes where the eventual verdict of a test execution log has to be computed by a comprehensive analysis of diverse test execution logs.
UTP 2 can be seen as a kind of test information integration language that enables the specification and implementation of analyses over a set of originally seperated test execution logs.

Double-check that the LoginExample copes with the latest version of the spec.

Deferred due to lack of time

This issue is still valid, but will be addressed in a future RTF due to lack of time.

UTP offers a concept to bind arbitration specifications (henceforth, the acronym AS is used) to test sets, test cases and procedural elements. The binding comes in three flavors (see page 136):

• default arbitration specification realized by an implicit binding
• static arbitration specification realized by a static binding directly by the respective test set, test case or procedural element
• dynamic arbitration specification (dynamic binding) overrides any previously existing binding in the context of a procedure invocation

Albeit these three binding kinds makes definitely sense (and should not be removed from the spec) the problem is the technical representation of how bindings shall be expressed, i.e., directly by the test set, test case, procedural element or procedure invocation for dynamic binding. The main issue is that the technical binding mechanism does not scale for large test sets that exhibits exhaustive modularization and reuse of test procedures. In theory, there can be an unlimited number of arbitration specifications set for a test set, test case or procedural element. The classification into static and dynamic binding is a mere synthetic (and unnecessarily complex) concept. In fact, there can only be one static binding being present at any point in time and any further binding must be made via dynamic binding, which in turn requires the use of a <<ProcedureInvocation>> and, thus, a TestExecutionSchedule to actually invoke a test set/test case to dynamically bind the respective AS. This imposes a certain methodology on the user, which is too prescriptive.

The entire mechanism becomes more complicated when procedural element AS shall be dynamically bound. This can only be achieved by simply listing procedural element AS in the set of 'ProcedureInvocation.dynamicArbitrationSpecification' and the use of the boolean field 'transitiveOverride'. It is not possible to explicitly bind a certaint procedural element AS to just a single procedural element (or a subset thereof). Furthermore, it is not easily possible to specify whether a transitiveOverride shall be applied to all deeply nested ProcedureInvocations or not.

Eventually, it must be stated that the approach to bind different AS to the very same test set/test case/procedural element is a good idea (that should be preserved), but as it is realized now, not usable and excessively complex.

A possible solution to that inflexible and overly complex binding mechanism could be to extract the binding from test set/test case/procedural element into a dedicated concept that allow for expressing these bindings external to the test set/test case/procedural element. This would have two advantages if done properly:

1. The test set/test case/procedural element would remain clean of any AS information and would only represent the information required to eventually execute a test case. Test sets/test cases/procedural elements would be easier to maintain after all.

2. An external binding mechanism would be able to explicitly specify which AS shall be set for which element in a certain context. This holds true even for deeply nested test procedures and procedural elements, which is not the case with the current binding mechanism. A kind of ArbitrationDirective would enable a flexible and unrestricted mechanism to define how the verdict for a given test set/test case/procedural element shall be calculated without imposing certain design decision (like the use of TestExecutionSchedules and ProcedureInvocations) on the test designer. This is in turn another fact that positively impacts maintainability of existing test specifications. Practically, an unlimited number of ArbitrationDirectives could be defined for any test set/test case, which would make the AS mechanism most flexible.

If extracted, there would only be two kinds of binding remain:

• implicit binding: if no additional AS information is specified for a certain test set/test case/procedural element, the default AS would take over (like it is right now)
• explicit binding: (Parts of) the AS bindings for test sets/test cases and procedural elements are explicitly bound and can be put into effect by using a certain ArbitrationDirectve. Additionally, other ArbitrationDirectives can be defined to declare further AS bindings, without the need to change (or even touch), the existing test set/test case/procedural elements.

Arbitration specification bindings will be cleaned up from technical debts

The FTF agree that the mechanism to bind multiple Arbitration Specifications to test cases, test sets and procedural elements is incomplete and blocking implementations. Parts of the specification needs to be cleaned up from techhnical debts.

Furthermore, the FTF agreed that UTP 2 would benefit from a mechanism that offers consistent and scalable way to bind different arbitration specifications for different analysis purposes. Idealy, the elements that shall be arbitrated and the corresponding arbitration specifications shall be loosely coupled by a dedicated and reliable UTP 2 capability for arbitration specification binding.

Adding such a new capability is outside the scope of the FTF. Therefore, the FTF will submit another issue for the upcoming RTF to incorporate that specific functionality.

Since dynamic binding of arbitration specifications is not anymore part of the specification, for it was never finished properly, editorial parts of the specification that deals with the different binding mechanisms must be revised to remove any description about the dynamic arbitration specification binding.

A few mistakes were made during the elaboration of the issue resolutions that led to a situation where editing instructions were fogotten to be stated, however, the changes have been applied to the language. The FTF had created an errata sheet to keep track of these supplementing editing instructions. Due to OMG policies, the changes described by the errata sheet need to be voted on as well. This was requested by the AB reviewer.
Therefore, the following list summarizes incomplete editing instructions of already resolved FTF issues (the errata sheet that was submitted with the first FTF submission is attached to the issues):

• Issue UMLTP2-14
  Summary of the correction: The predefined verdict 'fault' was removed from the UTP 2 Types Library. The corresponding element 'Fault' of the conceptual model (Clause 7.6 Arbitration Specification) was not removed by the resolution. This concept must be removed to keep the conceptual model and the profile specification in synch.

• Issue UMLTP2-24
  Summary of the correction: An obsolete entry 'Detailed Semantics' of 8.7.1.1.2.2 ArbitrationSpecification needs to be removed from the specification. Additionally, the Figure 8.1 Language Architecture need to reflect the removal of the UTP Arbitration library (old Clause 9.3, which is removed by the resolution of Issue UMLTP2-24)

• Issue UMLTP2-27
  Summary of the correction: The newly introduced stereotype "ArbitrationResult" needs to be listed in the profile summary table in Clause 8.2 Because of the introduction of a new stereotype, the numbering of the sub-clauses in clause 8.7.1.1.2 Stereotype Specifications needed to be updated. The new numbers that
  mentioned by the original issue resolution had a typo and were not correct. This is corrected by this errata document.

• Issue UMLTP2-29
  Summary of the correction: The newly introduced attribute 'expectationKind' of Clause 8.5.3.4.4 ExpectResponseAction was omitted by the issue resolution.

Corrections of incomplete editing instructions listed by the FTF errata sheet

Apply corrections for incomplete editing instructions of the following issues:

• Issue UMLTP2-14
  Summary of the correction: The predefined verdict 'fault' was removed from the UTP 2 Types Library. The corresponding element 'Fault' of the conceptual model (Clause 7.6 Arbitration Specification) was not removed by the resolution. This concept must be removed to keep the conceptual model and the profile specification in synch.

• Issue UMLTP2-24
  Summary of the correction: An obsolete entry 'Detailed Semantics' of 8.7.1.1.2.2 ArbitrationSpecification needs to be removed from the specification. Additionally, the Figure 8.1 Language Architecture need to reflect the removal of the UTP Arbitration library (old Clause 9.3, which is removed by the resolution of Issue UMLTP2-24)

• Issue UMLTP2-27
  Summary of the correction: The newly introduced stereotype "ArbitrationResult" needsto be listed in the profile summary table in Clause 8.2. Because of the introduction of a new stereotype, the numbering of the sub-clauses in clause 8.7.1.1.2 Stereotype Specifications needed to be updated. The new numbers that
  mentioned by the original issue resolution had a typo and were not correct. This is corrected by this errata document.

• Issue UMLTP2-29
  Summary of the correction: The newly introduced attribute 'expectationKind' of Clause 8.5.3.4.4 ExpectResponseAction was omitted by the issue resolution.

The stereotype ProcedureInvocation allows fro invoking test-specific procedures such as TestExecutionSchedules, TestCases and TestProcedures, but also simple libraries.

It is currently specializes CompoundProceduralElement, which seems to be erroneous. The invokation of a test-specific procedure is an atomic instruction, and as such, it should specialize AtomicProceduraleElement instead.

The corresponding ProcedureInvocationArbitrationSpecification specializes AtomicProceduralElementArbitrationSpecification, though. So, what ever decision will be made with respect to ProcedureInvocation, the corresponding arbitration specification might require some re-work, too.

Change ProcedureInvocation to specialize AtomicProceduralElement instead

The FTF decided that ProcedureInvocation is not a CompoundProceduralElement, but an AtomicProceduralElement. This is already reflected in the editorial parts of the specification (i.e., the conceptual model and the arbitration specification section of clause 8), but was obviously incorrectly stated in the diagrams of both the conceptual model and the profile specification (maybe as a relic from the initial submission).

This issues will be resolved together with proposal #54 for issue #24

Rules DRTC01 (should be "DRTR01") and DRTC01 have the same identifier.

Rename rule DRTC01 to DRTR01

The rule in clause 7.2.1 has accidently named the same way as a rule in clause 7.3.1.1. This must be fixed simply by renaming the rule.

UTP should better distinguish the origin of the terms in the vocabulary. E.g., there are concepts and definitions, which are solely listed in the conceptual model or the profile implementation vice versa. The UTP vocabulary should indicate where this term originated from.

A potential solution is to add a introductory sentence to section 4 to explain that the terms in section 4 are a summary of the conceptual model of UTP 2 (section 7). Furthermore, improve section 6.2 (Typographical Conventions) with respect to the kids of cross references in the spec (to the definition, to the conceptual model to stereotypes)

Clarify the nature of Clause 4

This issue will be merged with the issue #6 and resolved in the course of the resolution of issue #6 with is done with the proposal #47.

This issue was raised by Manfred Koethe as UTP 2 revised submission reviewer

All OCL constraints were removed from the specification

During the resolution of UMLTP2-16, it was decided to firstly removed the entire OCL code from the specification for it was highly erroneous and written in a rush. This means that also the incorrect references to UML 2.5 will removed in the course of the resolution of issue #16, thus, this issue will be resolved by proposal #UMLTP2-39.

This issue has been raised by Manfred Koethe as UTP 2 revised submission reviewer.

Clarify the nature of Clause 4

The terms and definitions found in clause 4 are a summary of the conceptual model that is defined in clause 7. Examples and further details can be found in the respective sub-section of clause 7.

Therfore, the FTF agreed to add an introduction to clause 4 that helps the reader to understand the nature of clause 4 and refer to clause 7 for further examples and details.

Furhter, the FTF decided that the definitions of test type and test level suffice as a summary, for further examples and details for both definitions are given in clause 7. Clause 6.1 "How to read this document" shall be updated accordingly.

In the course of this discussion, the FTF found out that the definition of test objective can be improved for clarity.

The description of the enumeration ProcedurePhaseKind is not part of the specification.

Incorporate ProcedurePhaseKind into the specification

The FTF decided that is an editorial issue that has to be fixed.

The tag definition specification of a TestObejctive and TestRequirement is currently typed by String. Although this would most likely suffice in many situation, it prevents tools to specify other forms than plain text (e.g., structured text with formal semantics such as ETSI's TPLan). If this tag definition would be of type ValueSpecification, tools/implementers could decide to go for a more capabale representation of a 'specification' such as OpaqueExpressions or Expression. Eventually, plain text is again supported by the use of LiteralString.

Replace String attributes 'specification' with ValueSpecification

As the issue description mentiones, replacing the String-typed attribute 'specification' of TestRequirement and TestObjective with ValueSpecification widens the possibilities of how to express these information.

The ID attribute offered by the TestSet and TestContext should be 0..1 similar to TestConfiguration, TestCase etc.
The reason is that those IDs shall act as traceability element in context of a testing tools landscapes, in particular for synchonization with test management and test reporting tools.
If such a traceability link is not required, models shall not become invalid.

Change multiplicity of attribute ID of TestSet and TestContext to 0..1

As recommended by the issue description, the mutliplicity of the attribute 'ID' should be changed to 0..1.
This proposal incorporates also the resolution of proposal UMLTP2-38. As agreed in that proposal, the tag definitions 'testLevel' and 'testType' of TestContext ought to have an upper bound of *.

The two tag definitions 'testLevel' and 'testType' determine the corresponding test item level and the test item's quality criteria that are targeted by the artifacts of the test context. Currently, this is set to an optional multiplicity of 1, but ought to be 0..*

Change upper bound to 0..*

The issue is correct; it is easily possible that tester want to reuse the test artifacts assembled by the test context for more than just one test level or test type.

In fact, the way it is written in the spec (plural), it seems that it was intended from the very beginning to assign an upper bound of * to both tag definitions.

The resolution of this issue was done in conjunction with proposal UMLTP2-34.

The formal specification of ChoiceOfValues is incomplete. These paragraphs are relics from an approach to provide a precise semantics of data value extensions.

Suggestion: Remove these paragraphs.

Remove incomplete formal specification of ChoiceOfValue

The paragraphs mentioned in the issue descriptions are relics of the attempt to state the semantics of some ValueSpecifications extensions more precise during the revised submission phase. It shall simply be removed from the description.

The last two paragraphs of AnyValue description are incomplete.

Suggest resolution: Remove those two paragraphs

Remove incomplete formal specification of AnyValue

The paragraphs mentioned in the issue descriptions are relics of the attempt to state the semantics of some ValueSpecifications extensions more precise during the revised submission phase. It shall simply be removed from the description.

Most changes specified in the errata to the beta specification (ad/17-06-01) have not yet been included (except for the changes on the cover page). These changes must be included in the final specification.

Resolve errata issues that are still valid

The FTF decided to check each errata issue for validity and to resolve them in case they have not yet been resolved in course of the production of the beta specification.

The following proposal compares the errata (OMG doc# ad/2017-06-01) issues (by number) with the published document with doc# ptc/2017-09-29. In case the errata issue has already been resolved during the production of the Beta specification (ptc/2017-09-29) na action will be taken and the corresponding errata issue will declared as obsolete.

UTP 2 provides the predefined verdict instances none, pass, inconclusive, fail, error. These instances are currenty represented by LiteralStrings of type 'verdict'. Using LiteralStrings is fine for the UTP test logging facility for TestCaseLog and TestSetLog simply refers to ValueSpecifications of type 'verdict'. But in situations where those predefined verdict instances shall be used in a different context e.g. in the context of types of Properties of a StructuredClassifier and InstanceSpecifications thereof, LiteralStrings cannot be really reused for Slot values. Therefore, the LiteralString should be reworked into InstanceSpecifications of type 'verdict' with their specification set to LiteralStrings (of type 'verdict') with name and value set to the 5 predefined verdicts.

Replace LiteralString verdict instances with InstanceSpecification verdict instances

As the issue description indicates the use of LiteralStrings as predefined verdict instances may prevent reuse of these predefined verdict instances in certain situations.
If one of the predefined verdict instances shall be used in a context other than associating test logs with these predefined verdicts, it happens that reuse is not possible because of composite associations between a UML metaclass and ValueSpecification.
For example, if there is a user-defined complex verdict, expressed as StructuredClassifer with Properties and a predefined verdict shall be assigned to one of the Properties in the context of an InstanceSpecification's Slot, it happens that Slot.value is a composite association to ValueSpecification. This means that the Slot physically owns the ValueSpecification, which is not possible then in the context of a predefined (i.e., read-only) model library.
Therefore, the proposal is to use InstanceSpecifications instead to express the predefined verdicts. By doing so, these verdict instances can be reused by means of InstanceValues.
The specification shall not, however, prevent using other ValueSpecifications as verdict instances in case of user-specific test models. It must be clearly stated that the use of InstanceSpecifications is simply the default UTP way to express verdict instances.

The relationship between "8.7.1 Arbitration Specifications" and "9.3 UTP Arbitration" is not clear. Probably, 9.3 is only the (informal?) semantics of 8.7.1?

Natural language description of default arbitration specification suffices for the moment

During the meeting on 19th April, the group agreed to do the following:

• Remove complete clause 9.3 UTP Arbitration from the spec for it represents an informative part of the spec. The classes provided by that library were never meant to be implemented by a vendor; they serve the mere purpose to unambiguously define the semantics of the default arbitration specifications.
• Provide a natrual language decscription of the default arbitration specification as normative part of the stereotype descriptions

Removal of clause 9.3 is the easiest way to solve this issue. Having an executable specification is a nice add-on, especially for tool vendors. But this should be a self-contained clause of mere informative nature. It is therefore heavily recommended and indorsed by the FTF to re-incorporate the precise yet executable semantics of the default arbitration specifications back into the document.

This issues was raised by Manfred Koethe as UTP 2 revised submission reviewer

All OCL constraints were removed from the specification

During the resolution of UMLTP2-16, it was decided to firstly removed the entire OCL code from the specification for it was highly erroneous and written in a rush. This means that also the incorrect references to UTP stereotypes in the OCL constraints were removed, thus, this issue was resolved by proposal UMLTP2-39.

This issue was raised by Manfred Koethe as UTP 2 revised submission reviewer.

Duplicated submission - see UMLTP2-6

This issue is a duplicated submission of UMLTP2-6

Most of the intra-document links (the blue underlined words) do not refer to the correct target. This seems to be caused by the transformation process between the original specification model generated out of the model and the final PDF document published as the beta version of UTP.

Obsolete without manual transformation of generated document

This issue should become obsolete without manual transformation of generated document

The concpetual model (Clause 7) and profile specification (Clause 8) uses the term 'procedure' as parent concept for the test-specific procedures introduced by UTP, which are TestCase, TestExecutionSchedule and TestProcedure.

While the test-specific procedures are reflected by stereotype definitions in Clause 8, the rather abstract concept 'procedure' is not represented as stereotype.

No added value by a stereotype Procedure

Having a stereotype <<Procedure>> would not add any value to the language.
The UML representation of the concept 'Procedure' is given by Behavior. A stereotype <<Procedure>> would not add any additional property. Therefore, the FTF decided to close this issue with no change.

Instead of creating a new stereotype, the FTF decided to introduce an additional derived tag definition into <<ProcedurInvocation>> of type Behavior that retrieves the invoked Behavior from the underlying UML metaclasses.

UTP allows for capturing the most important test execution information for test cases and test sets. Those information are the minimum required information to document and reasoning over coverage of test requirements or test objectives.

In theory and practice, it is possible and required to calculate verdicts by different verdict arbitrations for the very same test case or test set. A test log should only contain the mere information about the exchange of message between the test environment and the test item. Information about comparison or verdict arbitration should not be part of a test log, but of a dedicated ArbitrationResult concept.

The extraction of verdict into a dedicated ArbitrationResult is also necessary to avoid introducing redundant information into the model by copying the test log elements for the mere purpose of verdict association. Currently, it would not introduce much redundancy by copying just a single TestLog element (and the respective stereotype, of courde), but it is expected that UTP will introduce a fine-grained log model in a future version that is capable of recording all relevant information of test executions. If verdict still remained to test log or the test log entries later on, it would be necessary to potentially copy entire (and long) test log information with dozens of actual procedural elements, just for the sake of additional verdict arbitration. This situation would not scale at all for large test projects.

ArbitrationResult is an important concept to avoid redundancy

The concept ArbitrationResult will be integrated into the Profile Specification to avoid redundancy in case the very same test case shall be checked by different ArbitrationSpecifications which may produce different verdicts.
No change of the conceptual model required, for the ArbitrationResult is simply a technical solution to avaoid redundancy, not a conceptual change.
The solution will be based on UML's InstanceSpecification concept, for an ArbitrationResult is actually an instance of the execution of an ArbitrationSpecification.
ArbitrationResult will contain at least the attribute ‘verdict’. Additional information might be added by means of the pre-defined auxiliary library (later on)

Structural aspects other than the predefined ones for the stereotype TestLog can be added by means of a <<TestLogStructure>> Classifier. Currently, UTP restricts a TestLog to be classified by at most one TestLogStructure. But it may easily happen that a TestLog is supposed to capture the information form more than just one <<TestLogStructure>> Classifier. Therefore, the multiplicity of TestLog.instanceOf should be 0..*.

Allow for multiple <<TestLogStructure>> Classifiers

The abstract stereotype <<TestLog>> has a derived association to TestLogStructure. It is derived from the set of <<TestLogStructure>>Classifiers of the underlying InstanceSpecification (see InstanceSpecification.classifier). It should be possible to have more than just one <<TestLogStructure>>Classifier for any concrete <<TestLog>> to remain flexible and extendible for user needs.

This issue will be resolved in the course of resolving issue UMLTP2-27 with proposal UMLTP2-36.

When designing test cases, the formalization of expected responses of the SUT in accordance to previously sent stimuli are of utmost importance for eventual verdict calculation. The expectations of the SUT's behavior can be expressed by at least two sets, the set of expected responses and the set of forbidden elements. In some situations it accelerates the test design if the expected responses are specified (when X is submitted Y is expected), in some situations it is easier to simply specify what is not expected (when X is submitted everything else but not Y is expected). Which set shall rather be used cannot be answered and depends heavily on the context of the test case. These two sets are complemented by the set of responses that can be ignored as a response of a stimulus. Ignoring elements (i.e., excluding them from verdict arbitration) is an important capability in automated testing.

Currently, ExpectResponseAction explicitly defines the set of expected and forbidden elements. The set of ignored elements is implicitly represented by the complement set of the union of expected and forbidden elements.

The current semantics of forbiddenElements says that elements that are forbidden (i.e., that expected to not being sent by the SUT at all) need to be explicitly declared, and if no elements are specified, no elements are forbidden. This definition has a big disadvantage for testing, for it is prone to result in 'false negatives' during test execution. Let's assume that a tester makes a mistake while specifying the set of forbidden elements. Response Y should actually be forbidden, but the tester forgot to explicitly assign Y to the set of forbidden elements. Let's further assume that during test case execution Y occurs in a situation where it ought to not occur. Since it is implicitly (but erroneously) accepted as at least ignored element, the test case would conclude with a 'false negative' verdict. That means that an existing failure of the SUT was not detected as such. 'False negatives' are to be avoided under all circumstances for they provide a wrong assumption about the quality of the SUT.

Therefore, UTP should remove the set of forbidden elements from ExpectResponseAction and replace it by the set of ignored elements. By doing so, a tester would have to explicitly specify what is expected and what shall be ignored. All other elements, i.e., the complement set of the union of expected and ignored elements, are implicitly treated as forbidden elements. This would prevent the creation of implicit 'false negatives', for everything that the tester does not explicitly allow or ignore is considered to be forbidden.

A second alternative could be to enable the specification of all three sets for ExpectResponseAction and to introduce an enumeration ImplicitExpectationKind with literals ImplicitIgnore, ImplicitForbid (default) and ImplicitExpect. This enables the tester to define exactly those sets that are most appropriate for the respective ExpectResponseAction.

The three sets shall be made explicit

Flexibility is of utmost importance for a standard like UTP 2. Since it is not possible to foresee which way is the ideal way to deal with the three different sets, UTP 2 will make the three sets explicit.

In addition, to these three sets, a mechanism will be introduced that determines which set will be implicitly merged (union) with all other received responses. By default, the set of forbidden element will be implicitly extended. This prevents the occurrence of false negative results in case an actually forbidden element was forgotten to be specified as such.

Many Constraints of stereotypes are inconsistent with respect to OCL and natural language.

See section 8.5.1.2.2 TestCase:

• Constraint 'Use of BehavioredClassifier' is inconsistent
• Constraint /* DRTC02: It is necessary that each test execution schedule requires at most one preconditon. */ does not belong to the stereotype test case at all.

This is the case for most of the Constraints in section 8. A thorough check would be required to clean that up.

Remove Constraints and OCL syntax of natural langaue description if necessary

The formalization of natural language constraints into OCL code was rather done in a rush for the revised submission. As a result, many OCL constraints were not correctly written. In these situation, the OCL code shall be removed in the first place.
In some cases, entire Constraints were too restrictive and will be removed.

The predefined verdict fault and error are synonyms. Fault shall be removed. Error is more common in the test automation community to describe situations where some technical exception occured in the test environment.

Remove predefined 'fault' verdict

The predefined verdict 'fault' has no clear meaning to the TF. Therefore, it shall be removed from the specification entirely.

This issue shall be resolved by the resolution of issue UMLTP2-14.

Double-check that the LoginExample copes with the latest version of the spec.

Correct the details Login example in a later RTF

The example is based on an earlier submission (i.e., the initial submission) of the UTP 2. Even though the example is not anymore in synch with the current version of the profile specification, it still helps understanding how to apply the UTP ina dynamic test process.

Due to limited time, it was decided to defer the correction of the Login example to a later RTF.

The last two paragraphs represents an incomplete formal specification of the semantics of ComplementedValue. It should be removed for does not have an added value in the current state.

Remove incomplete formal specification of ComplementedValue

The paragraphs mentioned in the issue descriptions are relics of the attempt to state the semantics of some ValueSpecifications extensions more precise during the revised submission phase. It shall simply be removed from the description.

UTP 2 offers currently rudimentary test log capabilities. It is possible to capture the minimal test execution information for test cases and test sets. These information are sufficient for the sake of coverage calculation or high-level test reporting, however, the details of a test case execution (e.g., the execution of CreateStimulusActions or ExpectedResponseAction) are completely omitted. This wastes a lot of potential UTP could offer for building test automation architectures like post-execution comparison/verdict calculation, incorporation of test execution traces from heterogeneous test tools in order to process the captured information solely on model level (raising the level of abstraction for automated results checking), comprehensibility or high-level failure analysis and test execution comparison (trends etc.).

As high-level test modelling language, UTP ought to provide at least a facility to enable sophisticated activities on fine-grained test execution logs by capturing the execution on procedural element level.

Will be addressed by a later RTF due to timing reasons

During the F2F in Santander, the FTF agreed that this capability is an important aspect for UTP 2. It will be resolved in a subsequent RTF.

The current TestDesignFacility og UTP 2 is based on the stereotypes TestDesignDirective, TestDesignTechnique, TestDesignDirectiveStructure and TestDesignTechniqueStructure. These elements are tightly integrated by means of the Profile's abstract syntax. So far, this is fine for the test design purposes. As such, the offered concepts support the definition of platform-independent instructions how to derive test cases, test data, test sets, test procedures and/or test execution schedules. These instruction can be put into effect by either a manual test designer or translated into the inpurt for a test generator.

UTP 2, however, is capable of describing test automation architectures (ISTQB TAA) or the architecture of automated testware - both terms are considered to be synonyms. In a test automation architecture, further activities need to be specified such as the generation of executable test code, the configuration and automated setup of the test execution environment and tool (setup and teardown of the automated test environment; compilation of test scripts, automated scheduling of test cases in a test set according to a given test execution schedule etc.), the configuration of the logging facility (e.g., depths and structure of logged information), the configuration of the arbitration specification (which arbitration specifications shall be used, linked to test cases, test sets and procedural elements, etc.).

All these configurations are essentials parts of an automatedt testware and can be used in combination or separately from each other (e.g., specifying a specific arbitration specification for verdict calculation on already executed set of test cases in a post-execution manner).

UTP is currently too restrictive and only offers concepts for configuring the test design activities. As a test modelling language designed in particular for test automation, UTP should offer also concepts for the other activities that need to be performed in an test automation solution.

Therfore, UTP should abstract from TestDesignDirective and TestDesignTechnqiue into abstract classes TestArchitectureDirective and TestArchitectureCapability (and the respective structures). That would, one hand, enables user-specific extensions in order to cope with the aforementioned, not yet supported concepts of test script generation, test execution, test logging and verdict arbitration, on the other hand prepare UTP 2 for potential improvements in that area in further revisions of the language.

Wil be addressed by a later RTF due to time reasons

The FTF agreed that this is an important capability of which UTP 2 would benefit from. But it is not a implementation bocking issue. It would also not break backward compatibilitry, therefore, it will be deferred to a later RTF.