The empty sections 8.7.1 Arbitration Specifications and section 8.7.1.1 Arbitration Facility are relicts from UTP 2.1. They should be removed from the editorial contents of the spec. This will cause a renumbering of current section 7.2 to 7.1 and current section 7.3 to 7.2. The renumbering affects all subsection of these two sections transitively.

The arbitration specification binding mechanism that was introduced in UTP 2.2 supersedes the former binding of arbitration specifications directly from the stereotypes. This direct binding is deprecated (since it does not scale) and should be marked as such. It should be kept nonetheless for backward compatibility

The caption says "estSetArbitrationSpecification" but it should be "TestSetArbitrationSpecification".

The icons for the stereotypes should be added to the specification as a non-normative attachment.
Files for Icons would allow the specification to have a default set of common icons associated with the stereotypes, enabling users to use them without creating their own icons. This would also create a de facto style to aid in the adoption and consistency of implementations.

Note that icons should be provided as SVG. There is also a mechanism to include these icons in the XMI, but the SVG should also be a zipped attachment to the public non-normative attachments.

[ES20311903]

the correct link is

https://www.etsi.org/deliver/etsi_es/203100_203199/20311903/01.05.01_60/es_20311903v010501p.pdf

Graphics do not align between the UTP spec and the XMI.

Decision table testing is a major and frequently used black-box test design technique. In system and acceptance testing as well as testing of business processes, decision table testing is used to identify the test conditions within business rules by representing these rules in a tabluar format.

The Decision Modeling Notation (DMN) provides a MOF-based metamodel for modeling decision tables. Within the DMN, decition tables are an isolated part, well suited for being extracted as profile as part of UTP.

Incorporating a decision table profile into UTP, inspired by the DMN MOF-based metamodel, would improve UTP in the area of business rule testing and make UTP more complete regarding specification-oriented test design techniques.

StateCoverage and TransitionCoverage have same ISTQB quote... I am pretty sure that is wrong

In the current spec, non-navigable association ends (i.e., association ends that shall belong to the association) are still listed in the section 'Associations' of the stereotype on the other side.

See for example in section 8.4.2.5 TestConfigurationRole the following entry in the sub-section 'Associations':
: TestConfiguration

The spec should be cleaned up in that regard.

Curently, UTP2 supports capturing actual results of a Test Set or a Test Case execution via its Test Log capability. A TestLog is an instance of a Test Log Structure that shall specify, what exacty is to be logged. However, the current UTP2 specification does not specify, how such a Test Log Structure is to be expressed. Furthermore, when one or more Test Logs have been instantiated (after executing some Test Sets or Test Cases), the tester may whish to extract only those parts of an existing (potentially huge) Test Log that are relevant to investigate a specific problem. This is not supported by the current UTP2 specification.

The current document generation process used to produce the UTP 2 specs is limited with respect to the illustration of property/association end modifiers (i.e., unique, ordered, redefines, subsets, read-only, union). According to UML, those modifiers shall be illustrated after der name and type and multiplicity of a property. The following grammar rule (taken from UML 2.5, Clause 9.54) specifies the textual notation of properties:
<property> ::= [<visibility>] [‘/’] <name> [‘:’ <prop-type>] [‘[‘ <multiplicity-range> ‘]’] [‘=’ <default>] [‘

’]

In the UTP 2 document generation framework the illutration of modifiers was circumvented in a way that is is close to, but different to the notation prescribed by UML. Although, the aligned notation of UTP 2 is okay from a readers point of view, it appears that the modifiers have not been consistenly illustrated. See two examples from the spec:

In section 8.3.2.7.17 TestDesignDirective , TestDesignDirective modifies are displayed before the name of the propery:

: RoleConfiguration [*]

Even though both ways are still good enough to comprehend, from an editorial point of view, one way should be preferred to other.

In the current spec, the abstract syntax diagrams reside in dedictated sub-sections directly underneath the conceptual context section (e.g., 8.7.2. Test Logging) they belong to. Stereotype definitions of ths conceptual context are grouped in a dedicated sub-section 'stereotype definitions'.

In UTP 2.1 RTF turned out that this structure does not support maintainability and modifiability of the spec. Everytime an additional diagram is added to the conceptual context section (maybe due to enhacements, clean-ups, better seperation and illustration of complex diagrams), the entire stereotype definitions need to be renumbered, because the respective diagram sub-sections and the 'stereotype specification' sub-section reside on the same level.

If the abstract syntax diagrams would be sub-sections of a section called 'Abstract Syntax', there would be no need to renumber stereotypes.

Therefore, I propose to make the spec more maintainable and stable by incoporating the following structure for every concepual context section in the spec.

8.7. Test Evaluation
8.7.1 ArbitrationSpecifications
8.7.2 Test Logging
8.7.2.1 Abstract Syntax
8.7.2.2 Stereotype Specfications

Why is the specification for TestObjective a value specification? What is the use case for making it a value specification versus a primitive string and are there any case studies/examples demonstrating the application of the an expression or opaque expression.

Figure 7.2 seems to imply that test requirements are derived from System requirements. Can we clarify the relationship between SysML requirements and UTP test requirements?

Test planning should incorporate schedule and budget. Should it have the equivalent of a Gantt chart? How do we incorporate a time phased approach and get that embedded into the standard? TestExecutionSchedule helps allude to schedule, but the spec would benefit from more thought on this.

Is there a missing stereotype for instances of data ? In the conceptual section there is a mention of actual data pool which would be an instance specification of the data pool or data partition

Given the constraints applied to the <<Alternative>> construct why would we not just simply have it extend the ConditionalNode instead of StructuredActivityNode?

See section 8.5.2.3.1

Same for <<Loop>> should be restricted to loop node metaclass

<<Parallel>> seems to have a contradiction between the description and the constraint "Application in Activities
In an Activity, «Parallel» must only be applied to SequenceNode". We think that this is meant to say a Conditional Node not a sequence node.

We feel that this definitional rule should be applied to the arbitration result not the specification. 

DRAS01 It is necessary that an arbitration specification determines exactly one verdict.

We feel like there should be a relationship defined between a test case and a test objective in which the role is defined as "acceptanceCriteria" (this may be captured by the arbitration specifications and underlying expected response and check property actions) so perhaps a better role definition would be "testCaseObjective" or "goal". 

Another idea is that if we make it best practice to use test objectives to capture initial requirement success criteria then the Arbitration Specification would refine or be a lower level of abstraction in some way.

Why is it that there are definitional rules for limits number of preconditions to AT MOST one, couldn't there be more than one?

There are definitional rules limiting the number of test levels and test types of a test context to AT MOST one,  think of scenarios where a test context may cover more than one type and level (the XMI shows the upper limit value to be * therefore not enforcing this rule).

There are predefined test design techniques in the spec but don't have predefined test design technique structures to be the classifiers of the instances

Believe there is an error in the definition here " 8.6.1.2.4 DataProvider
Description DataProvider: A test component that is able to deliver (i.e., either select and/or generate) data according to a data specification.
The stereotype «DataProvider» is a specialization of stereotype «TestComponent». Such a test component is used to provide a data partition, represented as a Constraint extended by the stereotype «DataPartition» (SHOULD READ classifier extended by <<Data Partition>>), by generating some new data or by selecting some existing data from another data partition or a data pool according to some data specifications (represented as a Constraint extended by the stereotype 
«DataSpecification»)."

Morphing should be between data pool and data partitions for a true structure morphing mapping from one format of data to another, look into other OMG specifications to bridge concepts

DRTD04 contradicts the definition of the constraint

How do we handle that a test item is a part of a particular system? How can we point to a part property of a composite structure? Can we use dot notation to say my test item is not just the type OpticalSensor but the UAV.PayloadCage.OpticalSensor ?

Why does the UTP not specify InstanceSpecifications of test configurations?  Perhaps we handle this through the test log structure to have test log entries that are instances of test configurations.

It seems that this definitional rule should not be defined as a constraint on the test procedure but actually on the TestCase

DRTP04: It is necessary that each test case invokes at least one test procedure as a main procedure invocation.

Definitional rules don't have unique identifiers in the spec. For clarification, it would be good for these not to have repeated numbers.

This was true for v2.1 but have not confirmed that they still exist in the xmi for v2.2

The following elements are found in the XMI file but not defined within the pdf spec:

• OpaqueProceduralElementArbitrationSpecification
• ProceduralElementASBinding
• TestCaseASBinding
• TestSetASBinding

Made the relationship between the arbitration result relationship to the testlogentry item, currently this is a directional relationship but made it a derived property specification for bidirectional linking therefore a tag value for verdict appears in every log entry type. We did this to make metachain queries in Cameo easier for generating a RVTM. 

Does SysmL2 drop the <<TestCase>> stereotype and to UTP2, or will it be maintained.  What’s the expectation between rectifying potential conflicts/confusion in using both in the same model?

Error when attempting to open UTP 2.1 Library and Types Library. The same error message is presented when opening either file

"error on line 3 at column 80: xmlns:StandardProfile: ' http://www.omg.org/spec/UML/20161101/StandardProfile' is not a valid URI"

Will SysML2 <<verify>> and UTP2 <<verifies>> stereotypes become the same eventually? This is a stereotype that seems like it could become the same in both specs if worked between groups. The different stereotypes lead to unnecessary complexity which hurts adoption for users trying to use both specs.

Section 8.3.1.4.5 describes verifies.

The .xmi files for the UTP2 have links that do not work. Even when separately going to the OMG UML site and downloading the StandardProfile.xmi and the PrimitiveTypes.xmi, unable to load the utp2.xmi profile. Tool vendor (Cameo Systems Modeler)--they suggest that the problem is in the references in the profile. In the utp2.xmi import may be getting stuck on the UML.xmi "used project". It will not let me manually select the UML.xmi file as it says "NOT A USED PROJECT!" Many permutations of installation have been attempted. This is preventing effective use of the standard in its application of test to our system model.

The current document generation process used to produce the UTP 2 specs is limited with respect to the illustration of property/association end modifiers (i.e., unique, ordered, redefines, subsets, read-only, union). According to UML, those modifiers shall be illustrated after der name and type and multiplicity of a property. The following grammar rule (taken from UML 2.5, Clause 9.54) specifies the textual notation of properties:
<property> ::= [<visibility>] [‘/’] <name> [‘:’ <prop-type>] [‘[‘ <multiplicity-range> ‘]’] [‘=’ <default>] [‘

’]

In the UTP 2 document generation framework the illutration of modifiers was circumvented in a way that is is close to, but different to the notation prescribed by UML. Although, the aligned notation of UTP 2 is okay from a readers point of view, it appears that the modifiers have not been consistenly illustrated. See two examples from the spec:

In section 8.3.2.7.17 TestDesignDirective , TestDesignDirective modifies are displayed before the name of the propery:

: RoleConfiguration [*]

Even though both ways are still good enough to comprehend, from an editorial point of view, one way should be preferred to other.

Deferred due to lack of time

The RTF agreed that this a valid, but pure editorial issues. As explained in the issue description, the reason for these varying property modifiers lies in the document generation framework. Since all required information is available, though, the RTF agreed to resolve this issue in a future RTF.

In the current spec, the abstract syntax diagrams reside in dedictated sub-sections directly underneath the conceptual context section (e.g., 8.7.2. Test Logging) they belong to. Stereotype definitions of ths conceptual context are grouped in a dedicated sub-section 'stereotype definitions'.

In UTP 2.1 RTF turned out that this structure does not support maintainability and modifiability of the spec. Everytime an additional diagram is added to the conceptual context section (maybe due to enhacements, clean-ups, better seperation and illustration of complex diagrams), the entire stereotype definitions need to be renumbered, because the respective diagram sub-sections and the 'stereotype specification' sub-section reside on the same level.

If the abstract syntax diagrams would be sub-sections of a section called 'Abstract Syntax', there would be no need to renumber stereotypes.

Therefore, I propose to make the spec more maintainable and stable by incoporating the following structure for every concepual context section in the spec.

8.7. Test Evaluation
8.7.1 ArbitrationSpecifications
8.7.2 Test Logging
8.7.2.1 Abstract Syntax
8.7.2.2 Stereotype Specfications

Deferred due to lack of time

The issues requests a pure editorial change for better maintainability of the specifcation. The RTF agreed that the proposed change makes sense, but due to lack of time, the resolution of that issue will be resolved in a future RTF.

In the current spec, non-navigable association ends (i.e., association ends that shall belong to the association) are still listed in the section 'Associations' of the stereotype on the other side.

See for example in section 8.4.2.5 TestConfigurationRole the following entry in the sub-section 'Associations':
: TestConfiguration

The spec should be cleaned up in that regard.

Deferred due to lack of time

The RTF agreed that this is indeed a valid, yet pure editorial issue. The reason for showing non-navigable association ends in the stereotype specifications lies in the document generation framework that is used to produce the specification document. It is, at the moment, not possible to spare non-navigable association ends from the generation process. In the next RTF, it will be resolved by either enhancing the document generation framework in a way so that it can spare non-navigable association ends or by adding an editorial explanation to clause 6.2 Typographical conventions.

Decision table testing is a major and frequently used black-box test design technique. In system and acceptance testing as well as testing of business processes, decision table testing is used to identify the test conditions within business rules by representing these rules in a tabluar format.

The Decision Modeling Notation (DMN) provides a MOF-based metamodel for modeling decision tables. Within the DMN, decition tables are an isolated part, well suited for being extracted as profile as part of UTP.

Incorporating a decision table profile into UTP, inspired by the DMN MOF-based metamodel, would improve UTP in the area of business rule testing and make UTP more complete regarding specification-oriented test design techniques.

Deferred due to lack of time

The RTF agreed that there should be a tighter integration between decision tables and utp2 to perform decision table testing. Due to lack of time, the RTF agreed to defer the resolution of this issue to a subsequence RTF

Curently, UTP2 supports capturing actual results of a Test Set or a Test Case execution via its Test Log capability. A TestLog is an instance of a Test Log Structure that shall specify, what exacty is to be logged. However, the current UTP2 specification does not specify, how such a Test Log Structure is to be expressed. Furthermore, when one or more Test Logs have been instantiated (after executing some Test Sets or Test Cases), the tester may whish to extract only those parts of an existing (potentially huge) Test Log that are relevant to investigate a specific problem. This is not supported by the current UTP2 specification.

Deferred due to lack of time

RTF agreed that this issues is a helpful extension to the language. It will, thus, be addressed in a future revision due to lack of time.

"Figure 8.15 - Test-specific actions Overview" needs to be cleaned up. Stereotype shapes are overlapping.

Replace Figure 8.15 with new figure

RTF agreed to correct this pure editorial issue.

The specification documents (both ptc/23-04-02 and ptc/23-04-03) that were submitted with the UTP 2.2 RTF report (doc#: ptc/23-04-01) contained some issue with editiorial content. Some words that were set in (bold) blue do not represent links, but the typographical convention for these words are not given in the specification. Furthermore, the newly added clauses made not use of those links at all.

Complementary editorial/typographical changes from AB comments

The following resolution and the editing instructions represent complementary editorial corrections of the already submitted specification documents (both ptc/23-04-02 and ptc/23-04-03) because of the Ab review comments recieved after submission of the UTP 2.2 in May 2023 (doc# ptc/23-04-01).
The editing instructions describe only the delta of changes, thus, all the changes reported by the editing instructions of the formert report (doc# ptc/23-04-01) are present in the new specifications documents. In addition, the new submitted specification documents (ptc/24-07-02 and ptc/23-07-03) contain editing instructions as a result of the AB review of the former submission (ptc/23-04-01).

The machine-readable files submitted with UTP 2.2 RTF report (doc#: ptc/23-04-01) contained some syntactical issues that prevented the import of the machine readable files into compliant tools.

Minor editorial changes to the XMI

The machine-readable file for the UTP2 metamodel (utp2.xm, ptc/23-04-06) contained the following issues that prevent import/validity:

Issue in Line 1091
<ownedEnd xmi:type="uml:Property" xmi:id="_1CSfwPAuEeW2VZXfgcC2cA" name="referencedBy" type="_omHdQKJVEeS8gLY2208_PA" isDerived="true"association="_1CR4sPAuEeW2VZXfgcC2cA">

Corrected to:
<ownedEnd xmi:type="uml:Property" xmi:id="_1CSfwPAuEeW2VZXfgcC2cA" name="referencedBy" type="_omHdQKJVEeS8gLY2208_PA" isDerived="true" association="_1CR4sPAuEeW2VZXfgcC2cA">

Issue in line 1231:
<ownedLiteral xmi:type="uml:EnumerationLiteral" xmi:id="_MPcSIFeHEeihnpddrZbjug" name="implicitExpect"i/>

Corrected to:
<ownedLiteral xmi:type="uml:EnumerationLiteral" xmi:id="_MPcSIFeHEeihnpddrZbjug" name="implicitExpect"/>

Issue in Line 1566:
<ownedEnd xmi:type="uml:ExtensionEnd" xmi:id="KAmGUHJKEemyx_XW_ukewg" name="extension_TestTechniqueStructure" type="_90Z0HJJEemyx_XW_ukewg" aggregation="composite"association="_KAc8YHJKEemyx_XW_ukewg"/>

Corrected to:
<ownedEnd xmi:type="uml:ExtensionEnd" xmi:id="KAmGUHJKEemyx_XW_ukewg" name="extension_TestTechniqueStructure" type="_90Z0HJJEemyx_XW_ukewg" aggregation="composite" association="_KAc8YHJKEemyx_XW_ukewg"/>

There is an obvious misplaced change bar marker in the beginning of section "8.6.1 Data Specification". It needs to be removed.

Changebar not present anymore in UTP 2.1 official document

This pure editotial issue has been already fixed with the release of the UTP 2.1 document

UTP 2 defines for each element that may produce a verdict (i.e. test set, test case and all procedural elements), default arbitration specifications that prescribe how to calculate that verdict. The default arbitration specifications are optimized for functional testing, though. In case of testing non-functional aspects such as fault tolerance, performance or security, the rules to determine functional verdicts do usually not suffice anymore. Therfore, UTP 2 enables the definiton of proprietary arbitration specification (mainfested as stereotype <<ArbitrationSpecification>>) to grant the hightest flexibility possible to the user.

However, the current mechanism to associate a test set, test case or procedural element with a dedicated arbitration specification is restricted to only one arbitration specification. Even though this would be sufficient in many situations, the problem with the current binding mechanism is that the test set, test case or procedural element is literally changed by the binding. This may lead to situations where elements are copied only for the sake of associating a different arbitration specification to a that element. This is a situation that should be avoided under all circumstances.

We suggest adding a loosely coupled arbitration specification binding mechanism that keeps the elements and the arbitration specifications that ought to be bound to that element separated from each other. This would bring the benefit that test sets, test cases and procedural elements stay clean of any arbitration specification. This would also foster reusability of these elements.

The loosely coupled mechanism to bind arbitration specifications to the test sets, test cases and procedural elements could be based on a new concept 'ArbitrationDirective' , similar to the 'TestDesignDirective' concept that already exists in the language and proved its applicability. By doing so, users could define and assign an arbitrary number of arbitration specifications to a test set, test case or procedural element without technically modifying any of these elements.

Introducing an arbitration specification binding mechanism

The RTF agreed that the current arbitration facility is not flexile enough and will result in a number of duplicated model elements just for the purpose of assigning different pass/fail criteria. The RTF has developed an arbitration specification binding mechanism that loosely couples arbitration targets with arbitration specifications. Bindings can be overridden in a cascading manner that fosters reuse of existing arbitration specification.

The last paragraph of Clause 8.6.2 Data Values is an unintended left over from the revised submission. In the revised submission, there was the attempt to precisely define the semantics of matching ValueSpecifications. This attempt ought to be entirely removed. Thus, this paragraph should be removed.
For the sake of clarity:

Both stimuli and expected responses yield data values for distinct signature elements. A signature element is defined
as instance of either a Parameter or Property (i.e., this specification introduces a virtual metaclass SignatureElement that is the joint superclass of Property and Parameter and has at least the following attributes: type : UML::Type,
lower : Integer, upper : UnlimitedNatural). Given by UML [UML25], a "... Type specifies a set of allowed values
known as the instances of the Type." This specification denotes this set in the context of a SignatureElement
expressed as type(se), with type(se) as SignatureElement.type, and use T as abbreviation for type(se).
We specify
with se instance of SignatureElement and lower(se) as SignatureElement.lower and denote it by SE type.
A ValueSpecification V as an argument for a SignatureElement is specified as
These basic definitions are further used for the specific ValueSpecification matching mechanism extensions
introduced by UTP.

Remove last paragraph

RTF agreed to remove the last paragraph as it represents a relict from the initial submission.

Figure 9.3 shows a contained package "UTP Types Library" which is a relict from the initial submission. The UTP Types Library is a stand-alone library, technically unrelated to the UTP Auxiliary Library. This is also reflected in the machine readable files. Figure 9.3 needs to be corrected in that regard.

Replace Figure 9.3 with new figure

Task force agreed that this minor, pure editorial issue should be resolved by replacing the erroneous figure with a corrected one

This linebreak can be removed.

Issue not present anymore

The reported issue is not anymore present in the document with number formal/20-08-05.

This is an urgent issue, raised by the AB with respect to the correctness of the XMI files as well as the URI of the XMI files and the correct URIs mentioned on the cover page of the UTP 2.1 specification.

Correct XMI files and URI of machine-readable files for UTP 2.1

As discussed during the AB meeting at the Amsterdam meeting, the XMI file needed some cleanup and also the URI of the machine-readable files need to be corrected on the cover page.

The ID attribute offered by the TestSet and TestContext should be 0..1 similar to TestConfiguration, TestCase etc.
The reason is that those IDs shall act as traceability element in context of a testing tools landscapes, in particular for synchonization with test management and test reporting tools.
If such a traceability link is not required, models shall not become invalid.

Change multiplicity of attribute ID of TestSet and TestContext to 0..1

As recommended by the issue description, the mutliplicity of the attribute 'ID' should be changed to 0..1.
This proposal incorporates also the resolution of proposal UMLTP2-38. As agreed in that proposal, the tag definitions 'testLevel' and 'testType' of TestContext ought to have an upper bound of *.

The two tag definitions 'testLevel' and 'testType' determine the corresponding test item level and the test item's quality criteria that are targeted by the artifacts of the test context. Currently, this is set to an optional multiplicity of 1, but ought to be 0..*

Change upper bound to 0..*

The issue is correct; it is easily possible that tester want to reuse the test artifacts assembled by the test context for more than just one test level or test type.

In fact, the way it is written in the spec (plural), it seems that it was intended from the very beginning to assign an upper bound of * to both tag definitions.

The resolution of this issue was done in conjunction with proposal UMLTP2-34.