${taskforce.name} Avatar
  1. OMG Task Force

UML Testing Profile 2 FTF — All Issues

  • Key: UMLTP2
  • Issues Count: 29
Open Closed All
All Issues

Issues Summary

Key Issue Reported Fixed Disposition Status
UMLTP2-24 Explicit arbitration specification binding does not scale UTP 2.0b1 UTP2 2.0 Resolved closed
UMLTP2-57 Corrections to resolutions summarized by the FTF errata sheet UTP 2.0b1 UTP2 2.0 Resolved closed
UMLTP2-46 ProcedureInvocation should specialize AtomicProceduralElement UTP 2.0b1 UTP2 2.0 Duplicate or Merged closed
UMLTP2-12 Non-unique Rule Identifier UTP 2.0b1 UTP2 2.0 Resolved closed
UMLTP2-11 Clarification of vocabularies UTP 2.0b1 UTP2 2.0 Duplicate or Merged closed
UMLTP2-8 Some references to "UML 2.5" in the constraints in the Profile Specification need to be cleaned-up. UTP 2.0b1 UTP2 2.0 Duplicate or Merged closed
UMLTP2-6 The definitions of test level and test type are useless/not understandable. UTP 2.0b1 UTP2 2.0 Resolved closed
UMLTP2-53 Enumeration ProcedurePhaseKind is not listed in the specification UTP 2.0b1 UTP2 2.0 Resolved closed
UMLTP2-30 TestObejctive.specification and TestRequirement.specification should be ValueSpecifications UTP 2.0b1 UTP2 2.0 Resolved closed
UMLTP2-33 Multiplicity of ID for TestSet and TestContext should be 0..1 UTP2 1.0b1 UTP2 2.0 Resolved closed
UMLTP2-37 TestContext.testLevel and TestContext.testType ought to allow for an upper bound of * UTP2 1.0b1 UTP2 2.0 Duplicate or Merged closed
UMLTP2-3 Formal specification of ChoiceOfValues incomplete UTP 2.0a1 UTP2 2.0 Resolved closed
UMLTP2-2 Clean-up description of AnyValue UTP 2.0a1 UTP2 2.0 Resolved closed
UMLTP2-1 Include changes specified by errata UTP 2.0b1 UTP2 2.0 Resolved closed
UMLTP2-14 Predefined verdicts are not always reusable UTP 2.0b1 UTP2 2.0 Resolved closed
UMLTP2-13 Arbitration Specification Semantics UTP 2.0b1 UTP2 2.0 Duplicate or Merged closed
UMLTP2-9 Some references to "UTP" in the description in the Profile Specification need to be cleaned-up. UTP 2.0b1 UTP2 2.0 Duplicate or Merged closed
UMLTP2-7 The definitions of test level and test type are useless/not understandable. UTP 2.0b1 UTP2 2.0 Duplicate or Merged closed
UMLTP2-4 Invalid links UTP 2.0b1 UTP2 2.0 Closed; No Change closed
UMLTP2-25 A stereotype Procedure should be incorporated as superclass of TestCase, TestExecutionSchedule and TestProcedure UTP 2.0b1 UTP2 2.0 Duplicate or Merged closed
UMLTP2-27 TestLog.verdict should be extracted into a dedicated ArbitrationResult concept UTP 2.0b1 UTP2 2.0 Resolved closed
UMLTP2-28 TestLog.instanceOf should allow for many TestLogStructures UTP 2.0b1 UTP2 2.0 Duplicate or Merged closed
UMLTP2-29 Clarification of the sets expectedElement, forbiddenElement and ignoredElement in ExpectResponseAction UTP 2.0b1 UTP2 2.0 Resolved closed
UMLTP2-16 OCL code and natural language description of Constraints inconsistent UTP 2.0b1 UTP2 2.0 Resolved closed
UMLTP2-15 Predefined verdict fault has no meaning UTP 2.0b1 UTP2 2.0 Duplicate or Merged closed
UMLTP2-10 LoginExample is not in synch with latest Profile Specification UTP 2.0b1 UTP2 2.0 Deferred closed
UMLTP2-5 Formal specification of ComplementedValue incomplete UTP 2.0b1 UTP2 2.0 Resolved closed
UMLTP2-26 UTP test log facility ought to enable logging on procedural element level UTP 2.0b1 UTP2 2.0 Deferred closed
UMLTP2-23 Abstract TestDesignDirective concepts into abstract TestDirective UTP 2.0b1 UTP2 2.0 Deferred closed

Issues Descriptions

Explicit arbitration specification binding does not scale

  • Key: UMLTP2-24
  • Status: closed   Implementation work Blocked
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    UTP offers a concept to bind arbitration specifications (henceforth, the acronym AS is used) to test sets, test cases and procedural elements. The binding comes in three flavors (see page 136):

    • default arbitration specification realized by an implicit binding
    • static arbitration specification realized by a static binding directly by the respective test set, test case or procedural element
    • dynamic arbitration specification (dynamic binding) overrides any previously existing binding in the context of a procedure invocation

    Albeit these three binding kinds makes definitely sense (and should not be removed from the spec) the problem is the technical representation of how bindings shall be expressed, i.e., directly by the test set, test case, procedural element or procedure invocation for dynamic binding. The main issue is that the technical binding mechanism does not scale for large test sets that exhibits exhaustive modularization and reuse of test procedures. In theory, there can be an unlimited number of arbitration specifications set for a test set, test case or procedural element. The classification into static and dynamic binding is a mere synthetic (and unnecessarily complex) concept. In fact, there can only be one static binding being present at any point in time and any further binding must be made via dynamic binding, which in turn requires the use of a <<ProcedureInvocation>> and, thus, a TestExecutionSchedule to actually invoke a test set/test case to dynamically bind the respective AS. This imposes a certain methodology on the user, which is too prescriptive.

    The entire mechanism becomes more complicated when procedural element AS shall be dynamically bound. This can only be achieved by simply listing procedural element AS in the set of 'ProcedureInvocation.dynamicArbitrationSpecification' and the use of the boolean field 'transitiveOverride'. It is not possible to explicitly bind a certaint procedural element AS to just a single procedural element (or a subset thereof). Furthermore, it is not easily possible to specify whether a transitiveOverride shall be applied to all deeply nested ProcedureInvocations or not.

    Eventually, it must be stated that the approach to bind different AS to the very same test set/test case/procedural element is a good idea (that should be preserved), but as it is realized now, not usable and excessively complex.

    A possible solution to that inflexible and overly complex binding mechanism could be to extract the binding from test set/test case/procedural element into a dedicated concept that allow for expressing these bindings external to the test set/test case/procedural element. This would have two advantages if done properly:

    1. The test set/test case/procedural element would remain clean of any AS information and would only represent the information required to eventually execute a test case. Test sets/test cases/procedural elements would be easier to maintain after all.

    2. An external binding mechanism would be able to explicitly specify which AS shall be set for which element in a certain context. This holds true even for deeply nested test procedures and procedural elements, which is not the case with the current binding mechanism. A kind of ArbitrationDirective would enable a flexible and unrestricted mechanism to define how the verdict for a given test set/test case/procedural element shall be calculated without imposing certain design decision (like the use of TestExecutionSchedules and ProcedureInvocations) on the test designer. This is in turn another fact that positively impacts maintainability of existing test specifications. Practically, an unlimited number of ArbitrationDirectives could be defined for any test set/test case, which would make the AS mechanism most flexible.

    If extracted, there would only be two kinds of binding remain:

    • implicit binding: if no additional AS information is specified for a certain test set/test case/procedural element, the default AS would take over (like it is right now)
    • explicit binding: (Parts of) the AS bindings for test sets/test cases and procedural elements are explicitly bound and can be put into effect by using a certain ArbitrationDirectve. Additionally, other ArbitrationDirectives can be defined to declare further AS bindings, without the need to change (or even touch), the existing test set/test case/procedural elements.
  • Reported: UTP 2.0b1 — Wed, 24 Jan 2018 09:58 GMT
  • Disposition: Resolved — UTP2 2.0
  • Disposition Summary:

    Arbitration specification bindings will be cleaned up from technical debts

    The FTF agree that the mechanism to bind multiple Arbitration Specifications to test cases, test sets and procedural elements is incomplete and blocking implementations. Parts of the specification needs to be cleaned up from techhnical debts.

    Furthermore, the FTF agreed that UTP 2 would benefit from a mechanism that offers consistent and scalable way to bind different arbitration specifications for different analysis purposes. Idealy, the elements that shall be arbitrated and the corresponding arbitration specifications shall be loosely coupled by a dedicated and reliable UTP 2 capability for arbitration specification binding.

    Adding such a new capability is outside the scope of the FTF. Therefore, the FTF will submit another issue for the upcoming RTF to incorporate that specific functionality.

    Since dynamic binding of arbitration specifications is not anymore part of the specification, for it was never finished properly, editorial parts of the specification that deals with the different binding mechanisms must be revised to remove any description about the dynamic arbitration specification binding.

  • Updated: Fri, 5 Oct 2018 08:49 GMT
  • Attachments:

Corrections to resolutions summarized by the FTF errata sheet

  • Key: UMLTP2-57
  • Status: closed  
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    A few mistakes were made during the elaboration of the issue resolutions that led to a situation where editing instructions were fogotten to be stated, however, the changes have been applied to the language. The FTF had created an errata sheet to keep track of these supplementing editing instructions. Due to OMG policies, the changes described by the errata sheet need to be voted on as well. This was requested by the AB reviewer.
    Therefore, the following list summarizes incomplete editing instructions of already resolved FTF issues (the errata sheet that was submitted with the first FTF submission is attached to the issues):

    • Issue UMLTP2-14
      Summary of the correction: The predefined verdict 'fault' was removed from the UTP 2 Types Library. The corresponding element 'Fault' of the conceptual model (Clause 7.6 Arbitration Specification) was not removed by the resolution. This concept must be removed to keep the conceptual model and the profile specification in synch.
    • Issue UMLTP2-24
      Summary of the correction: An obsolete entry 'Detailed Semantics' of 8.7.1.1.2.2 ArbitrationSpecification needs to be removed from the specification. Additionally, the Figure 8.1 Language Architecture need to reflect the removal of the UTP Arbitration library (old Clause 9.3, which is removed by the resolution of Issue UMLTP2-24)
    • Issue UMLTP2-27
      Summary of the correction: The newly introduced stereotype "ArbitrationResult" needs to be listed in the profile summary table in Clause 8.2 Because of the introduction of a new stereotype, the numbering of the sub-clauses in clause 8.7.1.1.2 Stereotype Specifications needed to be updated. The new numbers that
      mentioned by the original issue resolution had a typo and were not correct. This is corrected by this errata document.
    • Issue UMLTP2-29
      Summary of the correction: The newly introduced attribute 'expectationKind' of Clause 8.5.3.4.4 ExpectResponseAction was omitted by the issue resolution.
  • Reported: UTP 2.0b1 — Thu, 14 Jun 2018 09:29 GMT
  • Disposition: Resolved — UTP2 2.0
  • Disposition Summary:

    Corrections of incomplete editing instructions listed by the FTF errata sheet

    Apply corrections for incomplete editing instructions of the following issues:

    • Issue UMLTP2-14
      Summary of the correction: The predefined verdict 'fault' was removed from the UTP 2 Types Library. The corresponding element 'Fault' of the conceptual model (Clause 7.6 Arbitration Specification) was not removed by the resolution. This concept must be removed to keep the conceptual model and the profile specification in synch.
    • Issue UMLTP2-24
      Summary of the correction: An obsolete entry 'Detailed Semantics' of 8.7.1.1.2.2 ArbitrationSpecification needs to be removed from the specification. Additionally, the Figure 8.1 Language Architecture need to reflect the removal of the UTP Arbitration library (old Clause 9.3, which is removed by the resolution of Issue UMLTP2-24)
    • Issue UMLTP2-27
      Summary of the correction: The newly introduced stereotype "ArbitrationResult" needsto be listed in the profile summary table in Clause 8.2. Because of the introduction of a new stereotype, the numbering of the sub-clauses in clause 8.7.1.1.2 Stereotype Specifications needed to be updated. The new numbers that
      mentioned by the original issue resolution had a typo and were not correct. This is corrected by this errata document.
    • Issue UMLTP2-29
      Summary of the correction: The newly introduced attribute 'expectationKind' of Clause 8.5.3.4.4 ExpectResponseAction was omitted by the issue resolution.
  • Updated: Wed, 3 Oct 2018 14:16 GMT
  • Attachments:

ProcedureInvocation should specialize AtomicProceduralElement

  • Key: UMLTP2-46
  • Status: closed  
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    The stereotype ProcedureInvocation allows fro invoking test-specific procedures such as TestExecutionSchedules, TestCases and TestProcedures, but also simple libraries.

    It is currently specializes CompoundProceduralElement, which seems to be erroneous. The invokation of a test-specific procedure is an atomic instruction, and as such, it should specialize AtomicProceduraleElement instead.

    The corresponding ProcedureInvocationArbitrationSpecification specializes AtomicProceduralElementArbitrationSpecification, though. So, what ever decision will be made with respect to ProcedureInvocation, the corresponding arbitration specification might require some re-work, too.

  • Reported: UTP 2.0b1 — Tue, 24 Apr 2018 18:11 GMT
  • Disposition: Duplicate or Merged — UTP2 2.0
  • Disposition Summary:

    Change ProcedureInvocation to specialize AtomicProceduralElement instead

    The FTF decided that ProcedureInvocation is not a CompoundProceduralElement, but an AtomicProceduralElement. This is already reflected in the editorial parts of the specification (i.e., the conceptual model and the arbitration specification section of clause 8), but was obviously incorrectly stated in the diagrams of both the conceptual model and the profile specification (maybe as a relic from the initial submission).

    This issues will be resolved together with proposal #54 for issue #24

  • Updated: Wed, 3 Oct 2018 14:16 GMT

Non-unique Rule Identifier

  • Key: UMLTP2-12
  • Status: closed  
  • Source: KnowGravity Inc. ( Mr. Markus Schacher)
  • Summary:

    Rules DRTC01 (should be "DRTR01") and DRTC01 have the same identifier.

  • Reported: UTP 2.0b1 — Tue, 16 Jan 2018 10:32 GMT
  • Disposition: Resolved — UTP2 2.0
  • Disposition Summary:

    Rename rule DRTC01 to DRTR01

    The rule in clause 7.2.1 has accidently named the same way as a rule in clause 7.3.1.1. This must be fixed simply by renaming the rule.

  • Updated: Wed, 3 Oct 2018 14:16 GMT

Clarification of vocabularies

  • Key: UMLTP2-11
  • Status: closed  
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    UTP should better distinguish the origin of the terms in the vocabulary. E.g., there are concepts and definitions, which are solely listed in the conceptual model or the profile implementation vice versa. The UTP vocabulary should indicate where this term originated from.

    A potential solution is to add a introductory sentence to section 4 to explain that the terms in section 4 are a summary of the conceptual model of UTP 2 (section 7). Furthermore, improve section 6.2 (Typographical Conventions) with respect to the kids of cross references in the spec (to the definition, to the conceptual model to stereotypes)

  • Reported: UTP 2.0b1 — Tue, 16 Jan 2018 09:46 GMT
  • Disposition: Duplicate or Merged — UTP2 2.0
  • Disposition Summary:

    Clarify the nature of Clause 4

    This issue will be merged with the issue #6 and resolved in the course of the resolution of issue #6 with is done with the proposal #47.

  • Updated: Wed, 3 Oct 2018 14:16 GMT

Some references to "UML 2.5" in the constraints in the Profile Specification need to be cleaned-up.

  • Key: UMLTP2-8
  • Status: closed  
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    This issue was raised by Manfred Koethe as UTP 2 revised submission reviewer

  • Reported: UTP 2.0b1 — Tue, 16 Jan 2018 09:13 GMT
  • Disposition: Duplicate or Merged — UTP2 2.0
  • Disposition Summary:

    All OCL constraints were removed from the specification

    During the resolution of UMLTP2-16, it was decided to firstly removed the entire OCL code from the specification for it was highly erroneous and written in a rush. This means that also the incorrect references to UML 2.5 will removed in the course of the resolution of issue #16, thus, this issue will be resolved by proposal #UMLTP2-39.

  • Updated: Wed, 3 Oct 2018 14:16 GMT

The definitions of test level and test type are useless/not understandable.

  • Key: UMLTP2-6
  • Status: closed  
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    This issue has been raised by Manfred Koethe as UTP 2 revised submission reviewer.

  • Reported: UTP 2.0b1 — Tue, 16 Jan 2018 09:07 GMT
  • Disposition: Resolved — UTP2 2.0
  • Disposition Summary:

    Clarify the nature of Clause 4

    The terms and definitions found in clause 4 are a summary of the conceptual model that is defined in clause 7. Examples and further details can be found in the respective sub-section of clause 7.

    Therfore, the FTF agreed to add an introduction to clause 4 that helps the reader to understand the nature of clause 4 and refer to clause 7 for further examples and details.

    Furhter, the FTF decided that the definitions of test type and test level suffice as a summary, for further examples and details for both definitions are given in clause 7. Clause 6.1 "How to read this document" shall be updated accordingly.

    In the course of this discussion, the FTF found out that the definition of test objective can be improved for clarity.

  • Updated: Wed, 3 Oct 2018 14:16 GMT

Enumeration ProcedurePhaseKind is not listed in the specification

  • Key: UMLTP2-53
  • Status: closed  
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    The description of the enumeration ProcedurePhaseKind is not part of the specification.

  • Reported: UTP 2.0b1 — Thu, 26 Apr 2018 10:49 GMT
  • Disposition: Resolved — UTP2 2.0
  • Disposition Summary:

    Incorporate ProcedurePhaseKind into the specification

    The FTF decided that is an editorial issue that has to be fixed.

  • Updated: Wed, 3 Oct 2018 14:16 GMT

TestObejctive.specification and TestRequirement.specification should be ValueSpecifications

  • Key: UMLTP2-30
  • Status: closed  
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    The tag definition specification of a TestObejctive and TestRequirement is currently typed by String. Although this would most likely suffice in many situation, it prevents tools to specify other forms than plain text (e.g., structured text with formal semantics such as ETSI's TPLan). If this tag definition would be of type ValueSpecification, tools/implementers could decide to go for a more capabale representation of a 'specification' such as OpaqueExpressions or Expression. Eventually, plain text is again supported by the use of LiteralString.

  • Reported: UTP 2.0b1 — Tue, 23 Jan 2018 10:05 GMT
  • Disposition: Resolved — UTP2 2.0
  • Disposition Summary:

    Replace String attributes 'specification' with ValueSpecification

    As the issue description mentiones, replacing the String-typed attribute 'specification' of TestRequirement and TestObjective with ValueSpecification widens the possibilities of how to express these information.

  • Updated: Wed, 3 Oct 2018 14:16 GMT
  • Attachments:

Multiplicity of ID for TestSet and TestContext should be 0..1

  • Key: UMLTP2-33
  • Status: closed  
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    The ID attribute offered by the TestSet and TestContext should be 0..1 similar to TestConfiguration, TestCase etc.
    The reason is that those IDs shall act as traceability element in context of a testing tools landscapes, in particular for synchonization with test management and test reporting tools.
    If such a traceability link is not required, models shall not become invalid.

  • Reported: UTP2 1.0b1 — Mon, 19 Feb 2018 14:22 GMT
  • Disposition: Resolved — UTP2 2.0
  • Disposition Summary:

    Change multiplicity of attribute ID of TestSet and TestContext to 0..1

    As recommended by the issue description, the mutliplicity of the attribute 'ID' should be changed to 0..1.
    This proposal incorporates also the resolution of proposal UMLTP2-38. As agreed in that proposal, the tag definitions 'testLevel' and 'testType' of TestContext ought to have an upper bound of *.

  • Updated: Wed, 3 Oct 2018 14:16 GMT
  • Attachments:

TestContext.testLevel and TestContext.testType ought to allow for an upper bound of *

  • Key: UMLTP2-37
  • Status: closed  
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    The two tag definitions 'testLevel' and 'testType' determine the corresponding test item level and the test item's quality criteria that are targeted by the artifacts of the test context. Currently, this is set to an optional multiplicity of 1, but ought to be 0..*

  • Reported: UTP2 1.0b1 — Mon, 26 Mar 2018 11:47 GMT
  • Disposition: Duplicate or Merged — UTP2 2.0
  • Disposition Summary:

    Change upper bound to 0..*

    The issue is correct; it is easily possible that tester want to reuse the test artifacts assembled by the test context for more than just one test level or test type.

    In fact, the way it is written in the spec (plural), it seems that it was intended from the very beginning to assign an upper bound of * to both tag definitions.

    The resolution of this issue was done in conjunction with proposal UMLTP2-34.

  • Updated: Wed, 3 Oct 2018 14:16 GMT

Formal specification of ChoiceOfValues incomplete

  • Key: UMLTP2-3
  • Status: closed  
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    The formal specification of ChoiceOfValues is incomplete. These paragraphs are relics from an approach to provide a precise semantics of data value extensions.

    Suggestion: Remove these paragraphs.

  • Reported: UTP 2.0a1 — Mon, 15 Jan 2018 13:42 GMT
  • Disposition: Resolved — UTP2 2.0
  • Disposition Summary:

    Remove incomplete formal specification of ChoiceOfValue

    The paragraphs mentioned in the issue descriptions are relics of the attempt to state the semantics of some ValueSpecifications extensions more precise during the revised submission phase. It shall simply be removed from the description.

  • Updated: Wed, 3 Oct 2018 14:16 GMT
  • Attachments:

Clean-up description of AnyValue

  • Key: UMLTP2-2
  • Status: closed  
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    The last two paragraphs of AnyValue description are incomplete.

    Suggest resolution: Remove those two paragraphs

  • Reported: UTP 2.0a1 — Mon, 15 Jan 2018 13:25 GMT
  • Disposition: Resolved — UTP2 2.0
  • Disposition Summary:

    Remove incomplete formal specification of AnyValue

    The paragraphs mentioned in the issue descriptions are relics of the attempt to state the semantics of some ValueSpecifications extensions more precise during the revised submission phase. It shall simply be removed from the description.

  • Updated: Wed, 3 Oct 2018 14:16 GMT

Include changes specified by errata

  • Key: UMLTP2-1
  • Status: closed  
  • Source: KnowGravity Inc. ( Mr. Markus Schacher)
  • Summary:

    Most changes specified in the errata to the beta specification (ad/17-06-01) have not yet been included (except for the changes on the cover page). These changes must be included in the final specification.

  • Reported: UTP 2.0b1 — Mon, 15 Jan 2018 11:45 GMT
  • Disposition: Resolved — UTP2 2.0
  • Disposition Summary:

    Resolve errata issues that are still valid

    The FTF decided to check each errata issue for validity and to resolve them in case they have not yet been resolved in course of the production of the beta specification.

    The following proposal compares the errata (OMG doc# ad/2017-06-01) issues (by number) with the published document with doc# ptc/2017-09-29. In case the errata issue has already been resolved during the production of the Beta specification (ptc/2017-09-29) na action will be taken and the corresponding errata issue will declared as obsolete.

  • Updated: Wed, 3 Oct 2018 14:16 GMT
  • Attachments:

Predefined verdicts are not always reusable

  • Key: UMLTP2-14
  • Status: closed   Implementation work Blocked
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    UTP 2 provides the predefined verdict instances none, pass, inconclusive, fail, error. These instances are currenty represented by LiteralStrings of type 'verdict'. Using LiteralStrings is fine for the UTP test logging facility for TestCaseLog and TestSetLog simply refers to ValueSpecifications of type 'verdict'. But in situations where those predefined verdict instances shall be used in a different context e.g. in the context of types of Properties of a StructuredClassifier and InstanceSpecifications thereof, LiteralStrings cannot be really reused for Slot values. Therefore, the LiteralString should be reworked into InstanceSpecifications of type 'verdict' with their specification set to LiteralStrings (of type 'verdict') with name and value set to the 5 predefined verdicts.

  • Reported: UTP 2.0b1 — Tue, 16 Jan 2018 13:10 GMT
  • Disposition: Resolved — UTP2 2.0
  • Disposition Summary:

    Replace LiteralString verdict instances with InstanceSpecification verdict instances

    As the issue description indicates the use of LiteralStrings as predefined verdict instances may prevent reuse of these predefined verdict instances in certain situations.
    If one of the predefined verdict instances shall be used in a context other than associating test logs with these predefined verdicts, it happens that reuse is not possible because of composite associations between a UML metaclass and ValueSpecification.
    For example, if there is a user-defined complex verdict, expressed as StructuredClassifer with Properties and a predefined verdict shall be assigned to one of the Properties in the context of an InstanceSpecification's Slot, it happens that Slot.value is a composite association to ValueSpecification. This means that the Slot physically owns the ValueSpecification, which is not possible then in the context of a predefined (i.e., read-only) model library.
    Therefore, the proposal is to use InstanceSpecifications instead to express the predefined verdicts. By doing so, these verdict instances can be reused by means of InstanceValues.
    The specification shall not, however, prevent using other ValueSpecifications as verdict instances in case of user-specific test models. It must be clearly stated that the use of InstanceSpecifications is simply the default UTP way to express verdict instances.

  • Updated: Wed, 3 Oct 2018 14:16 GMT
  • Attachments:

Arbitration Specification Semantics

  • Key: UMLTP2-13
  • Status: closed  
  • Source: KnowGravity Inc. ( Mr. Markus Schacher)
  • Summary:

    The relationship between "8.7.1 Arbitration Specifications" and "9.3 UTP Arbitration" is not clear. Probably, 9.3 is only the (informal?) semantics of 8.7.1?

  • Reported: UTP 2.0b1 — Tue, 16 Jan 2018 12:52 GMT
  • Disposition: Duplicate or Merged — UTP2 2.0
  • Disposition Summary:

    Natural language description of default arbitration specification suffices for the moment

    During the meeting on 19th April, the group agreed to do the following:

    • Remove complete clause 9.3 UTP Arbitration from the spec for it represents an informative part of the spec. The classes provided by that library were never meant to be implemented by a vendor; they serve the mere purpose to unambiguously define the semantics of the default arbitration specifications.
    • Provide a natrual language decscription of the default arbitration specification as normative part of the stereotype descriptions

    Removal of clause 9.3 is the easiest way to solve this issue. Having an executable specification is a nice add-on, especially for tool vendors. But this should be a self-contained clause of mere informative nature. It is therefore heavily recommended and indorsed by the FTF to re-incorporate the precise yet executable semantics of the default arbitration specifications back into the document.

  • Updated: Wed, 3 Oct 2018 14:16 GMT

Some references to "UTP" in the description in the Profile Specification need to be cleaned-up.

  • Key: UMLTP2-9
  • Status: closed  
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    This issues was raised by Manfred Koethe as UTP 2 revised submission reviewer

  • Reported: UTP 2.0b1 — Tue, 16 Jan 2018 09:14 GMT
  • Disposition: Duplicate or Merged — UTP2 2.0
  • Disposition Summary:

    All OCL constraints were removed from the specification

    During the resolution of UMLTP2-16, it was decided to firstly removed the entire OCL code from the specification for it was highly erroneous and written in a rush. This means that also the incorrect references to UTP stereotypes in the OCL constraints were removed, thus, this issue was resolved by proposal UMLTP2-39.

  • Updated: Wed, 3 Oct 2018 14:16 GMT

The definitions of test level and test type are useless/not understandable.

  • Key: UMLTP2-7
  • Status: closed  
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    This issue was raised by Manfred Koethe as UTP 2 revised submission reviewer.

  • Reported: UTP 2.0b1 — Tue, 16 Jan 2018 09:08 GMT
  • Disposition: Duplicate or Merged — UTP2 2.0
  • Disposition Summary:

    Duplicated submission - see UMLTP2-6

    This issue is a duplicated submission of UMLTP2-6

  • Updated: Wed, 3 Oct 2018 14:16 GMT

Invalid links

  • Key: UMLTP2-4
  • Status: closed  
  • Source: KnowGravity Inc. ( Mr. Markus Schacher)
  • Summary:

    Most of the intra-document links (the blue underlined words) do not refer to the correct target. This seems to be caused by the transformation process between the original specification model generated out of the model and the final PDF document published as the beta version of UTP.

  • Reported: UTP 2.0b1 — Mon, 15 Jan 2018 14:03 GMT
  • Disposition: Closed; No Change — UTP2 2.0
  • Disposition Summary:

    Obsolete without manual transformation of generated document

    This issue should become obsolete without manual transformation of generated document

  • Updated: Wed, 3 Oct 2018 14:16 GMT

A stereotype Procedure should be incorporated as superclass of TestCase, TestExecutionSchedule and TestProcedure

  • Key: UMLTP2-25
  • Status: closed  
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    The concpetual model (Clause 7) and profile specification (Clause 8) uses the term 'procedure' as parent concept for the test-specific procedures introduced by UTP, which are TestCase, TestExecutionSchedule and TestProcedure.

    While the test-specific procedures are reflected by stereotype definitions in Clause 8, the rather abstract concept 'procedure' is not represented as stereotype.

  • Reported: UTP 2.0b1 — Wed, 24 Jan 2018 08:36 GMT
  • Disposition: Duplicate or Merged — UTP2 2.0
  • Disposition Summary:

    No added value by a stereotype Procedure

    Having a stereotype <<Procedure>> would not add any value to the language.
    The UML representation of the concept 'Procedure' is given by Behavior. A stereotype <<Procedure>> would not add any additional property. Therefore, the FTF decided to close this issue with no change.

    Instead of creating a new stereotype, the FTF decided to introduce an additional derived tag definition into <<ProcedurInvocation>> of type Behavior that retrieves the invoked Behavior from the underlying UML metaclasses.

  • Updated: Wed, 3 Oct 2018 14:16 GMT

TestLog.verdict should be extracted into a dedicated ArbitrationResult concept

  • Key: UMLTP2-27
  • Status: closed   Implementation work Blocked
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    UTP allows for capturing the most important test execution information for test cases and test sets. Those information are the minimum required information to document and reasoning over coverage of test requirements or test objectives.

    In theory and practice, it is possible and required to calculate verdicts by different verdict arbitrations for the very same test case or test set. A test log should only contain the mere information about the exchange of message between the test environment and the test item. Information about comparison or verdict arbitration should not be part of a test log, but of a dedicated ArbitrationResult concept.

    The extraction of verdict into a dedicated ArbitrationResult is also necessary to avoid introducing redundant information into the model by copying the test log elements for the mere purpose of verdict association. Currently, it would not introduce much redundancy by copying just a single TestLog element (and the respective stereotype, of courde), but it is expected that UTP will introduce a fine-grained log model in a future version that is capable of recording all relevant information of test executions. If verdict still remained to test log or the test log entries later on, it would be necessary to potentially copy entire (and long) test log information with dozens of actual procedural elements, just for the sake of additional verdict arbitration. This situation would not scale at all for large test projects.

  • Reported: UTP 2.0b1 — Tue, 23 Jan 2018 20:11 GMT
  • Disposition: Resolved — UTP2 2.0
  • Disposition Summary:

    ArbitrationResult is an important concept to avoid redundancy

    The concept ArbitrationResult will be integrated into the Profile Specification to avoid redundancy in case the very same test case shall be checked by different ArbitrationSpecifications which may produce different verdicts.
    No change of the conceptual model required, for the ArbitrationResult is simply a technical solution to avaoid redundancy, not a conceptual change.
    The solution will be based on UML's InstanceSpecification concept, for an ArbitrationResult is actually an instance of the execution of an ArbitrationSpecification.
    ArbitrationResult will contain at least the attribute ‘verdict’. Additional information might be added by means of the pre-defined auxiliary library (later on)

  • Updated: Wed, 3 Oct 2018 14:16 GMT
  • Attachments:

TestLog.instanceOf should allow for many TestLogStructures

  • Key: UMLTP2-28
  • Status: closed  
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    Structural aspects other than the predefined ones for the stereotype TestLog can be added by means of a <<TestLogStructure>> Classifier. Currently, UTP restricts a TestLog to be classified by at most one TestLogStructure. But it may easily happen that a TestLog is supposed to capture the information form more than just one <<TestLogStructure>> Classifier. Therefore, the multiplicity of TestLog.instanceOf should be 0..*.

  • Reported: UTP 2.0b1 — Tue, 23 Jan 2018 19:56 GMT
  • Disposition: Duplicate or Merged — UTP2 2.0
  • Disposition Summary:

    Allow for multiple <<TestLogStructure>> Classifiers

    The abstract stereotype <<TestLog>> has a derived association to TestLogStructure. It is derived from the set of <<TestLogStructure>>Classifiers of the underlying InstanceSpecification (see InstanceSpecification.classifier). It should be possible to have more than just one <<TestLogStructure>>Classifier for any concrete <<TestLog>> to remain flexible and extendible for user needs.

    This issue will be resolved in the course of resolving issue UMLTP2-27 with proposal UMLTP2-36.

  • Updated: Wed, 3 Oct 2018 14:16 GMT
  • Attachments:

Clarification of the sets expectedElement, forbiddenElement and ignoredElement in ExpectResponseAction

  • Key: UMLTP2-29
  • Status: closed   Implementation work Blocked
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    When designing test cases, the formalization of expected responses of the SUT in accordance to previously sent stimuli are of utmost importance for eventual verdict calculation. The expectations of the SUT's behavior can be expressed by at least two sets, the set of expected responses and the set of forbidden elements. In some situations it accelerates the test design if the expected responses are specified (when X is submitted Y is expected), in some situations it is easier to simply specify what is not expected (when X is submitted everything else but not Y is expected). Which set shall rather be used cannot be answered and depends heavily on the context of the test case. These two sets are complemented by the set of responses that can be ignored as a response of a stimulus. Ignoring elements (i.e., excluding them from verdict arbitration) is an important capability in automated testing.

    Currently, ExpectResponseAction explicitly defines the set of expected and forbidden elements. The set of ignored elements is implicitly represented by the complement set of the union of expected and forbidden elements.

    The current semantics of forbiddenElements says that elements that are forbidden (i.e., that expected to not being sent by the SUT at all) need to be explicitly declared, and if no elements are specified, no elements are forbidden. This definition has a big disadvantage for testing, for it is prone to result in 'false negatives' during test execution. Let's assume that a tester makes a mistake while specifying the set of forbidden elements. Response Y should actually be forbidden, but the tester forgot to explicitly assign Y to the set of forbidden elements. Let's further assume that during test case execution Y occurs in a situation where it ought to not occur. Since it is implicitly (but erroneously) accepted as at least ignored element, the test case would conclude with a 'false negative' verdict. That means that an existing failure of the SUT was not detected as such. 'False negatives' are to be avoided under all circumstances for they provide a wrong assumption about the quality of the SUT.

    Therefore, UTP should remove the set of forbidden elements from ExpectResponseAction and replace it by the set of ignored elements. By doing so, a tester would have to explicitly specify what is expected and what shall be ignored. All other elements, i.e., the complement set of the union of expected and ignored elements, are implicitly treated as forbidden elements. This would prevent the creation of implicit 'false negatives', for everything that the tester does not explicitly allow or ignore is considered to be forbidden.

    A second alternative could be to enable the specification of all three sets for ExpectResponseAction and to introduce an enumeration ImplicitExpectationKind with literals ImplicitIgnore, ImplicitForbid (default) and ImplicitExpect. This enables the tester to define exactly those sets that are most appropriate for the respective ExpectResponseAction.

  • Reported: UTP 2.0b1 — Tue, 23 Jan 2018 10:40 GMT
  • Disposition: Resolved — UTP2 2.0
  • Disposition Summary:

    The three sets shall be made explicit

    Flexibility is of utmost importance for a standard like UTP 2. Since it is not possible to foresee which way is the ideal way to deal with the three different sets, UTP 2 will make the three sets explicit.

    In addition, to these three sets, a mechanism will be introduced that determines which set will be implicitly merged (union) with all other received responses. By default, the set of forbidden element will be implicitly extended. This prevents the occurrence of false negative results in case an actually forbidden element was forgotten to be specified as such.

  • Updated: Wed, 3 Oct 2018 14:16 GMT
  • Attachments:

OCL code and natural language description of Constraints inconsistent

  • Key: UMLTP2-16
  • Status: closed   Implementation work Blocked
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    Many Constraints of stereotypes are inconsistent with respect to OCL and natural language.

    See section 8.5.1.2.2 TestCase:

    • Constraint 'Use of BehavioredClassifier' is inconsistent
    • Constraint /* DRTC02: It is necessary that each test execution schedule requires at most one preconditon. */ does not belong to the stereotype test case at all.

    This is the case for most of the Constraints in section 8. A thorough check would be required to clean that up.

  • Reported: UTP 2.0b1 — Tue, 16 Jan 2018 15:35 GMT
  • Disposition: Resolved — UTP2 2.0
  • Disposition Summary:

    Remove Constraints and OCL syntax of natural langaue description if necessary

    The formalization of natural language constraints into OCL code was rather done in a rush for the revised submission. As a result, many OCL constraints were not correctly written. In these situation, the OCL code shall be removed in the first place.
    In some cases, entire Constraints were too restrictive and will be removed.

  • Updated: Wed, 3 Oct 2018 14:16 GMT

Predefined verdict fault has no meaning

  • Key: UMLTP2-15
  • Status: closed   Implementation work Blocked
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    The predefined verdict fault and error are synonyms. Fault shall be removed. Error is more common in the test automation community to describe situations where some technical exception occured in the test environment.

  • Reported: UTP 2.0b1 — Tue, 16 Jan 2018 13:13 GMT
  • Disposition: Duplicate or Merged — UTP2 2.0
  • Disposition Summary:

    Remove predefined 'fault' verdict

    The predefined verdict 'fault' has no clear meaning to the TF. Therefore, it shall be removed from the specification entirely.

    This issue shall be resolved by the resolution of issue UMLTP2-14.

  • Updated: Wed, 3 Oct 2018 14:16 GMT

LoginExample is not in synch with latest Profile Specification

  • Key: UMLTP2-10
  • Status: closed  
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    Double-check that the LoginExample copes with the latest version of the spec.

  • Reported: UTP 2.0b1 — Tue, 16 Jan 2018 09:17 GMT
  • Disposition: Deferred — UTP2 2.0
  • Disposition Summary:

    Correct the details Login example in a later RTF

    The example is based on an earlier submission (i.e., the initial submission) of the UTP 2. Even though the example is not anymore in synch with the current version of the profile specification, it still helps understanding how to apply the UTP ina dynamic test process.

    Due to limited time, it was decided to defer the correction of the Login example to a later RTF.

  • Updated: Wed, 3 Oct 2018 14:16 GMT

Formal specification of ComplementedValue incomplete

  • Key: UMLTP2-5
  • Status: closed  
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    The last two paragraphs represents an incomplete formal specification of the semantics of ComplementedValue. It should be removed for does not have an added value in the current state.

  • Reported: UTP 2.0b1 — Mon, 15 Jan 2018 15:50 GMT
  • Disposition: Resolved — UTP2 2.0
  • Disposition Summary:

    Remove incomplete formal specification of ComplementedValue

    The paragraphs mentioned in the issue descriptions are relics of the attempt to state the semantics of some ValueSpecifications extensions more precise during the revised submission phase. It shall simply be removed from the description.

  • Updated: Wed, 3 Oct 2018 14:16 GMT
  • Attachments:

UTP test log facility ought to enable logging on procedural element level

  • Key: UMLTP2-26
  • Status: closed  
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    UTP 2 offers currently rudimentary test log capabilities. It is possible to capture the minimal test execution information for test cases and test sets. These information are sufficient for the sake of coverage calculation or high-level test reporting, however, the details of a test case execution (e.g., the execution of CreateStimulusActions or ExpectedResponseAction) are completely omitted. This wastes a lot of potential UTP could offer for building test automation architectures like post-execution comparison/verdict calculation, incorporation of test execution traces from heterogeneous test tools in order to process the captured information solely on model level (raising the level of abstraction for automated results checking), comprehensibility or high-level failure analysis and test execution comparison (trends etc.).

    As high-level test modelling language, UTP ought to provide at least a facility to enable sophisticated activities on fine-grained test execution logs by capturing the execution on procedural element level.

  • Reported: UTP 2.0b1 — Tue, 23 Jan 2018 20:30 GMT
  • Disposition: Deferred — UTP2 2.0
  • Disposition Summary:

    Will be addressed by a later RTF due to timing reasons

    During the F2F in Santander, the FTF agreed that this capability is an important aspect for UTP 2. It will be resolved in a subsequent RTF.

  • Updated: Wed, 3 Oct 2018 14:16 GMT

Abstract TestDesignDirective concepts into abstract TestDirective

  • Key: UMLTP2-23
  • Status: closed  
  • Source: Fraunhofer FOKUS ( Mr. Marc-Florian Wendland)
  • Summary:

    The current TestDesignFacility og UTP 2 is based on the stereotypes TestDesignDirective, TestDesignTechnique, TestDesignDirectiveStructure and TestDesignTechniqueStructure. These elements are tightly integrated by means of the Profile's abstract syntax. So far, this is fine for the test design purposes. As such, the offered concepts support the definition of platform-independent instructions how to derive test cases, test data, test sets, test procedures and/or test execution schedules. These instruction can be put into effect by either a manual test designer or translated into the inpurt for a test generator.

    UTP 2, however, is capable of describing test automation architectures (ISTQB TAA) or the architecture of automated testware - both terms are considered to be synonyms. In a test automation architecture, further activities need to be specified such as the generation of executable test code, the configuration and automated setup of the test execution environment and tool (setup and teardown of the automated test environment; compilation of test scripts, automated scheduling of test cases in a test set according to a given test execution schedule etc.), the configuration of the logging facility (e.g., depths and structure of logged information), the configuration of the arbitration specification (which arbitration specifications shall be used, linked to test cases, test sets and procedural elements, etc.).

    All these configurations are essentials parts of an automatedt testware and can be used in combination or separately from each other (e.g., specifying a specific arbitration specification for verdict calculation on already executed set of test cases in a post-execution manner).

    UTP is currently too restrictive and only offers concepts for configuring the test design activities. As a test modelling language designed in particular for test automation, UTP should offer also concepts for the other activities that need to be performed in an test automation solution.

    Therfore, UTP should abstract from TestDesignDirective and TestDesignTechnqiue into abstract classes TestArchitectureDirective and TestArchitectureCapability (and the respective structures). That would, one hand, enables user-specific extensions in order to cope with the aforementioned, not yet supported concepts of test script generation, test execution, test logging and verdict arbitration, on the other hand prepare UTP 2 for potential improvements in that area in further revisions of the language.

  • Reported: UTP 2.0b1 — Tue, 23 Jan 2018 10:00 GMT
  • Disposition: Deferred — UTP2 2.0
  • Disposition Summary:

    Wil be addressed by a later RTF due to time reasons

    The FTF agreed that this is an important capability of which UTP 2 would benefit from. But it is not a implementation bocking issue. It would also not break backward compatibilitry, therefore, it will be deferred to a later RTF.

  • Updated: Wed, 3 Oct 2018 14:16 GMT
  • Attachments: