1. OMG Mailing List
  2. UML Testing Profile 2.1 Revision Task Force

Open Issues

  • Issues not resolved
  • Name: utp2-rtf
  • Issues Count: 14

Issues Descriptions

UTP test log facility ought to enable logging on procedural element level

  • Key: UMLTP21-3
  • Status: open  
  • Source: Fraunhofer FOKUS ( Marc-Florian Wendland)
  • Summary:

    UTP 2 offers currently rudimentary test log capabilities. It is possible to capture the minimal test execution information for test cases and test sets. These information are sufficient for the sake of coverage calculation or high-level test reporting, however, the details of a test case execution (e.g., the execution of CreateStimulusActions or ExpectedResponseAction) are completely omitted. This wastes a lot of potential UTP could offer for building test automation architectures like post-execution comparison/verdict calculation, incorporation of test execution traces from heterogeneous test tools in order to process the captured information solely on model level (raising the level of abstraction for automated results checking), comprehensibility or high-level failure analysis and test execution comparison (trends etc.).

    As high-level test modelling language, UTP ought to provide at least a facility to enable sophisticated activities on fine-grained test execution logs by capturing the execution on procedural element level.

  • Reported: UTP 2.0b1 — Tue, 23 Jan 2018 20:30 GMT
  • Updated: Sat, 29 Jun 2019 00:02 GMT
  • Attachments:

Correct XMI files and URI of machine-readable files for UTP 2.1

  • Key: UMLTP21-19
  • Status: open  
  • Source: Fraunhofer FOKUS ( Marc-Florian Wendland)
  • Summary:

    This is an urgent issue, raised by the AB with respect to the correctness of the XMI files as well as the URI of the XMI files and the correct URIs mentioned on the cover page of the UTP 2.1 specification.

  • Reported: UTP2 1.0b1 — Thu, 20 Jun 2019 15:11 GMT
  • Updated: Sat, 29 Jun 2019 00:02 GMT

Abstract TestDesignDirective concepts into abstract TestDirective

  • Key: UMLTP21-2
  • Status: open  
  • Source: Fraunhofer FOKUS ( Marc-Florian Wendland)
  • Summary:

    The current TestDesignFacility og UTP 2 is based on the stereotypes TestDesignDirective, TestDesignTechnique, TestDesignDirectiveStructure and TestDesignTechniqueStructure. These elements are tightly integrated by means of the Profile's abstract syntax. So far, this is fine for the test design purposes. As such, the offered concepts support the definition of platform-independent instructions how to derive test cases, test data, test sets, test procedures and/or test execution schedules. These instruction can be put into effect by either a manual test designer or translated into the inpurt for a test generator.

    UTP 2, however, is capable of describing test automation architectures (ISTQB TAA) or the architecture of automated testware - both terms are considered to be synonyms. In a test automation architecture, further activities need to be specified such as the generation of executable test code, the configuration and automated setup of the test execution environment and tool (setup and teardown of the automated test environment; compilation of test scripts, automated scheduling of test cases in a test set according to a given test execution schedule etc.), the configuration of the logging facility (e.g., depths and structure of logged information), the configuration of the arbitration specification (which arbitration specifications shall be used, linked to test cases, test sets and procedural elements, etc.).

    All these configurations are essentials parts of an automatedt testware and can be used in combination or separately from each other (e.g., specifying a specific arbitration specification for verdict calculation on already executed set of test cases in a post-execution manner).

    UTP is currently too restrictive and only offers concepts for configuring the test design activities. As a test modelling language designed in particular for test automation, UTP should offer also concepts for the other activities that need to be performed in an test automation solution.

    Therfore, UTP should abstract from TestDesignDirective and TestDesignTechnqiue into abstract classes TestArchitectureDirective and TestArchitectureCapability (and the respective structures). That would, one hand, enables user-specific extensions in order to cope with the aforementioned, not yet supported concepts of test script generation, test execution, test logging and verdict arbitration, on the other hand prepare UTP 2 for potential improvements in that area in further revisions of the language.

  • Reported: UTP 2.0b1 — Tue, 23 Jan 2018 10:00 GMT
  • Updated: Sat, 29 Jun 2019 00:02 GMT
  • Attachments:

LoginExample is not in synch with latest Profile Specification

  • Key: UMLTP21-1
  • Status: open  
  • Source: Fraunhofer FOKUS ( Marc-Florian Wendland)
  • Summary:

    Double-check that the LoginExample copes with the latest version of the spec.

  • Reported: UTP 2.0b1 — Tue, 16 Jan 2018 09:17 GMT
  • Updated: Sat, 29 Jun 2019 00:02 GMT

Erroneous Figure 9.3 The UTP auxiliary library

  • Key: UMLTP21-20
  • Status: open  
  • Source: Fraunhofer FOKUS ( Marc-Florian Wendland)
  • Summary:

    Figure 9.3 shows a contained package "UTP Types Library" which is a relict from the initial submission. The UTP Types Library is a stand-alone library, technically unrelated to the UTP Auxiliary Library. This is also reflected in the machine readable files. Figure 9.3 needs to be corrected in that regard.

  • Reported: UTP2 2.0 — Thu, 20 Jun 2019 16:34 GMT
  • Updated: Thu, 20 Jun 2019 16:34 GMT

Harmonize illustration of property modifiers in the spec

  • Key: UMLTP21-14
  • Status: open  
  • Source: Fraunhofer FOKUS ( Marc-Florian Wendland)
  • Summary:

    The current document generation process used to produce the UTP 2 specs is limited with respect to the illustration of property/association end modifiers (i.e., unique, ordered, redefines, subsets, read-only, union). According to UML, those modifiers shall be illustrated after der name and type and multiplicity of a property. The following grammar rule (taken from UML 2.5, Clause 9.54) specifies the textual notation of properties:
    <property> ::= [<visibility>] [‘/’] <name> [‘:’ <prop-type>] [‘[‘ <multiplicity-range> ‘]’] [‘=’ <default>] [‘

    {‘<prop-modifier > [‘,’ <prop-modifier >]* ’}

    ’]

    In the UTP 2 document generation framework the illutration of modifiers was circumvented in a way that is is close to, but different to the notation prescribed by UML. Although, the aligned notation of UTP 2 is okay from a readers point of view, it appears that the modifiers have not been consistenly illustrated. See two examples from the spec:

    In section 8.3.2.7.17 TestDesignDirective , TestDesignDirective modifies are displayed before the name of the propery:

    {read-only, union} subDirective : TestDesignDirective [*]

    In section 8.4.2.5 TestConfigurationRole, modifiers are displayed after the name of the property:
    /roleConfiguration {read-only, union}

    : RoleConfiguration [*]

    Even though both ways are still good enough to comprehend, from an editorial point of view, one way should be preferred to other.

  • Reported: UTP2 1.0b1 — Fri, 10 May 2019 10:31 GMT
  • Updated: Thu, 20 Jun 2019 12:17 GMT

Unnecessary linebreak between first and second paragraph of Clause 8.7.1 Arbitration Specifications

  • Key: UMLTP21-18
  • Status: open  
  • Source: Fraunhofer FOKUS ( Marc-Florian Wendland)
  • Summary:

    This linebreak can be removed.

  • Reported: UTP2 1.0b1 — Wed, 19 Jun 2019 13:59 GMT
  • Updated: Wed, 19 Jun 2019 13:59 GMT

Last paragraph of Clause 8.6.2 Data Values needs to be removed

  • Key: UMLTP21-17
  • Status: open  
  • Source: Fraunhofer FOKUS ( Marc-Florian Wendland)
  • Summary:

    The last paragraph of Clause 8.6.2 Data Values is an unintended left over from the revised submission. In the revised submission, there was the attempt to precisely define the semantics of matching ValueSpecifications. This attempt ought to be entirely removed. Thus, this paragraph should be removed.
    For the sake of clarity:

    Both stimuli and expected responses yield data values for distinct signature elements. A signature element is defined
    as instance of either a Parameter or Property (i.e., this specification introduces a virtual metaclass SignatureElement that is the joint superclass of Property and Parameter and has at least the following attributes: type : UML::Type,
    lower : Integer, upper : UnlimitedNatural). Given by UML [UML25], a "... Type specifies a set of allowed values
    known as the instances of the Type." This specification denotes this set in the context of a SignatureElement
    expressed as type(se), with type(se) as SignatureElement.type, and use T as abbreviation for type(se).
    We specify
    with se instance of SignatureElement and lower(se) as SignatureElement.lower and denote it by SE type.
    A ValueSpecification V as an argument for a SignatureElement is specified as
    These basic definitions are further used for the specific ValueSpecification matching mechanism extensions
    introduced by UTP.

  • Reported: UTP2 1.0b1 — Wed, 19 Jun 2019 13:56 GMT
  • Updated: Wed, 19 Jun 2019 13:56 GMT

Change bar in section 8.6.1 Data Specifications need to be removed

  • Key: UMLTP21-16
  • Status: open  
  • Source: Fraunhofer FOKUS ( Marc-Florian Wendland)
  • Summary:

    There is an obvious misplaced change bar marker in the beginning of section "8.6.1 Data Specification". It needs to be removed.

  • Reported: UTP2 1.0b1 — Wed, 19 Jun 2019 13:50 GMT
  • Updated: Wed, 19 Jun 2019 13:50 GMT

Cleanup Figure 8.15 - Test-specific actions Overview

  • Key: UMLTP21-15
  • Status: open  
  • Source: Fraunhofer FOKUS ( Marc-Florian Wendland)
  • Summary:

    "Figure 8.15 - Test-specific actions Overview" needs to be cleaned up. Stereotype shapes are overlapping.

  • Reported: UTP2 1.0b1 — Wed, 19 Jun 2019 13:46 GMT
  • Updated: Wed, 19 Jun 2019 13:46 GMT

Group abstract syntax diagram in separate sub-section for better changeability of the spec

  • Key: UMLTP21-13
  • Status: open  
  • Source: Fraunhofer FOKUS ( Marc-Florian Wendland)
  • Summary:

    In the current spec, the abstract syntax diagrams reside in dedictated sub-sections directly underneath the conceptual context section (e.g., 8.7.2. Test Logging) they belong to. Stereotype definitions of ths conceptual context are grouped in a dedicated sub-section 'stereotype definitions'.

    In UTP 2.1 RTF turned out that this structure does not support maintainability and modifiability of the spec. Everytime an additional diagram is added to the conceptual context section (maybe due to enhacements, clean-ups, better seperation and illustration of complex diagrams), the entire stereotype definitions need to be renumbered, because the respective diagram sub-sections and the 'stereotype specification' sub-section reside on the same level.

    If the abstract syntax diagrams would be sub-sections of a section called 'Abstract Syntax', there would be no need to renumber stereotypes.

    Therefore, I propose to make the spec more maintainable and stable by incoporating the following structure for every concepual context section in the spec.

    8.7. Test Evaluation
    8.7.1 ArbitrationSpecifications
    8.7.2 Test Logging
    8.7.2.1 Abstract Syntax
    8.7.2.2 Stereotype Specfications

  • Reported: UTP2 1.0b1 — Fri, 10 May 2019 10:15 GMT
  • Updated: Fri, 10 May 2019 10:15 GMT

Arbitration specification binding mechanism restricted to only one arbitration specification

  • Key: UMLTP21-4
  • Status: open  
  • Source: Fraunhofer FOKUS ( Marc-Florian Wendland)
  • Summary:

    UTP 2 defines for each element that may produce a verdict (i.e. test set, test case and all procedural elements), default arbitration specifications that prescribe how to calculate that verdict. The default arbitration specifications are optimized for functional testing, though. In case of testing non-functional aspects such as fault tolerance, performance or security, the rules to determine functional verdicts do usually not suffice anymore. Therfore, UTP 2 enables the definiton of proprietary arbitration specification (mainfested as stereotype <<ArbitrationSpecification>>) to grant the hightest flexibility possible to the user.

    However, the current mechanism to associate a test set, test case or procedural element with a dedicated arbitration specification is restricted to only one arbitration specification. Even though this would be sufficient in many situations, the problem with the current binding mechanism is that the test set, test case or procedural element is literally changed by the binding. This may lead to situations where elements are copied only for the sake of associating a different arbitration specification to a that element. This is a situation that should be avoided under all circumstances.

    We suggest adding a loosely coupled arbitration specification binding mechanism that keeps the elements and the arbitration specifications that ought to be bound to that element separated from each other. This would bring the benefit that test sets, test cases and procedural elements stay clean of any arbitration specification. This would also foster reusability of these elements.

    The loosely coupled mechanism to bind arbitration specifications to the test sets, test cases and procedural elements could be based on a new concept 'ArbitrationDirective' , similar to the 'TestDesignDirective' concept that already exists in the language and proved its applicability. By doing so, users could define and assign an arbitrary number of arbitration specifications to a test set, test case or procedural element without technically modifying any of these elements.

  • Reported: UTP2 1.0b1 — Fri, 5 Oct 2018 08:48 GMT
  • Updated: Fri, 10 May 2019 10:06 GMT
  • Attachments:

Non-navigable associations ends shall be removed from the spec

  • Key: UMLTP21-12
  • Status: open  
  • Source: Fraunhofer FOKUS ( Marc-Florian Wendland)
  • Summary:

    In the current spec, non-navigable association ends (i.e., association ends that shall belong to the association) are still listed in the section 'Associations' of the stereotype on the other side.

    See for example in section 8.4.2.5 TestConfigurationRole the following entry in the sub-section 'Associations':
    : TestConfiguration

    The spec should be cleaned up in that regard.

  • Reported: UTP2 2.0 — Fri, 10 May 2019 10:05 GMT
  • Updated: Fri, 10 May 2019 10:05 GMT

Generators for and Queries on Test Logs

  • Key: UMLTP21-5
  • Status: open  
  • Source: KnowGravity Inc. ( Markus Schacher)
  • Summary:

    Curently, UTP2 supports capturing actual results of a Test Set or a Test Case execution via its Test Log capability. A TestLog is an instance of a Test Log Structure that shall specify, what exacty is to be logged. However, the current UTP2 specification does not specify, how such a Test Log Structure is to be expressed. Furthermore, when one or more Test Logs have been instantiated (after executing some Test Sets or Test Cases), the tester may whish to extract only those parts of an existing (potentially huge) Test Log that are relevant to investigate a specific problem. This is not supported by the current UTP2 specification.

  • Reported: UTP2 2.0b2 — Tue, 18 Dec 2018 15:50 GMT
  • Updated: Fri, 18 Jan 2019 13:32 GMT