A master issue for the rework of test architecture package. Several issues will be merged into this issue to improve readability and understandability of the resolutions

Incorporate the issues 15644, 15648 and 15910. The first two issues targets the fact the Scheduler is never part of the test model by definition. Therefore, it represents a pure tooling concept which has nothing in common with a domain-specific profile. The resolution for this is to remove the Scheduler from the profile section and to put it into the annex.
The last issue deals with a semantically necessary precision of a TestContext. Currently, a TestContext is only required to be an instance of uml::StructuredClassifier. Since a TestContext must have the ability to contain operations and classifier behavior, the UTP has to ensure that each instance of a TestContext is both a StructuredClassifier and BehavioredClassifier (e.g. Class, Component…).

The following profile elements do not provide a semantics section:

• SUT
• TestContext
• TestLog
• TestLogApplication
• TestObjective

For a precise and unambiguous interpretation, a description of the semantics is important and should be supplemented.

Due to the new structure of the specification, we figured out that some semantics sections were not defined. The following semantics descriptions will be incorporated to complete the profile.
The semantics section of test context was already changed by resolution of issue 15910.

A standard for attributes supporting review processes out of / with UTP based test design (e.g. author, change history for the TestCase element) is missing in the current version of UTP.

Especially for manual tests and in regulatory domains like Medical IT or Automotive additional test execution information has to be logged (e.g. tester's identification).
A standard is necessary for potential process support coming out of an UTP test model.

There is missing support of test management issues within the current version of UTP in contradiction to the fact, that test frameworks usually are supported by a tool based or manual test management process.
One of the major tasks of test management is to decide, which test cases must be executed. In safety-critical environments, good rationale shall be given if test cases are ommited. To establish corresponding process interfaces from model based test design to test management, the test models have to be enriched with test management features.
Currently, UTP test cases are supposed to be independent of the specific configuration of the SUT. If product variants are tested, specific test cases may be required that only apply to one or several variants (but not all).
This is very important to fulfill basic regulatory affairs, e.g. IEEE 826 defining the content of test plan and test design.
Example 1: A test case attribute "priority" in the model can be used, to include the design of prioritized test sets within the model used for stakeholder review and later for test suite management in the test management tool used.
Example 2: Explicite tags for requirements as model attributes to test cases enable a traceable and efficient requirements based test design with UTP models and can be used as a basis for tool interfaces / synchronization tools between requirements engineering and test design.

Test management aspects become more and more important. UTP should target those aspects as well. Since test management is not in scope of the initial RFP of the UTP, the RTF decided to define a basic set of test management attributes in an additional non-normative annex.
The RTF defined a common set of important attributes which are reusable for a lot of test management aspects (like version, description…). Those information are grouped in a newly created stereotype called ManagedElement. As part of the resolution, an issue will be submitted to the UML, since most of those information are general purpose information and should be included in UML as well.
Additionally, some more test-specific attributes are added directly into already existing concepts to enable them for test management purposes.

Rational:
UTP spec, 1.0, section 6.3.1 TestArchitecture, subsection TestContext, subsubsection Description mentiones the following:
"A structured classifier acting as a grouping mechanism for a set of test cases. The composite structure of a test context is
referred to as test configuration. The classifier behavior of a test context is used for test control."

Issue:
Since a test context extends UML::StructuredClassifier solely, a test context will never have a classifier behavior or operations, representing the test cases of that test context. StructuredClassifiers do not care about behavior but about structure. In order to be able to add a classifier behavior to a test context, it is required, that test context stereotypes only those classifiers that are subclasses of UML::StructuredClassifier AND UML::BehavioredClassifier (context TestContext inv: self.base_StructuredClassifier.oclIsKindOf(StructuredClassifier) and self.base_StructuredClassifier.oclIsKindOf(BehavioredClassifier).

Even if in the current UML specification (haven't checked this thoroughly) all concrete subclasses of StructuredClassifier are subclasses of BehavioredClassifers, too, UTP should formally define this constraint since it is a crucial aspect of test context semantics

No Data Available

Within the descriptional sections, the specification often refers to stereotype names, without showing the typical stereotype brackets («»).
For example:

"A data partition stereotyped classifier..." may be changed to
"A «DataPartition» stereotyped classifier.."

or

"The notation for data partition is a classifier with stereotype data partition."
may be changed to
"The notation for data partition is a classifier with stereotype «DataPartition»."

This should be consequently done along the descriptional sections to increase redeability and consistency, also regarding other UML profile specifications.

The sections need to be harmonized with each other. The contents of the stereotype sections vary from section to section. Sometimes stereotypes are written in upper-case letters (e.g. GetTimezoneAction), sometines in lower-case letters (e.g. default, default application, test component). The same counts for the notation sections of each stereotype. The resolution of this issue aims at aligning the stereotype's section with each other by using the following scheme:
All UML metaclasses are written in camel-case style starting with upper-case letters (e.g. StructuredClassifier)
All UTP concepts are written in lower-case letters (e.g. default application, get timezone action) in the description and semantics section. Whenever a dedidacted UTP concept is referenced, the stereotype style is used (e.g. see «ValidationAction» for further details). In case the concept does not represent a stereotype, the letter-case variant is used (e.g. No additional notation for arbiter defined). - UTP concepts in the examples sections are written in stereotype style («ValidationAction») since the examples section addresses the application of the technical concept, i.e. the stereotype.
UTP concepts in the notation sections are written in stereotype style («ValidationAction») since the notation section addresses the visualization of the technical concept, i.e. the stereotype. When no additional notation is defined for the stereotype, the standardized paragraph “No additional notation for «StereotypeName» defined.” is used.
In the constraint section, the stereotype style («TestContext») is used when the constraints required a particular stereotype to be applied on a UML metaclass (e.g. “a classifier with «TestContext» applied” instead of “a classifier with test context stereotype applied”).

The second constraint of TestLogApplication seems to be wrong. Currently, it says:

"[2] The supplier of a test log must be a named element with a test log stereotype applied"

This must rather be:
"[2] The supplier of a test log application must be a named element with a test log stereotype applied" Rational: A dependency going from a «TestLog» to a «TestCase» or a «TestContext».

Discussion: The direction of <<TestLogApplication>> currently points from a <<TestCase>> to a <<TestLog>>. Given the UML semantics of UML::Dependency, this would be interpreted as the test case is depending on the test log. Actually, this must be switched, since a test case specification is complete and not depending on the test log elements. Rather, a test log is depending on a test case/test context, expressing that this test log was created due to the test case specification it points to.
[1] The client of a test log application must be a named element with a test case or test context stereotype applied.
[2] The supplier of a test log application must be a named element with a test log stereotype applied

A test case should never be logically dependent on its logs. Flip the direction of the test log application as proposed in the summary

In section 6.6.1 Money Example, Figure 6.19 shows a generalization relationship between IMoney and Money/MoneyBag.

The prefix "I" supposes the IMoney classifier to be an interface. If so, the generalization relationship is wrongly used. Instead, this should rather be a InterfaceRealization (dashed arrow).

But if so IMoney classifier is supposed to be a class, the prefix "I" should removed. This would require the class to be renamed, to avoid name clashes with the specialized class "Money".

Agreed, the relationship must be an InterfaceRealization

Chapter 4 “Terms and Definitions” copes with defining a common terminology for the further specification. Its subsections reflects the logical packages of the UTP, hence it should be rather a subsection of chapter 6.

Chapter 6 “The UML Testing Profile” deals with the profiles and its elements exclusively, so adding a subsection for defining a common terminology and providing an overall overview of the subsequently discussed profile and MOF-based metamodel (which are structured by the same logical packages) fits into the intention of chapter 6. It also concentrates the domain-specific knowledge of the profile’s concepts in the core chapter, what may help reader to find and retrieve information on a particular element.

Additionally, chapter 4 must be thoroughly revised regarding its content. Currently, there is no agreement of what is included in the sections. For example, section 4.1 „Test Architecture“ contains all profile element‘s (and additional, logical concepts being no part of the profile) whereas subsection 4.4 „Timer Concepts“ just discuss a subset of the profile element‘s.

The Terms and Definitions section will be merged into just one section, so the distinction between the logical packages of UTP will be removed. To avoid redundancy, whenever a term is represented as a dedicated concept (a stereotype) in the UTP a reference to the more elaborated descriptions and semantics of the stereotype will be inserted. The information of the definition is not lost and was partially merged by the resolution of issue 15772 and 16105.

The specification is ambiguous regarding the usage of its correct name within the descriptional sections. Sometimes, "UML Testing Profile" is mentioned, sometines "Testing Profile" is mentioned.

In order to increase editorial consistency and cohasion this should be consequently changed to either "UML Testing Profile" or "UTP"

Agreed. “UML Testing Profile” shall be consequently used.

The subsection 6.3.1 ­ 6.3.4 define the profiles concepts, semantics and descriptions. Those descriptions are the main section of the whole specification, though it must be ideally free from ambiguities, redundancies and impreciseness.
Before introducing the profile’s elements, an introduction section is given.

Unfortunately, those introductions have been used to describe also detailed descriptions of an element’s objective and semantic. Even if an introduction into the following problem space is needful and helpful for the reader, the way how it was done in UTP is confusing, since the semantical descriptions are scattered over several subsections and sometimes even redundant.

This affects the following elements and their descriptions:

• 6.3.1: Arbiter, Scheduler, SUT
• 6.3.2: Verdict, Default, FinishAction, LogAction, determAlt, TestLog
• 6.3.3: Data Pool, Data Partition, Data Selector, Coding Rules
• 6.3.4: Timer, TimeOut, Timezone

In order to keep the specification clean and easy to read and maintain, it should be precisely identified, what parts of the description should rather be positioned in the element’s subsections and what could remain in the introduction.

The resolution of this issue will be done by a complete editorial review of the profile section, where the introductory sections of UTP concepts will be merged with the stereotype sections. Complete redundant information will be removed.
Furthermore, the stereotype sections will be checked for redundancy as well. If necessary, the redundant information will be either merged or removed

UTP specification, section 6.4.2 Test Behavior, subsection DefaultApplication (p.19) say the following:
"A default application is a dependency used to apply a default behavior to a unit of testing on a test component. That unit of test behavior must be one of the following: Package, Classifier, Behavior, InteractionFragment, State, Region, or Activity."

Additionally, subsubsection Notation says the following:
"The notation for a default is identical to a Comment (i.e., a rectangle with a bent corner). The text in the comment symbol has the following syntax:
default default-identifier [continue | repeat [repetition-count] ]
If nothing is given following the default identifier, continue is assumed. If no repetition-count is given, infinity is assumed."

Description
##########################################
The information, given in the notation section, determine how to proceed if a applied default is executed. Continue means that the process flow will continue after the behavior of the default was executed once. In contrast, repeat indicates the number of re-invocations of the default, in a case, when a previous execution has not matched.
Those information are crucial parts of «DefaultApplication», hence, introducing them just as a remark in the notation section is not enough. Since the continuation information are currently just defined as “textual syntax”, meaning, there are no metamodel concepts for expressing the continuation behavior, it is not clear where (to what property of the extended metaclass “Dependency”) to add these information. Additionally, a parser is required to obtain the information from the text.
To easy both the applicability and understandability those information should be part of «DefaultApplication» as property. By doing so, access to the information is easy to get and to evaluate.

Introduce a repetition attribute that determines the number of repetitions of the default behavior. The default value is unlimited .
For further formalization a new constraint will be introduced, stating the target of the default application must be a behavior marked as <<Default>>.

The structure of the specification is erroneous in a way, that is not clear why some parts, belonging to the same topic, have been extracted into an own subsection:

Issues with section 6.4 and 6.5:
Section 6.4 “MOF-based Metamodel” consists of the subsection 6.4.1 “Test Architecture and Behavior” solely. Except that using a single, numbered subsection is not a good style, the sibling section 6.5 “Test Data” must belong to section 6.4, henceforth referred as subsection 6.4.2 “Test Data”.
Additionally, the subsection 6.5.1 “Time Concepts” is rather a sibling subsection to 6.4.1 and 6.4.2, resulting in 6.4.3 “Time Concepts”, so that the logically more correct structure is like:
6.4 “MOF-based Metamodel”
-6.4.1 Test Architecture and Test Behavior
-6.4.2 Test Data
-6.4.3 Timer Concepts

Issues with section 6.6 and 6.7:
Section 6.6 “Examples” describes a comprehensive usage example of UTP. Since chapter 6 should be dedicated to the definition of the profile’s concepts solely, section 6.6 should be extracted either to a new chapter 7 or to the appendix (as it is often done in other MOF-related specifications).
Section 6.7 “Mapping” should remain as section 6.6, since it definitely belongs to the specification of the profile.

The RTF agreed on that the current outline of the specification is not ideal to read and to maintain. In order to improve both, a new structure for the UTP 1.1 specification was proposed, targeting

• the elimination of redundant sections
• a clear numbering of the sections so that they can be directly and precisely referred
• to shift any accompanying material in the specification from the normative sections into the non-normative sections (Annexes)
• the introduction of a common schema for the meta element descriptions, like it is done in UML, SysML or MARTE, which is used for each meta element

Issue: There is no clarity concerning the official acronym of the profile. In the namespace of the XMI definition file, u2tp is used as shortcuut. However, since UML 2 is established in both research and industry, upcoming profiles are considered to be profiles for UML 2. UTP as typical 3-characters acronym should be sufficient and contemporary.

Since UML 2 is well established today, UML Testing Profile will be acronymed by UTP.
Include the acronym UTP at the cover page in parenthesis after the specification title, like it was done in OCL and UML specifications.
Replace any occurrence of U2TP or u2tp in the specification with UTP or u2tp.

Rational:
UTP specification, Section 6.3.4, Timer Concepts, Subsections TimeOut, TimeOutMessages, TimeOut Action says the following:
—
TimeOut: A TimeOut is generated by a timer when it expires and may trigger an associated behavior. The TimeOut is placed in the input pool of the object owning the timer.

TimeOutMessage: A TimeOutMessage is generated by a timer when it expires. The timeout message is sent to the active class
that owns the timer.

A timeout is enabled when the timer expires. It may trigger an associated activity. The TimeOutAction occurs, when all input conditions for that activity are satisfied (including the TimeOutAction).

No Data Available

Rational: UTP specification 1.0, Section 6.3.4 Time Concepts, Subsection TimOut says the following:
—
A «TimeOut» is generated by a timer when it expires and may trigger an associated behavior (e.g., a transition in a state machine).
—

Issue: It is pretty obvious that this is an error resulting from the fact that UTP 1.0 was finalized before UML 2.0 was finalized. UML::TimeTrigger is not a legal, existing metaclass of the UML specification, but there is a metaclass called UML::TimeEvent that can be used instead.

Extend TimeEvent instead of non-existent metaclass TimeTrigger

Rational: Section 6.3.2 Test Behavior, Subsection FinishAction (Stereotyp description) says the following:
—
A FinishAction is an action that determines that the test component will finish its execution of the test case immediately. This does not mean that the test component is terminated.
—

Issue: Extending the abstract metaclass UML::Action (the root of any defined Action within UML superstructure), a tester has to select a concrete subclass of Action of his flavor. Extending a metaclass in terms of a stereotype is not comparable to the object-oriented concepts of extension (i.e. specialization). As a result, «FinishAction» could be assigned to each concrete action in a model, what is too ambiguous. With respect to test model interchange a more precise semantics and metaclass should be defined. It would be better, even for the understanding, to extend a concrete subclass of UML::Action (e.g. UML::OpaqueAction or similar).

Use UML::OpaqueAction and UML::InvocationAction as base classes for «FinishAction», due to the following reasons:

• UML::OpaqueAction is a concrete subclass of UML::Action, allowing the definition of user-/domain-specific descriptions as well as to add a various number of input and output parameter to and from the action. A user can configure the action to his liking. By using a domain-specific OpaqueAction, the user has the possibility to model the «FinishAction» in a declarative way.
• UML::InvocationAction is the abstract super class of all invocation actions (SendObjectAction, CallBehaviorAction. CallOperationAction, BroadcastSignalAction). The user might want to execute a particular behavior that represents the finish behavior. In order to be as generic as necessary but concrete as possible, UML::InvocationAction is used

Issue: UTP specification 1.0 does not have an ongoing numbering of subsection within the main chapters. This leads to imprecise references of particular subsections. Other specifications of OMG exhibit such an ongoing numbering.

For example, Section 6.3.1 Test Architecture of the UTP spec contains two subsections "Arbiter" on the same hierarchy level, so that it is not clear how references to either of them should be distinguished.

No Data Available

Issue: For better a recognition and placement inside the UML 2 specification family, a UTP logo should be created.

Put the logo on the cover page of UTP as it was done by other OMG UML-related specifications

Rational: A «TestLog » represents the actual trace of an executed test case. It contains the messages and stimuli sent to the SUT as well as the result of the SUT. Additionally, the final verdict of each test component and each explicit log statement is present.

Issue: The name «TestLog» is confusing in a way that only the «LogAction» results seem to be included in the resulting behavior. In fact, it is a detailed and exact trace of all events executed in the test case.

RTF agreed that TestLog is an appropriate name for the concept.
Disposition: Closed

Rational: Section 6.3.2 Test Behavior, Subsection LogAction says the following:
—
The LogAction defines the tester‘s desire to log certain aspects during the execution of a test case.
—

Issue: Since «LogAction» stereotype enhances UML::SendObjectAction, the receiver of the object to be logged must be explicitly mentioned. It is not clear who receives the sent object.

Commonly, a log is sent to the execution environment which accepts and persists the entry. This seems to be a tool concepts, too, comparable to Scheduler semantics. It might make sense to introduce another additional technical interface for logging at test execution system level. «LogAction» might extend uml::OpaqueAction, so that there is no explicit receiver to be specified on model level. Vendors must ensure that logs are transported properly to the log interface.

Actually, the issue does not target a technical error in the specification, since the multiplicity of the target pin of SendObjectAction is not defined and left open. Therefore, it is possible to neglect the receiver or to add one if needed.
However, subsection semantics and constraints are stated more precisely

Rational: Section 6.3.2 Test Behavior, Subsection InteractionOperator says the following:
—
The deterministic alternative is a shorthand for an Alternative where the operands are evaluated in sequence such that it is deterministic which operand is chosen given the value of the guards, regardless of the fact that the guard for more than one operand may be true
—

Issue: utp::InteractionOperator::determAlt is embedded in an UTP enumeration and therefore not applicable to UML models. UML::CombinedFragment refers to UML::InteractionOperator instead, so that it seems as utp::InteractionOperator is an extension to UM::InteractionOperator. Extending metaclasses is restricted by the UML Profile design requirements (UML::Superstructure, Section 18.1.2, Item 1):
—
8. […] UML Profiles should form a metamodel extension mechanism that imposes certain restrictions on how the UML metamodel can be modified. The reference metamodel is considered as a “read only” model, that is extended without changes by profiles. It is therefore forbidden to insert new metaclasses in the UML metaclass hierarchy (i.e., new super-classes for standard UML metaclasses) or to modify the standard UML metaclass definitions (e.g., by adding meta-associations). Such restrictions do not apply in a MOF context where in principle any metamodel can be reworked in any direction.
—

UTP::InteractionOperator::determAlt must be completely redesigned in order to be applicable to UML models.

Change the determAlt enumeration into a «DetermAlt» stereotype, enhancing CombinedFragment instead. The idea is to declare the whole combined fragment as a deterministic alternative. This would address another shortcoming of the current description of determAlt. In contrast to the normal operand selection semantics of UML for alternative combined fragments, the determAlt combined fragment is intended to work slightly different: The decision whether an operand should be entered is resolved by the first occurrence specification of a test component lifeline first. If the actual occurrence specification matches the expected one during execution, than the guard condition is going to be evaluated. The guard condition expresses restrictions on data instances, potentially accompanying a receive message. If the guard is left empty, this is a shortcut for “true”. Only if both conditions are met, the operand will be entered.

Rational: Arbiter is a predefined interface defining operations used for arbitration of tests. Test cases, test contexts, and the runtime system can use realizations of this interface to assign verdicts of tests and to retrieve the current verdict of a test (verdicts are discussed in Section 6.3.2, “Test Behavior,” on page 14)

Issue: Section 6.3.2 Test Behavior, Subsection Verdict mentions the following:
—
The final verdict of a test case is determined by an arbiter. Every test context has an arbiter and the tool vendor will provide a default arbiter if the test context does not explicitly specify one [… ]
—
Section 6.3.1 "TestArchitecture" (page 10) says:
—
Every test context must have an implementation of the arbiter interface, and the tool vendor constructing tools based on
the Testing Profile will provide a default arbiter to be used if one is not explicitly defined in the test context.
—

It seems to be a logical contradiction. Since the multiplictity of a test context's arbiter reference (see Fig. 6.1 TestArchitecture) is exactly one (so it has to be set in any case), why is it necessary to have a default arbiter in addition? It can never be used by a test context. In the case, a test context wants to use a kind of default arbiter, it has to define it anyway.
In order to avoid confusion on the readers, this must be clarified and probably simplified.

No Data Available

Rational: Scheduler is a predefined interface defining operations used for controlling the tests and the test components. None of the operations of the Scheduler are available to the UML specifier.

Issue: Section 6.3.1 Test Architecture, Subsection TestContext, Subsubsection Constraints states the following:
—
A test context must contain exactly one property realizing the Scheduler interface
—
A scheduler represents a pure tool concept and is not visible to any UML element in the test model. Implementing this interface in context of test model is not possible.

No Data Available

Rational: Scheduler is a predefined interface defining operations used for controlling the tests and the test components. None of the operations of the Scheduler are available to the UML specifier.

Issue: The usage of the scheduler is not clear. Scheduler seems to be a pure tool concept. Section 6.3.1 Test Architecture, Subsection Scheduler says the following:

Scheduler is a predefined interface defining operations used for controlling the tests and the test components. None of the operations of the Scheduler are available to the UML specifier. The implementation of the predefined interface will determine the detailed semantics. The implementation must make sure that the scheduler has enough information to keep track of the existence and participation of the test components in every test case. The test context itself will ensure the creation of a scheduler.

Hence, the scheduler cannot be used within the test model as it is not visible to the UML elements. The specification should separate between model elements and interfaces to the test execution system, which can be seen as technical issues.

No Data Available

What is the type/class of the instance specification called "self" in figure 6.25?

As UML says, self always refers to the context classifier of an interaction. Being applied to hwe lifeline (6.29), this implies HWEComponent being the context classifier. However, this issue leads to another one when using “self” notation: The «DefaultApplication» may point from «Defaults» among others to Package. As long as the supplier («Default») is one of Classifier, Behavior, InteractionFragment, State, Region, Activity, there is no problem. By attaching it to Package, the general idea is that such a «Default» will be applied to any test component (what represents a generic declaration and powerful application of «Default») being indirectly contained in that package. When using the “self” notation the type of such lifeline is ambiguous, because a Package never has a context classifier. Implicitly, it is seen as a generic type declaration, representing any test component/test context.

Source: Lockheed Martin, Jon Hagar
Specification: UML Testing Profile
Section: sec 4
Nature: Issue
Severity: med
Summary:

Section 4 defines a series of definition and terms unique to the UTP, but the UTP uses other testing terms within the document, e.g. test, verification, validation, test plan, and other. These do not always have comment usage in industry.

Resolution:

We suggest a common list of test terms be provided either directly or by reference to an ISO or IEEE type of standard

No Data Available

Source: Lockheed Martin, Jon Hagar

Specification: UML Testing Profile
Section: all

Nature: Issue
Severity: med
Summary:

testcase or TestCase
In the UML testing profile the stereotype is «TestCase» in the SysML profile it defines «testCase». If both profiles include it, which one is should be used? In the same model, you do not want one team using one and another using the other.

Resolution:

Coordinate the profiles so there is one Test Case stereotype for both, the SysML profile and UTP, i.e., update UTP to match SysML profile.

Revised Text:
Actions taken:

No Data Available

Change «SUT»
The word system is too overloaded and causes us to think of an entire system. Much of the testing is not done on the entire system but on parts of the system as subsystems, components or units before they are integrated. In fact, there can be multiple «SUT» items defined in a single test context.

Resolution:

Suggest changing it to «IUT» “Item Under Test”.

No Data Available

Spell out all acronyms at first use with the document, e.g. see first page, UML, MOF, and XMI

Resolution of issue 15770 already introduced a new section 5 Symbols and Acronyms with empty content. This resolution fills the section with content. The RTF agreed to include only technical symbols.

UML Testing Profile defines that “the notation for coding rule is identical to a comment in UML”. Can coding rule be displayed as stereotype attribute (tagged value) using notation given by UML 2.0 specification?

Since the UTP spec says (Section 6.3.3 Coding Rule, subsection Description) “Coding Rules are defined outside of the Testing Profile and referred to within Testing Profile specifications.” the coding rule can surely be displayed as ordinary tagged value of the Stereotype. The string does not define the coding rule itself, but rather an identifier for a coding rule that is part of the test execution system.
We propose to change this issue with no change.
Disposition: Closed

<<FinishAction>> stereotype extends Action metaclass. However FinishAction chapter, Notation section specifies how FinishAction should be represented on Activity, Interaction and StateMachine diagrams. In that case we have conflict with UML 2.0 specification: according to UML specification actions cannot be used on Interaction and StateMachine diagrams. Actually for interactions UML 2.0 defines ActionExecutionSpecification that specifies execution of action within the Lifeline. However, then <<FinishAction>> stereotype should extend ActionExecutionSpecification metaclass.

No Data Available

Default chapter, Notation section (p.28): “a default is behavior and has no special notation”. DefaultApplication chapter, Description section (p.28): “a default application is a dependency used to apply a default behavior to a unit of test component”. However, in Notation section of DefaultApplication chapter there is written that default should be represented as comment. The same notation is used in examples on Figures 29 and 31. Since information in Default and DefaultApplication chapters contradicts each other we need clarification what notation should be used for Default and DefaultApplication. Also it is unclear how repeat and continue parameters should be represented. According to UML specification the detailed semantics of behavior is determined by it subtypes. Each subtype (Interaction, Activity, StateMachine) has it own notation. Since Default can also be described on Interaction, Activity or StateMachine the most convenient notation for Default would be behavior subtype notation with <<Default>> stereotype and for DefaultApplication – dependency with stereotype <<DefaultApplication>>.

No Data Available

SysML also defines a stereotype «TestCase» on "Operation" and "Behavior" as well as a stereotype «Verify» between requirements and test cases. This should be reflected in the UTP specification as well.

The RTF decided not to create a hard-copy of the SysML verify concept. Instead, an example will be shown how UTP could be combined with SysML in orde to reuse the verify dependency.
These concepts have been introduced into UTP by the resolution of issue #15931
Disposition: See issue 15931 for disposition

Test Objective
1. In the OMG Testing Profile (OTP) specification a Test Objective is defined both as a stereotype on a dependency, «TestObjective», and as an artifact elsewhere. The «TestObjective» stereotype does not apply to this artifact, just to the dependency.

2. The Test Objective dependency relationship is intended to show the dependency from a Test Context or a Test Case to a model element that contains the textual string stating the objective of the test, the test objective artifact.

Resolution:

Test Objective as a dependency:

a. Change Suggestion 1 - No longer use the stereotype «TestObjective» for this dependency relationship. A dependency relationship stereotype name is typically a verb, like verify, satisfy, refine, and copy. Using a noun phrase like “TestObjectiveon” to name a dependency does not provide the declaration that is needed, i.e. subject verb object, e.g. “BlockA satisfies requirement 1”, or “Use Case A refines requirement 1”.

b. Change Suggestion 2 – Use the verb “Validate” as the name of the stereotype that is associated with a dependency. This stereotyped dependency can be used from a Test Case or Test Context to a Test Objective but, can also be used between a test case and model elements that state objectives/goals/capabilities of the item being validated.

c. Why use validate? – Others were considered such as satisfy, realize and implement.

i. The validation definition used by test is “Validation is the set of activities ensuring and gaining confidence that a system is able to accomplish its intended use, goals and objectives (i.e. meet stakeholder requirements) in the intended operational environment” , or “Did we build the right thing?” These system uses, goals and objectives are represented in the model in the form of use cases, MOEs, TPMs, etc.

ii. There is a clear distinction between verification and validation. Based on this, there will be unique Test Contexts for verification and validation.

iii. Just as we need to show what test cases will be verifying what requirements we will also need to show what test cases will be validating objectives and goals of the system (or the thing being validated). Therefore, this relationship will ultimately be needed just as verify is used today to verify requirements.

iv. This relationship can also be used to show what test cases are validating the objectives stated for a Test Context. It is the same relationship pattern.

Test Objective part 2
1. A Test Objective artifact is a textual statement stating a goal or objective of a test context. The OTP specification doesn’t define the type of model element to use for this artifact.

Resolution:

1. Test Objective artifact suggestion –

a. We have selected a comment stereotyped «TestObjective» to capture the Test Objective artifact for a Test Context. However, other model elements could be used, such as an attribute of the Test Context. The advantage of a comment is that it can be reused across test contexts

Test Objective as a dependency:
a. Change Suggestion 1 - No longer use the stereotype «TestObjective» for this dependency relationship. A dependency relationship stereotype name is typically a verb, like verify, satisfy, refine, and copy. Using a noun phrase like “TestObjectiveon” to name a dependency does not provide the declaration that is needed, i.e. subject verb object, e.g. “BlockA satisfies requirement 1”, or “Use Case A refines requirement 1”.
b. Change Suggestion 2 – Use the verb “Validate” as the name of the stereotype that is associated with a dependency. This stereotyped dependency can be used from a Test Case or Test Context to a Test Objective but, can also be used between a test case and model elements that state objectives/goals/capabilities of the item being validated.
c. Why use validate? – Others were considered such as satisfy, realize and implement.
i. The validation definition used by test is “Validation is the set of activities ensuring and gaining confidence that a system is able to accomplish its intended use, goals and objectives (i.e. meet stakeholder requirements) in the intended operational environment” , or “Did we build the right thing?” These system uses, goals and objectives are represented in the model in the form of use cases, MOEs, TPMs, etc.
ii. There is a clear distinction between verification and validation. Based on this, there will be unique Test Contexts for verification and validation.
iii. Just as we need to show what test cases will be verifying what requirements we will also need to show what test cases will be validating objectives and goals of the system (or the thing being validated). Therefore, this relationship will ultimately be needed just as verify is used today to verify requirements.
iv. This relationship can also be used to show what test cases are validating the objectives stated for a Test Context. It is the same relationship pattern.
Test Objective part 2
1. A Test Objective artifact is a textual statement stating a goal or objective of a test context. The OTP specification doesn’t define the type of model element to use for this artifact.
Resolution:
1. Test Objective artifact suggestion –
a. We have selected a comment stereotyped «TestObjective» to capture the Test Objective artifact for a Test Context. However, other model elements could be used, such as an attribute of the Test Context. The advantage of a comment is that it can be reused across test contexts
Resolution:
The RTF discussed a lot on the test objective issue and we agreed entirely that a dependency as test objective is quite unintuitive, confusing and misleading. We had a long way to go until the RTF agreed on a resolution that suits all and is most accepted in the testing area in the industry. Finally, we decided to change the metaclass of the current «TestObjective» from Dependency to Class and to put tag definitions into the stereotype that are pertinent to test management, test planning and test schedule.
The new «TestObjective» is conceptually compatible with the concept Objective from the Business Motivation Metamodel (BMM), although BMM does not provide a UML profile yet and is therefore not a normative reference for the UML Testing Profile.

Most test execution systems allow the definition of a reason why a particular test cases has failed or passed.

In TTCN-3, for example, the user can attach a string message to the setverdict() operation indicating the reason that describes verdict from a logical and human-readable perspective.

In JUnit the same can be done within the Assert.XXX methods.

Such a human-readable message guides the analysis commonly performed by testers and developers, by providing a logical explanation (logical in the sense of what does the verdict state with regard to the intention of the test case) to the technical verdict.

A message "The system was expected to respond y and when X was submitted" is more obvious and can be understood even by people outside of the testing/development process than a simple FAIL, where the logical reason behind that fail has to be derived from the stimuli and corresponding reponse within the test case execution trace.

To support this, ValidationAction should have an additional attribute 'reason : String [0..1]'

This would just be consequent, since a TestLog in (the extended version) UTP is able to expose the reason why the final verdict has beedn assigned to the test log.

The RTF decided to extend the ValidationAction with a new tag definition reason. That tag definition allows a user to directly log arbitrary information directly while setting a verdict. This is required by relevant industry standards like ISO 29119, but is part of almost any test execution tool. UTP should cope with that concepts, whereas the RTF decided to not predefine how such a reason has to be specified, nor how many reasons might be logged for a ValidationAction, nor what reasons are actually logged as the final verdictReason in TestLog. Having this premise, the RTF agreed on having an optional tag definition in ValidationAction with an unbound upper multiplicity. The type of reason is ValueSpecification, since this allows the highest flexibility to the user. A user can then decide to either log plain messages (LiteralString) or even complex InstanceSpecifications (InstanceValue).

IN both changebarred / no-changebar documents for UTP (i.e., 11-06-10 and 11-06-09) do the following:

Keep:

• [OCL] Object Constraint Language (OCL), version 2.2, formal/2010-02-01, 2010

Replace:

[UMLi] Unified Modeling Language (UML) Specification: Infrastructure, version 2.3, formal/2010-05-03, 2010
• [UMLs] Unified Modeling Language (UML) Specification: Superstructure, version 2.3, formal/2010-05-05, 2010
• [MOF] Meta Object Facility (MOF) Core Specification, version 2.0, formal/2006-01-01, 2010

with corresponding names for:

http://www.omg.org/spec/UML/2.4.1
http://www.omg.org/spec/MOF/2.4.1
http://www.omg.org/spec/XMI/2.4.1

I have to produce the SysML1.3 normative / SMSC-kosher files in XMI anyway, I'll do the same for your profile.
I'll send you XMI2.4.1 profiles for UTP normative & informative files that extend UML2.4.1.

Change the inventory such that the following files are: informative machine-readable artifacts:

http://www.omg.org/cgi-bin/doc?ptc/11-05-15
http://www.omg.org/cgi-bin/doc?ptc/11-05-14

In the inventory, add:

http://www.omg.org/cgi-bin/doc?ptc/xx-yy-zz (the XMI2.4.1 / UML2.4.1 version of 11-05-14 – normative – without test case management
http://www.omg.org/cgi-bin/doc?ptc/xx-yy-zz+1 (the XMI2.4.1 / UML2.4.1 version of 11-05-15 – informative – with test case management

Get the numbers from Juergen. I don't know if you want new numbers for the whole series of UTP or if Juergen can get you 11-05-XX/XX+1 numbers for the two additional artifacts & the new versions of the UTP documents
that say UTP1.1 extends the 2.4.1 series of UML/XMI/MOF.

Finally, add another resolution to explain the changes in your updated report (i.e., 11-06-08)

Discussion:
Not relevant for UTP 1.2 anymore.
Disposition: Closed

The UTP contains a stereotype <<TestObjective>>, connects either a test context of test case to any model element, expressing the purpose/objective of the test case or the test context. However, most standards related to testing define standalone specifications of such a test case's target in forms of test purposes (ETSI) or test requirements (IEEE/ISQTB).

It would be good to have a dedicated stereotype <<TestRequirement>> to express the test case descriptions, similar to system requirements (for example stereotype <<Requirement>> in SysML).

It should have atleast an ID, a name, a description and a priority

Merged with issue #15931
Disposition: see resolution of issue #15931 for disposition

UTP 1.0 provides little support specification of large datasets that are of complex structure. Such data set are typical in test cases for information systems that process complex business objects (e.g. contracts). Furthermore the concepts "DataPool", "DataPartition", and "DataSelector" are based on a programmer's procedural paradigm and are currently not specified proper stereotypes.

The current data package must doubtlessly be thoroughly revised. However, data partitions are quite useful to express equivalence classes for a given base type (currently done with generalizations, but this does not scale). The issue rather points to the inflexible extension or specification capabilities of instance specifications in the UML. Currently, instance specifications cannot participate in a generalization dependency, preventing reusage of partially defines instance specification in order to avoid redundancy. This seems to be an issue for UML, however, since instance specification are more vital for testing pruposes than for architecture design purposes, UTP should address a refinement mechanism for instance specification in the next minor revisions.
Necessary concepts:
Extension of already defined instance specification
Refinement of slots
Overriding of slots
These concepts have been introduced into UTP by the resolution of issue #16878
Disposition: See issue 16878 for disposition

The repetition attribute has not a clear semantics with respect to the statements given in the Default section. Repetition actually represents how often the control flow will jump back to the unit of testing that actually applies the default.

The RTF agreed that this is indeed a not clear description of the semantics. In addition, the default value for the repetition shall be changed to 0, since the default behavior of the control flow should be to continue when an alternative in a default behavior become activated.

Source: Lockheed Martin, Jon Hagar

Specification: UML Testing Profile
Section: tbd section

Nature: Issue
Severity: med
Summary:

Verify Relationship
The verify relationship, a «verify» stereotype dependency, is defined in SysML (section 16.3.2.7) and is used from a test case to a requirement. It can also be used between a test context and a requirement. There is nothing specified in the Testing Profile.

Resolution:

The UTP should state that it uses the «verify» relationship from the SysML specification, and have sections/diagram created to show this.

The UTP should state that it uses the «verify» relationship from the SysML specification, and have sections/diagram created to show this.
Discussion:
The RTF decided not to create a hard-copy of the SysML verify concept. Instead, an example will be shown how UTP could be combined with SysML in orde to reuse the verify dependency.
These concepts have been introduced into UTP by the resolution of issue #15931
Disposition: See issue 15931 for disposition