This task describes how to identify which tests should be executed together.
Disciplines: Test
Purpose
The purpose of this task is to do the following:
Assemble collections of tests to be executed together to capture test logs of value
Facilitate appropriate breadth and depth of test coverage by exercising interesting combinations of tests
Relationships
RolesPrimary Performer:
Tester
Additional Performers:
InputsMandatory:
Build
Test Strategy
Optional:
Development Infrastructure
Implementation Model
Test Suite
Outputs
Test Suite
Process Usage
Test > Verify Test Approach > Implement Test Suite
Test > Improve Test Assets > Implement Test Suite
Test > Test and Evaluate > Implement Test Suite
Steps
Examine Candidate Test Suites
Purpose:
To understand the Test Suites and select which candidates will be implemented
Start by reviewing any existing Test Suite outlines, and determine which Test Suites are good candidates for
implementation at the current time. Use the Iteration Test Plan, Test-Ideas List and any additional test definition
work products as a basis for making your decision.
Examine Related Tests and Target Test Items
Purpose:
To understand the relationships between the planned Tests and the Target Test Items
For each Test Suite you have selected for implementation, identify what Target Test Items and associated Tests are
candidates for inclusion in the scope of the Test Suite.
Identify Test Dependencies
Purpose:
To identify any dependencies the Tests have in terms of other Tests, and in general terms in relation to
system state
Begin by considering the Test Environment Configuration and specific system start state. Consider what specific setup
requirements there will be, such as the starting data set for dependent databases. Where one Target Environment
Configuration will be used for various Test Suites, identify any configuration settings that may need to be managed for
each Test Suite, such as the screen resolution of video displays or the regional operating system settings.
Now determine any specific relationships between the Tests. Look for dependencies where the execution of one Test
included in the Test Suite will result in a system state change required as a precondition of another Test.
Once you've identified the relevant dependencies, determine the correct sequence of execution for the dependent Tests.
Identify Opportunities for Reuse
Purpose:
To improve Test Suite maintainability, both by reusing existing assets and consolidating new assets
One of the main challenges in maintaining a Test Suite-especially an automated one-is ensuring that ongoing changes are
easy to make. It's a good idea when possible and deemed useful to maintain a central point of modification for elements
that are used in multiple places. That's especially true if those same elements are likely to change.
While the Tests themselves form natural units of modularity, assembly of the Tests into a Test Suite often identifies
duplicate procedural elements across multiple Tests that could be more effectively maintained if the were consolidated.
Take the opportunity to identify any general mechanics of the Tests that might potentially be refactored into a
standard routine to assist ongoing maintenance.
Apply Necessary Infrastructure Utilities
Purpose:
To factor out complex implementation detail that is required in support of the test as simplified utility
functions
Most test efforts require the use of one or more "utilities" that generate, gather, diagnose, convert and compare
information used during test execution. These utilities typically simplify both complex and laborious tasks that would
be prone to error if performed manually. This step relates to applying existing utility functions within the Test
Suite, and identifying new utilities that are required.
It's a good idea to simplify the interfaces to these utilities, encapsulating as much complexity as possible within the
private implementation of the utility. It's also a good idea to develop the utility in such a way that it can be reused
where required for both manual and automated test efforts.
We recommend you don't hide the information that characterizes an individual test within these utilities: instead,
limit the utility to the complex mechanics of gathering information, or comparing actual values to expected results
etc. but where possible, pass the specific characteristics of each individual test in as input from-and return the
individual actual results an output to-a controlling Test or Test Suite.
Determine Recovery Requirements
Purpose:
To enable Test Suites to be recovered without requiring the complete re-execution of the Test Suite
Determine the appropriate points within the Test Suite to provide recovery if the Test Suite fails during execution.
This step gains importance where the Test Suite will contain a large number of Tests, or will run for an extended
period of time-often unattended. While most often identified as a requirement for automated Test Suites, it is also
important to consider recovery points for manually executed Test Suites.
In addition to recovery or restart points you may also want-in the case of automated Test Suites-to consider automated
Test Suite recovery. Two approaches to auto-recovery are 1) basic recovery where the existing Test Suite can
self-recover from a minor error that occurs in one of it's Tests, typically recovering execution at the next Test in
the Test Suite or 2) sophisticated recovery that cleans up after the failed Test, resetting appropriate system state
including operating system reboot and data restoration if necessary. As in the first approach, the Test Suite then
determines the Test that failed and selects the next Test to execute.
Implement Recovery Requirements
Purpose:
To implement and verify that the recovery process works as required
Depending on the level of sophistication required, it will require effort to implement and stabilize recovery
processing. You'll need to allow time to simulate a number of likely (and a few unlikely) failures to prove the
recovery processing works.
In the case of automated recovery, both approaches outlined in the previous step have strengths and weaknesses. You
should consider carefully the cost of sophisticated automated recovery, both in terms of initial development but also
ongoing maintenance effort. Sometimes manual recovery is good enough.
Stabilize the Test Suite
Purpose:
To resolve any dependency problems both in terms of System State and Test execution sequences
You should take time to stabilize the Test Suite by one or more trial test executions where possible. The difficulty in
achieving stability increases proportionally to the complexity of the Test Suite, and where there is excessively tight
coupling between unrelated and and low cohesion between related Tests.
There is the possibility of errors occurring when Tests are executed together within a given Test Suite, that were not
encountered when the individual Tests were executed independently. These errors are often the most difficult to track
down and diagnose, especially when the are encountered halfway though a length automated test run. Where practical,
it's a good idea to rerun the Test Suite regularly as you add additional Tests. This will help you isolate a small
number of potential candidate Tests to be diagnosed to identify the problem.
Maintain Traceability Relationships
Purpose:
To enable impact analysis and assessment reporting to be performed on the traced items
Using the Traceability requirements outlined in the Test Plan, update the traceability relationships as necessary. Test
Suites might be traced to defined Test Cases or to Test Ideas. Optionally, they may be traced to Use Cases, software
specification elements, Implementation Model elements and to one or more measures of Test Coverage.
Evaluate and Verify Your Results
Purpose:
To verify that the task has been completed appropriately and that the resulting work products are
acceptable
Now that you have completed the work, it is beneficial to verify that the work was of sufficient value, and that you
did not simply consume vast quantities of paper. You should evaluate whether your work is of appropriate quality, and
that it is complete enough to be useful to those team members who will make subsequent use of it as input to their
work. Where possible, use the checklists provided in RUP to verify that quality and completeness are "good enough".
Have the people performing the downstream tasks that rely on your work as input take part in reviewing your interim
work. Do this while you still have time available to take action to address their concerns. You should also evaluate
your work against the key input work products to make sure you have represented them accurately and sufficiently. It
may be useful to have the author of the input work product review your work on this basis.
Try to remember that that RUP is an iterative delivery process and that in many cases work products evolve over time.
As such, it is not usually necessary-and is often counterproductive-to fully-form a work product that will only be
partially used or will not be used at all in immediately subsequent work. This is because there is a high probability
that the situation surrounding the work product will change-and the assumptions made when the work product was created
proven incorrect-before the work product is used, resulting in wasted effort and costly rework. Also avoid the trap of
spending too many cycles on presentation to the detriment of content value. In project environments where presentation
has importance and economic value as a project deliverable, you might want to consider using an administrative resource
to perform presentation tasks.
More Information
Tool Mentors
Creating Performance Test Suites with Rational TestManager
Executing a Test Suite Using Rational TestFactory
Implementing an Automated Test Suite Using Rational TestManager
Implementing Developer Tests using Rational Test RealTime
Implementing Generated Test Scripts Using Rational TestFactory