Task: Execute Test Suite









var backPath = './../../';
var imgPath = './../../images/';
var nodeInfo=[{view: "view:_LVCagP5WEdmAzesbYywanQ", path: ["_LVCagP5WEdmAzesbYywanQ", "_mp7z0DIDEdqwaNnSEheSAg", "_1yMJAN7YEdmjRZts2c4ZjQ", "{A719CEAE-35FB-42F6-A076-F501B83E5B85}", "{6B420194-5EB7-4A12-A061-09FAA5DC18DE}"]}, {view: "view:_LVCagP5WEdmAzesbYywanQ", path: ["_LVCagP5WEdmAzesbYywanQ", "_mp7z0DIDEdqwaNnSEheSAg", "_u2yEADIEEdqwaNnSEheSAg", "_yd3EzdnmEdmO6L4XMImrsA", "{6B420194-5EB7-4A12-A061-09FAA5DC18DE}"]}, {view: "view:_FCx1oN7CEdmsEI4YDGX2ag", path: ["_FCx1oN7CEdmsEI4YDGX2ag", "_f6_YwN7DEdmsEI4YDGX2ag", "_1yMJAN7YEdmjRZts2c4ZjQ", "{A719CEAE-35FB-42F6-A076-F501B83E5B85}", "{6B420194-5EB7-4A12-A061-09FAA5DC18DE}"]}, {view: "view:_FCx1oN7CEdmsEI4YDGX2ag", path: ["_FCx1oN7CEdmsEI4YDGX2ag", "_kC0pcN7GEdm8G6yT7-Wdqw", "_yd3EzdnmEdmO6L4XMImrsA", "{6B420194-5EB7-4A12-A061-09FAA5DC18DE}"]}];
contentPage.preload(imgPath, backPath, nodeInfo, '', true, false, false);










Task: Execute Test Suite















This task describes how to execute the appropriate collections of tests required to evaluate product quality amd how to capture test results that facilitate ongoing assessment of the product.


Disciplines: Test






Relationships



RolesPrimary Performer:



Tester


Additional Performers:



InputsMandatory:



Build


Test Suite



Optional:



Development Infrastructure


Iteration Plan






Outputs


Test Log






Process Usage


Test
 > 
Validate Build Stability
 > 
Execute Test Suite


Test
 > 
Test and Evaluate
 > 
Execute Test Suite






Steps




Setup Test Environment to Known State



Purpose:  To accurately establish the test environment in preparation for Test Suite execution. Setup the test environment to ensure that all the required components (hardware, software, tools, data, etc.) have been established, and are available and ready in the test environment in the correct state to enable the tests to be conducted. Typically this will involve some form of basic environment reset (e.g. Registry and other configuration files), restoration of underlying databases to the require state, and the setup of any peripheral devices (e.g. such as loading paper into printers). While some tasks can be performed automatically, some aspects typically require human attention. The use of environment support tools such as those that enable hard-disk image capture and restoration are extremely valuable in managing this effort effectively.



Set Execution Tool Options



Purpose:  To appropriately configure the tools used in Test Suite execution. Set the execution options of the supporting tools. Depending on the sophistication of the tool, this may be many options to consider. Failing to set these options appropriately may reduce the usefulness and value of the resulting Test Logs and other outputs. Where possible, you should try to store these tool options and settings so that they can be reloaded easily based on one or more predetermined profiles. In the case of automated test execution tools, there may be many different settings to be considered, such as the speed at which execution should be performed. In the case of manual testing, it is often simply a matter of logging into issue or changes request tracking systems, or partitioning a new unique entry in a support system for logging results. You should give some thought to concerns such as the name, location and state of the Test Log to be written to.



Schedule Test Suite Execution



Purpose:  To determine the appropriate time for test execution to begin. In many cases where test execution can be attended, the Test Suite can be executed relatively on demand. In these cases, scheduling will likely need to take into account considerations such as the work of other testers, other team members as well as different test teams that share the test environment. In these cases, test execution will typically need to work around infrequent environment resets. However, in cases where unattended execution of automated tests is desired, or where the execution of many tests running concurrently on different machines must be coordinated, some form of automated scheduling mechanism may be required. Either use the features of your automated test execution tool or develop your own utility functions to enable the required scheduling.



Execute Test Suite



Purpose:  To conduct the tests enclosed in the Test Suite and to monitor their completion. Executing the Test Suite will vary dependent upon whether testing is conducted automatically or manually. In either case, the test suites developed during the test implementation tasks are used to either execute the tests automatically, or guide the manual execution of the tests.



Evaluate Execution of Test Suite



Purpose:  To determine whether the Test Suite executed to completion or halted abnormally, and make an assessment concerning whether corrective action is required. The execution of testing ends or terminates in one of two conditions: Normal: all the Tests execute as intended to completion.  Abnormal or premature: the Tests did not execute completely as intended. When testing ends abnormally, the Test Logs from which subsequent Test Results are derived may be unreliable. The cause of the abnormal termination needs to be identified, and if necessary, the fault corrected and the tests re-executed.



Recover from Halted Tests



Purpose:  To determine the appropriate corrective action to recover from a halted Test Suite execution, and if required correct the problem, recover, and re-execute the Test Suite. To recover from halted tests, do the following: Inspect the Test Logs and other output Correct errors Schedule and execute Test Suite again Reevaluate execution of Test Suite Inspect the Test Logs and other output Inspect the Test Logs and other output for completeness and accuracy. Identify where errors have occurred and inspect them. When test automation is being employed, there are two categories of halted tests that it is important to be aware of: Fatal errors-the system fails (network failures, hardware crashes, etc.) Test failures-this is when some part of a Test within a Test Suite cannot be executed as intended. When either category of abnormal behavior occurs during test execution, they may exhibit the following symptoms: a large number of (ongoing occurrence) unexpected actions or unexpected windows occur while the Test Suite is executing the test environment appears unresponsive, is slow or is in an undesirable state (such as hung or crashed). Work through the symptoms until you can determine the root cause of the problem. Correct errors Errors may be found in the input data consumed by the test, the test itself or other aspects of the test such as the test environment or runtime tool settings. It's common for a fix to an error in one aspect of the test to require the correct state to be present in all often aspects of the test. Once you have finished investigating problems, you may have discovered one or more faults that needs correction. To make permanent corrections to the environment, test data or the test itself, it is a good practice to restore each aspect of the test again to a known state before applying any permanent corrections. This ensures that no additional unwanted or invalid changes find their way into the known-state environment. After making the necessary changes, save the Test and backup or save the accompanying input data and test environment as required. Schedule and execute Test Suite again Reschedule and re-execute the Test Suite. Depending on what recovery process is available (if any), you may be able to restart the test suite from an interim point rather than starting from the beginning. Note that to enable recovery of test execution from a point part-way through the test run, typically necessitates the implementation and ongoing maintenance of some form of partial recovery procedure. Reevaluate execution of Test Suite Confirm the Test Suite now runs to completion. If there are still problems, work through these subsections that make up Recover from halted tests again until all of the problems are resolved.



Inspect the Test Logs for Completeness and Accuracy



Purpose:  To determine if the Test Suite execution generated worthwhile test information and if not, to identify appropriate corrective action. When test execution initially completes, the Test Logs should be reviewed to ensure that the logs are reliable and reported failures, warnings, or unexpected results were not caused by external influences (to the target-of-test), such as improper environment setup or invalid input data for the test. For GUI-driven automated Tests, common Test failures include: Test verification failures-this occurs when the actual result and the expected result do not match. Verify that the verification method(s) used focus only on the essential items and / or properties and modify if necessary. Unexpected GUI windows-this occurs for several reasons. The most common is when a GUI window other than the expected one is active or the number of displayed GUI windows is greater than expected. Ensure that the test environment has been setup and initialized as intended for proper test execution. Missing GUI windows-this failure is noted when a GUI window is expected to be available (but not necessarily active) and is not. Ensure that the test environment has been setup and initialized as intended for proper test execution. Verify that the actual missing windows are / were removed from the target-of-test. If the reported failures are due to errors identified in the test work products, or due to problems with the test environment, the appropriate corrective action should be taken and the testing re-executed. If the Test Log enables you to determine that the failures are due to genuine failures in the Target Test Items, then the execution portion of the task is complete.



Restore Test Environment to Known State



Purpose:  To ensure the environment is properly reset after Test Suite execution. (See the first step) Next you should restore the environment back to it's original state. Typically this will involve some form of basic environment reset (e.g. Registry and other configuration files), restoration of underlying databases to known state, and so forth in addition to tasks such as loading paper into printers. While some tasks can be performed automatically, some aspects typically require human attention.



Maintain Traceability Relationships



Purpose:  To enable impact analysis and assessment reporting to be performed on the traced items.  Using the Traceability requirements outlined in the Test Plan, update the traceability relationships as required. A good starting point is to consider traceability in terms of measuring the extent of testing or test coverage. As a general rules, we recommend basing the measurement of the extent of testing against the motivators you discovered during the test planning activities. Test Suites might also be traced to the defined Test Cases they realize. They may also be traced to elements of the requirements, software specification, design or implementation. Whatever relationships you have decided are important to trace, you will need to update the status of the relationships that were established during implementation of the Test Suite.



Evaluate and Verify Your Results



Purpose:  To verify that the task has been completed appropriately and that the resulting work products are acceptable. Now that you have completed the work, it is beneficial to verify that the work was of sufficient value, and that you did not simply consume vast quantities of paper. You should evaluate whether your work is of appropriate quality, and that it is complete enough to be useful to those team members who will make subsequent use of it as input to their work. Where possible, use the checklists provided in RUP to verify that quality and completeness are "good enough". Have the people performing the downstream tasks that rely on your work as input take part in reviewing your interim work. Do this while you still have time available to take action to address their concerns. You should also evaluate your work against the key input work products to make sure you have represented them accurately and sufficiently. It may be useful to have the author of the input work product review your work on this basis. Try to remember that that RUP is an iterative delivery process and that in many cases work products evolve over time. As such, it is not usually necessary-and is often counterproductive-to fully-form a work product that will only be partially used or will not be used at all in immediately subsequent work. This is because there is a high probability that the situation surrounding the work product will change-and the assumptions made when the work product was created proven incorrect-before the work product is used, resulting in wasted effort and costly rework. Also avoid the trap of spending too many cycles on presentation to the detriment of content value. In project environments where presentation has importance and economic value as a project deliverable, you might want to consider using an administrative resource to perform presentation tasks.







More Information



Tool Mentors


Evaluating Test Coverage Using Rational TestFactory


Evaluating the Results of Executing a Test Suite Using Rational TestFactory


Executing a Test Suite Using Rational TestFactory


Executing a Test Suite Using Rational TestManager


Executing Developer Tests Using Rational Test RealTime


Executing Test Suites Using Rational Robot


Executing Test Suites Using the Rational PurifyPlus Tools (Windows and UNIX)


Setting Up the Test Environment in Rational Robot


Using Rational TestFactory to Measure and Evaluate Code-based Test Coverage on Rational Robot Test Scripts








©  Copyright IBM Corp. 1987, 2006.  All Rights Reserved.







contentPage.onload();