Page 1

Testing induction

Testing Essentials Unit 9 & 10 Test Process Management QSIT- Quality Solutions for Information Technology Pvt. Ltd., A 806 & 807, Mittal Tower, BANGALORE- 560 001,INDIA URL: PH: +91 80 41134334 Fax: +91 80 25597445

Software Testing Essentials


Test Management •


Test Organization

Test Planning and Estimation

Test Progress monitoring and control

Incident Management

Configuration Management

Risk and Testing 

Project Risk, Product Risk

Software Testing Essentials

QSIT Copyright



Testing induction Levels of Independent Testing   


The Developer Independent Testers ceded to the development team Independent permanent test team with in the organization (IVS) Independent testers or test team provided by the operational business units Specialist testers such as usability, security and performance testers Outsourced test team or testers Eg: Contractors or other organization



Software Testing Essentials


Test Management Benefits

Isolation from the development team (if treated as totally independent on the test basis to understand what it is the tester is testing

The tester is unbiased

Tester may be seen as bottleneck as independent test execution is normally the last stage and affected by any delays earlier in the process of testing

The tester can see what has been built rather than what the developer thought had been built

Developers lose a sense of responsibility for quality as it may be assumed that they need not worry about errors because test team exists

The tester makes no assumptions regarding quality

Fully independent view sets developers and testers on either side of an invisible fence. This can hinder communication. It can also mean that developers are seen to throw the software over the fence

Software Testing Essentials

QSIT Copyright


The tester sees other and different defects too than the author



Testing induction Typical tasks of a Tester may include • Reviewing and contributing to the development of test plans • Analyzing, reviewing and assessing user requirement and models for testability • Creating test specification for test basis • Setting up test environment (often coordinating with system administrator) • In some organization setting up and management of test environment could be centrally controlled in this situation tester would directly liaise with environment management to ensure test environment is delivered in time • Preparing and acquiring/creating test data • Implementing tests on test level, executing and logging the tests • Evaluating the results and documenting the deviation in the form of defects • Automating tests (may be supported by a developer or test automation expert) • Reviewing tests developed by other testers

Software Testing Essentials


Typical tasks of a Test Leader may include • Coordinating the development of test strategy and plan with project manager and others • Writing/Reviewing test strategy produced for the project and test policies produced for the organization •Planning the development of required tests. •Managing the specification, preparation, implementation and execution of tests, including monitoring and control of all specification and execution • Taking required action including adopting the planning, based on test results and progress • Ensuring the adequate configuration management of test ware is in place and that the test ware is fully traceable • Putting in place suitable metrics for measuring test progress and evaluating the quality of testing •Agreeing what should be automated, to what degree and how •Scheduling all testing activity •End of the project writing a test summary report based on the information gathered during testing

Software Testing Essentials

QSIT Copyright



Testing induction Testing Strategies Introduction to Strategy  Approach to Software Testing Strategy – Defect Prevention Strategy – Defect Detection Strategy  Developing Strategy for New Project  Developing Strategy for Maintenance project  Inputs and Outputs of Strategy  Issues and Risks to be considered while doing Strategy

Software Testing Essentials


Testing Process Testing Process has following steps: Planning and Control Analysis and Design Implementation and execution Evaluating exit Criteria and reporting Test Closure activities

Software Testing Essentials

QSIT Copyright



Testing induction Introduction Test Strategy 

Defines the scope and general direction for testing in the project

The intention of Test Strategy is to establish framework for testing the system and work products.

Aim is to find most important errors as early as possible hence reducing the cost of fixing the errors

To define the level of participation for the relevant stakeholders.

Should identify types of testing to address client’s requirements.

Test strategy should be considered as a prerequisite and overview to the testing design. Software Testing Essentials


Approach to Software Test Strategy 

Defect Prevention Strategies – Examination of constraints –

Early test involvement

Use of Standards

Defect Detection Strategies – Inspections and Walkthroughs –

Testing Product Deliverables

Design Testability in to the application

Use of Automated tools

Adherence to a test process

Inspection and Walkthroughs Quality Gates

Software Testing Essentials

QSIT Copyright



Testing induction Developing Strategy for New Project

To Clarify expectations with the user, sponsor and bidders

To Describe the details of how the testing team will evaluate the work products, system, and testing activities and results

To describe approach to all testing phases and the activities for which they are responsible

Software Testing Essentials


Developing Strategy for Maintenance Project

Includes a greater focus on regression testing an keeping the users informed of specific fixes or changes that were requested

Test process should be described in terms of the periodic release cycles that are part of the change control process

Also describe a set of minimum tests to be performed when emergency fixes are needed (for instance, due to failed hardware or recovering from a database crash)

Software Testing Essentials

QSIT Copyright



Testing induction Test Strategy: Inputs & Deliverables

Time Required for Testing

Priority & Criticality

No. & Levels of Resources

Types of Application

Rounds of Testing

Test Deliverables Strategy

Exit Criteria Test Suspension Criteria

Project Success Criteria

Resumption Criteria

Software Testing Essentials


Test Risks Poor Requirements Stakeholder Participation

Test Staffing

Requirement Testability Schedule Compression

Test Related & Risks & Considerations

Performance and Stress Testing

Testing of COTS

External Interfaces


Insufficient Resources Software Testing Essentials

QSIT Copyright



Testing induction Typical Test Issues

Test Participation Performances and Capacity Requirement/Testing

Test Environments

Approach of Testing COTS products

Test Issues

Scope of Acceptance Testing

Pilot of Field Testing Criteria for Acceptance of the System

Software Testing Essentials

Approach to Testing External Interfaces

Verification of Un-testable Requirements



Test Strategy Must answer the following: – – – – – – – – –

When will testing occur? What kind of testing will occur? What are the risks? What are the critical success factors? What are the testing objectives? What are the trade-off? Who will conduct the testing? How much testing will be done? What tools will be used?

Software Testing Essentials

QSIT Copyright



Testing induction Purpose of Test Plan 

Provides guidelines to Management to understand technical effort needed to support test program

Establishes nature and extent of tests necessary to achieve test goals and objectives

Outlines the orderly schedule of events and activities testing

Outlines efficient use of resources – Personal, financial, equipment and facility resources

Software Testing Essentials


Purpose of Test Plan 

Provides assurance of level of test coverage through traceability matrix

Outlines the contents of Test procedure scripts and its execution

Clearly details the Test Completion Criteria along with list of supporting documents.

Software Testing Essentials

QSIT Copyright



Testing induction Purpose of Test Plan 

Provides assurance of level of test coverage through traceability matrix

Outlines the contents of Test procedure scripts and its execution

Clearly details the Test Completion Criteria along with list of supporting documents.

Software Testing Essentials


Test Planning and Estimation This involves following activities: 

Test Plan Identifier


Test Items

Features to be tested

Features not to be tested


Item Pass/Fail criteria. Suspension /Resumption Criteria

Schedule/ Mile Stone

Entry/Exit Criteria

Testing Deliverables and Risk and Contingencies

Resources & Responsibility and Approvals

Software Testing Essentials

QSIT Copyright

IEEE testplan template



Testing induction Test Plan – Exit Criteria Purpose : to define when to stop testing Typical Exit Criteria might be : • All tests planned have been run • A certain level of requirements coverage has been achieved • No high –priority or severe defects are left outstanding • All high risk areas have been fully tested, with only minor residual risks left outstanding • Cost – when the budget has been spent • The schedule has been achieved Eg: the release date has been reached and the product has to go live Software Testing Essentials


Test Metrics

Software Testing Essentials

QSIT Copyright



Testing induction Test Metrics Collection and Analysis 

Scope of process:





Goals and objectives are set relative to the product and software (test) management process

Measurements are defined and selected to ascertain the degree to which the goals and objectives are being met

The data collection process and recording mechanism are defined and used.

Software Testing Essentials


Test Metrics Collection and Analysis 

Scope of Metrics process (Cont.):



Measurements and reports are part of a closed-loop system that provides current (operational) and historical information to technical staff and management.

Data on post-software product life measurement are retained for analysis that could lead to improvements for future product and process management.

Software Testing Essentials

QSIT Copyright




Testing induction White Box Testing Metrics or Depth Coverage Analysis 

Source Code Analysis – Helps discern code quality –

Determines whether an algorithm works

Function is being called too frequently

Identifies complex areas of the source code

Quantifies Fault density ,Fd = Nd/KSLOC,  where Nd is the number of defects, and  KSLOC is the number of non-comment lines of source code.

Software Testing Essentials


Black Box Testing Metrics or Breadth Coverage Analysis

Metrics collection focuses on the breadth of testing, such as the amount of demonstrated functionality and the amount of testing that has been performed. Categories of testing metrics to be collected during the black-box testing : – Coverage, –



Software Testing Essentials

QSIT Copyright



Testing induction Black Box Testing Metrics - Coverage Metrics

Test Coverage

System Coverage Analysis

Functional Test Coverage

Software Testing Essentials


Black Box Testing Metrics - Progress Metrics

Test procedure execution status

Error Discovery Rate

Defect Aging

Defect Fix Retest

Defect Trend Analysis

Software Testing Essentials

QSIT Copyright



Testing induction Black Box Testing Metrics - Quality Metrics

Test Success Index

Defect Density

Defect Trend Analysis

Test Effectiveness

Problem Report-Acceptance Criteria Metrics

Test Automation Metric Software Testing Essentials


Test Reporting Test Reporting is concerned with summarizing information about the testing endeavor • What happened during a period of testing such as dates when exit criteria were met • Analyzed information and metrics to support recommendation and decision about future action •An assessment of defects remaining • economic benefit of continued testing • outstanding risks • level of confidence in the tested software • Metrics should be collected during and at the end of a test level to assess • the adequacy of the test objective for that level • adequacy of the test approach taken • Effectiveness of testing with respect to its objectives

Software Testing Essentials

QSIT Copyright



Testing induction Test Summary – IEEE 829 To

summarize the results of the designated testing activities and to provide evaluation based on these results A test summary report shall have the following structure • Test summary report identifier • summary • Variances • comprehensive assessment • summary of results • evaluation • summary of activities • approvals

Software Testing Essentials


Test Control Test control describes any guidance or corrective action taken as a result of information and metrics gathered and reported

Examples of Test Control actions • Reprioritize test when an identified project risk occurs (eg. Software delivered late) • Change the test schedule due to availability of test environment • Set an entry criterion requiring fixes to be retested by developer before accepting them in to a build (this is particularly useful when defect fixes continually fail) • Review of product risks and perhaps changing the risk rating to meet the target • Adjusting the scope of testing (amount of tests to be run) to manage the testing of late change requests

Software Testing Essentials

QSIT Copyright



Testing induction Incident Management An Incident is any significant, unplanned event that occurs during testing that requires subsequent investigation or correction • Discrepancies between actual and expected outcomes need to be logged as incident • Incident should be tracked from discovery and classification to correction and confirmation of the solution • In order to manage all incidents to completion, an organization should establish a process and rules for classification

Software Testing Essentials


Incident Management A tester or reviewer typically logs the following information regarding the incident        

Date of issue, issuing organinsation, author, approvals and status Scope, severity and priority of the incident References, including the identity of the test case specification that reveled the problem. Date the incident was discovered Urgency or priority to fix Severity of the impact on the system Degree of impact on stakeholders interest Software or system life cycle process in which the incident was observed

Software Testing Essentials

QSIT Copyright



Testing induction Incident Management : Incident Report outline Sl No




Test Incident report identifier

The unique identifier assigned to this test incident report



A summary of the incident, detailing where expected and actual results differ, identifying at a high level the items that are affected and the steps leading up the recognition of the incident.


Incident description

A detailed description of the incident which should include – Inputs – Expected results – Actual results – Anomalies – Date and time – Procedure step – Environment – Attempts to repeat – Tester’s / Observer’s comments Should also include any information regarding possible causes and solution


Impact Software Testing Essentials

If known, document what impact the incident has on progress 35

Introduction to Software Configuration Management

Software Testing Essentials

QSIT Copyright



Testing induction Software Configuration Management What is SCM – Software Configuration Management? A management tool that applies an engineering discipline to manage the evolution of a system from concept through to system retirement The primary focus of this discipline is to ensure Repeatability Traceability and Integrity of the system being developed and produced Ensures that everyone know what is supposed to be built, what is being built and what has been built

Software Testing Essentials


Software Configuration Management Purpose of SCM     

Maintain product integrity Make product evolution more manageable and controlled Provide product status during evolution Ensure product meets customer requirements Actively control changes – What changes were made to the software? – Why were the changes made? – When were the changes made? – Who made the changes?

Software Testing Essentials

QSIT Copyright



Testing induction Software Configuration Management Need for SCM      

Increased software control Integration of millions of lines of code from multiple subcontractors Complex co-ordination of product’s evolution over years by many people Avoid rework due to lack of understanding of all the components that make up a software product Many variants of main product to reach out to a diversified market Reduction in cost of rework caused by changes and defects

Software Testing Essentials


SCM – Process Dependencies

Software Testing Essentials

QSIT Copyright



Testing induction Software Configuration Management Common issues in SCM Developers view this as a hindrance to product improvements due to overhead associated with the change control function of SCM Some issues that are faced due to not having a good SCM Practice  The latest version of source code cannot be found  A difficult defect that was fixed at great expense suddenly reappears  A developed and tested feature is mysteriously missing  A fully tested program suddenly does not work  The wrong version of the code was tested  There is no traceability between the software requirements, documentation and code  Programmers are working on the wrong version of the code  The wrong versions of the configuration items are being baselined  No one knows which modules comprise the software system delivered to the customer

Software Testing Essentials


Software Configuration Management What is a build? Identifying a set of source items (features, fixes, changes etc) and through some means of compilation or generation, producing a set of executables and other files which can be used to perform a runtime function

What is a Package? Includes output from the build and other items like scripts, database items, documents, etc that create a set of deliverables that run in a test or production environment

Software Testing Essentials

QSIT Copyright



Testing induction Software Configuration Management Version Numbering: 

Build version number will have the following components

Product name Eg: CMM, BSC, FIN,..

Major Number - Identifies major milestones in the product

Minor Number – Identifies intermediate milestones on the product road map

Upgrade Number - These are typically incremental on the intermediate milestones and are built on demand

Patch Number – Identified ad-hoc fixes made for immediate deployment to customer sites. These are not planned releases

Software Testing Essentials


Configuration Management Functions




Versions and Configurations

Audit History



Project Context Information Repository




System Model

Access Control


Change Requests


Bug Tracking


Change Propagation











Impact analysis Regeneration

Software Testing Essentials

QSIT Copyright



Test Process  

Test Process