Page 1

Testing induction

Testing Essentials Unit 9 & 10 Test Process Management QSIT- Quality Solutions for Information Technology Pvt. Ltd., A 806 & 807, Mittal Tower, BANGALORE- 560 001,INDIA URL: www.qsitglobal.com PH: +91 80 41134334 Fax: +91 80 25597445

Software Testing Essentials

1

www.qsitglobal.com

Test Management •

Overview

Test Organization

Test Planning and Estimation

Test Progress monitoring and control

Incident Management

Configuration Management

Risk and Testing 

Project Risk, Product Risk

Software Testing Essentials

QSIT Copyright

2

www.qsitglobal.com

Page:1


Testing induction Levels of Independent Testing   





 

The Developer Independent Testers ceded to the development team Independent permanent test team with in the organization (IVS) Independent testers or test team provided by the operational business units Specialist testers such as usability, security and performance testers Outsourced test team or testers Eg: Contractors or other organization

Low

High

Software Testing Essentials

3

www.qsitglobal.com

Test Management Benefits

Isolation from the development team (if treated as totally independent on the test basis to understand what it is the tester is testing

The tester is unbiased

Tester may be seen as bottleneck as independent test execution is normally the last stage and affected by any delays earlier in the process of testing

The tester can see what has been built rather than what the developer thought had been built

Developers lose a sense of responsibility for quality as it may be assumed that they need not worry about errors because test team exists

The tester makes no assumptions regarding quality

Fully independent view sets developers and testers on either side of an invisible fence. This can hinder communication. It can also mean that developers are seen to throw the software over the fence

Software Testing Essentials

QSIT Copyright

Drawbacks

The tester sees other and different defects too than the author

4

www.qsitglobal.com

Page:2


Testing induction Typical tasks of a Tester may include • Reviewing and contributing to the development of test plans • Analyzing, reviewing and assessing user requirement and models for testability • Creating test specification for test basis • Setting up test environment (often coordinating with system administrator) • In some organization setting up and management of test environment could be centrally controlled in this situation tester would directly liaise with environment management to ensure test environment is delivered in time • Preparing and acquiring/creating test data • Implementing tests on test level, executing and logging the tests • Evaluating the results and documenting the deviation in the form of defects • Automating tests (may be supported by a developer or test automation expert) • Reviewing tests developed by other testers

Software Testing Essentials

5

www.qsitglobal.com

Typical tasks of a Test Leader may include • Coordinating the development of test strategy and plan with project manager and others • Writing/Reviewing test strategy produced for the project and test policies produced for the organization •Planning the development of required tests. •Managing the specification, preparation, implementation and execution of tests, including monitoring and control of all specification and execution • Taking required action including adopting the planning, based on test results and progress • Ensuring the adequate configuration management of test ware is in place and that the test ware is fully traceable • Putting in place suitable metrics for measuring test progress and evaluating the quality of testing •Agreeing what should be automated, to what degree and how •Scheduling all testing activity •End of the project writing a test summary report based on the information gathered during testing

Software Testing Essentials

QSIT Copyright

6

www.qsitglobal.com

Page:3


Testing induction Testing Strategies Introduction to Strategy  Approach to Software Testing Strategy – Defect Prevention Strategy – Defect Detection Strategy  Developing Strategy for New Project  Developing Strategy for Maintenance project  Inputs and Outputs of Strategy  Issues and Risks to be considered while doing Strategy

Software Testing Essentials

7

www.qsitglobal.com

Testing Process Testing Process has following steps: Planning and Control Analysis and Design Implementation and execution Evaluating exit Criteria and reporting Test Closure activities

Software Testing Essentials

QSIT Copyright

8

www.qsitglobal.com

Page:4


Testing induction Introduction Test Strategy 

Defines the scope and general direction for testing in the project



The intention of Test Strategy is to establish framework for testing the system and work products.



Aim is to find most important errors as early as possible hence reducing the cost of fixing the errors



To define the level of participation for the relevant stakeholders.



Should identify types of testing to address client’s requirements.



Test strategy should be considered as a prerequisite and overview to the testing design. Software Testing Essentials

9

www.qsitglobal.com

Approach to Software Test Strategy 

Defect Prevention Strategies – Examination of constraints –

Early test involvement

Use of Standards

Defect Detection Strategies – Inspections and Walkthroughs –

Testing Product Deliverables

Design Testability in to the application

Use of Automated tools

Adherence to a test process

Inspection and Walkthroughs Quality Gates

Software Testing Essentials

QSIT Copyright



10

www.qsitglobal.com

Page:5


Testing induction Developing Strategy for New Project



To Clarify expectations with the user, sponsor and bidders



To Describe the details of how the testing team will evaluate the work products, system, and testing activities and results



To describe approach to all testing phases and the activities for which they are responsible

Software Testing Essentials

11

www.qsitglobal.com

Developing Strategy for Maintenance Project



Includes a greater focus on regression testing an keeping the users informed of specific fixes or changes that were requested



Test process should be described in terms of the periodic release cycles that are part of the change control process



Also describe a set of minimum tests to be performed when emergency fixes are needed (for instance, due to failed hardware or recovering from a database crash)

Software Testing Essentials

QSIT Copyright

12

www.qsitglobal.com

Page:6


Testing induction Test Strategy: Inputs & Deliverables

Time Required for Testing

Priority & Criticality

No. & Levels of Resources

Types of Application

Rounds of Testing

Test Deliverables Strategy

Exit Criteria Test Suspension Criteria

Project Success Criteria

Resumption Criteria

Software Testing Essentials

13

www.qsitglobal.com

Test Risks Poor Requirements Stakeholder Participation

Test Staffing

Requirement Testability Schedule Compression

Test Related & Risks & Considerations

Performance and Stress Testing

Testing of COTS

External Interfaces

Acceptance

Insufficient Resources Software Testing Essentials

QSIT Copyright

14

www.qsitglobal.com

Page:7


Testing induction Typical Test Issues

Test Participation Performances and Capacity Requirement/Testing

Test Environments

Approach of Testing COTS products

Test Issues

Scope of Acceptance Testing

Pilot of Field Testing Criteria for Acceptance of the System

Software Testing Essentials

Approach to Testing External Interfaces

Verification of Un-testable Requirements

15

www.qsitglobal.com

Summary



Test Strategy Must answer the following: – – – – – – – – –

When will testing occur? What kind of testing will occur? What are the risks? What are the critical success factors? What are the testing objectives? What are the trade-off? Who will conduct the testing? How much testing will be done? What tools will be used?

Software Testing Essentials

QSIT Copyright

16

www.qsitglobal.com

Page:8


Testing induction Purpose of Test Plan 

Provides guidelines to Management to understand technical effort needed to support test program



Establishes nature and extent of tests necessary to achieve test goals and objectives



Outlines the orderly schedule of events and activities testing



Outlines efficient use of resources – Personal, financial, equipment and facility resources

Software Testing Essentials

17

www.qsitglobal.com

Purpose of Test Plan 

Provides assurance of level of test coverage through traceability matrix



Outlines the contents of Test procedure scripts and its execution



Clearly details the Test Completion Criteria along with list of supporting documents.

Software Testing Essentials

QSIT Copyright

18

www.qsitglobal.com

Page:9


Testing induction Purpose of Test Plan 

Provides assurance of level of test coverage through traceability matrix



Outlines the contents of Test procedure scripts and its execution



Clearly details the Test Completion Criteria along with list of supporting documents.

Software Testing Essentials

19

www.qsitglobal.com

Test Planning and Estimation This involves following activities: 

Test Plan Identifier



Introduction



Test Items



Features to be tested



Features not to be tested



Approach



Item Pass/Fail criteria. Suspension /Resumption Criteria



Schedule/ Mile Stone



Entry/Exit Criteria



Testing Deliverables and Risk and Contingencies



Resources & Responsibility and Approvals

Software Testing Essentials

QSIT Copyright

IEEE testplan template

20

www.qsitglobal.com

Page:10


Testing induction Test Plan – Exit Criteria Purpose : to define when to stop testing Typical Exit Criteria might be : • All tests planned have been run • A certain level of requirements coverage has been achieved • No high –priority or severe defects are left outstanding • All high risk areas have been fully tested, with only minor residual risks left outstanding • Cost – when the budget has been spent • The schedule has been achieved Eg: the release date has been reached and the product has to go live Software Testing Essentials

21

www.qsitglobal.com

Test Metrics

Software Testing Essentials

QSIT Copyright

22

www.qsitglobal.com

Page:11


Testing induction Test Metrics Collection and Analysis 

Scope of process:

Metrics

Collection

and

analysis

Goals and objectives are set relative to the product and software (test) management process

Measurements are defined and selected to ascertain the degree to which the goals and objectives are being met

The data collection process and recording mechanism are defined and used.

Software Testing Essentials

23

www.qsitglobal.com

Test Metrics Collection and Analysis 

Scope of Metrics process (Cont.):

and

analysis

Measurements and reports are part of a closed-loop system that provides current (operational) and historical information to technical staff and management.

Data on post-software product life measurement are retained for analysis that could lead to improvements for future product and process management.

Software Testing Essentials

QSIT Copyright

Collection

24

www.qsitglobal.com

Page:12


Testing induction White Box Testing Metrics or Depth Coverage Analysis 

Source Code Analysis – Helps discern code quality –

Determines whether an algorithm works

Function is being called too frequently

Identifies complex areas of the source code

Quantifies Fault density ,Fd = Nd/KSLOC,  where Nd is the number of defects, and  KSLOC is the number of non-comment lines of source code.

Software Testing Essentials

25

www.qsitglobal.com

Black Box Testing Metrics or Breadth Coverage Analysis





Metrics collection focuses on the breadth of testing, such as the amount of demonstrated functionality and the amount of testing that has been performed. Categories of testing metrics to be collected during the black-box testing : – Coverage, –

Progress

Quality

Software Testing Essentials

QSIT Copyright

26

www.qsitglobal.com

Page:13


Testing induction Black Box Testing Metrics - Coverage Metrics



Test Coverage



System Coverage Analysis



Functional Test Coverage

Software Testing Essentials

27

www.qsitglobal.com

Black Box Testing Metrics - Progress Metrics



Test procedure execution status



Error Discovery Rate



Defect Aging



Defect Fix Retest



Defect Trend Analysis

Software Testing Essentials

QSIT Copyright

28

www.qsitglobal.com

Page:14


Testing induction Black Box Testing Metrics - Quality Metrics



Test Success Index



Defect Density



Defect Trend Analysis



Test Effectiveness



Problem Report-Acceptance Criteria Metrics



Test Automation Metric Software Testing Essentials

29

www.qsitglobal.com

Test Reporting Test Reporting is concerned with summarizing information about the testing endeavor • What happened during a period of testing such as dates when exit criteria were met • Analyzed information and metrics to support recommendation and decision about future action •An assessment of defects remaining • economic benefit of continued testing • outstanding risks • level of confidence in the tested software • Metrics should be collected during and at the end of a test level to assess • the adequacy of the test objective for that level • adequacy of the test approach taken • Effectiveness of testing with respect to its objectives

Software Testing Essentials

QSIT Copyright

30

www.qsitglobal.com

Page:15


Testing induction Test Summary – IEEE 829 To

summarize the results of the designated testing activities and to provide evaluation based on these results A test summary report shall have the following structure • Test summary report identifier • summary • Variances • comprehensive assessment • summary of results • evaluation • summary of activities • approvals

Software Testing Essentials

31

www.qsitglobal.com

Test Control Test control describes any guidance or corrective action taken as a result of information and metrics gathered and reported

Examples of Test Control actions • Reprioritize test when an identified project risk occurs (eg. Software delivered late) • Change the test schedule due to availability of test environment • Set an entry criterion requiring fixes to be retested by developer before accepting them in to a build (this is particularly useful when defect fixes continually fail) • Review of product risks and perhaps changing the risk rating to meet the target • Adjusting the scope of testing (amount of tests to be run) to manage the testing of late change requests

Software Testing Essentials

QSIT Copyright

32

www.qsitglobal.com

Page:16


Testing induction Incident Management An Incident is any significant, unplanned event that occurs during testing that requires subsequent investigation or correction • Discrepancies between actual and expected outcomes need to be logged as incident • Incident should be tracked from discovery and classification to correction and confirmation of the solution • In order to manage all incidents to completion, an organization should establish a process and rules for classification

Software Testing Essentials

33

www.qsitglobal.com

Incident Management A tester or reviewer typically logs the following information regarding the incident        

Date of issue, issuing organinsation, author, approvals and status Scope, severity and priority of the incident References, including the identity of the test case specification that reveled the problem. Date the incident was discovered Urgency or priority to fix Severity of the impact on the system Degree of impact on stakeholders interest Software or system life cycle process in which the incident was observed

Software Testing Essentials

QSIT Copyright

34

www.qsitglobal.com

Page:17


Testing induction Incident Management : Incident Report outline Sl No

Heading

Details

1

Test Incident report identifier

The unique identifier assigned to this test incident report

2

Summary

A summary of the incident, detailing where expected and actual results differ, identifying at a high level the items that are affected and the steps leading up the recognition of the incident.

3

Incident description

A detailed description of the incident which should include – Inputs – Expected results – Actual results – Anomalies – Date and time – Procedure step – Environment – Attempts to repeat – Tester’s / Observer’s comments Should also include any information regarding possible causes and solution

4

Impact Software Testing Essentials

If known, document what impact the incident has on progress 35

www.qsitglobal.com

Introduction to Software Configuration Management

Software Testing Essentials

QSIT Copyright

36

www.qsitglobal.com

Page:18


Testing induction Software Configuration Management What is SCM – Software Configuration Management? A management tool that applies an engineering discipline to manage the evolution of a system from concept through to system retirement The primary focus of this discipline is to ensure Repeatability Traceability and Integrity of the system being developed and produced Ensures that everyone know what is supposed to be built, what is being built and what has been built

Software Testing Essentials

37

www.qsitglobal.com

Software Configuration Management Purpose of SCM     

Maintain product integrity Make product evolution more manageable and controlled Provide product status during evolution Ensure product meets customer requirements Actively control changes – What changes were made to the software? – Why were the changes made? – When were the changes made? – Who made the changes?

Software Testing Essentials

QSIT Copyright

38

www.qsitglobal.com

Page:19


Testing induction Software Configuration Management Need for SCM      

Increased software control Integration of millions of lines of code from multiple subcontractors Complex co-ordination of product’s evolution over years by many people Avoid rework due to lack of understanding of all the components that make up a software product Many variants of main product to reach out to a diversified market Reduction in cost of rework caused by changes and defects

Software Testing Essentials

39

www.qsitglobal.com

SCM – Process Dependencies

Software Testing Essentials

QSIT Copyright

40

www.qsitglobal.com

Page:20


Testing induction Software Configuration Management Common issues in SCM Developers view this as a hindrance to product improvements due to overhead associated with the change control function of SCM Some issues that are faced due to not having a good SCM Practice  The latest version of source code cannot be found  A difficult defect that was fixed at great expense suddenly reappears  A developed and tested feature is mysteriously missing  A fully tested program suddenly does not work  The wrong version of the code was tested  There is no traceability between the software requirements, documentation and code  Programmers are working on the wrong version of the code  The wrong versions of the configuration items are being baselined  No one knows which modules comprise the software system delivered to the customer

Software Testing Essentials

41

www.qsitglobal.com

Software Configuration Management What is a build? Identifying a set of source items (features, fixes, changes etc) and through some means of compilation or generation, producing a set of executables and other files which can be used to perform a runtime function

What is a Package? Includes output from the build and other items like scripts, database items, documents, etc that create a set of deliverables that run in a test or production environment

Software Testing Essentials

QSIT Copyright

42

www.qsitglobal.com

Page:21


Testing induction Software Configuration Management Version Numbering: 

Build version number will have the following components



Product name Eg: CMM, BSC, FIN,..



Major Number - Identifies major milestones in the product



Minor Number – Identifies intermediate milestones on the product road map



Upgrade Number - These are typically incremental on the intermediate milestones and are built on demand



Patch Number – Identified ad-hoc fixes made for immediate deployment to customer sites. These are not planned releases

Software Testing Essentials

43

www.qsitglobal.com

Configuration Management Functions

Team

Components

Process

Versions and Configurations

Audit History

Baselines

Traceability

Project Context Information Repository

Logging

Structure

Control

System Model

Access Control

Interfaces

Change Requests

Relationships

Bug Tracking

Selection

Change Propagation

Consistency

Partitioning

Construction

Accounting

Building

Statistics

Snapshots

Status

Optimization

Reports

Impact analysis Regeneration

Software Testing Essentials

QSIT Copyright

44

www.qsitglobal.com

Page:22

Test Process  

Test Process

Advertisement