Testing induction
Testing Essentials Unit 12 Introduction to the Automation QSIT- Quality Solutions for Information Technology Pvt. Ltd., A 806 & 807, Mittal Tower, BANGALORE- 560 001,INDIA URL: www.qsitglobal.com PH: +91 80 41134334 Fax: +91 80 25597445
Software Testing Essentials
1
www.qsitglobal.com
Basic Automation Glossary
Test Case – Definition – Set of Predefined inputs to Produce a definitive output, organized into a sequential execution
Types of Test case – Manual – Automation
Software Testing Essentials
QSIT Copyright
2
www.qsitglobal.com
Page:1
Testing induction Manual Testing
Draw backs: – Hard to repeat – Not always reliable – Costly – Time consuming – Certain scenarios not practically possible
Software Testing Essentials
3
www.qsitglobal.com
Test Automation
Need for Automation: –
– –
Develop additional test cases to achieve better coverage Time available for Ad-hoc Testing Process improvement in Manual testing
Software Testing Essentials
QSIT Copyright
4
www.qsitglobal.com
Page:2
Testing induction Automation Testing
Benefits of Automation: – Reliable – Good return on investment – Run attended anytime – Able to simulate scenarios
Software Testing Essentials
5
www.qsitglobal.com
Test Automation
Software test automation refers to –
Activities and efforts that intend to automate engineering tasks and operations in a software test process.
Software test automation uses –
Well-defined strategies and systematic solutions.
Software Testing Essentials
QSIT Copyright
6
www.qsitglobal.com
Page:3
Testing induction Objectives of Test Automation
To free engineers from tedious and redundant manual testing operations
To speed up a software testing process, and to reduce software testing cost and time during a software life cycle
To increase the quality and effectiveness of a software test process by achieving pre-defined adequate test criteria in a limited schedule
To reduce manual testing activities and redundant test operations using a systematic solution to achieve a better testing coverage.
Software Testing Essentials
7
www.qsitglobal.com
Test Automation Process
Plan Software Test Automation Design Test Automation Strategies & Solutions
Select and Evaluate Available Software Testing Tools
Develop & Implement Test Automation Solutions Introduce and Deploy Test Automation Solutions Review and Evaluate Software Test Automation
Software Testing Essentials
QSIT Copyright
8
www.qsitglobal.com
Page:4
Testing induction Test Automation Process
Step #1: Test automation planning – This is the initial step in software test automation. The major task here is to come out a plan that specifies the identified test automation focuses, objectives, strategies, requirements, schedule and budget.
–
Step #2: Test automation Design – The primary objective of this step is to draw out the detailed test automation solutions to achieve the major objectives and meet the given requirements in a test automation plan.
Software Testing Essentials
9
www.qsitglobal.com
Test Automation Process
Step #3: Test tool development – At this step, the designed test automation solutions are developed and tested as quality tools and facilities. The key in this step is to make sure that the developed tools are reliable and reusable with good documentation.
Step #4: Test tool deployment – Similar to commercial tools, the developed test tools and facilities must be introduced and deployed into a project or onto a product line. At this step, basic user training is essential, and proper user support is necessary.
Software Testing Essentials
QSIT Copyright
10
www.qsitglobal.com
Page:5
Testing induction Test Automation Process Step #5: Review and evaluation – Whenever a new tool is deployed, a review should be conducted to identify its issues and limitations, and evaluate its provided features. The review results will provide valuable feedback to the test automation group for further improvements and enhancements.
Software Testing Essentials
11
www.qsitglobal.com
Test Process Tools Costs Automation involved in Automation
Test team shall research to ascertain and help shape management expectation. – 1.Improved requirements definition. – 2.Improved performance testing. – 3.Improved load/stress testing. – 4. Quality measurements and test optimization. – 5. Improved partnership with development team. – 6.Improved system development life cycle.
Outline phased implementation – Effort distributed – Cost Distributed
Get the large purchase discount.
Identify project where the tool can be used.
Number of licenses required concurrently.
Software Testing Essentials
QSIT Copyright
12
www.qsitglobal.com
Page:6
Testing induction Conclusion First , the test team could select a less expensive tool that supports test requirements adequately for the near term Second, it could outline the cost savings or performance enhancing benefits in a way that convinces management that the tool is worth the upfront investment Third, it could scale down the implementation of the test tool and plan for additional implementation during the next budget period.
Software Testing Essentials
13
www.qsitglobal.com
Classification of Software Test Tools
White Box Testing Tools Functionality Testing Tools Test Management Tools Test-Case Generation Tools Performance Testing Tools
Software Testing Essentials
QSIT Copyright
14
www.qsitglobal.com
Page:7
Testing induction Some test case for different types of testing
Software Testing Essentials
Functionality Reliability
Load / stress testing
Performance
Internationalization
Check whether log works Repeat log in operation in loop for 48 hrs Perform log in from 10000 clients Measure time taken for log in operations in different conditions Run log in operation from a machine running japanese language
15
www.qsitglobal.com
Skills required for automation Automation-First generation
Automation-second generation
Automation-third generation
Skills for test case
Skills for test case automation
Skills for test case automation
Skills for Framework
Scripting languages
Scripting languages
Scripting languages
Programming languages
Record-playback tools usage
Programming languages
Programming languages
Designing and architecture skills for framework creation
Knowledge of data generation techniques
Designing and architecture of the product under test
Generic test requirement for multiple products
Usage of the product under test
Usage of the framework
automation
Software Testing Essentials
QSIT Copyright
16
www.qsitglobal.com
Page:8
Testing induction Basic Principles of the test Automation tools
What is test Script? What are the verification Point? What are synchronization points?
Software Testing Essentials
17
www.qsitglobal.com
Verification points
Verification points verify that a certain action has taken place, or verify the state of an object
When you create a verification point‌ – You are capturing information about an object in the application to establish this as baseline information for comparison during playback
Software Testing Essentials
QSIT Copyright
18
www.qsitglobal.com
Page:9
Testing induction Test Script
A test script is a short program written in a programming language used to test part of the functionality of a software system
A written set of steps that should be performed automatically can also be called a test script, however this is more correctly called a test case
Software Testing Essentials
19
www.qsitglobal.com
Synchronization points
synchronization means the coordination of simultaneous threads or processes to complete a task in order to get correct runtime order and avoid unexpected race conditions
Software Testing Essentials
QSIT Copyright
20
www.qsitglobal.com
Page:10
Testing induction Tool classification
Object Based automation tools Bitmap Based automation tools
Software Testing Essentials
21
www.qsitglobal.com
Approaches for test automation
Record and playback Scripting Framework
Software Testing Essentials
QSIT Copyright
22
www.qsitglobal.com
Page:11
Testing induction Integration of the functional testing tools with test management tools
Best practices – – – – – – – – – – –
Document the process Manage expectations Use a pilot project Verify test tool compatibility Verify test tool upgrades Baseline the system setup Configuration before installing a new tool Avoid incompatible test tools Avoid unnecessary software installation in an already-base lined Environment Testing environment Understand the overall test program objectives
Software Testing Essentials
23
www.qsitglobal.com
Best practices cont…..
Remember that not everything should be automated Keep automation simple Follow test procedure design and development standards Conduct automated versus manual test analysis Conduct reuse analysis Encourage test team partnership with all teams involved in the system development life cycle Keep the communication flowing Ensure schedule compatibility Involve the customer from the beginning of the system development life cycle
Software Testing Essentials
QSIT Copyright
24
www.qsitglobal.com
Page:12
Testing induction Best practices cont‌.
Document and report defects Use an automated test expert or tool advocate to sell the benefits of the automated tool within the project team Clarify test team assignments Participate in user group meetings and testing discussions on the Web Suggest test tool improvements to the vendor Become a beta tester Take advantage of the knowledge of the specialty topic experts (tool experts)
Software Testing Essentials
25
www.qsitglobal.com
Myths of Automation‌
Automated testing has the advantage over manual testing in that it is easily repeatable This is flavored when doing regression testing This however is not always the case as automated tests may be poorly written and can break during playback Since most systems are designed with human interaction in mind, it is good practice that a human tests the system at some point
Software Testing Essentials
QSIT Copyright
26
www.qsitglobal.com
Page:13
Testing induction White Box Testing Tools
Tests – Coding Standards – Memory Leaks – Code Complexity – Code Coverage
Tools available – Rational Purifier – C++ Test from Parasoft – Rational Quantifier – Jtest from Parasoft – Rational Purecoverage
Software Testing Essentials
27
www.qsitglobal.com
Functionality Testing Tools
Features: – Record and Playback capability – Object Recognition & Object Repository – Synchronization – Parameterization – Checkpoints or Verification points (Standard, Bitmap, Database etc) – Batch Testing – Error Handling – Configurable Options – Initialization Scripts – Support Client server and Web Based Applications
Software Testing Essentials
QSIT Copyright
28
www.qsitglobal.com
Page:14
Testing induction Functionality Testing Tools (Contd…) Functionality Testing Tools contd.
Features for web based applications – Functions to know when the page has finished loading. – Making the tool to wait until an image appears on the page – Can I extract data from the web page itself? E.g. the title? A hidden form element? – Testing Links – State of the web based objects (enabled, disabled, contain data etc) – Are there facilities that will allow me to programmatically look for objects of a certain type on a web page or locate a specific object?
Software Testing Essentials
29
www.qsitglobal.com
Functionality Testing Tools available today…
Rational Robot from IBM Win-Runner, Quick Test Professional from HP Silk-test from Borland Test-Partner from Compuware Etc…
Software Testing Essentials
QSIT Copyright
30
www.qsitglobal.com
Page:15
Testing induction Functionality Testing Tools available today… Test Management Tools
Test Management Tools address – Requirements – Managing manual test suites, scripts – Managing Test Automation Scripts – Test Execution Activities – Generation of various reports on execution – Defect Tracking Tools available… – Rational Test-Manager from Rational – Test Director from Mercury etc
Software Testing Essentials
31
www.qsitglobal.com
Test Case Generation Tool
Generate Test Cases based on inputs passed on to the tool.
Tools available… – Caliber RBT
Software Testing Essentials
QSIT Copyright
32
www.qsitglobal.com
Page:16
Testing induction Performance Testing Tools
Capacity – Capacity means whether the web application is able to handle large number of concurrent users. Throughput – Throughput means whether the number of clients requests processes within a certain time. Response Time – Response time is the time from when the browser has finished sending a request to when it starts receiving the response from the server. – Response time and capacity both are inversely proportional to each other.
Software Testing Essentials
33
www.qsitglobal.com
Performance Testing Tools contd.
Load-Runner from HP Rational Performance from IBM Silk-Performer from Borland Etc..
Software Testing Essentials
QSIT Copyright
34
www.qsitglobal.com
Page:17
Testing induction Essential needs of Test Automation •
A dedicated work force for test automation
•
The commitment from senior managers and engineers
•
The dedicated budget and project schedule
•
A well-defined plan and strategy
•
Talent engineers and cost-effective testing tools
•
Maintenance of automated software tests and tools
Software Testing Essentials
35
www.qsitglobal.com
Issues in Software Test Automation ♦ ♦
Poor manually performed software test process Late engagement of software test automation in a software product life cycle
♦
Unrealistic goals and unreasonable expectations
♦
Organization issues
♦
Lack of good understanding and experience of software test automation
Software Testing Essentials
QSIT Copyright
36
www.qsitglobal.com
Page:18
Testing induction Benefits of Test Automation
Speed and Accuracy – It can be as much as 50 times faster, depending upon the speed of the driver machine and the speed of the application to process information
Accessibility – Automation tools allow access to objects, data, communication protocols, and operating systems that manual testers cannot access
Accumulation – Once tests are developed, long-term benefits are derived through reuse
Software Testing Essentials
37
www.qsitglobal.com
Benefits of Test Automation contd.
Manageability – Ability to manage artifacts through automation tools
Discovery of issues – Automated testing assists with the discovery of issues early in the development process, reducing costs
Repeatability – An automation suite provides a repeatable process for verifying functionality on the functional side and scalability on the performance side
Software Testing Essentials
QSIT Copyright
38
www.qsitglobal.com
Page:19
Testing induction Intangible Benefits of Test Automation (contd..)
Formal process – Automation forces a more formal process on test teams, due to the nature of the explicitness of the artifacts and the flow of information that is needed
Retention of customers – Performing correct and systematic automated testing helps assure a quality experience for the customer – both internal and external
Software Testing Essentials
39
www.qsitglobal.com
Intangible Benefits of Test Automation (contd..)
Greater job satisfaction for Testers – The Test Engineers no longer manually execute the same test cases over and over – They would utilize a programming-like IDE and language that is more challenging, rewarding and portable to other positions
Software Testing Essentials
QSIT Copyright
40
www.qsitglobal.com
Page:20
Testing induction Causes for failure in Test Automation Lack of structured automation methodology Test automation is not treated as a project with proper project planning (i.e. scope, resources, time-to-market) Testing is performed at the end of the development cycle (the waterfall method) No modularization (use of functions) in automation scripts Test engineers are untrained in tool interface and programming techniques After initially creating automation suite, customer does not maintain the suite for future builds
Software Testing Essentials
41
www.qsitglobal.com
Solutions to the reasons for failure
Implement a pragmatic approach to testing that is – Manageable, such that the project can be decomposed into modular, defined tasks with assigned resources and timelines –
Repeatable, such that others can easily carry forward the process that has been defined
–
Measurable, such that the effort is quantifiable how many defects found in each stage
what is trend of different severities of defects
how close is the testing cycle to completion
how long does a transaction take to complete
Software Testing Essentials
QSIT Copyright
42
www.qsitglobal.com
Page:21
Testing induction Solutions to the reasons for failure
Implement a pragmatic approach to testing that is – Improvable, such that each build becomes more efficient in producing defects – The goal of this measurable and improvable process is to produce more defects in the testing life cycle so that less are found in production –
Automated, to build a data-driven regression and scalability/performance suite that takes advantage of the best-of-breed testing software
Software Testing Essentials
43
www.qsitglobal.com
Myths and realities of automation
An automated test tool cannot be expected to support 100% of the test requirements of any given test effort Automated testing my increase depth and breadth of test coverage, yet there is still not enough time or recourses to perform a 100% exhaustive test
Software Testing Essentials
QSIT Copyright
44
www.qsitglobal.com
Page:22
Testing induction Different types of tools : Cross-Platform tools Eggplant : Automated user-interface testing of a device running on any platform that uses the Virtual Network Computing (VNC) protocol Squish : Automated GUI testing tool on Windows, Linux/Unix, Mac OS X, embedded Linux for Qt, Web/HTML, Java/SWT, Java/Swing, Mac OS X, Tk, 4JS, XView, ... Silk Test: Is a leading functional GUI testing tool for Windows, and several flavors of UNIX
Software Testing Essentials
45
www.qsitglobal.com
Testing Tools Windows test tools IBM Rational Functional Tester (RFT) is a leading Functional GUI testing tool, supports testing Java, .NET, Web Apps(IE, Firefox, Mozilla, Netscape), Siebel and SAP GUI environments. Badboy is an inexpensive automated testing tool targeted specifically at testing of web applications. KeepLiquid TestAutomation is a keyword and data driven GUI test automation tool. NUnitForms is an open source extension for unit testing. QuickTest Professional (QTP) is a leading functional GUI testing tool. Ranorex is a free GUI test and automation Library for C++, Python and for all .NET languages. SilkTest is a leading functional GUI testing tool. TestDrive-Gold, code-free testing of GUI and browser applications. TestPartner an automated functional and regression testing tool from Compuware Corporation. WinRunner is a leading functional GUI testing tool. Wintask is an automation and regression test tool for GUI. TestComplete is functional GUI testing tool. Visual_Test from Microsoft / Rational was the first automation testing tool. Visual Studio Team Test is the successor to Visual_Test from Microsoft.
Software Testing Essentials
QSIT Copyright
46
www.qsitglobal.com
Page:23
Testing induction Testing Tools Web only test tools Selenium - Selenium tests run directly in a browser. Runs in IE, Mozilla and Firefox on Windows, Linux, and Macintosh. Watir - An opensource webtesting framework written in Ruby. Works only under Windows/IE6. WET Web Tester - An opensource webtesting framework inspired by Watir.Includes a script generation utility Sahi - Open source, java based testing tool for automated testing of web/html/ajax applications Mercury Interactive Tools - Win Runner, QTP Compuware Tools - QA Tester Empirix Tools - eTester [edit] Related tools AutoHotkey is a free GUI automation/scripting tool for windows applications, source available under the GPL. CodePro AnalytiX - Automated Java testing and sofware quality tools for Eclipse. Jameleon is an automated testing framework that uses plug-ins to test database, desktop, web, and mainframe software applications. Used mainly for integration, regression, functional, and acceptance testing at the code/object level.
Software Testing Essentials
QSIT Copyright
47
www.qsitglobal.com
Page:24