Automate Data Warehouse ETL Testing & Migration Testing The Agile Way

Fast forward Data Warehouse/ETL Testing Migration Testing The Agile Way

ETL Testing & Monitoring Platform

Author: Sandesh Gawande

About: Torana, Inc.

CTO -iCEDQ

 Year established 2005

Torana, Inc.

 Stamford, CT

Email: Sandesh.g@ToranaInc.com

 Developers of Big Data Integration & Data Migration, Data warehouse/ETL Testing Software

Office: 203 666 4442 Twitter: @sandesh_gawande Skype: Sandesh.Gawande.ToranaInc LinkedIn: https://www.linkedin.com/in/sandeshgawande-1a25757

 Fortune 500 customers in Banking, Insurance, Healthcare, e-commerce, Manufacturing

10,000x

Find the issues early or the costs are too High

100x

1x Development & QA Testing

Customer Finds It Post Release

Leads to Litigation, Fines, Reputation Loss

Finding Issues in QA Stage is the Best, but QA is…  Not Agile

Not Agile

 With waterfall approch its too late...

 Not Automated  Manual data checks → Wasted Time  No repeatability or consistency

Manual & Slow Not Collaborative Or Feedback

 No way to test millions of rows  Wrong focus on creating scripts rather than the business problem

 Cannot reconcile data across systems (e.g. Files vs.

Database)

 Not Collobrative  QA teams work in isolation  No feedabck to developers or business users

No Transparency or Compliance

 Disoragnized

 No Transparancy  Late Discovery of Issues  Project Failure or High Costs

But Why is it so Difficult to Automate ETL Testing?  ETL Processes don’t have screens  Conventional QA Automation product were designed for screen based testing

 New Concepts  Source Data + Transformation = Target Data  Quality of an ETL Process = Expected Data vs. Actual Data  Most developers are from traditional software development  New to concepts such as data reconciliation for ETL Testing  Mix up of QA/QC concept with Data Quality

 High Volume of Data (Millions of rows)  Since the source data and target data could be in two different systems reconciliation is difficult

Introducing iCEDQ…

An automation platform for: A.Data Warehouse Testing • ETL Testing • MDM Testing • Data Integration Testing

B.Data Migration Testing C.Data Monitoring

Automated & Fast

Agile

Collaborative Feedback & Transparency

iCEDQ has in-Memory Rules Engine

 It Tests ETL Transformations by  Validating the output data generated by ETL

Validation Rule

ETL  It Tests ETL Transformations by  Reconciling Source Data Vs. Target Data  …

Reconciliation Rule

A

Data Warehouse/ETL Test Automation Data Sources

ETL

Data Warehouse Biz Reconciliation Test

Tech Reconciliation Test Tech Validation Test

Biz Validation Test

Validation Tests Technical Validation Rule Validate incoming data before processing. Test for…

• Data format • Nulls • Data types • more

Business Validate Rules Business rules based validation will indicate if there is an data issue becuase of ETL processes, Source data or wrong requirements...

• Check if Net Amount =? Gross Amout – (Taxes + Fees + Commisions)

Reconciliation Tests Technical Reconciliation Rules These rules test specific to an ETL process which is doing transformation…

• An ETL processes calculating end of day balances from daily transactions can be tested. Sum of todays transactions =? Today’s End of Day balance – Yesterday’s End of Day balance

Business Reconciliation Rules These tests are designed to test the overall system independent of the ETL processes, Source data or business requirements

B

Data Migration Test Automation Legacy System

ETL

New System

Initial Reconciliation Test

Post Reconciliation Test Initial Migration Testing

Post Migration Testing

1st Create the data structures in the target system. Ex. Table, columns. 2nd copy the initial data from the legacy system to the new database • iCEDQ can validate the tables, columns, data types & precision • Reconcile the legacy vs. target data to make sure they have the same initial state

Once the initial state is populated & tested. The post migration phase involves. Feeding the same data or triggering of same business processes in legacy system and the new system.

• iCEDQ can reconcile the data to make sure the after running the business processes the data generated same • Because regardless of the system change, unless there is a business rule change the net output from business point of view must be same

C

Production Data Monitoring Automation Source→ Stage

Star t Process Load Stage Customer

Process Load Stage Policy

Process Load Stage Claims

Stage→ Data Warehouse

Process Load Dim Customer Process Load Month Policy

Data Warehouse →Data Marts

Data Marts →Reports / Extracts

Process Load Dim Customer

Sto p

Process Load P&L Process Load Month Claims

Process Load Daily Claims iCEDQ

Monitoring in Series Embed iCEDQ Rules in the batch process • If Audit Fails the users are notified and the process can be stopped automatically

Monitoring in Parallel The Audit Rules are run in parallel to the batch process • If Audit Fails the users are notified but the process is not stopped automatically

iCEDQ is Agile  Test processes in parallel to the development pipeline  No reasons to wait!

Development Pipeline User Story

Tech Requirements

Mapping Document

iCEDQ Rule 1

QA Pipeline Audit Requirements

ETL Process

Test Case

iCEDQ Rule 2

…

iCEDQ-Central Repository & Collaborative  Centralized Repository for Rules Library  An collaborative environment to work together

 Work together regardless of the  Location  time  Role

iCEDQ-Feedback & Transparency

 Dashboard  Fails & Custom Reports  Integration with ALM & Issue management  Auto Notification  Ability to drill down to an defect  Audit Logs & execution histroy…

iCEDQ-What changed?

Before

After NO

Very Complicated 1000…

Reconcile Across Files & database

SQL

Low

NO SQL or Simple SQL Millions…

Test millions of rows Cost

100%

YES

60% High

Test Coverage

NO

Repeatability & Consistency

YES

NO

Scheduling

YES

Desktop Based NO High

Test Execution Transparency & Reporting Cost of Defect

Server Based YES Low

NO

Regression Testing & Audit

YES

NO

Production Monitoring

YES

Who uses iCEDQ?



Stock Exchange



Banks



Insurance



Manufacturing



Healthcare



E-Commerce



Manufacturing



…

iCEDQ Healthcare Client iCEDQ Usage  iCEDQ was used for Migration Testing  Test provider data migration from Mainframe to MDM

 iCEDQ Enterprise Data Warehouse Testing  Test Members Data, Enrolment data, Plans Data, Claims Data load from Legacy to (Enterprise Data warehouse)EDW & Health Rules to EDW

 iCEDQ to Validate External Feeds  Test data feeds to State of Maryland, CMS (Centers for Medicare & Medicaid Services)

iCEDQ Feedback  Helped Finalize Requirements  It found anomalies in the requirements and mapping documents and provided feedback

 Helped Test Automation  It was able to automatically reconcile feeds from legacy as well as new system.  This was impossible to test manually

 Transparency to management  It linked with defect management system and auto generated status

ETL Testing & Monitoring Platform

Fast forward Data Warehouse & Migration Testing

Request 30 Day Free Trial