DATA ENGINEERING ECOSYSTEM EVOLUTION: FROM RDBMS TO CLOUD MIGRATION

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 12 Issue: 07 | Jul 2025

p-ISSN: 2395-0072

www.irjet.net

DATA ENGINEERING ECOSYSTEM EVOLUTION: FROM RDBMS TO CLOUD MIGRATION Promod Kumar B M 1, Shruthi V P 2 1 Assistant Professor, Department of Computer Science and Engineering, P.E.S. College of Engineering, Mandya,

Karnataka, India.

2 Department of Computer Science and Engineering, P.E.S. College of Engineering, Mandya, Karnataka, India.

---------------------------------------------------------------------***---------------------------------------------------------------------

Abstract - In the era of digital transformation, the

Key Words: Cloud Migration, RDBMS, Data Engineering, Azure Databricks, ADLS, Data Masking, Banking Sector, AI/ML Integration.

evolution of data engineering ecosystems has fundamentally reshaped how organizations manage, process, and analyze data. This research delves into the transition from traditional on-premises Relational Database Management Systems (RDBMS) to modern, cloud-native architectures, with a specific focus on India's growing in fluence as a global IT and data engineering hub. The study shows how cloud-based platforms empower enterprises to unlock the full potential of their data, fostering real-time analytics, automation, and innovation. Leveraging cutting-edge technologies such as Microsoft PowerApps, Azure Data Lake Storage (ADLS) Gen2, Azure Data Factory (ADF), Databricks, Power BI, and Artificial Intelligence/Machine Learning (AI/ML), this project builds an end-to-end scalable, secure, and highly efficient data pipeline. The pipeline ensures robust data ingestion, transformation, and visualization while incorporating advanced data governance techniques, including dynamic and static data masking, encryption-atrest, and role-based access control (RBAC) to safeguard Personally Identifiable Information (PII) and ensure regulatory compliance with frameworks like GDPR and RBI guidelines. Using the Indian banking sector as a case study— a sector undergoing rapid digital adoption— the research evaluates the economic and technological impact of cloud migration. Key benefits identified include significant operational efficiency, infrastructure cost reduction, improved scalability and elasticity, enhanced data security, and enriched customer experiences through data-driven decision-making. Migration also enables seamless integration of disparate data sources, fostering unified analytics and intelligent automation through AI/ML model deployment within Databricks notebooks. Moreover, the study highlights how the democratization of cloud technologies levels the playing field for small and medium enterprises (SMEs), allowing them to compete on a global scale with enterprise-grade capabilities. Challenges such as data silos, latency in processing, regulatory compliance, and legacy system compatibility are addressed through modular architecture, containerization, CI/CD pipelines, and metadata-driven orchestration in ADF.

© 2025, IRJET

|

Impact Factor value: 8.315

1.INTRODUCTION The rapid evolution of data engineering ecosystems has transformed the landscape of enterprise data management. As businesses increasingly rely on data for strategic decision-making, the limitations of traditional Relational Database Management Systems (RDBMS) have become more pronounced. While RDBMS platforms have long served as the backbone of data storage and management, they often struggle with scalability, data silos, performance bottlenecks, and lack of support for real-time analytics and big data processing. These constraints have catalyzed a global shift toward cloud-based data platforms that offer elasticity, pay-as-you-go pricing models, seamless integration capabilities, and high availability.

Cloud computing has emerged as a foundational enabler for modern data engineering, allowing organizations to construct resilient, scalable, and cost-effective data pipelines.

Cloud platforms such as Microsoft Azure provide a comprehensive suite of services—ranging from storage (Azure Data Lake Storage Gen2) and compute (Databricks) to orchestration (Azure Data Factory) and visualization (Power BI)—that collectively support the end-to-end lifecycle of data engineering. These services are further enhanced by native integration with AI/ML tools and frameworks, enabling businesses to operationalize advanced analytics models for real-time insights.

India, recognized globally for its robust IT capabilities and a growing ecosystem of cloud professionals, is uniquely positioned to lead this digital revolution. The country's extensive talent pool, supportive policy frameworks (e.g., Digital India initiative), and widespread digital adoption have accelerated cloud migration across sectors. Among these, the banking and financial services industry (BFSI) stands out as a prime candidate due to the vast volume and sensitivity of data it handles. From real-time transaction processing and loan approvals to regulatory reporting and personalized customer engagement, banks require agile and

|

ISO 9001:2008 Certified Journal

|

Page 645

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 12 Issue: 07 | Jul 2025

p-ISSN: 2395-0072

www.irjet.net

secure data architectures that can adapt to evolving business needs.

enhanced scalability, operational agility, improved data security, and cost efficiency. It emphasizes how cloud solutions help banks respond to changing customer expectations and competitive pressures through rapid deployment of digital services. The study also identifies challenges related to the migration of legacy infrastructure, particularly around data integration, vendor lock-in, and ensuring compliance with stringent regulatory frameworks like Basel III and GDPR.

This research explores the design and implementation of a modern, cloud-native data engineering framework tailored to the Indian banking sector. The proposed solution leverages PowerApps for business process automation, Azure Data Lake Storage (ADLS) Gen2 for scalable and secure data storage, Databricks for large-scale data transformation and machine learning workflows, and Power BI for rich, interactive analytics. Azure Data Factory orchestrates the entire data pipeline with metadata-driven workflows and error handling capabilities.

Disadvantage: The research predominantly centers on Tier 1 and multinational financial institutions, offering limited insight into the cloud adoption barriers encountered by small- and medium-sized banks, such as budget constraints, limited cloud expertise, and integration complexity with legacy systems.

A critical aspect of the solution is data privacy and compliance. Banks must comply with stringent global and regional regulations such as the General Data Protection Regulation (GDPR), Reserve Bank of India (RBI) cybersecurity guidelines, and the Personal Data Protection Bill (PDPB). This research emphasizes the implementation of advanced data masking techniques, including both dynamic and static masking, tokenization, and role-based access controls to protect Personally Identifiable Information (PII) and prevent unauthorized data exposure.

These techniques ensure that sensitive information such as account numbers, PAN, Aadhaar details, and customer names is obfuscated or anonymized during analytics and reporting, without impacting data utility for business insights.

[2] Oliver Wyman, Economic Impact of Cloud Computing in India. Oliver Wyman, 2021.

This report analyses the macroeconomic benefits of cloud computing across India's digital ecosystem. It estimates that the adoption of cloud technologies could contribute approximately 8% to India’s GDP by 2026, driven by enhanced productivity, digital innovation, and the growth of emerging technologies like AI and IoT. Disadvantage: While economically insightful, the study lacks granular technical details on the implementation of cloud infrastructure in banking environments. It does not address operational challenges such as data migration, architectural complexity, and cloud security enforcement.

Furthermore, the integration of AI/ML models—developed and deployed using Databricks—adds a predictive layer to the system, enabling use cases like fraud detection, credit risk scoring, and churn prediction. The scalable nature of the cloud environment ensures that these models can be trained and retrained efficiently on large datasets with minimal infrastructure overhead.

[3] Gartner Research, Data Engineering Ecosystem Evolution, 2023.

This research outlines the transition from legacy onpremises data architectures to modern, cloud-native data engineering platforms. The study underscores the increasing role of technologies such as Azure Data Lake Storage Gen2, Databricks, and Power BI in enabling scalable, modular, and high-performance data pipelines. It also elaborates on the importance of data democratization, real-time processing, and metadata-driven data orchestration in modern enterprises.

Disadvantage: The publication lacks detailed use cases and fails to provide hands-on implementation examples or best practices from real-world deployments, making it less actionable for technical practitioners.

In conclusion, this research presents a comprehensive and future-proof data engineering architecture that not only modernizes legacy systems but also empowers banks to harness the full value of their data assets. The proposed

solution serves as a blueprint for digital transformation in the BFSI domain and holds significant relevance for other industries facing similar challenges in data scalability, security, and compliance.

2. LITERATURE REVIEW [1] Capgemini Research Institute, Cloud Adoption in the Banking Sector. Capgemini, 2022.

[4] McKinsey & Company, AI/ML in Banking: Fraud Detection and Risk Assessment, 2022.

This study provides a comprehensive overview of the increasing adoption of cloud technologies within the banking and financial services sector. The research outlines key drivers behind this transformation, including the need for

© 2025, IRJET

|

Impact Factor value: 8.315

This report delves into the growing role of Artificial Intelligence and Machine Learning in transforming the banking sector. It discusses how ML algorithms are used for real-time fraud detection, credit risk scoring, loan

|

ISO 9001:2008 Certified Journal

|

Page 646

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 12 Issue: 07 | Jul 2025

p-ISSN: 2395-0072

www.irjet.net

approval automation, and customer segmentation. The report stresses the necessity of integrating AI/ML workflows into cloud-based data platforms to leverage real-time insights and predictive analytics at scale.

parallel processing, Delta Lake support, and auto-scaling clusters for on-demand resource provisioning.

Use Case: Easily process hundreds of millions of banking transactions per day across regions without downtime or performance degradation.

Disadvantage: Although insightful on business use cases, the study does not explore the technical complexity of implementing and maintaining AI/ML models in a production-grade cloud infrastructure. Topics like model versioning, data drift monitoring, ML Ops, and cloud resource optimization are not addressed.

2. Cost Efficiency via Serverless and Pay-as-You-Go Cloud-native components operate under elastic resource allocation, ensuring that compute and storage resources are billed only when used.

Databricks supports job clusters and automated job scheduling, eliminating the need for persistent infrastructure.

[5] National Institute of Standards and Technology (NIST) – Special Publication 800-122, Data Masking Techniques for Privacy Preservation in Cloud Environments,2020.

Impact: Reduces capital expenditure (CapEx) and operational costs (OpEx), while improving cost forecasting and resource utilization.

This publication presents a framework for protecting Personally Identifiable Information (PII) in cloud environments. It outlines key data protection methods, including Static Data Masking (SDM), Dynamic Data Masking (DDM), Tokenization, and Format-Preserving Encryption (FPE). The guidelines emphasize the trade-off between data usability and privacy, advocating for rolebased access control, encryption-at-rest, and data anonymization strategies in compliance with global regulations such as GDPR, HIPAA, and India’s PDP Bill.

Disadvantage: While comprehensive in masking techniques, the document does not explore integration strategies with advanced analytics or AI/ML pipelines, where masked or anonymized data may impact model accuracy or lead to training inconsistencies.

3. Real-Time Analytics & AI-Driven Insights Power BI enables seamless visualization of banking KPIs such as credit risk scores, loan defaults, fraud detection alerts, and customer segmentation.

ML Models on Databricks allow for streaming analytics, predictive modeling, and integration with tools like Azure Machine Learning or MLflow for experimentation and model lifecycle management.

Example: Real-time fraud detection by scoring transactions as they flow through the pipeline using trained ML models.

3. METHODOLOGY

4. Enhanced Security & Regulatory Compliance

The proposed system transitions from traditional RDBMS to a cloud-native data engineering ecosystem, integrating cutting-edge services and tools including PowerApps, Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Databricks, and Power BI. This architecture is designed to overcome the challenges of legacy systems by ensuring scalability, security, real-time analytics, and regulatory compliance — all critical to modern banking infrastructure.

RBAC (Role-Based Access Control) ensures that only authorized users can access sensitive datasets, based on roles (e.g., Compliance Officer, Data Analyst, Risk Manager).

Data Encryption: Both at rest (Azure Storage Encryption) and in transit (TLS 1.2+) to protect customer data such as Aadhaar numbers and account details.

Multi-Factor Authentication (MFA) and Azure Active Directory integration safeguard login credentials and access.

Key Components & Features: 1. Scalability with ADLS Gen2 and Azure Databricks

Auditing & Logging: Activities are tracked via Azure Monitor and Databricks audit logs, essential for forensic analysis and compliance.

ADLS Gen2: Provides hierarchical namespace storage on top of Blob Storage. It enables massive-scale data ingestion (structured, semi-structured, and unstructured data) with optimized throughput and cost-effective storage tiers (Hot, Cool, Archive).

Compliance: Aligned with standards such as GDPR, RBI IT Guidelines, and PCI-DSS.

Azure Databricks: Built on Apache Spark, it supports distributed computing for big data pipelines. It enables

© 2025, IRJET

|

Impact Factor value: 8.315

|

ISO 9001:2008 Certified Journal

|

Page 647

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 12 Issue: 07 | Jul 2025

p-ISSN: 2395-0072

www.irjet.net

5. Integrated Data Masking for Privacy Preservation Model

Format-Preserving Encryption (FPE): Encrypts data while retaining its original format (e.g., masking a PAN number ABCDE1234F to XYZWQ5678R).

Static Data Masking (SDM): Applied to backup or nonproduction environments for PII redaction (e.g., replacing real PAN numbers with dummy values).

Tokenization: Replaces sensitive data with non-sensitive tokens (e.g., substituting actual account numbers with tokens in analytics workflows).

Dynamic Data Masking (DDM): Used for production analytics where role-based views are necessary. For example, a data analyst may see masked account numbers, while a financial controller sees full data.

Compliance: Ensures alignment with GDPR, RBI’s data localization norms, and SOC 2 standards.

4. PROJECT CONTEXT AND PROTOTYPE APPROACH

Integration with AI/ML: Masked datasets retain referential integrity, allowing model training without exposing real PII.

This project aims to demonstrate the feasibility and functionality of a robust data engineering ecosystem, conceptually mirroring the architecture of a productiongrade cloud solution like those built on Microsoft Azure.

Given the significant investment typically required for such an Azure-based system, especially for preliminary proof-ofconcept, we have opted to utilize readily available Google tools for this prototype model.

3.1 FLOWCHART Data Collection: PowerApps captures real-time banking transaction data, storing it in SharePoint.

Cloud Migration: Azure Data Factory (ADF) ingests structured and semi-structured data into ADLS Gen2.

This approach allows us to:

Data Processing & Transformation: Databricks processes raw data, performs ETL operations, applies data masking (SDM/DDM/FPE), and enhances data security.



AI/ML Integration: AI/ML models analyse masked transaction patterns for fraud detection and risk assessment.

Minimize Initial Costs: To demonstrate the core data flow and functionality without incurring substantial upfront expenses.



Data Storage & Distribution: Processed data is stored in ADLS Gen2, enabling seamless access for business applications.

Rapidly Prototype and Iterate: To quickly build and test the data pipeline, allowing for agile adjustments and refinement.



Focus on Conceptual Understanding: To emphasize the logical flow of data, transformation processes, and analytical outputs, rather than the specifics of a particular vendor's platform.



Provide Proof of Concept: Before any large-scale investments are made, provide a functional demonstration of the proposed system.

Visualization & AI: Power BI dashboards and AI-driven insights support data-driven decision-making.

[PowerApps] ↓ [Azure Data Factory] --> [ADLS Gen2] ↓ [Azure Databricks] (ETL/ELT, ML Models, Masking) ↓ [Delta Lake] ↓ [Power BI Dashboards]

It's important to note that this prototype, while functionally representative, is not intended to be a production-ready system. Google tools, while powerful for prototyping, have limitations in scalability, performance, and security compared to enterprise-grade cloud platforms like Azure.

Specifically, we have replaced the following Azure components with their Google equivalents:

3.2 Data Masking Implementation: Static Data Masking (SDM): Irreversibly anonymizes PII in non-production environments (e.g., replacing actual customer names with pseudonyms).

Dynamic Data Masking (DDM): Restricts real-time exposure of sensitive fields (e.g., masking credit card numbers as ********-****-1234 for unauthorized users).

© 2025, IRJET

|

Impact Factor value: 8.315

|



PowerApps: Replaced with Google Forms, enhanced with Google Apps Script for custom functionality.



Azure Data Lake Storage Gen2 (ADLS Gen2): Replaced with Google Sheets for initial data viewing and a simulated data lake storage.

ISO 9001:2008 Certified Journal

|

Page 648





International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 12 Issue: 07 | Jul 2025

p-ISSN: 2395-0072

www.irjet.net 3. Rapid Prototyping:

Azure Databricks: Replaced with Google Colab, leveraging PySpark and Spark SQL for data processing and masking.

o

Power BI: Replaced with Looker Studio for data visualization and analysis.

4. Learning and Experimentation:

This approach allows us to demonstrate the core concepts of data ingestion, processing, masking, and visualization using a cost-effective and accessible toolset. The insights gained from this prototype will serve as a foundation for future development and implementation in a production environment, potentially utilizing Azure or a similar platform."

o

Google tools provide a safe and costeffective environment to learn and experiment with data engineering concepts.

o

You can explore different data processing and visualization techniques without financial risk.

Key Points to Emphasize: 

Purpose of the Prototype: Clearly state that it's a proof of concept.



Cost-Effectiveness: Highlight the financial benefits of using Google tools.



Conceptual Representation: Stress that the prototype demonstrates the overall architecture.



Limitations: Acknowledge the limitations of Google tools compared to enterprise-grade solutions.



Future Scalability: Indicate that the prototype will guide future production deployments.

5. Collaboration: o

o

o

Azure services (PowerApps, ADLS Gen2, Databricks, Power BI) are enterprise-grade but come with significant costs, especially for large-scale deployments.

Google tools (Forms, Sheets, Colab, Looker Studio) are free for basic use, making them ideal for prototyping and proof-of-concept projects.

o

Google tools have a low barrier to entry.

They are user-friendly and require minimal setup.

o

Colab provides a pre-configured environment for Python and Spark, eliminating the need for complex installations.

|

Before investing heavily in Azure, it is best to provide the business a proof of concept.

Limitations of Google Tools:

2. Accessibility and Ease of Use:

© 2025, IRJET

For individual projects, or for smaller companies, the cost of the Azure ecosystem could be prohibitive.

7. Proof of Concept:

1. Cost Savings:

o

Google tools facilitate easy collaboration.

You can share forms, sheets, and Colab notebooks with team members.

6. Resource Constraints:

Why Use Google Tools Instead of Paid Software?

o

Google tools allow for quick iteration and development. You can rapidly build and test your data pipeline without waiting for procurement or complex configurations.

Impact Factor value: 8.315



Scalability: Google Sheets and Colab limitations in handling massive datasets.



Performance: Colab's performance may not match dedicated cloud services like Databricks.



Security: Google tools may not meet the security requirements for highly sensitive data.



Features: Paid tools offer more advanced features and integrations.



Enterprise support: Google tools do not have the same level of enterprise level support that a paid Azure service would have.

have

In essence, Google tools are excellent for prototyping and learning, but paid cloud platforms like Azure are better suited for production environments that require scalability, performance, and robust security.

|

ISO 9001:2008 Certified Journal

|

Page 649

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 12 Issue: 07 | Jul 2025

p-ISSN: 2395-0072

Prototype Development (Detailed Steps):

with

Google

www.irjet.net

Tools

2. Create Google Form (Enhanced): o

Sections: Use form sections to organize questions logically. For example, a section for customer information and another for transaction details.

o

Conditional Logic: Implement conditional logic to show or hide questions based on user responses. For instance, show "Merchant" only if TransactionType is "Debit."

4.1 Phase 1: Data Collection (Google Forms & Scripting) 1. Define Data Schema (Elaborated): o

Data Dictionary: Create a data dictionary.

This will contain all the fields, data types, validation rules, and descriptions. This helps to maintain consistency and clarity.

o

Data Integrity: Consider data integrity constraints. For example:

o

TransactionID should be unique.

Implement checks or validations for uniqueness.

Descriptions: Add detailed descriptions to questions to provide context and instructions.

o

Required Fields: Mark essential fields as "required" to ensure data completeness.



o



CustomerID should reference an existing customer. (You can simulate this using a dropdown with predefined customer IDs).



Amount should be a positive number.



TransactionType should limited to predefined values.

be

Data Sensitivity: Identify sensitive data fields that will require masking (e.g., CustomerID, potentially Location or Merchant).

Fig 2: User Interface for Axis Bank Data Collection Form 3. Google Apps Script (Advanced): o

Fig 1: Axis Bank Customer Form Creation using Google Apps Scripts

© 2025, IRJET

|

Impact Factor value: 8.315

|

External Data Lookup: If you need to populate data from external sources (e.g., a customer database), use Google Apps Script to fetch data and populate form fields.

ISO 9001:2008 Certified Journal

|

Page 650

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 12 Issue: 07 | Jul 2025

p-ISSN: 2395-0072

o

www.irjet.net

Real-time Validation: Implement real-time validation as the user types, providing immediate feedback.

o

Automated Notifications: Send email notifications upon form submission.

o

Data Logging: Log form submission details to a separate sheet for auditing purposes.

o

Error Handling: Implement robust error handling to prevent script failures.

o

Data Relationships (Simulated): 

If you are simulating related tables, use VLOOKUP or INDEX/MATCH functions to create relationships between sheets.



Create a separate sheet for customer data and use VLOOKUP to retrieve customer information based on CustomerID.

4. Testing (Thorough): o

o

o

Edge Cases: Test with various edge cases, such as: 

Invalid data types.



Missing data.



Boundary amounts).



Long text inputs.

values

(e.g.,

zero

Performance: Test with a large number of simulated submissions to assess the form's performance.

Fig 3: Google Sheets-Axis Bank Customer Form Responses Data

User Experience: Evaluate the form's usability and make adjustments as needed.

2.

4.2 Phase 2: Data Storage and Initial View (Google Sheets) 1.

Google Sheet as Data View (Detailed): o

o

© 2025, IRJET

Organization (Advanced): o

Named Ranges: Use named ranges to make formulas more readable and maintainable.

o

Data Slicers: Use data slicers to create interactive filters.

o

Data Grouping: Group related columns for better organization.

Data Organization: 

Use data filters to quickly sort and filter data.



Apply conditional formatting to highlight important data points.



Freeze header rows and key columns for easy navigation.

4.3 Phase 3: Data Processing and Masking (Google Colab & PySpark) Google Colab Setup (Detailed): o

Environment Setup: Ensure that your Colab runtime is configured for Python 3.

Data Validation: 

Use Google Sheets' data validation features to enforce data integrity.

o

Dependency Management: Use requirements.txt to manage dependencies.



Create dropdown predefined values.

o

Version Control (Simulated): If you are working with a team, use Google Drive versioning to track changes.



Set data type restrictions.

|

Impact Factor value: 8.315

lists

for

|

ISO 9001:2008 Certified Journal

|

Page 651

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 12 Issue: 07 | Jul 2025

p-ISSN: 2395-0072

www.irjet.net

Connect to Google Sheets (Robust): o

Service Accounts: Use Google Cloud service accounts for more secure authentication.

o

Error Handling: Implement error handling for network issues and API errors.

o

Retry Logic: Add retry logic to handle transient errors.

o

User Defined Functions (UDFs): Create UDFs for complex data transformations.

Initialize Spark (Configured): o

Spark Configuration: Configure Spark settings for optimal performance.

o

Logging: Configure Spark logging to capture error messages.

Fig 5: Tabular Display of Axis Bank Customer Data in Google Colab Data Masking (Advanced):

Fig 4: Python Script in Google Colab for Reading Axis Bank CSV Data

o

Parameterization: Parameterize masking functions for flexibility.

o

Consistent Masking: Ensure consistent masking across different datasets.

o

Masking Metadata: Store masking metadata (e.g., masking rules) for auditing.

o

Tokenization Mapping: Store token mapping in a database or file for secure management.

Pandas to Spark DataFrame (Efficient): o

Schema Inference: Explicitly define the schema to avoid Spark's schema inference, which can be inefficient.

Data Processing (Comprehensive): o

Data Quality Checks: Implement data quality checks to identify and handle data errors.

o

Data Enrichment: Enrich data with external sources (e.g., using APIs).

o

Window Functions: Use Spark window functions for advanced analytics.

© 2025, IRJET

|

Impact Factor value: 8.315

Fig 6: PySpark Script for PII Masking and Output Table Visualization

|

ISO 9001:2008 Certified Journal

|

Page 652

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 12 Issue: 07 | Jul 2025

p-ISSN: 2395-0072

www.irjet.net

Save Output (Flexible): o

Partitioning: Partition the output data for efficient querying.

o

Data Formats: Use optimized data formats like Parquet.

o

Google Drive Integration: Save data to Google Drive folders for organization.

o

Storytelling: Create data stories communicate insights effectively.

to

Fig 8: Dashboard Analytics for Axis Bank Customer Financial Overview

Fig 7: Exporting Financial Analysis Results to CSV using PySpark and Pandas

5. CONCLUSION A Paradigm Shift in Secure Cloud-Driven Innovation

4.4 Phase 4: Data Visualization (Looker Studio) 1.

2.

3.

This project serves as a compelling demonstration of the profound impact that strategic cloud migration, coupled with advanced data masking techniques, can have on revolutionizing India's IT infrastructure and fostering a secure, data-driven ecosystem. In an era where data is the lifeblood of modern enterprises, particularly within the highly regulated banking sector, the ability to balance data utility with stringent privacy requirements is paramount.

Connect to Data Source (Reliable): o

Data Refresh: Configure data refresh schedules.

o

Data Blending: Blend data from multiple sources.

Create Visualizations (Interactive): o

Drill-down Reports: Create drill-down reports for detailed analysis.

o

Calculated Fields: Use calculated fields for custom metrics.

o

Interactive Filters: Create interactive filters and slicers.

o

Custom Visualizations: Explore custom visualizations.

The core strength of the proposed system lies in its comprehensive approach to data security, integrating a suite of masking technologies—Static Data Masking (SDM), Dynamic Data Masking (DDM), and Tokenization. By meticulously implementing these techniques, the system achieves a critical objective: enabling secure data sharing for advanced analytics while rigorously adhering to global data privacy regulations such as GDPR, RBI's data localization norms, and SOC 2. This multi-layered security framework not only mitigates the risks associated with data breaches but also fosters a culture of trust and transparency, essential for building robust customer relationships and maintaining regulatory compliance.

Analysis (Insightful): o

Trend Analysis: Analyze data trends over time.

o

Comparative Analysis: Compare data across different dimensions.

o

Anomaly Detection: Use Looker Studio's features to detect anomalies.

© 2025, IRJET

|

Impact Factor value: 8.315

The banking sector case study presented within this project provides a tangible illustration of how masked data can retain its intrinsic analytical value. Specifically, the system demonstrates the feasibility of leveraging masked transaction data for AI/ML-driven fraud detection and risk assessment. By employing sophisticated machine learning models on anonymized datasets, financial institutions can effectively identify and mitigate fraudulent activities without

|

ISO 9001:2008 Certified Journal

|

Page 653

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 12 Issue: 07 | Jul 2025

p-ISSN: 2395-0072

www.irjet.net

compromising customer privacy. This delicate balance between privacy preservation and innovation is a testament to the system's design and its potential to unlock new frontiers in data-driven decision-making.

Furthermore, this research underscores the compelling economic and technical alignment of cloud adoption with India's strategic objectives. By embracing cloud-based solutions, organizations can achieve significant cost efficiencies, enhance scalability, and accelerate innovation cycles. The pay-as-you-go model offered by cloud providers, coupled with the ability to dynamically scale resources, allows businesses to optimize their IT spending and invest in core competencies. The integration of advanced analytics tools, such as Power BI and Azure Databricks, empowers decision-makers with real-time insights, enabling them to respond swiftly to market dynamics and customer needs.

Beyond the immediate benefits to individual organizations, this project highlights India's potential to emerge as a global leader in the IT landscape. By fostering a culture of innovation and embracing cutting-edge technologies, India can attract foreign investment, stimulate economic growth, and create high-skilled jobs. The development of robust data protection frameworks, such as the one proposed in this project, is crucial for building trust and attracting international partnerships.

3.

Capgemini – “91% of banks and insurers have initiated cloud adoption.” (Illustrates industrywide cloud adoption trends)

© 2025, IRJET

|

Impact Factor value: 8.315

Wikipedia – “Unified Payments Interface (UPI).”

(Contextualizes the Indian digital payments landscape)

6.

NIST Special Publication 800-122 – “Guide to Protecting the Confidentiality of Personally Identifiable Information (PII).” (Provides foundational guidance on data masking and PII protection)

7.

Microsoft Azure Documentation – “Data Masking in Azure SQL Database and Azure Synapse Analytics.” (Specific guidance on Azure's data masking capabilities)

8.

Armbrust, M., Ghodsi, A., Xin, R. S., Zaharia, M., Franklin, M. J., Shenker, S., & Stoica, I. (2010). A Berkeley view of cloud computing. (Seminal paper on cloud computing concepts)

9.

Ghodsi, A., Zaharia, M., Xin, R. S., Davidson, A., Armbrust, M., Franklin, M. J., ... & Stoica, I. (2011). Matei zaharia: Spark: cluster computing with working sets. (Explains the foundation of Apache Spark, central to Databricks)

11. Microsoft Azure Documentation - Azure Data Lake Storage Gen2 (Official Microsoft Documentation) 12. Microsoft Azure Documentation - Azure Databricks Documentation (Official Microsoft Documentation) 13. Microsoft Azure Documentation - Azure Data Factory Documentation (Official Microsoft Documentation)

Statista – “IT industry in India - statistics & facts.” (Provides general market data and growth trends) Reuters – “India cenbank plans 2025 launch of cloud services.” (Focuses on regulatory changes and cloud adoption in finance)

5.

10. Databricks. (n.d.). What is a Data Lakehouse?

(Official Databricks resource defining the lakehouse architecture)

6. REFERENCES

2.

Oliver Wyman – “Future of cloud and its economic impact.” (Analyzes the broader economic benefits of cloud computing)

Cloud Computing and Data Lakehouse:

In conclusion, this project serves as a blueprint for organizations seeking to modernize their IT infrastructure while prioritizing data security and regulatory compliance.

By leveraging the power of cloud computing and advanced data masking techniques, India can unlock new opportunities for innovation and position itself as a frontrunner in the global digital economy. The insights gleaned from this research have far-reaching implications, not only for the banking sector but also for a wide range of industries seeking to harness the transformative potential of data-driven decision-making. The future of India's IT landscape hinges on its ability to embrace these technologies and cultivate a secure, innovative, and data-centric ecosystem.

1.

4.

Data Security and Privacy: 14. Schneier, B. (1996). Applied cryptography: protocols, algorithms, and source code in C. (Classic cryptography text, relevant to FPE and encryption) 15. Cavoukian, A. (2009). Privacy by design: The 7 foundational principles. (Framework for embedding privacy into system design)

|

ISO 9001:2008 Certified Journal

|

Page 654

International Research Journal of Engineering and Technology (IRJET)

e-ISSN: 2395-0056

Volume: 12 Issue: 07 | Jul 2025

p-ISSN: 2395-0072

www.irjet.net

16. GDPR (General Data Protection Regulation).

(2016). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC. (The official GDPR text)

Indian IT and Financial Sector: 28. NASSCOM (National Association of Software and Services Companies). (Various reports and publications). (Provides industry insights and data on the Indian IT sector) 29. IBA (Indian Banks' Association). (Various reports and publications). (Provides information on the Indian banking sector)

17. RBI (Reserve Bank of India). (2018). Storage of Payment System Data. (RBI guidelines on data localization)

30. MeitY (Ministry of Electronics and Information Technology, India). (Various reports and publications). (Provides information on Indian government IT policies)

18. ISO/IEC 27001:2022. Information security, cybersecurity and privacy protection — Information security management systems — Requirements. (International standard for information security management) 19. CCPA (California Consumer Privacy Act). (2018).

(California's data privacy law) 20. HIPAA (Health Insurance Portability and Accountability Act). (1996). (U.S. health data privacy law, if applicable to health-related banking data) 21. FIPS 140-2. Security Requirements for Cryptographic Modules. (U.S. government standard for cryptographic modules) AI/ML and Data Analytics: 22. Bishop, C. M. (2006). Pattern recognition and machine learning. (Fundamental text on machine learning algorithms) 23. Goodfellow, I., Bengio, Y., & Courville, A. (2016).

Deep learning. (Comprehensive book on deep learning) 24. Microsoft Azure Documentation - Power BI Documentation (Official Microsoft Documentation) 25. MLflow Documentation (Official Documentation for the MLflow platform) 26. Han, J., Kamber, M., & Pei, J. (2011). Data mining: concepts and techniques. (Textbook on data mining principles) 27. Provost, F., & Fawcett, T. (2013). Data science for business. (Focuses on the application of data science in business contexts)

© 2025, IRJET

|

Impact Factor value: 8.315

|

ISO 9001:2008 Certified Journal

|

Page 655