What is AWS DMS Replication and Changed Data Capture (CDC)

WHAT IS AWS DMS REPLICATION AND CHANGED DATA CAPTURE (CDC)

AWS DMS – Amazon Web Service Data Migration Service – is a cloud provider that ensures easy migration of data warehouses, relational databases, NoSQL databases, and more. It migrates data from on-site systems to the cloud, from one cloud provider to another, and in very rare cases from the cloud to an on-site set-up, often called reverse migration.

AWS DMS is also used to undertake one-time migration and replicate changes so that the data at the source and target locations are always in sync. Since Data Migration Service (DMS) is a part of AWS, all the benefits of the latter are available to DMS including speed to market, cost-efficiency, flexibility, and security.

The Change Data Capture (CDC) component in AWS DMS CDC identifies and captures the changes that have been made in the databases, production data, or metadata after a one-time migration or replication of databases has been done.

The changes are programmed to be updated after specific periods, often in minutes and seconds. Replication software then copies the changes into a separate repository. The CDC batch load replication ensures the target database is always in sync with source database after each load is completed. CDC replication software can copy data from one single source to multiple targets too.





While migrating data an AWS DMS CDC task has to be created for capturing changes made to the source data. This process has to be initiated after the full migration to a supported target has been completed. This is called “ongoing replication” and is used when replicating concurrent changes from a source data store after collecting changes to the database logs through the utilization of the database engine’s native API. The migration process has only tables from the source regardless of whether it is a CDC-only task or a full-load one which starts CDC after it is completed. In the full-load only task, views, or a combination of views may be migrated.

For reading current changes from the source database, AWS DMS CDC uses engine-centric API actions from the source engine’s transaction logs.

Following is a snapshot of how it is usually done. · For Microsoft SQL Server, AWS DMS uses MS-Replication or MS-CDC to enter information to the SQL Server transaction log. · For Oracle, AWS DMS CDC uses either the Oracle LogMiner API or the binary reader API to track ongoing changes. AWS DMS reads ongoing changes from the online or archived redo logs based on the SCN (System Change Number).

· For MySQL, AWS DMS reads changes from the row-based binary logs and loads them to the target. · For PostgreSQL, AWS DMS uses the test_decoding plugin and sets up logical replication slots to track changes from the source and migrate them to the target. · For Amazon RDS it is advisable to ensure AWS DMS CDC in backups.

This is the functioning of CDC in AWS DMS