[Bug] Flink Mongo CDC can not fetch incremental data after snapshot dump success, but it can fetch incremental data when set copy.existing=false

Search before asking

[X] I searched in the issues and found nothing similar.

Flink version
1.15.2
Flink CDC version
2.4.0 snapshot
Database and its version
Mongo 4.4
Hudi 0.12.1
Minimal reproduce step
Flink SQL:
CREATE TABLE mongo_cdc_test (
_id string,
...
...
PRIMARY KEY(_id) NOT ENFORCED
)
WITH (
'connector' = 'mongodb-cdc',
'poll.max.batch.size' = '100',
'hosts' = 'xxx:27017',
'username' = 'xx',
'password' = 'xx',
'database' = 'xx',
'collection' = 'xx',
'heartbeat.interval.ms' = '60000',
'scan.incremental.snapshot.enabled' = 'true',
'copy.existing' = 'true'
)
CREATE TABLE hudi_test(
_id string,
...
...
PRIMARY KEY(_id) NOT ENFORCED
)
WITH (
'connector' = 'hudi',
'path' = 'xx',
'table.type' = 'MERGE_ON_READ',
'index.type' = 'BUCKET',
'hoodie.datasource.write.recordkey.field' = '_id',
'hoodie.bucket.index.hash.field' = '_id',
'hoodie.bucket.index.num.buckets' = '64',
'changelog.enabled'= 'true',
'compaction.async.enabled'='true',
'compaction.tasks'= 'xx',
'compaction.trigger.strategy'= 'time_elapsed',
'compaction.delta_seconds'= '3600',
'compaction.max_memory'= '1024',
'write.option' = 'upsert',
'read.streaming.enabled'= 'true',
'read.streaming.check-interval'= '4',
'hive_sync.enable' = 'true',
'hive_sync.mode' = 'hms',
'hive_sync.metastore.uris' = 'thrift://xx:9083',
'hive_sync.table'='xx',
'hive_sync.auto_create_database'='false',
'hive_sync.table.strategy'='ALL',
'hive_sync.db'='default',
'write.tasks'='xx'
)
INSERT INTO hudi_test SELECT * FROM mongo_cdc_test
What did you expect to see?
I expect to use flink Mongo CDC to sync snapshot and incremental data from mongo and write it to Hudi
What did you see instead?
Flink Mongo CDC can not fetch incremental data after snapshot dump success, but it can fetch incremental data when set copy.existing=false
mongo cdc source is busy

mongo cdc source can not fetch any change streams from mongodb

hudi deltacommit instant is in a earlier time

Anything else?
No response
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

Thanks @newsbreak-tonglin to report this problem.
Can you help provide some log file?

I'm glad to help investigating this.

@Jiabao-Sun @yuxiqian  Thank you for your reply, I will try to reproduce this bug

@newsbreak-tonglin Could you please kindly provide the log file?

@yuxiqian  I have tried to reproduce it, here is the detail job manager logs

2023-07-10 07:58:54,477 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: mongo_cdc_test[1] received split request from parallel task 51
2023-07-10 07:58:54,510 INFO  com.ververica.cdc.connectors.base.source.enumerator.IncrementalSourceEnumerator [] - The enumerator receives finished split offsets FinishedSnapshotSplitsReportEvent{finishedOffsets={staticFeature.document:144719={resumeToken={"_data": "8264ABBA37000000192B022C0100296E5A1004B807779924DA402AB13486B3F67B6102463C5F6964003C306E4B6B6A41744E000004"}, timestamp=7254096370196283417}}} from subtask 35.
2023-07-10 07:58:54,510 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: mongo_cdc_test[1] received split request from parallel task 35
2023-07-10 07:58:54,598 INFO  com.ververica.cdc.connectors.base.source.enumerator.IncrementalSourceEnumerator [] - The enumerator receives finished split offsets FinishedSnapshotSplitsReportEvent{finishedOffsets={staticFeature.document:144716={resumeToken={"_data": "8264ABBA38000000032B0229296E04"}, timestamp=7254096374491250691}}} from subtask 27.
2023-07-10 07:58:54,598 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: mongo_cdc_test[1] received split request from parallel task 27
2023-07-10 07:58:56,013 INFO  com.ververica.cdc.connectors.base.source.enumerator.IncrementalSourceEnumerator [] - The enumerator receives finished split offsets FinishedSnapshotSplitsReportEvent{finishedOffsets={staticFeature.document:144723={resumeToken={"_data": "8264ABBA3A000000062B0229296E04"}, timestamp=7254096383081185286}}} from subtask 21.
2023-07-10 07:58:56,013 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: mongo_cdc_test[1] received split request from parallel task 21
2023-07-10 07:58:57,171 INFO  com.ververica.cdc.connectors.base.source.enumerator.IncrementalSourceEnumerator [] - The enumerator receives finished split offsets FinishedSnapshotSplitsReportEvent{finishedOffsets={staticFeature.document:144728={resumeToken={"_data": "8264ABBA3C0000000A2B022C0100296E5A1004B807779924DA402AB13486B3F67B6102463C5F6964003C306E4C4647655553000004"}, timestamp=7254096391671119882}}} from subtask 29.
2023-07-10 07:58:57,171 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: mongo_cdc_test[1] received split request from parallel task 29
2023-07-10 07:58:57,279 INFO  com.ververica.cdc.connectors.base.source.enumerator.IncrementalSourceEnumerator [] - The enumerator receives finished split offsets FinishedSnapshotSplitsReportEvent{finishedOffsets={staticFeature.document:144729={resumeToken={"_data": "8264ABBA3C0000000A2B022C0100296E5A1004B807779924DA402AB13486B3F67B6102463C5F6964003C306E4C4647655553000004"}, timestamp=7254096391671119882}}} from subtask 39.
2023-07-10 07:58:57,279 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: mongo_cdc_test[1] received split request from parallel task 39
2023-07-10 07:58:57,517 INFO  com.ververica.cdc.connectors.base.source.enumerator.IncrementalSourceEnumerator [] - The enumerator receives finished split offsets FinishedSnapshotSplitsReportEvent{finishedOffsets={staticFeature.document:144725={resumeToken={"_data": "8264ABBA3C000000022B0229296E04"}, timestamp=7254096391671119874}}} from subtask 40.
2023-07-10 07:58:57,517 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: mongo_cdc_test[1] received split request from parallel task 40
2023-07-10 07:58:57,527 INFO  com.ververica.cdc.connectors.base.source.enumerator.IncrementalSourceEnumerator [] - The enumerator receives finished split offsets FinishedSnapshotSplitsReportEvent{finishedOffsets={staticFeature.document:144727={resumeToken={"_data": "8264ABBA3C0000000A2B022C0100296E5A1004B807779924DA402AB13486B3F67B6102463C5F6964003C306E4C4647655553000004"}, timestamp=7254096391671119882}}} from subtask 31.
2023-07-10 07:58:57,527 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: mongo_cdc_test[1] received split request from parallel task 31
2023-07-10 07:58:58,339 INFO  com.ververica.cdc.connectors.base.source.enumerator.IncrementalSourceEnumerator [] - The enumerator receives finished split offsets FinishedSnapshotSplitsReportEvent{finishedOffsets={staticFeature.document:144731={resumeToken={"_data": "8264ABBA3C0000000C2B0229296E04"}, timestamp=7254096391671119884}}} from subtask 28.
2023-07-10 07:58:58,339 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: mongo_cdc_test[1] received split request from parallel task 28
2023-07-10 07:58:58,439 INFO  com.ververica.cdc.connectors.base.source.enumerator.IncrementalSourceEnumerator [] - The enumerator receives finished split offsets FinishedSnapshotSplitsReportEvent{finishedOffsets={staticFeature.document:144732={resumeToken={"_data": "8264ABBA3C0000000A2B022C0100296E5A1004B807779924DA402AB13486B3F67B6102463C5F6964003C306E4C4647655553000004"}, timestamp=7254096391671119882}}} from subtask 4.
2023-07-10 07:58:58,439 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: mongo_cdc_test[1] received split request from parallel task 4
2023-07-10 07:58:58,559 INFO  com.ververica.cdc.connectors.base.source.enumerator.IncrementalSourceEnumerator [] - The enumerator receives finished split offsets FinishedSnapshotSplitsReportEvent{finishedOffsets={staticFeature.document:144726={resumeToken={"_data": "8264ABBA3C000000022B0229296E04"}, timestamp=7254096391671119874}}} from subtask 41.
2023-07-10 07:58:58,559 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: mongo_cdc_test[1] received split request from parallel task 41
2023-07-10 07:58:58,623 INFO  com.ververica.cdc.connectors.base.source.enumerator.IncrementalSourceEnumerator [] - The enumerator receives finished split offsets FinishedSnapshotSplitsReportEvent{finishedOffsets={staticFeature.document:144724={resumeToken={"_data": "8264ABBA3B000000052B0229296E04"}, timestamp=7254096387376152581}}} from subtask 0.
2023-07-10 07:58:58,623 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: mongo_cdc_test[1] received split request from parallel task 0
2023-07-10 07:58:58,634 INFO  com.ververica.cdc.connectors.base.source.enumerator.IncrementalSourceEnumerator [] - The enumerator receives finished split offsets FinishedSnapshotSplitsReportEvent{finishedOffsets={staticFeature.document:144722={resumeToken={"_data": "8264ABBA3B000000012B022C0100296E5A1004B807779924DA402AB13486B3F67B6102463C5F6964003C306E4C46474F5977000004"}, timestamp=7254096387376152577}}} from subtask 24.
2023-07-10 07:58:58,635 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: mongo_cdc_test[1] received split request from parallel task 24
2023-07-10 07:58:58,649 INFO  com.ververica.cdc.connectors.base.source.enumerator.IncrementalSourceEnumerator [] - The enumerator receives finished split offsets FinishedSnapshotSplitsReportEvent{finishedOffsets={staticFeature.document:144734={resumeToken={"_data": "8264ABBA3C0000000A2B022C0100296E5A1004B807779924DA402AB13486B3F67B6102463C5F6964003C306E4C4647655553000004"}, timestamp=7254096391671119882}}} from subtask 26.
2023-07-10 07:58:58,649 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: mongo_cdc_test[1] received split request from parallel task 26
2023-07-10 07:58:58,691 INFO  com.ververica.cdc.connectors.base.source.enumerator.IncrementalSourceEnumerator [] - The enumerator receives finished split offsets FinishedSnapshotSplitsReportEvent{finishedOffsets={staticFeature.document:144735={resumeToken={"_data": "8264ABBA3C0000000A2B022C0100296E5A1004B807779924DA402AB13486B3F67B6102463C5F6964003C306E4C4647655553000004"}, timestamp=7254096391671119882}}} from subtask 8.
2023-07-10 07:58:58,691 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: mongo_cdc_test[1] received split request from parallel task 8
2023-07-10 07:58:58,963 INFO  com.ververica.cdc.connectors.base.source.enumerator.IncrementalSourceEnumerator [] - The enumerator receives finished split offsets FinishedSnapshotSplitsReportEvent{finishedOffsets={staticFeature.document:144730={resumeToken={"_data": "8264ABBA3C0000000C2B022C0100296E5A1004B807779924DA402AB13486B3F67B6102463C5F6964003C306E4C4647655553000004"}, timestamp=7254096391671119884}}} from subtask 34.
2023-07-10 07:58:58,963 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: mongo_cdc_test[1] received split request from parallel task 34
2023-07-10 07:58:59,076 INFO  com.ververica.cdc.connectors.base.source.enumerator.IncrementalSourceEnumerator [] - The enumerator receives finished split offsets FinishedSnapshotSplitsReportEvent{finishedOffsets={staticFeature.document:144733={resumeToken={"_data": "8264ABBA3D0000000B2B0229296E04"}, timestamp=7254096395966087179}}} from subtask 42.
2023-07-10 07:58:59,076 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: mongo_cdc_test[1] received split request from parallel task 42
2023-07-10 07:59:01,198 INFO  com.ververica.cdc.connectors.base.source.enumerator.IncrementalSourceEnumerator [] - The enumerator receives finished split offsets FinishedSnapshotSplitsReportEvent{finishedOffsets={staticFeature.document:144736={resumeToken={"_data": "8264ABBA3E000000042B0229296E04"}, timestamp=7254096400261054468}}} from subtask 5.
2023-07-10 07:59:01,198 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: mongo_cdc_test[1] received split request from parallel task 5
2023-07-10 08:00:18,912 INFO  com.ververica.cdc.connectors.base.source.enumerator.IncrementalSourceEnumerator [] - The enumerator receives finished split offsets FinishedSnapshotSplitsReportEvent{finishedOffsets={staticFeature.document:144737={resumeToken={"_data": "8264ABBA640000002E2B0229296E04"}, timestamp=7254096563469811758}}} from subtask 54.
2023-07-10 08:00:18,912 INFO  com.ververica.cdc.connectors.base.source.assigner.SnapshotSplitAssigner [] - Snapshot split assigner received all splits finished, waiting for a complete checkpoint to mark the assigner finished.
2023-07-10 08:00:18,912 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Source Source: mongo_cdc_test[1] received split request from parallel task 54
2023-07-10 08:00:33,110 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 103 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @<PHONE_NUMBER>109 for job 40c88e0009d80a880af321102ab3c577.
2023-07-10 08:00:33,297 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [taking checkpoint 103] success!
2023-07-10 08:00:33,950 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,951 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,961 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,965 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,967 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,967 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,968 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,969 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,969 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,970 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,972 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,972 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,972 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,974 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,974 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,975 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,976 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,977 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,977 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,978 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,980 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,980 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,981 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,981 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,983 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:33,986 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:37,038 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:37,104 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:37,116 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:37,120 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:37,138 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:37,158 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:37,231 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:37,408 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:37,426 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:37,469 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:37,483 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:37,802 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:37,847 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:38,374 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:40,041 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:40,067 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:40,110 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:40,207 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:40,238 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:40,321 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:40,355 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:40,420 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:40,678 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:41,214 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:42,871 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:42,992 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:43,260 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:43,514 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:44,022 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:44,119 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:44,228 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:44,298 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:45,119 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:45,442 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:45,498 INFO  org.apache.flink.runtime.state.filesystem.FsCheckpointMetadataOutputStream [] - Cannot create recoverable writer due to Recoverable writers on Hadoop are only supported for HDFS, will use the ordinary writer.
2023-07-10 08:00:46,122 INFO  com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream  [] - close closed:false s3://xxx/hudi_test_checkpoint/40c88e0009d80a880af321102ab3c577/chk-103/_metadata
2023-07-10 08:00:46,343 INFO  com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream  [] - uploadPart: partNum 2 of 's3://xxx/hudi_test_checkpoint/40c88e0009d80a880af321102ab3c577/chk-103/_metadata' from local file '/mnt/s3/emrfs-7213784681553450334/0000000001', 18029334 bytes in 221 ms, md5: CSrBs8qzQs57q6m62s6U0Q== md5hex: 092ac1b3cab342ce7baba9badace94d1
2023-07-10 08:00:46,985 INFO  com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream  [] - uploadPart: partNum 1 of 's3://xxx/hudi_test_checkpoint/40c88e0009d80a880af321102ab3c577/chk-103/_metadata' from local file '/mnt/s3/emrfs-7213784681553450334/0000000000', 134217728 bytes in 946 ms, md5: huwe0k8uzA23jimZntdE/w== md5hex: 86ec1ed24f2ecc0db78e29999ed744ff
2023-07-10 08:00:47,219 INFO  com.amazon.ws.emr.hadoop.fs.s3.upload.dispatch.DefaultMultipartUploadDispatcher [] - Completed multipart upload of 2 parts 152247062 bytes
2023-07-10 08:00:51,075 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 103 for job 40c88e0009d80a880af321102ab3c577<PHONE_NUMBER>35 bytes, checkpointDuration=14110 ms, finalizationTime=3856 ms).
2023-07-10 08:00:51,076 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Marking checkpoint 103 as completed for source Source: mongo_cdc_test[1].
2023-07-10 08:00:51,076 INFO  com.ververica.cdc.connectors.base.source.assigner.SnapshotSplitAssigner [] - Snapshot split assigner is turn into finished status.
2023-07-10 08:00:51,099 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Committing<PHONE_NUMBER>5809416 action deltacommit
2023-07-10 08:00:51,099 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading HoodieTableMetaClient from s3a://xxx/hudi_test19
2023-07-10 08:00:51,151 INFO  org.apache.hudi.common.table.HoodieTableConfig               [] - Loading table properties from s3a://xxx/hudi_test19/.hoodie/hoodie.properties
2023-07-10 08:00:51,182 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from s3a://xxx/hudi_test19
2023-07-10 08:00:51,182 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading Active commit timeline for s3a://xxx/hudi_test19
2023-07-10 08:00:51,209 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230710075809416__deltacommit__INFLIGHT]}
2023-07-10 08:00:51,210 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating View Manager with storage type :REMOTE_FIRST
2023-07-10 08:00:51,210 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating remote first table view
2023-07-10 08:00:51,210 INFO  org.apache.hudi.common.util.CommitUtils                      [] - Creating  metadata for null numWriteStats:64 numReplaceFileIds:0
2023-07-10 08:00:51,210 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Committing<PHONE_NUMBER>5809416 action deltacommit
2023-07-10 08:00:51,259 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading HoodieTableMetaClient from s3a://xxx/hudi_test19
2023-07-10 08:00:51,290 INFO  org.apache.hudi.common.table.HoodieTableConfig               [] - Loading table properties from s3a://xxx/hudi_test19/.hoodie/hoodie.properties
2023-07-10 08:00:51,314 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from s3a://xxx/hudi_test19
2023-07-10 08:00:51,314 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading HoodieTableMetaClient from s3a://xxx/hudi_test19
2023-07-10 08:00:51,341 INFO  org.apache.hudi.common.table.HoodieTableConfig               [] - Loading table properties from s3a://xxx/hudi_test19/.hoodie/hoodie.properties
2023-07-10 08:00:51,358 INFO  com.ververica.cdc.connectors.base.source.enumerator.IncrementalSourceEnumerator [] - Assign split StreamSplit{splitId='stream-split', offset={resumeToken={"_data": "8264AB7311000000222B022C0100296E5A1004B807779924DA402AB13486B3F67B6102463C5F6964003C306E4C374F58304F000004"}, timestamp=7254018141661954082}, endOffset={resumeToken=null, timestamp=9223372034707292159}} to subtask 0
2023-07-10 08:00:51,402 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from s3a://xxx/hudi_test19
2023-07-10 08:00:51,402 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading Active commit timeline for s3a://xxx/hudi_test19
2023-07-10 08:00:51,430 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230710075809416__deltacommit__INFLIGHT]}
2023-07-10 08:00:51,430 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating View Manager with storage type :REMOTE_FIRST
2023-07-10 08:00:51,430 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating remote first table view
2023-07-10 08:00:51,431 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Marking instant complete [==>20230710075809416__deltacommit__INFLIGHT]
2023-07-10 08:00:51,431 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Checking for file exists ?s3a://xxx/hudi_test19/.hoodie/20230710075809416.deltacommit.inflight
2023-07-10 08:00:51,628 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Create new file for toInstant ?s3a://xxx/hudi_test19/.hoodie/20230710075809416.deltacommit
2023-07-10 08:00:51,628 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Completed [==>20230710075809416__deltacommit__INFLIGHT]
2023-07-10 08:00:51,628 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading HoodieTableMetaClient from s3a://xxx/hudi_test19
2023-07-10 08:00:51,733 INFO  org.apache.hudi.common.table.HoodieTableConfig               [] - Loading table properties from s3a://xxx/hudi_test19/.hoodie/hoodie.properties
2023-07-10 08:00:51,759 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from s3a://xxx/hudi_test19
2023-07-10 08:00:51,759 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading Active commit timeline for s3a://xxx/hudi_test19
2023-07-10 08:00:51,790 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[20230710075809416__deltacommit__COMPLETED]}
2023-07-10 08:00:51,791 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating View Manager with storage type :REMOTE_FIRST
2023-07-10 08:00:51,791 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating remote first table view
2023-07-10 08:00:52,426 INFO  org.apache.hudi.common.fs.FSUtils                            [] - Removed directory at s3a://xxx/hudi_test19/.hoodie/.temp/20230710075809416
2023-07-10 08:00:52,426 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Start to archive synchronously.
2023-07-10 08:00:52,459 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[20230710075809416__deltacommit__COMPLETED]}
2023-07-10 08:00:52,459 INFO  org.apache.hudi.client.HoodieTimelineArchiver                [] - No Instants to archive
2023-07-10 08:00:52,459 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Committed<PHONE_NUMBER>5809416
2023-07-10 08:00:52,605 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Commit instant [20230710075809416] success!
2023-07-10 08:00:52,605 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Scheduling table service COMPACT
2023-07-10 08:00:52,605 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Scheduling compaction at instant time :20230710080052605
2023-07-10 08:00:52,605 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading HoodieTableMetaClient from s3a://xxx/hudi_test19
2023-07-10 08:00:52,638 INFO  org.apache.hudi.common.table.HoodieTableConfig               [] - Loading table properties from s3a://xxx/hudi_test19/.hoodie/hoodie.properties
2023-07-10 08:00:52,663 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from s3a://xxx/hudi_test19
2023-07-10 08:00:52,663 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading Active commit timeline for s3a://xxx/hudi_test19
2023-07-10 08:00:52,691 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[20230710075809416__deltacommit__COMPLETED]}
2023-07-10 08:00:52,692 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating View Manager with storage type :REMOTE_FIRST
2023-07-10 08:00:52,692 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating remote first table view
2023-07-10 08:00:52,692 INFO  org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - Checking if compaction needs to be run on s3a://xxx/hudi_test19
2023-07-10 08:00:52,692 INFO  org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - The elapsed time >=3600s, trigger compaction scheduler.
2023-07-10 08:00:52,692 INFO  org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - Generating compaction plan for merge on read table s3a://xxx/hudi_test19
2023-07-10 08:00:52,692 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating remote view for basePath s3a://xxx/hudi_test19. Server=<IP_ADDRESS>:38901, Timeout=300
2023-07-10 08:00:52,692 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating InMemory based view for basePath s3a://xxx/hudi_test19
2023-07-10 08:00:52,692 INFO  org.apache.hudi.common.table.view.AbstractTableFileSystemView [] - Took 0 ms to read  0 instants, 0 replaced file groups
2023-07-10 08:00:52,756 INFO  org.apache.hudi.common.util.ClusteringUtils                  [] - Found 0 files in pending clustering operations
2023-07-10 08:00:52,756 INFO  org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView [] - Sending request : (http://<IP_ADDRESS>:38901/v1/hoodie/view/compactions/pending/?basepath=s3a%3A%2F%2Fuser-devs%2Flintong%2Fhudi_test19&lastinstantts=20230710075809416&timelinehash=c68ce2f836da0367a80f96ed0f1df3ffd4824506e7ce35595b53fb552e358172)
2023-07-10 08:00:52,757 INFO  org.apache.hudi.timeline.service.RequestHandler              [] - Syncing view as client passed last known instant<PHONE_NUMBER>5809416 as last known instant but server has the following last instant on timeline :Option{val=[20230710075510186__deltacommit__COMPLETED]}
2023-07-10 08:00:52,788 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[20230710075809416__deltacommit__COMPLETED]}
2023-07-10 08:00:52,788 INFO  org.apache.hudi.common.table.view.AbstractTableFileSystemView [] - Took 0 ms to read  0 instants, 0 replaced file groups
2023-07-10 08:00:52,833 INFO  org.apache.hudi.common.util.ClusteringUtils                  [] - Found 0 files in pending clustering operations
2023-07-10 08:00:52,837 INFO  org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView [] - Sending request : (http://<IP_ADDRESS>:38901/v1/hoodie/view/clustering/pending/?basepath=s3a%3A%2F%2Fuser-devs%2Flintong%2Fhudi_test19&lastinstantts=20230710075809416&timelinehash=c68ce2f836da0367a80f96ed0f1df3ffd4824506e7ce35595b53fb552e358172)
2023-07-10 08:00:52,838 INFO  org.apache.hudi.table.action.compact.HoodieCompactor         [] - Compacting s3a://xxx/hudi_test19 with commit<PHONE_NUMBER>0052605
2023-07-10 08:00:53,147 INFO  org.apache.hudi.table.action.compact.HoodieCompactor         [] - Compaction looking for files to compact in [] partitions
2023-07-10 08:00:53,147 INFO  org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView [] - Sending request : (http://<IP_ADDRESS>:38901/v1/hoodie/view/slices/partition/latest/?partition=&basepath=s3a%3A%2F%2Fuser-devs%2Flintong%2Fhudi_test19&lastinstantts=20230710075809416&timelinehash=c68ce2f836da0367a80f96ed0f1df3ffd4824506e7ce35595b53fb552e358172)
2023-07-10 08:00:53,148 INFO  org.apache.hudi.common.table.view.AbstractTableFileSystemView [] - Building file system view for partition ()
2023-07-10 08:00:53,931 INFO  org.apache.hudi.common.table.view.AbstractTableFileSystemView [] - addFilesToView: NumFiles=2806, NumFileGroups=64, FileGroupsCreationTime=550, StoreTimeTaken=0
2023-07-10 08:00:54,371 INFO  org.apache.hudi.table.action.compact.HoodieCompactor         [] - Total of 0 compactions are retrieved
2023-07-10 08:00:54,371 INFO  org.apache.hudi.table.action.compact.HoodieCompactor         [] - Total number of latest files slices 0
2023-07-10 08:00:54,371 INFO  org.apache.hudi.table.action.compact.HoodieCompactor         [] - Total number of log files 0
2023-07-10 08:00:54,371 INFO  org.apache.hudi.table.action.compact.HoodieCompactor         [] - Total number of file slices 0
2023-07-10 08:00:54,396 WARN  org.apache.hudi.table.action.compact.HoodieCompactor         [] - After filtering, Nothing to compact for s3a://xxx/hudi_test19
2023-07-10 08:00:54,396 INFO  org.apache.hudi.common.util.CleanerUtils                     [] - Cleaned failed attempts if any
2023-07-10 08:00:54,396 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading HoodieTableMetaClient from s3a://xxx/hudi_test19
2023-07-10 08:00:54,428 INFO  org.apache.hudi.common.table.HoodieTableConfig               [] - Loading table properties from s3a://xxx/hudi_test19/.hoodie/hoodie.properties
2023-07-10 08:00:54,452 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from s3a://xxx/hudi_test19
2023-07-10 08:00:54,452 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading Active commit timeline for s3a://xxx/hudi_test19
2023-07-10 08:00:54,485 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[20230710075809416__deltacommit__COMPLETED]}
2023-07-10 08:00:54,485 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating View Manager with storage type :REMOTE_FIRST
2023-07-10 08:00:54,485 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating remote first table view
2023-07-10 08:00:54,485 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Generate a new instant time:<PHONE_NUMBER>0054485 action: deltacommit
2023-07-10 08:00:54,485 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Creating a new instant [==>20230710080054485__deltacommit__REQUESTED]
2023-07-10 08:00:54,586 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Checking for file exists ?s3a://xxx/hudi_test19/.hoodie/20230710080054485.deltacommit.requested
2023-07-10 08:00:54,715 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Create new file for toInstant ?s3a://xxx/hudi_test19/.hoodie/20230710080054485.deltacommit.inflight
2023-07-10 08:00:54,981 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Create instant [20230710080054485] for table [hudi_test] with type [MERGE_ON_READ]
2023-07-10 08:00:54,981 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [commits the instant<PHONE_NUMBER>5809416] success!
2023-07-10 08:00:54,982 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading HoodieTableMetaClient from s3a://xxx/hudi_test19
2023-07-10 08:00:55,014 INFO  org.apache.hudi.common.table.HoodieTableConfig               [] - Loading table properties from s3a://xxx/hudi_test19/.hoodie/hoodie.properties
2023-07-10 08:00:55,037 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from s3a://xxx/hudi_test19
2023-07-10 08:00:55,037 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading Active commit timeline for s3a://xxx/hudi_test19
2023-07-10 08:00:55,071 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230710080054485__deltacommit__INFLIGHT]}
2023-07-10 08:00:55,072 INFO  org.apache.hadoop.hive.metastore.HiveMetaStoreClient         [] - Trying to connect to metastore with URI thrift://ip-172-31-176-221.us-west-2.compute.internal:9083
2023-07-10 08:00:55,073 INFO  org.apache.hadoop.hive.metastore.HiveMetaStoreClient         [] - Opened a connection to metastore, current connections: 1
2023-07-10 08:00:55,073 INFO  org.apache.hadoop.hive.metastore.HiveMetaStoreClient         [] - Connected to metastore.
2023-07-10 08:00:55,073 INFO  org.apache.hadoop.hive.metastore.RetryingMetaStoreClient     [] - RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=hadoop (auth:SIMPLE) retries=1 delay=1 lifetime=0
2023-07-10 08:00:55,073 INFO  org.apache.hudi.hive.HiveSyncTool                            [] - Syncing target hoodie table with hive table(default.mongo_cdc_hudi_lintong_test19). Hive metastore URL :thrift://data.emr.nb.com:9083, basePath :s3a://xxx/hudi_test19
2023-07-10 08:00:55,073 INFO  org.apache.hudi.hive.HiveSyncTool                            [] - Trying to sync hoodie table mongo_cdc_hudi_lintong_test19_ro with base path s3a://xxx/hudi_test19 of type MERGE_ON_READ
2023-07-10 08:00:55,124 ERROR org.apache.hudi.hive.ddl.HMSDDLExecutor                      [] - Failed to create database default
org.apache.hadoop.hive.metastore.api.AlreadyExistsException: Database default already exists
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_database_result$create_database_resultStandardScheme.read(ThriftHiveMetastore.java:39325) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_database_result$create_database_resultStandardScheme.read(ThriftHiveMetastore.java:39311) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_database_result.read(ThriftHiveMetastore.java:39245) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_database(ThriftHiveMetastore.java:1106) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_database(ThriftHiveMetastore.java:1093) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createDatabase(HiveMetaStoreClient.java:809) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at sun.reflect.GeneratedMethodAccessor124.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_372]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_372]
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at com.sun.proxy.$Proxy92.createDatabase(Unknown Source) ~[?:?]
at sun.reflect.GeneratedMethodAccessor124.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_372]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_372]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2773) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at com.sun.proxy.$Proxy92.createDatabase(Unknown Source) ~[?:?]
at org.apache.hudi.hive.ddl.HMSDDLExecutor.createDatabase(HMSDDLExecutor.java:95) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.hive.HoodieHiveSyncClient.createDatabase(HoodieHiveSyncClient.java:224) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:187) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:158) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:142) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.doSyncHive(StreamWriteOperatorCoordinator.java:335) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_372]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_372]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_372]
2023-07-10 08:00:55,125 WARN  org.apache.hudi.hive.HiveSyncTool                            [] - Unable to create database
org.apache.hudi.hive.HoodieHiveSyncException: Failed to create database default
at org.apache.hudi.hive.ddl.HMSDDLExecutor.createDatabase(HMSDDLExecutor.java:98) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.hive.HoodieHiveSyncClient.createDatabase(HoodieHiveSyncClient.java:224) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:187) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:158) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:142) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.doSyncHive(StreamWriteOperatorCoordinator.java:335) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_372]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_372]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_372]
Caused by: org.apache.hadoop.hive.metastore.api.AlreadyExistsException: Database default already exists
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_database_result$create_database_resultStandardScheme.read(ThriftHiveMetastore.java:39325) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_database_result$create_database_resultStandardScheme.read(ThriftHiveMetastore.java:39311) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_database_result.read(ThriftHiveMetastore.java:39245) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_database(ThriftHiveMetastore.java:1106) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_database(ThriftHiveMetastore.java:1093) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createDatabase(HiveMetaStoreClient.java:809) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at sun.reflect.GeneratedMethodAccessor124.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_372]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_372]
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at com.sun.proxy.$Proxy92.createDatabase(Unknown Source) ~[?:?]
at sun.reflect.GeneratedMethodAccessor124.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_372]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_372]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2773) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at com.sun.proxy.$Proxy92.createDatabase(Unknown Source) ~[?:?]
at org.apache.hudi.hive.ddl.HMSDDLExecutor.createDatabase(HMSDDLExecutor.java:95) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
... 9 more
2023-07-10 08:00:55,270 INFO  org.apache.hadoop.fs.s3a.S3AInputStream                      [] - Switching to Random IO seek policy
2023-07-10 08:00:55,732 INFO  org.apache.hudi.hive.ddl.HMSDDLExecutor                      [] - Time taken to getTableSchema: 30 ms
2023-07-10 08:00:55,733 INFO  org.apache.hudi.hive.HiveSyncTool                            [] - No Schema difference for mongo_cdc_hudi_lintong_test19_ro
2023-07-10 08:00:55,733 INFO  org.apache.hudi.hive.HiveSyncTool                            [] - Schema sync complete. Syncing partitions for mongo_cdc_hudi_lintong_test19_ro
2023-07-10 08:00:55,756 INFO  org.apache.hudi.hive.HiveSyncTool                            [] - Last commit time synced was found to be null
2023-07-10 08:00:55,756 INFO  org.apache.hudi.sync.common.HoodieSyncClient                 [] - Last commit time synced is not known, listing all partitions in s3a://xxx/hudi_test19,FS :S3AFileSystem{uri=s3a://user-devs, workingDir=s3a://user-devs/user/hadoop, inputPolicy=normal, partSize=67108864, enableMultiObjectsDelete=true, maxKeys=5000, readAhead=65536, blockSize=33554432, multiPartThreshold=134217728, s3EncryptionAlgorithm='NONE', blockFactory=org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory@2abbd8ae, auditManager=Service NoopAuditManagerS3A in state NoopAuditManagerS3A: STARTED, metastore=NullMetadataStore, authoritativeStore=false, authoritativePath=[], useListV1=false, magicCommitter=true, boundedExecutor=BlockingThreadPoolExecutorService{SemaphoredDelegatingExecutor{permitCount=4096, available=4096, waiting=0}, activeCount=0}, unboundedExecutor=java.util.concurrent.ThreadPoolExecutor@767b3655[Running, pool size = 22, active threads = 0, queued tasks = 0, completed tasks = 2640], credentials=AWSCredentialProviderList[refcount= 1: [TemporaryAWSCredentialsProvider, SimpleAWSCredentialsProvider, EnvironmentVariableCredentialsProvider, org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider@bfd748d] last provider: org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider@bfd748d, delegation tokens=disabled, DirectoryMarkerRetention{policy='delete'}, instrumentation {S3AInstrumentation{}}, ClientSideEncryption=false}
2023-07-10 08:00:56,078 INFO  org.apache.hudi.hive.HiveSyncTool                            [] - Storage partitions scan complete. Found 1
2023-07-10 08:00:56,084 INFO  org.apache.hadoop.hive.metastore.HiveMetaStoreClient         [] - Closed a connection to metastore, current connections: 0
2023-07-10 08:00:56,084 ERROR org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [sync hive metadata for instant<PHONE_NUMBER>0054485] error
org.apache.hudi.exception.HoodieException: Got runtime exception when hive syncing mongo_cdc_hudi_lintong_test19
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:145) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.sink.StreamWriteOperatorCoordinator.doSyncHive(StreamWriteOperatorCoordinator.java:335) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.sink.utils.NonThrownExecutor.lambda$wrapAction$0(NonThrownExecutor.java:130) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_372]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_372]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_372]
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to sync partitions for table mongo_cdc_hudi_lintong_test19_ro
at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:341) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:232) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:158) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:142) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
... 5 more
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to get all partitions for table default.mongo_cdc_hudi_lintong_test19_ro
at org.apache.hudi.hive.HoodieHiveSyncClient.getAllPartitions(HoodieHiveSyncClient.java:180) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:317) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:232) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:158) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:142) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
... 5 more
Caused by: org.apache.hadoop.hive.metastore.api.NoSuchObjectException: @hive#default.mongo_cdc_hudi_lintong_test19_ro table not found
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result$get_partitions_resultStandardScheme.read(ThriftHiveMetastore.java) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.read(ThriftHiveMetastore.java) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions(ThriftHiveMetastore.java:2958) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions(ThriftHiveMetastore.java:2943) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitions(HiveMetaStoreClient.java:1368) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitions(HiveMetaStoreClient.java:1362) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at sun.reflect.GeneratedMethodAccessor125.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_372]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_372]
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at com.sun.proxy.$Proxy92.listPartitions(Unknown Source) ~[?:?]
at sun.reflect.GeneratedMethodAccessor125.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_372]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_372]
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2773) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at com.sun.proxy.$Proxy92.listPartitions(Unknown Source) ~[?:?]
at org.apache.hudi.hive.HoodieHiveSyncClient.getAllPartitions(HoodieHiveSyncClient.java:175) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:317) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:232) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:158) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:142) ~[blob_p-8126a52f21d07b448344e4277f6fb0837c921987-71810e5577eef736251ddb420010bd50:?]
... 5 more
2023-07-10 08:03:33,110 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 104 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @<PHONE_NUMBER>109 for job 40c88e0009d80a880af321102ab3c577.
2023-07-10 08:03:33,317 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [taking checkpoint 104] success!
2023-07-10 08:03:34,129 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,131 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,135 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,135 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,135 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,136 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,136 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,137 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,139 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,139 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,139 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,139 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,139 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,140 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,140 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,140 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,140 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,141 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,141 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,142 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,143 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,143 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,143 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,143 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,143 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,144 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,145 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,145 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,145 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,146 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,146 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,146 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,147 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,147 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,147 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,147 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,149 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,150 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,150 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,150 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,150 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,150 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,150 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,150 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,151 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,151 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,152 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,152 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,154 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,155 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,155 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,155 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,155 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,156 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,157 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,157 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,161 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,169 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,187 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:03:34,620 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:23,182 INFO  org.apache.flink.runtime.state.filesystem.FsCheckpointMetadataOutputStream [] - Cannot create recoverable writer due to Recoverable writers on Hadoop are only supported for HDFS, will use the ordinary writer.
2023-07-10 08:48:23,848 INFO  com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream  [] - close closed:false s3://xxx/hudi_test_checkpoint/40c88e0009d80a880af321102ab3c577/chk-104/_metadata
2023-07-10 08:48:24,092 INFO  com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream  [] - uploadPart: partNum 2 of 's3://xxx/hudi_test_checkpoint/40c88e0009d80a880af321102ab3c577/chk-104/_metadata' from local file '/mnt/s3/emrfs-6907486897091698972/0000000001', 17912860 bytes in 243 ms, md5: X7uLerjCTJCEFNNeYuNBJg== md5hex: 5fbb8b7ab8c24c908414d35e62e34126
2023-07-10 08:48:24,690 INFO  com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream  [] - uploadPart: partNum 1 of 's3://xxx/hudi_test_checkpoint/40c88e0009d80a880af321102ab3c577/chk-104/_metadata' from local file '/mnt/s3/emrfs-6907486897091698972/0000000000', 134217728 bytes in 955 ms, md5: JWkgNm5kAOlsmB39mPGH8Q== md5hex: 256920366e6400e96c981dfd98f187f1
2023-07-10 08:48:25,010 INFO  com.amazon.ws.emr.hadoop.fs.s3.upload.dispatch.DefaultMultipartUploadDispatcher [] - Completed multipart upload of 2 parts 152130588 bytes
2023-07-10 08:48:29,646 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 104 for job 40c88e0009d80a880af321102ab3c577<PHONE_NUMBER>79 bytes, checkpointDuration=2691901 ms, finalizationTime=4636 ms).
2023-07-10 08:48:29,648 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Marking checkpoint 104 as completed for source Source: mongo_cdc_test[1].
2023-07-10 08:48:29,712 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230710080054485__deltacommit__INFLIGHT]}
2023-07-10 08:48:29,713 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Scheduling table service COMPACT
2023-07-10 08:48:29,713 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Scheduling compaction at instant time :20230710075931950
2023-07-10 08:48:29,713 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading HoodieTableMetaClient from s3a://xxx/hudi_test19
2023-07-10 08:48:29,741 INFO  org.apache.hudi.common.table.HoodieTableConfig               [] - Loading table properties from s3a://xxx/hudi_test19/.hoodie/hoodie.properties
2023-07-10 08:48:29,787 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from s3a://xxx/hudi_test19
2023-07-10 08:48:29,787 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading Active commit timeline for s3a://xxx/hudi_test19
2023-07-10 08:48:29,828 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230710080054485__deltacommit__INFLIGHT]}
2023-07-10 08:48:29,828 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating View Manager with storage type :REMOTE_FIRST
2023-07-10 08:48:29,828 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating remote first table view
2023-07-10 08:48:29,828 INFO  org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - Checking if compaction needs to be run on s3a://xxx/hudi_test19
2023-07-10 08:48:29,829 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [commits the instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:30,227 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 105 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @<PHONE_NUMBER>647 for job 40c88e0009d80a880af321102ab3c577.
2023-07-10 08:48:30,362 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [taking checkpoint 105] success!
2023-07-10 08:48:31,007 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,017 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,018 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,018 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,018 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,019 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,019 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,020 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,020 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,021 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,021 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,021 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,021 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,022 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,023 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,023 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,024 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,024 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,024 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,024 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,025 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,025 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,025 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,025 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,026 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,026 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,026 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,026 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,028 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,028 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,028 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,028 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,029 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,029 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,029 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,030 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,030 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,030 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,031 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,031 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,032 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,032 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,032 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,033 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,033 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,033 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,033 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,034 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,034 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,034 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,034 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,035 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,035 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,036 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,036 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,037 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,037 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,038 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,041 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,048 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 08:48:31,450 INFO  org.apache.hudi.timeline.service.RequestHandler              [] - Syncing view as client passed last known instant<PHONE_NUMBER>5809416 as last known instant but server has the following last instant on timeline :Option{val=[20230710075809416__deltacommit__COMPLETED]}
2023-07-10 08:48:31,484 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230710080054485__deltacommit__INFLIGHT]}
2023-07-10 08:48:31,485 INFO  org.apache.hudi.common.table.view.AbstractTableFileSystemView [] - Took 0 ms to read  0 instants, 0 replaced file groups
2023-07-10 08:48:31,508 INFO  org.apache.hudi.common.util.ClusteringUtils                  [] - Found 0 files in pending clustering operations
2023-07-10 08:48:32,748 INFO  org.apache.hudi.common.table.view.AbstractTableFileSystemView [] - Building file system view for partition ()
2023-07-10 08:48:33,528 INFO  org.apache.hudi.common.table.view.AbstractTableFileSystemView [] - addFilesToView: NumFiles=2818, NumFileGroups=64, FileGroupsCreationTime=469, StoreTimeTaken=0

still stuck, no exception and all checkpoints success

2023-07-10 08:48:33,528 INFO  org.apache.hudi.common.table.view.AbstractTableFileSystemView [] - addFilesToView: NumFiles=2818, NumFileGroups=64, FileGroupsCreationTime=469, StoreTimeTaken=0
2023-07-10 09:30:18,852 INFO  org.apache.flink.runtime.state.filesystem.FsCheckpointMetadataOutputStream [] - Cannot create recoverable writer due to Recoverable writers on Hadoop are only supported for HDFS, will use the ordinary writer.
2023-07-10 09:30:19,472 INFO  com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream  [] - close closed:false s3://xxx/hudi_test_checkpoint/40c88e0009d80a880af321102ab3c577/chk-105/_metadata
2023-07-10 09:30:19,661 INFO  com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream  [] - uploadPart: partNum 2 of 's3://xxx/hudi_test_checkpoint/40c88e0009d80a880af321102ab3c577/chk-105/_metadata' from local file '/mnt/s3/emrfs-2307899688819587031/0000000001', 17912860 bytes in 189 ms, md5: T1Fh6HPC0W3drOs2XbuK6Q== md5hex: 4f5161e873c2d16dddaceb365dbb8ae9
2023-07-10 09:30:20,413 INFO  com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream  [] - uploadPart: partNum 1 of 's3://xxx/hudi_test_checkpoint/40c88e0009d80a880af321102ab3c577/chk-105/_metadata' from local file '/mnt/s3/emrfs-2307899688819587031/0000000000', 134217728 bytes in 1032 ms, md5: RDj5MgX4jBwYakxvonEzYw== md5hex: 4438f93205f88c1c186a4c6fa2713363
2023-07-10 09:30:20,593 INFO  com.amazon.ws.emr.hadoop.fs.s3.upload.dispatch.DefaultMultipartUploadDispatcher [] - Completed multipart upload of 2 parts 152130588 bytes
2023-07-10 09:30:24,674 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 105 for job 40c88e0009d80a880af321102ab3c577<PHONE_NUMBER>79 bytes, checkpointDuration=2510946 ms, finalizationTime=4081 ms).
2023-07-10 09:30:24,675 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Marking checkpoint 105 as completed for source Source: mongo_cdc_test[1].
2023-07-10 09:30:24,676 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 106 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @<PHONE_NUMBER>675 for job 40c88e0009d80a880af321102ab3c577.
2023-07-10 09:30:24,739 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230710080054485__deltacommit__INFLIGHT]}
2023-07-10 09:30:24,740 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Scheduling table service COMPACT
2023-07-10 09:30:24,740 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Scheduling compaction at instant time :20230710075931950
2023-07-10 09:30:24,740 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading HoodieTableMetaClient from s3a://xxx/hudi_test19
2023-07-10 09:30:24,769 INFO  org.apache.hudi.common.table.HoodieTableConfig               [] - Loading table properties from s3a://xxx/hudi_test19/.hoodie/hoodie.properties
2023-07-10 09:30:24,820 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from s3a://xxx/hudi_test19
2023-07-10 09:30:24,820 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading Active commit timeline for s3a://xxx/hudi_test19
2023-07-10 09:30:24,848 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230710080054485__deltacommit__INFLIGHT]}
2023-07-10 09:30:24,848 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating View Manager with storage type :REMOTE_FIRST
2023-07-10 09:30:24,848 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating remote first table view
2023-07-10 09:30:24,848 INFO  org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - Checking if compaction needs to be run on s3a://xxx/hudi_test19
2023-07-10 09:30:24,849 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [commits the instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:24,849 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [taking checkpoint 106] success!
2023-07-10 09:30:25,565 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,566 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,566 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,566 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,566 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,567 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,568 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,569 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,569 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,569 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,570 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,570 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,570 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,570 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,571 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,571 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,571 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,571 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,572 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,574 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,574 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,575 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,575 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,576 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,576 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,576 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,576 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,577 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,577 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,577 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,577 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,577 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,577 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,578 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,578 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,579 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,579 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,579 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,580 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,581 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,581 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,581 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,581 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,582 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,582 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,582 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,583 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,583 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,584 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,585 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,585 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,585 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,586 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,586 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,587 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,587 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,588 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,589 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,598 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 09:30:25,603 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:08,425 INFO  org.apache.flink.runtime.state.filesystem.FsCheckpointMetadataOutputStream [] - Cannot create recoverable writer due to Recoverable writers on Hadoop are only supported for HDFS, will use the ordinary writer.
2023-07-10 10:12:09,028 INFO  com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream  [] - close closed:false s3://xxx/hudi_test_checkpoint/40c88e0009d80a880af321102ab3c577/chk-106/_metadata
2023-07-10 10:12:09,200 INFO  com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream  [] - uploadPart: partNum 2 of 's3://xxx/hudi_test_checkpoint/40c88e0009d80a880af321102ab3c577/chk-106/_metadata' from local file '/mnt/s3/emrfs-1292721833969541913/0000000001', 17912860 bytes in 172 ms, md5: vVn70UUERV8/WAJ3FdrH3w== md5hex: bd59fbd14504455f3f58027715dac7df
2023-07-10 10:12:09,816 INFO  com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream  [] - uploadPart: partNum 1 of 's3://xxx/hudi_test_checkpoint/40c88e0009d80a880af321102ab3c577/chk-106/_metadata' from local file '/mnt/s3/emrfs-1292721833969541913/0000000000', 134217728 bytes in 886 ms, md5: YsfiHCOxq9s9CIptx6/YFA== md5hex: 62c7e21c23b1abdb3d088a6dc7afd814
2023-07-10 10:12:10,113 INFO  com.amazon.ws.emr.hadoop.fs.s3.upload.dispatch.DefaultMultipartUploadDispatcher [] - Completed multipart upload of 2 parts 152130588 bytes
2023-07-10 10:12:14,249 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 106 for job 40c88e0009d80a880af321102ab3c577<PHONE_NUMBER>79 bytes, checkpointDuration=2505438 ms, finalizationTime=4136 ms).
2023-07-10 10:12:14,250 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Marking checkpoint 106 as completed for source Source: mongo_cdc_test[1].
2023-07-10 10:12:14,252 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 107 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @<PHONE_NUMBER>251 for job 40c88e0009d80a880af321102ab3c577.
2023-07-10 10:12:14,308 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230710080054485__deltacommit__INFLIGHT]}
2023-07-10 10:12:14,308 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Scheduling table service COMPACT
2023-07-10 10:12:14,309 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Scheduling compaction at instant time :20230710075931950
2023-07-10 10:12:14,309 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading HoodieTableMetaClient from s3a://xxx/hudi_test19
2023-07-10 10:12:14,336 INFO  org.apache.hudi.common.table.HoodieTableConfig               [] - Loading table properties from s3a://xxx/hudi_test19/.hoodie/hoodie.properties
2023-07-10 10:12:14,380 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from s3a://xxx/hudi_test19
2023-07-10 10:12:14,380 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading Active commit timeline for s3a://xxx/hudi_test19
2023-07-10 10:12:14,407 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230710080054485__deltacommit__INFLIGHT]}
2023-07-10 10:12:14,407 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating View Manager with storage type :REMOTE_FIRST
2023-07-10 10:12:14,407 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating remote first table view
2023-07-10 10:12:14,408 INFO  org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - Checking if compaction needs to be run on s3a://xxx/hudi_test19
2023-07-10 10:12:14,408 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [commits the instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:14,408 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [taking checkpoint 107] success!
2023-07-10 10:12:15,119 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,132 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,134 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,134 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,134 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,134 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,136 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,138 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,138 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,139 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,139 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,140 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,140 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,140 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,141 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,141 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,141 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,141 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,142 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,142 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,142 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,143 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,144 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,145 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,145 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,145 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,146 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,146 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,146 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,146 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,146 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,146 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,147 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,148 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,149 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,149 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,149 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,156 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,156 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,156 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,156 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,156 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,157 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,157 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,157 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,157 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,157 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,157 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,157 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,157 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,157 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,157 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,157 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,157 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,157 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,157 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,158 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,159 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,161 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:12:15,165 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:54:19,306 INFO  org.apache.flink.runtime.state.filesystem.FsCheckpointMetadataOutputStream [] - Cannot create recoverable writer due to Recoverable writers on Hadoop are only supported for HDFS, will use the ordinary writer.
2023-07-10 10:54:19,903 INFO  com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream  [] - close closed:false s3://xxx/hudi_test_checkpoint/40c88e0009d80a880af321102ab3c577/chk-107/_metadata
2023-07-10 10:54:20,201 INFO  com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream  [] - uploadPart: partNum 2 of 's3://xxx/hudi_test_checkpoint/40c88e0009d80a880af321102ab3c577/chk-107/_metadata' from local file '/mnt/s3/emrfs-6436569385430530837/0000000001', 17912860 bytes in 298 ms, md5: dafiJ0vScEZACFXIiS2I5A== md5hex: 75a7e2274bd27046400855c8892d88e4
2023-07-10 10:54:20,616 INFO  com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream  [] - uploadPart: partNum 1 of 's3://xxx/hudi_test_checkpoint/40c88e0009d80a880af321102ab3c577/chk-107/_metadata' from local file '/mnt/s3/emrfs-6436569385430530837/0000000000', 134217728 bytes in 795 ms, md5: LL2HrbZ/m2cWmPOr33V+HA== md5hex: 2cbd87adb67f9b671698f3abdf757e1c
2023-07-10 10:54:21,021 INFO  com.amazon.ws.emr.hadoop.fs.s3.upload.dispatch.DefaultMultipartUploadDispatcher [] - Completed multipart upload of 2 parts 152130588 bytes
2023-07-10 10:54:25,262 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 107 for job 40c88e0009d80a880af321102ab3c577<PHONE_NUMBER>79 bytes, checkpointDuration=2526770 ms, finalizationTime=4241 ms).
2023-07-10 10:54:25,263 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Marking checkpoint 107 as completed for source Source: mongo_cdc_test[1].
2023-07-10 10:54:25,267 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 108 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @<PHONE_NUMBER>264 for job 40c88e0009d80a880af321102ab3c577.
2023-07-10 10:54:25,317 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230710080054485__deltacommit__INFLIGHT]}
2023-07-10 10:54:25,318 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Scheduling table service COMPACT
2023-07-10 10:54:25,318 INFO  org.apache.hudi.client.BaseHoodieWriteClient                 [] - Scheduling compaction at instant time :20230710075931950
2023-07-10 10:54:25,318 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading HoodieTableMetaClient from s3a://xxx/hudi_test19
2023-07-10 10:54:25,343 INFO  org.apache.hudi.common.table.HoodieTableConfig               [] - Loading table properties from s3a://xxx/hudi_test19/.hoodie/hoodie.properties
2023-07-10 10:54:25,384 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=PARQUET) from s3a://xxx/hudi_test19
2023-07-10 10:54:25,384 INFO  org.apache.hudi.common.table.HoodieTableMetaClient           [] - Loading Active commit timeline for s3a://xxx/hudi_test19
2023-07-10 10:54:25,414 INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline   [] - Loaded instants upto : Option{val=[==>20230710080054485__deltacommit__INFLIGHT]}
2023-07-10 10:54:25,414 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating View Manager with storage type :REMOTE_FIRST
2023-07-10 10:54:25,414 INFO  org.apache.hudi.common.table.view.FileSystemViewManager      [] - Creating remote first table view
2023-07-10 10:54:25,414 INFO  org.apache.hudi.table.action.compact.ScheduleCompactionActionExecutor [] - Checking if compaction needs to be run on s3a://xxx/hudi_test19
2023-07-10 10:54:25,415 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [commits the instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:54:25,415 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [taking checkpoint 108] success!
2023-07-10 10:54:26,233 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:54:26,233 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:54:26,234 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:54:26,234 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:54:26,235 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:54:26,235 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:54:26,235 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:54:26,236 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:54:26,237 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:54:26,238 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:54:26,238 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:54:26,238 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!
2023-07-10 10:54:26,238 INFO  org.apache.hudi.sink.StreamWriteOperatorCoordinator          [] - Executor executes action [handle write metadata event for instant<PHONE_NUMBER>0054485] success!