24/03/01 11:21:06 INFO FileScanRDD: Reading File path: file:///Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/sampledata/orders/bd14f873-5e9e-41b4-a767-830823f49e97_orders.csv, range: 0-13076, partition values: [empty row]
24/03/01 11:21:06 INFO HoodieSparkKeyGeneratorFactory: The value of hoodie.datasource.write.keygenerator.type is empty; inferred to be SIMPLE
24/03/01 11:21:06 INFO HoodieSparkKeyGeneratorFactory: The value of hoodie.datasource.write.keygenerator.type is empty; inferred to be SIMPLE
24/03/01 11:21:06 INFO HoodieSparkKeyGeneratorFactory: The value of hoodie.datasource.write.keygenerator.type is empty; inferred to be SIMPLE
24/03/01 11:21:06 INFO HoodieSparkKeyGeneratorFactory: The value of hoodie.datasource.write.keygenerator.type is empty; inferred to be SIMPLE
24/03/01 11:21:06 INFO HoodieSparkKeyGeneratorFactory: The value of hoodie.datasource.write.keygenerator.type is empty; inferred to be SIMPLE
24/03/01 11:21:06 INFO HoodieSparkKeyGeneratorFactory: The value of hoodie.datasource.write.keygenerator.type is empty; inferred to be SIMPLE
24/03/01 11:21:06 INFO HoodieSparkKeyGeneratorFactory: The value of hoodie.datasource.write.keygenerator.type is empty; inferred to be SIMPLE
24/03/01 11:21:06 INFO HoodieSparkKeyGeneratorFactory: The value of hoodie.datasource.write.keygenerator.type is empty; inferred to be SIMPLE
24/03/01 11:21:06 INFO Executor: Finished task 3.0 in stage 13.0 (TID 22). 1463 bytes result sent to driver
24/03/01 11:21:06 INFO Executor: Finished task 1.0 in stage 13.0 (TID 20). 1463 bytes result sent to driver
24/03/01 11:21:06 INFO Executor: Finished task 4.0 in stage 13.0 (TID 23). 1463 bytes result sent to driver
24/03/01 11:21:06 INFO TaskSetManager: Finished task 3.0 in stage 13.0 (TID 22) in 128 ms on soumils-mbp (executor driver) (1/8)
24/03/01 11:21:06 INFO TaskSetManager: Finished task 1.0 in stage 13.0 (TID 20) in 130 ms on soumils-mbp (executor driver) (2/8)
24/03/01 11:21:06 INFO TaskSetManager: Finished task 4.0 in stage 13.0 (TID 23) in 130 ms on soumils-mbp (executor driver) (3/8)
24/03/01 11:21:06 INFO Executor: Finished task 6.0 in stage 13.0 (TID 25). 1506 bytes result sent to driver
24/03/01 11:21:06 INFO Executor: Finished task 5.0 in stage 13.0 (TID 24). 1463 bytes result sent to driver
24/03/01 11:21:06 INFO Executor: Finished task 7.0 in stage 13.0 (TID 26). 1506 bytes result sent to driver
24/03/01 11:21:06 INFO TaskSetManager: Finished task 6.0 in stage 13.0 (TID 25) in 132 ms on soumils-mbp (executor driver) (4/8)
24/03/01 11:21:06 INFO Executor: Finished task 0.0 in stage 13.0 (TID 19). 1463 bytes result sent to driver
24/03/01 11:21:06 INFO TaskSetManager: Finished task 7.0 in stage 13.0 (TID 26) in 132 ms on soumils-mbp (executor driver) (5/8)
24/03/01 11:21:06 INFO TaskSetManager: Finished task 0.0 in stage 13.0 (TID 19) in 134 ms on soumils-mbp (executor driver) (6/8)
24/03/01 11:21:06 INFO TaskSetManager: Finished task 5.0 in stage 13.0 (TID 24) in 134 ms on soumils-mbp (executor driver) (7/8)
24/03/01 11:21:06 INFO Executor: Finished task 2.0 in stage 13.0 (TID 21). 1506 bytes result sent to driver
24/03/01 11:21:06 INFO TaskSetManager: Finished task 2.0 in stage 13.0 (TID 21) in 134 ms on soumils-mbp (executor driver) (8/8)
24/03/01 11:21:06 INFO TaskSchedulerImpl: Removed TaskSet 13.0, whose tasks have all completed, from pool
24/03/01 11:21:06 INFO DAGScheduler: ShuffleMapStage 13 (mapToPair at HoodieJavaRDD.java:149) finished in 0.139 s
24/03/01 11:21:06 INFO DAGScheduler: looking for newly runnable stages
24/03/01 11:21:06 INFO DAGScheduler: running: Set()
24/03/01 11:21:06 INFO DAGScheduler: waiting: Set(ResultStage 15, ShuffleMapStage 14)
24/03/01 11:21:06 INFO DAGScheduler: failed: Set()
24/03/01 11:21:06 INFO DAGScheduler: Submitting ShuffleMapStage 14 (MapPartitionsRDD[40] at distinct at HoodieJavaRDD.java:157), which has no missing parents
24/03/01 11:21:06 INFO MemoryStore: Block broadcast_16 stored as values in memory (estimated size 26.5 KiB, free 433.3 MiB)
24/03/01 11:21:06 INFO MemoryStore: Block broadcast_16_piece0 stored as bytes in memory (estimated size 12.5 KiB, free 433.3 MiB)
24/03/01 11:21:06 INFO BlockManagerInfo: Added broadcast_16_piece0 in memory on soumils-mbp:49385 (size: 12.5 KiB, free: 434.1 MiB)
24/03/01 11:21:06 INFO SparkContext: Created broadcast 16 from broadcast at DAGScheduler.scala:1535
24/03/01 11:21:06 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 14 (MapPartitionsRDD[40] at distinct at HoodieJavaRDD.java:157) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7))
24/03/01 11:21:06 INFO TaskSchedulerImpl: Adding task set 14.0 with 8 tasks resource profile 0
24/03/01 11:21:06 INFO TaskSetManager: Starting task 0.0 in stage 14.0 (TID 27) (soumils-mbp, executor driver, partition 0, NODE_LOCAL, 7170 bytes)
24/03/01 11:21:06 INFO TaskSetManager: Starting task 1.0 in stage 14.0 (TID 28) (soumils-mbp, executor driver, partition 1, NODE_LOCAL, 7170 bytes)
24/03/01 11:21:06 INFO TaskSetManager: Starting task 2.0 in stage 14.0 (TID 29) (soumils-mbp, executor driver, partition 2, NODE_LOCAL, 7170 bytes)
24/03/01 11:21:06 INFO TaskSetManager: Starting task 3.0 in stage 14.0 (TID 30) (soumils-mbp, executor driver, partition 3, NODE_LOCAL, 7170 bytes)
24/03/01 11:21:06 INFO TaskSetManager: Starting task 4.0 in stage 14.0 (TID 31) (soumils-mbp, executor driver, partition 4, NODE_LOCAL, 7170 bytes)
24/03/01 11:21:06 INFO TaskSetManager: Starting task 5.0 in stage 14.0 (TID 32) (soumils-mbp, executor driver, partition 5, NODE_LOCAL, 7170 bytes)
24/03/01 11:21:06 INFO TaskSetManager: Starting task 6.0 in stage 14.0 (TID 33) (soumils-mbp, executor driver, partition 6, NODE_LOCAL, 7170 bytes)
24/03/01 11:21:06 INFO TaskSetManager: Starting task 7.0 in stage 14.0 (TID 34) (soumils-mbp, executor driver, partition 7, NODE_LOCAL, 7170 bytes)
24/03/01 11:21:06 INFO Executor: Running task 1.0 in stage 14.0 (TID 28)
24/03/01 11:21:06 INFO Executor: Running task 0.0 in stage 14.0 (TID 27)
24/03/01 11:21:06 INFO Executor: Running task 2.0 in stage 14.0 (TID 29)
24/03/01 11:21:06 INFO Executor: Running task 4.0 in stage 14.0 (TID 31)
24/03/01 11:21:06 INFO Executor: Running task 3.0 in stage 14.0 (TID 30)
24/03/01 11:21:06 INFO Executor: Running task 5.0 in stage 14.0 (TID 32)
24/03/01 11:21:06 INFO Executor: Running task 7.0 in stage 14.0 (TID 34)
24/03/01 11:21:06 INFO Executor: Running task 6.0 in stage 14.0 (TID 33)
24/03/01 11:21:06 INFO ShuffleBlockFetcherIterator: Getting 8 (14.3 KiB) non-empty blocks including 8 (14.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:06 INFO ShuffleBlockFetcherIterator: Getting 8 (13.7 KiB) non-empty blocks including 8 (13.7 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:06 INFO ShuffleBlockFetcherIterator: Getting 8 (13.8 KiB) non-empty blocks including 8 (13.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:06 INFO ShuffleBlockFetcherIterator: Getting 8 (14.0 KiB) non-empty blocks including 8 (14.0 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:06 INFO ShuffleBlockFetcherIterator: Getting 8 (13.9 KiB) non-empty blocks including 8 (13.9 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:06 INFO ShuffleBlockFetcherIterator: Getting 8 (15.7 KiB) non-empty blocks including 8 (15.7 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
24/03/01 11:21:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:06 INFO ShuffleBlockFetcherIterator: Getting 8 (13.6 KiB) non-empty blocks including 8 (13.6 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
24/03/01 11:21:06 INFO ShuffleBlockFetcherIterator: Getting 8 (13.6 KiB) non-empty blocks including 8 (13.6 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
24/03/01 11:21:07 INFO MemoryStore: Block rdd_36_2 stored as values in memory (estimated size 18.1 KiB, free 433.3 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Added rdd_36_2 in memory on soumils-mbp:49385 (size: 18.1 KiB, free: 434.1 MiB)
24/03/01 11:21:07 INFO MemoryStore: Block rdd_36_5 stored as values in memory (estimated size 17.4 KiB, free 433.3 MiB)
24/03/01 11:21:07 INFO MemoryStore: Block rdd_36_1 stored as values in memory (estimated size 17.7 KiB, free 433.2 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Added rdd_36_5 in memory on soumils-mbp:49385 (size: 17.4 KiB, free: 434.1 MiB)
24/03/01 11:21:07 INFO MemoryStore: Block rdd_36_0 stored as values in memory (estimated size 17.7 KiB, free 433.2 MiB)
24/03/01 11:21:07 INFO MemoryStore: Block rdd_36_6 stored as values in memory (estimated size 17.2 KiB, free 433.2 MiB)
24/03/01 11:21:07 INFO MemoryStore: Block rdd_36_7 stored as values in memory (estimated size 20.0 KiB, free 433.2 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Added rdd_36_1 in memory on soumils-mbp:49385 (size: 17.7 KiB, free: 434.1 MiB)
24/03/01 11:21:07 INFO MemoryStore: Block rdd_36_4 stored as values in memory (estimated size 17.9 KiB, free 433.2 MiB)
24/03/01 11:21:07 INFO MemoryStore: Block rdd_36_3 stored as values in memory (estimated size 17.2 KiB, free 433.2 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Added rdd_36_6 in memory on soumils-mbp:49385 (size: 17.2 KiB, free: 434.1 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Added rdd_36_0 in memory on soumils-mbp:49385 (size: 17.7 KiB, free: 434.1 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Added rdd_36_7 in memory on soumils-mbp:49385 (size: 20.0 KiB, free: 434.0 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Added rdd_36_4 in memory on soumils-mbp:49385 (size: 17.9 KiB, free: 434.0 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Added rdd_36_3 in memory on soumils-mbp:49385 (size: 17.2 KiB, free: 434.0 MiB)
24/03/01 11:21:07 INFO Executor: Finished task 2.0 in stage 14.0 (TID 29). 1842 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 2.0 in stage 14.0 (TID 29) in 98 ms on soumils-mbp (executor driver) (1/8)
24/03/01 11:21:07 INFO Executor: Finished task 1.0 in stage 14.0 (TID 28). 1842 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 5.0 in stage 14.0 (TID 32). 1842 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 1.0 in stage 14.0 (TID 28) in 100 ms on soumils-mbp (executor driver) (2/8)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 5.0 in stage 14.0 (TID 32) in 100 ms on soumils-mbp (executor driver) (3/8)
24/03/01 11:21:07 INFO Executor: Finished task 0.0 in stage 14.0 (TID 27). 1842 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 0.0 in stage 14.0 (TID 27) in 102 ms on soumils-mbp (executor driver) (4/8)
24/03/01 11:21:07 INFO Executor: Finished task 6.0 in stage 14.0 (TID 33). 1842 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 6.0 in stage 14.0 (TID 33) in 102 ms on soumils-mbp (executor driver) (5/8)
24/03/01 11:21:07 INFO Executor: Finished task 7.0 in stage 14.0 (TID 34). 1842 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 7.0 in stage 14.0 (TID 34) in 102 ms on soumils-mbp (executor driver) (6/8)
24/03/01 11:21:07 INFO Executor: Finished task 3.0 in stage 14.0 (TID 30). 1842 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 3.0 in stage 14.0 (TID 30) in 103 ms on soumils-mbp (executor driver) (7/8)
24/03/01 11:21:07 INFO Executor: Finished task 4.0 in stage 14.0 (TID 31). 1842 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 4.0 in stage 14.0 (TID 31) in 103 ms on soumils-mbp (executor driver) (8/8)
24/03/01 11:21:07 INFO TaskSchedulerImpl: Removed TaskSet 14.0, whose tasks have all completed, from pool
24/03/01 11:21:07 INFO DAGScheduler: ShuffleMapStage 14 (distinct at HoodieJavaRDD.java:157) finished in 0.109 s
24/03/01 11:21:07 INFO DAGScheduler: looking for newly runnable stages
24/03/01 11:21:07 INFO DAGScheduler: running: Set()
24/03/01 11:21:07 INFO DAGScheduler: waiting: Set(ResultStage 15)
24/03/01 11:21:07 INFO DAGScheduler: failed: Set()
24/03/01 11:21:07 INFO DAGScheduler: Submitting ResultStage 15 (MapPartitionsRDD[42] at distinct at HoodieJavaRDD.java:157), which has no missing parents
24/03/01 11:21:07 INFO MemoryStore: Block broadcast_17 stored as values in memory (estimated size 6.3 KiB, free 433.2 MiB)
24/03/01 11:21:07 INFO MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 3.5 KiB, free 433.2 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Added broadcast_17_piece0 in memory on soumils-mbp:49385 (size: 3.5 KiB, free: 434.0 MiB)
24/03/01 11:21:07 INFO SparkContext: Created broadcast 17 from broadcast at DAGScheduler.scala:1535
24/03/01 11:21:07 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 15 (MapPartitionsRDD[42] at distinct at HoodieJavaRDD.java:157) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7))
24/03/01 11:21:07 INFO TaskSchedulerImpl: Adding task set 15.0 with 8 tasks resource profile 0
24/03/01 11:21:07 INFO TaskSetManager: Starting task 0.0 in stage 15.0 (TID 35) (soumils-mbp, executor driver, partition 0, NODE_LOCAL, 7181 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 1.0 in stage 15.0 (TID 36) (soumils-mbp, executor driver, partition 1, NODE_LOCAL, 7181 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 2.0 in stage 15.0 (TID 37) (soumils-mbp, executor driver, partition 2, NODE_LOCAL, 7181 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 3.0 in stage 15.0 (TID 38) (soumils-mbp, executor driver, partition 3, NODE_LOCAL, 7181 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 4.0 in stage 15.0 (TID 39) (soumils-mbp, executor driver, partition 4, NODE_LOCAL, 7181 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 5.0 in stage 15.0 (TID 40) (soumils-mbp, executor driver, partition 5, NODE_LOCAL, 7181 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 6.0 in stage 15.0 (TID 41) (soumils-mbp, executor driver, partition 6, NODE_LOCAL, 7181 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 7.0 in stage 15.0 (TID 42) (soumils-mbp, executor driver, partition 7, NODE_LOCAL, 7181 bytes)
24/03/01 11:21:07 INFO Executor: Running task 1.0 in stage 15.0 (TID 36)
24/03/01 11:21:07 INFO Executor: Running task 0.0 in stage 15.0 (TID 35)
24/03/01 11:21:07 INFO Executor: Running task 2.0 in stage 15.0 (TID 37)
24/03/01 11:21:07 INFO Executor: Running task 3.0 in stage 15.0 (TID 38)
24/03/01 11:21:07 INFO Executor: Running task 6.0 in stage 15.0 (TID 41)
24/03/01 11:21:07 INFO Executor: Running task 5.0 in stage 15.0 (TID 40)
24/03/01 11:21:07 INFO Executor: Running task 4.0 in stage 15.0 (TID 39)
24/03/01 11:21:07 INFO Executor: Running task 7.0 in stage 15.0 (TID 42)
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (640.0 B) non-empty blocks including 8 (640.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (821.0 B) non-empty blocks including 8 (821.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (656.0 B) non-empty blocks including 8 (656.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (749.0 B) non-empty blocks including 8 (749.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (803.0 B) non-empty blocks including 8 (803.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (648.0 B) non-empty blocks including 8 (648.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (664.0 B) non-empty blocks including 8 (664.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (750.0 B) non-empty blocks including 8 (750.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO Executor: Finished task 5.0 in stage 15.0 (TID 40). 1789 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 1.0 in stage 15.0 (TID 36). 1729 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 7.0 in stage 15.0 (TID 42). 1765 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 2.0 in stage 15.0 (TID 37). 1729 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 5.0 in stage 15.0 (TID 40) in 7 ms on soumils-mbp (executor driver) (1/8)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 1.0 in stage 15.0 (TID 36) in 7 ms on soumils-mbp (executor driver) (2/8)
24/03/01 11:21:07 INFO Executor: Finished task 4.0 in stage 15.0 (TID 39). 1765 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 0.0 in stage 15.0 (TID 35). 1784 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 2.0 in stage 15.0 (TID 37) in 8 ms on soumils-mbp (executor driver) (3/8)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 4.0 in stage 15.0 (TID 39) in 8 ms on soumils-mbp (executor driver) (4/8)
24/03/01 11:21:07 INFO Executor: Finished task 6.0 in stage 15.0 (TID 41). 1820 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 3.0 in stage 15.0 (TID 38). 1772 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 7.0 in stage 15.0 (TID 42) in 9 ms on soumils-mbp (executor driver) (5/8)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 0.0 in stage 15.0 (TID 35) in 11 ms on soumils-mbp (executor driver) (6/8)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 6.0 in stage 15.0 (TID 41) in 10 ms on soumils-mbp (executor driver) (7/8)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 3.0 in stage 15.0 (TID 38) in 10 ms on soumils-mbp (executor driver) (8/8)
24/03/01 11:21:07 INFO TaskSchedulerImpl: Removed TaskSet 15.0, whose tasks have all completed, from pool
24/03/01 11:21:07 INFO DAGScheduler: ResultStage 15 (collect at HoodieJavaRDD.java:177) finished in 0.014 s
24/03/01 11:21:07 INFO DAGScheduler: Job 11 is finished. Cancelling potential speculative or zombie tasks for this job
24/03/01 11:21:07 INFO TaskSchedulerImpl: Killing all running tasks in stage 15: Stage finished
24/03/01 11:21:07 INFO DAGScheduler: Job 11 finished: collect at HoodieJavaRDD.java:177, took 0.267255 s
24/03/01 11:21:07 INFO SparkContext: Starting job: collect at HoodieSparkEngineContext.java:150
24/03/01 11:21:07 INFO DAGScheduler: Got job 12 (collect at HoodieSparkEngineContext.java:150) with 56 output partitions
24/03/01 11:21:07 INFO DAGScheduler: Final stage: ResultStage 16 (collect at HoodieSparkEngineContext.java:150)
24/03/01 11:21:07 INFO DAGScheduler: Parents of final stage: List()
24/03/01 11:21:07 INFO DAGScheduler: Missing parents: List()
24/03/01 11:21:07 INFO DAGScheduler: Submitting ResultStage 16 (MapPartitionsRDD[44] at flatMap at HoodieSparkEngineContext.java:150), which has no missing parents
24/03/01 11:21:07 INFO MemoryStore: Block broadcast_18 stored as values in memory (estimated size 449.8 KiB, free 432.7 MiB)
24/03/01 11:21:07 INFO MemoryStore: Block broadcast_18_piece0 stored as bytes in memory (estimated size 157.2 KiB, free 432.6 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Added broadcast_18_piece0 in memory on soumils-mbp:49385 (size: 157.2 KiB, free: 433.8 MiB)
24/03/01 11:21:07 INFO SparkContext: Created broadcast 18 from broadcast at DAGScheduler.scala:1535
24/03/01 11:21:07 INFO DAGScheduler: Submitting 56 missing tasks from ResultStage 16 (MapPartitionsRDD[44] at flatMap at HoodieSparkEngineContext.java:150) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
24/03/01 11:21:07 INFO TaskSchedulerImpl: Adding task set 16.0 with 56 tasks resource profile 0
24/03/01 11:21:07 INFO TaskSetManager: Starting task 0.0 in stage 16.0 (TID 43) (soumils-mbp, executor driver, partition 0, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 1.0 in stage 16.0 (TID 44) (soumils-mbp, executor driver, partition 1, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 2.0 in stage 16.0 (TID 45) (soumils-mbp, executor driver, partition 2, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 3.0 in stage 16.0 (TID 46) (soumils-mbp, executor driver, partition 3, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 4.0 in stage 16.0 (TID 47) (soumils-mbp, executor driver, partition 4, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 5.0 in stage 16.0 (TID 48) (soumils-mbp, executor driver, partition 5, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 6.0 in stage 16.0 (TID 49) (soumils-mbp, executor driver, partition 6, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 7.0 in stage 16.0 (TID 50) (soumils-mbp, executor driver, partition 7, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 8.0 in stage 16.0 (TID 51) (soumils-mbp, executor driver, partition 8, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 9.0 in stage 16.0 (TID 52) (soumils-mbp, executor driver, partition 9, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 10.0 in stage 16.0 (TID 53) (soumils-mbp, executor driver, partition 10, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 11.0 in stage 16.0 (TID 54) (soumils-mbp, executor driver, partition 11, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 1.0 in stage 16.0 (TID 44)
24/03/01 11:21:07 INFO Executor: Running task 4.0 in stage 16.0 (TID 47)
24/03/01 11:21:07 INFO Executor: Running task 3.0 in stage 16.0 (TID 46)
24/03/01 11:21:07 INFO Executor: Running task 0.0 in stage 16.0 (TID 43)
24/03/01 11:21:07 INFO Executor: Running task 5.0 in stage 16.0 (TID 48)
24/03/01 11:21:07 INFO Executor: Running task 2.0 in stage 16.0 (TID 45)
24/03/01 11:21:07 INFO Executor: Running task 7.0 in stage 16.0 (TID 50)
24/03/01 11:21:07 INFO Executor: Running task 6.0 in stage 16.0 (TID 49)
24/03/01 11:21:07 INFO Executor: Running task 8.0 in stage 16.0 (TID 51)
24/03/01 11:21:07 INFO Executor: Running task 9.0 in stage 16.0 (TID 52)
24/03/01 11:21:07 INFO Executor: Running task 11.0 in stage 16.0 (TID 54)
24/03/01 11:21:07 INFO Executor: Running task 10.0 in stage 16.0 (TID 53)
24/03/01 11:21:07 INFO BlockManagerInfo: Removed broadcast_13_piece0 on soumils-mbp:49385 in memory (size: 35.8 KiB, free: 433.9 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Removed broadcast_17_piece0 on soumils-mbp:49385 in memory (size: 3.5 KiB, free: 433.9 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Removed broadcast_14_piece0 on soumils-mbp:49385 in memory (size: 35.8 KiB, free: 433.9 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Removed broadcast_15_piece0 on soumils-mbp:49385 in memory (size: 19.9 KiB, free: 433.9 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Removed broadcast_12_piece0 on soumils-mbp:49385 in memory (size: 123.8 KiB, free: 434.1 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Removed broadcast_16_piece0 on soumils-mbp:49385 in memory (size: 12.5 KiB, free: 434.1 MiB)
24/03/01 11:21:07 INFO BlockManager: Removing RDD 28
24/03/01 11:21:07 INFO Executor: Finished task 5.0 in stage 16.0 (TID 48). 809 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 1.0 in stage 16.0 (TID 44). 852 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 3.0 in stage 16.0 (TID 46). 809 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 12.0 in stage 16.0 (TID 55) (soumils-mbp, executor driver, partition 12, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 13.0 in stage 16.0 (TID 56) (soumils-mbp, executor driver, partition 13, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 12.0 in stage 16.0 (TID 55)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 14.0 in stage 16.0 (TID 57) (soumils-mbp, executor driver, partition 14, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 13.0 in stage 16.0 (TID 56)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 1.0 in stage 16.0 (TID 44) in 93 ms on soumils-mbp (executor driver) (1/56)
24/03/01 11:21:07 INFO Executor: Running task 14.0 in stage 16.0 (TID 57)
24/03/01 11:21:07 INFO Executor: Finished task 8.0 in stage 16.0 (TID 51). 809 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 10.0 in stage 16.0 (TID 53). 809 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 5.0 in stage 16.0 (TID 48) in 93 ms on soumils-mbp (executor driver) (2/56)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 15.0 in stage 16.0 (TID 58) (soumils-mbp, executor driver, partition 15, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 16.0 in stage 16.0 (TID 59) (soumils-mbp, executor driver, partition 16, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 15.0 in stage 16.0 (TID 58)
24/03/01 11:21:07 INFO Executor: Running task 16.0 in stage 16.0 (TID 59)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 10.0 in stage 16.0 (TID 53) in 93 ms on soumils-mbp (executor driver) (3/56)
24/03/01 11:21:07 INFO Executor: Finished task 6.0 in stage 16.0 (TID 49). 809 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 0.0 in stage 16.0 (TID 43). 895 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 11.0 in stage 16.0 (TID 54). 852 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 9.0 in stage 16.0 (TID 52). 852 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 4.0 in stage 16.0 (TID 47). 895 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 2.0 in stage 16.0 (TID 45). 852 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 17.0 in stage 16.0 (TID 60) (soumils-mbp, executor driver, partition 17, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 17.0 in stage 16.0 (TID 60)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 18.0 in stage 16.0 (TID 61) (soumils-mbp, executor driver, partition 18, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Finished task 7.0 in stage 16.0 (TID 50). 852 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 3.0 in stage 16.0 (TID 46) in 98 ms on soumils-mbp (executor driver) (4/56)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 8.0 in stage 16.0 (TID 51) in 98 ms on soumils-mbp (executor driver) (5/56)
24/03/01 11:21:07 INFO Executor: Running task 18.0 in stage 16.0 (TID 61)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 0.0 in stage 16.0 (TID 43) in 99 ms on soumils-mbp (executor driver) (6/56)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 6.0 in stage 16.0 (TID 49) in 98 ms on soumils-mbp (executor driver) (7/56)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 19.0 in stage 16.0 (TID 62) (soumils-mbp, executor driver, partition 19, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 19.0 in stage 16.0 (TID 62)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 20.0 in stage 16.0 (TID 63) (soumils-mbp, executor driver, partition 20, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 11.0 in stage 16.0 (TID 54) in 98 ms on soumils-mbp (executor driver) (8/56)
24/03/01 11:21:07 INFO Executor: Running task 20.0 in stage 16.0 (TID 63)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 21.0 in stage 16.0 (TID 64) (soumils-mbp, executor driver, partition 21, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 4.0 in stage 16.0 (TID 47) in 99 ms on soumils-mbp (executor driver) (9/56)
24/03/01 11:21:07 INFO Executor: Running task 21.0 in stage 16.0 (TID 64)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 9.0 in stage 16.0 (TID 52) in 99 ms on soumils-mbp (executor driver) (10/56)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 22.0 in stage 16.0 (TID 65) (soumils-mbp, executor driver, partition 22, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 22.0 in stage 16.0 (TID 65)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 23.0 in stage 16.0 (TID 66) (soumils-mbp, executor driver, partition 23, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 7.0 in stage 16.0 (TID 50) in 100 ms on soumils-mbp (executor driver) (11/56)
24/03/01 11:21:07 INFO Executor: Running task 23.0 in stage 16.0 (TID 66)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 2.0 in stage 16.0 (TID 45) in 101 ms on soumils-mbp (executor driver) (12/56)
24/03/01 11:21:07 INFO Executor: Finished task 12.0 in stage 16.0 (TID 55). 766 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 14.0 in stage 16.0 (TID 57). 766 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 24.0 in stage 16.0 (TID 67) (soumils-mbp, executor driver, partition 24, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 24.0 in stage 16.0 (TID 67)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 25.0 in stage 16.0 (TID 68) (soumils-mbp, executor driver, partition 25, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 14.0 in stage 16.0 (TID 57) in 37 ms on soumils-mbp (executor driver) (13/56)
24/03/01 11:21:07 INFO Executor: Running task 25.0 in stage 16.0 (TID 68)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 12.0 in stage 16.0 (TID 55) in 37 ms on soumils-mbp (executor driver) (14/56)
24/03/01 11:21:07 INFO Executor: Finished task 15.0 in stage 16.0 (TID 58). 766 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 26.0 in stage 16.0 (TID 69) (soumils-mbp, executor driver, partition 26, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 15.0 in stage 16.0 (TID 58) in 48 ms on soumils-mbp (executor driver) (15/56)
24/03/01 11:21:07 INFO Executor: Running task 26.0 in stage 16.0 (TID 69)
24/03/01 11:21:07 INFO Executor: Finished task 19.0 in stage 16.0 (TID 62). 766 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 27.0 in stage 16.0 (TID 70) (soumils-mbp, executor driver, partition 27, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 27.0 in stage 16.0 (TID 70)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 19.0 in stage 16.0 (TID 62) in 47 ms on soumils-mbp (executor driver) (16/56)
24/03/01 11:21:07 INFO Executor: Finished task 18.0 in stage 16.0 (TID 61). 809 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 28.0 in stage 16.0 (TID 71) (soumils-mbp, executor driver, partition 28, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 18.0 in stage 16.0 (TID 61) in 50 ms on soumils-mbp (executor driver) (17/56)
24/03/01 11:21:07 INFO Executor: Running task 28.0 in stage 16.0 (TID 71)
24/03/01 11:21:07 INFO Executor: Finished task 16.0 in stage 16.0 (TID 59). 766 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 29.0 in stage 16.0 (TID 72) (soumils-mbp, executor driver, partition 29, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 16.0 in stage 16.0 (TID 59) in 54 ms on soumils-mbp (executor driver) (18/56)
24/03/01 11:21:07 INFO Executor: Running task 29.0 in stage 16.0 (TID 72)
24/03/01 11:21:07 INFO Executor: Finished task 22.0 in stage 16.0 (TID 65). 766 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 30.0 in stage 16.0 (TID 73) (soumils-mbp, executor driver, partition 30, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 22.0 in stage 16.0 (TID 65) in 49 ms on soumils-mbp (executor driver) (19/56)
24/03/01 11:21:07 INFO Executor: Running task 30.0 in stage 16.0 (TID 73)
24/03/01 11:21:07 INFO Executor: Finished task 13.0 in stage 16.0 (TID 56). 766 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 17.0 in stage 16.0 (TID 60). 766 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 20.0 in stage 16.0 (TID 63). 766 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 31.0 in stage 16.0 (TID 74) (soumils-mbp, executor driver, partition 31, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 31.0 in stage 16.0 (TID 74)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 32.0 in stage 16.0 (TID 75) (soumils-mbp, executor driver, partition 32, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Finished task 23.0 in stage 16.0 (TID 66). 766 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 33.0 in stage 16.0 (TID 76) (soumils-mbp, executor driver, partition 33, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 32.0 in stage 16.0 (TID 75)
24/03/01 11:21:07 INFO Executor: Running task 33.0 in stage 16.0 (TID 76)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 34.0 in stage 16.0 (TID 77) (soumils-mbp, executor driver, partition 34, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 17.0 in stage 16.0 (TID 60) in 58 ms on soumils-mbp (executor driver) (20/56)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 20.0 in stage 16.0 (TID 63) in 54 ms on soumils-mbp (executor driver) (21/56)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 13.0 in stage 16.0 (TID 56) in 60 ms on soumils-mbp (executor driver) (22/56)
24/03/01 11:21:07 INFO Executor: Running task 34.0 in stage 16.0 (TID 77)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 23.0 in stage 16.0 (TID 66) in 53 ms on soumils-mbp (executor driver) (23/56)
24/03/01 11:21:07 INFO Executor: Finished task 21.0 in stage 16.0 (TID 64). 766 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 35.0 in stage 16.0 (TID 78) (soumils-mbp, executor driver, partition 35, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 21.0 in stage 16.0 (TID 64) in 59 ms on soumils-mbp (executor driver) (24/56)
24/03/01 11:21:07 INFO Executor: Running task 35.0 in stage 16.0 (TID 78)
24/03/01 11:21:07 INFO Executor: Finished task 25.0 in stage 16.0 (TID 68). 766 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 36.0 in stage 16.0 (TID 79) (soumils-mbp, executor driver, partition 36, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 36.0 in stage 16.0 (TID 79)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 25.0 in stage 16.0 (TID 68) in 56 ms on soumils-mbp (executor driver) (25/56)
24/03/01 11:21:07 INFO Executor: Finished task 24.0 in stage 16.0 (TID 67). 766 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 37.0 in stage 16.0 (TID 80) (soumils-mbp, executor driver, partition 37, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 24.0 in stage 16.0 (TID 67) in 60 ms on soumils-mbp (executor driver) (26/56)
24/03/01 11:21:07 INFO Executor: Running task 37.0 in stage 16.0 (TID 80)
24/03/01 11:21:07 INFO Executor: Finished task 26.0 in stage 16.0 (TID 69). 809 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 28.0 in stage 16.0 (TID 71). 809 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 38.0 in stage 16.0 (TID 81) (soumils-mbp, executor driver, partition 38, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 39.0 in stage 16.0 (TID 82) (soumils-mbp, executor driver, partition 39, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Finished task 27.0 in stage 16.0 (TID 70). 809 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Running task 38.0 in stage 16.0 (TID 81)
24/03/01 11:21:07 INFO Executor: Running task 39.0 in stage 16.0 (TID 82)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 26.0 in stage 16.0 (TID 69) in 65 ms on soumils-mbp (executor driver) (27/56)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 40.0 in stage 16.0 (TID 83) (soumils-mbp, executor driver, partition 40, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 28.0 in stage 16.0 (TID 71) in 62 ms on soumils-mbp (executor driver) (28/56)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 27.0 in stage 16.0 (TID 70) in 64 ms on soumils-mbp (executor driver) (29/56)
24/03/01 11:21:07 INFO Executor: Running task 40.0 in stage 16.0 (TID 83)
24/03/01 11:21:07 INFO Executor: Finished task 29.0 in stage 16.0 (TID 72). 809 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 30.0 in stage 16.0 (TID 73). 809 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 41.0 in stage 16.0 (TID 84) (soumils-mbp, executor driver, partition 41, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 42.0 in stage 16.0 (TID 85) (soumils-mbp, executor driver, partition 42, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 41.0 in stage 16.0 (TID 84)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 30.0 in stage 16.0 (TID 73) in 61 ms on soumils-mbp (executor driver) (30/56)
24/03/01 11:21:07 INFO Executor: Running task 42.0 in stage 16.0 (TID 85)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 29.0 in stage 16.0 (TID 72) in 63 ms on soumils-mbp (executor driver) (31/56)
24/03/01 11:21:07 INFO Executor: Finished task 33.0 in stage 16.0 (TID 76). 809 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 43.0 in stage 16.0 (TID 86) (soumils-mbp, executor driver, partition 43, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 43.0 in stage 16.0 (TID 86)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 33.0 in stage 16.0 (TID 76) in 60 ms on soumils-mbp (executor driver) (32/56)
24/03/01 11:21:07 INFO Executor: Finished task 32.0 in stage 16.0 (TID 75). 809 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 44.0 in stage 16.0 (TID 87) (soumils-mbp, executor driver, partition 44, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 32.0 in stage 16.0 (TID 75) in 65 ms on soumils-mbp (executor driver) (33/56)
24/03/01 11:21:07 INFO Executor: Running task 44.0 in stage 16.0 (TID 87)
24/03/01 11:21:07 INFO Executor: Finished task 31.0 in stage 16.0 (TID 74). 809 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 45.0 in stage 16.0 (TID 88) (soumils-mbp, executor driver, partition 45, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 31.0 in stage 16.0 (TID 74) in 68 ms on soumils-mbp (executor driver) (34/56)
24/03/01 11:21:07 INFO Executor: Running task 45.0 in stage 16.0 (TID 88)
24/03/01 11:21:07 INFO Executor: Finished task 35.0 in stage 16.0 (TID 78). 809 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 46.0 in stage 16.0 (TID 89) (soumils-mbp, executor driver, partition 46, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 46.0 in stage 16.0 (TID 89)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 35.0 in stage 16.0 (TID 78) in 66 ms on soumils-mbp (executor driver) (35/56)
24/03/01 11:21:07 INFO Executor: Finished task 34.0 in stage 16.0 (TID 77). 809 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 47.0 in stage 16.0 (TID 90) (soumils-mbp, executor driver, partition 47, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 34.0 in stage 16.0 (TID 77) in 75 ms on soumils-mbp (executor driver) (36/56)
24/03/01 11:21:07 INFO Executor: Running task 47.0 in stage 16.0 (TID 90)
24/03/01 11:21:07 INFO Executor: Finished task 36.0 in stage 16.0 (TID 79). 809 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 48.0 in stage 16.0 (TID 91) (soumils-mbp, executor driver, partition 48, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 36.0 in stage 16.0 (TID 79) in 58 ms on soumils-mbp (executor driver) (37/56)
24/03/01 11:21:07 INFO Executor: Running task 48.0 in stage 16.0 (TID 91)
24/03/01 11:21:07 INFO Executor: Finished task 37.0 in stage 16.0 (TID 80). 809 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 49.0 in stage 16.0 (TID 92) (soumils-mbp, executor driver, partition 49, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 49.0 in stage 16.0 (TID 92)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 37.0 in stage 16.0 (TID 80) in 60 ms on soumils-mbp (executor driver) (38/56)
24/03/01 11:21:07 INFO Executor: Finished task 38.0 in stage 16.0 (TID 81). 766 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 50.0 in stage 16.0 (TID 93) (soumils-mbp, executor driver, partition 50, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Finished task 42.0 in stage 16.0 (TID 85). 766 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 38.0 in stage 16.0 (TID 81) in 47 ms on soumils-mbp (executor driver) (39/56)
24/03/01 11:21:07 INFO Executor: Running task 50.0 in stage 16.0 (TID 93)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 51.0 in stage 16.0 (TID 94) (soumils-mbp, executor driver, partition 51, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 42.0 in stage 16.0 (TID 85) in 45 ms on soumils-mbp (executor driver) (40/56)
24/03/01 11:21:07 INFO Executor: Running task 51.0 in stage 16.0 (TID 94)
24/03/01 11:21:07 INFO Executor: Finished task 40.0 in stage 16.0 (TID 83). 766 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 52.0 in stage 16.0 (TID 95) (soumils-mbp, executor driver, partition 52, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 52.0 in stage 16.0 (TID 95)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 40.0 in stage 16.0 (TID 83) in 49 ms on soumils-mbp (executor driver) (41/56)
24/03/01 11:21:07 INFO Executor: Finished task 39.0 in stage 16.0 (TID 82). 766 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 53.0 in stage 16.0 (TID 96) (soumils-mbp, executor driver, partition 53, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 39.0 in stage 16.0 (TID 82) in 53 ms on soumils-mbp (executor driver) (42/56)
24/03/01 11:21:07 INFO Executor: Running task 53.0 in stage 16.0 (TID 96)
24/03/01 11:21:07 INFO Executor: Finished task 41.0 in stage 16.0 (TID 84). 766 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 54.0 in stage 16.0 (TID 97) (soumils-mbp, executor driver, partition 54, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 41.0 in stage 16.0 (TID 84) in 51 ms on soumils-mbp (executor driver) (43/56)
24/03/01 11:21:07 INFO Executor: Running task 54.0 in stage 16.0 (TID 97)
24/03/01 11:21:07 INFO Executor: Finished task 45.0 in stage 16.0 (TID 88). 766 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 43.0 in stage 16.0 (TID 86). 766 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 55.0 in stage 16.0 (TID 98) (soumils-mbp, executor driver, partition 55, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 45.0 in stage 16.0 (TID 88) in 43 ms on soumils-mbp (executor driver) (44/56)
24/03/01 11:21:07 INFO Executor: Running task 55.0 in stage 16.0 (TID 98)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 43.0 in stage 16.0 (TID 86) in 51 ms on soumils-mbp (executor driver) (45/56)
24/03/01 11:21:07 INFO Executor: Finished task 44.0 in stage 16.0 (TID 87). 766 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 44.0 in stage 16.0 (TID 87) in 47 ms on soumils-mbp (executor driver) (46/56)
24/03/01 11:21:07 INFO Executor: Finished task 46.0 in stage 16.0 (TID 89). 766 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 46.0 in stage 16.0 (TID 89) in 46 ms on soumils-mbp (executor driver) (47/56)
24/03/01 11:21:07 INFO Executor: Finished task 47.0 in stage 16.0 (TID 90). 809 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 47.0 in stage 16.0 (TID 90) in 49 ms on soumils-mbp (executor driver) (48/56)
24/03/01 11:21:07 INFO Executor: Finished task 48.0 in stage 16.0 (TID 91). 766 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 48.0 in stage 16.0 (TID 91) in 40 ms on soumils-mbp (executor driver) (49/56)
24/03/01 11:21:07 INFO Executor: Finished task 49.0 in stage 16.0 (TID 92). 766 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 49.0 in stage 16.0 (TID 92) in 38 ms on soumils-mbp (executor driver) (50/56)
24/03/01 11:21:07 INFO Executor: Finished task 52.0 in stage 16.0 (TID 95). 766 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 50.0 in stage 16.0 (TID 93). 809 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 52.0 in stage 16.0 (TID 95) in 35 ms on soumils-mbp (executor driver) (51/56)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 50.0 in stage 16.0 (TID 93) in 37 ms on soumils-mbp (executor driver) (52/56)
24/03/01 11:21:07 INFO Executor: Finished task 51.0 in stage 16.0 (TID 94). 809 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 51.0 in stage 16.0 (TID 94) in 38 ms on soumils-mbp (executor driver) (53/56)
24/03/01 11:21:07 INFO Executor: Finished task 55.0 in stage 16.0 (TID 98). 766 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 55.0 in stage 16.0 (TID 98) in 30 ms on soumils-mbp (executor driver) (54/56)
24/03/01 11:21:07 INFO Executor: Finished task 54.0 in stage 16.0 (TID 97). 766 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 53.0 in stage 16.0 (TID 96). 766 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 54.0 in stage 16.0 (TID 97) in 32 ms on soumils-mbp (executor driver) (55/56)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 53.0 in stage 16.0 (TID 96) in 33 ms on soumils-mbp (executor driver) (56/56)
24/03/01 11:21:07 INFO TaskSchedulerImpl: Removed TaskSet 16.0, whose tasks have all completed, from pool
24/03/01 11:21:07 INFO DAGScheduler: ResultStage 16 (collect at HoodieSparkEngineContext.java:150) finished in 0.345 s
24/03/01 11:21:07 INFO DAGScheduler: Job 12 is finished. Cancelling potential speculative or zombie tasks for this job
24/03/01 11:21:07 INFO TaskSchedulerImpl: Killing all running tasks in stage 16: Stage finished
24/03/01 11:21:07 INFO DAGScheduler: Job 12 finished: collect at HoodieSparkEngineContext.java:150, took 0.346630 s
24/03/01 11:21:07 INFO MapPartitionsRDD: Removing RDD 36 from persistence list
24/03/01 11:21:07 INFO BlockManager: Removing RDD 36
24/03/01 11:21:07 INFO AbstractTableFileSystemView: Took 0 ms to read  0 instants, 0 replaced file groups
24/03/01 11:21:07 INFO ClusteringUtils: Found 0 files in pending clustering operations
24/03/01 11:21:07 INFO SparkContext: Starting job: countByKey at HoodieJavaPairRDD.java:105
24/03/01 11:21:07 INFO DAGScheduler: Registering RDD 37 (mapToPair at HoodieJavaRDD.java:149) as input to shuffle 3
24/03/01 11:21:07 INFO DAGScheduler: Registering RDD 47 (mapToPair at HoodieJavaRDD.java:149) as input to shuffle 4
24/03/01 11:21:07 INFO DAGScheduler: Registering RDD 55 (countByKey at HoodieJavaPairRDD.java:105) as input to shuffle 5
24/03/01 11:21:07 INFO DAGScheduler: Got job 13 (countByKey at HoodieJavaPairRDD.java:105) with 8 output partitions
24/03/01 11:21:07 INFO DAGScheduler: Final stage: ResultStage 21 (countByKey at HoodieJavaPairRDD.java:105)
24/03/01 11:21:07 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 20)
24/03/01 11:21:07 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 20)
24/03/01 11:21:07 INFO DAGScheduler: Submitting ShuffleMapStage 18 (MapPartitionsRDD[37] at mapToPair at HoodieJavaRDD.java:149), which has no missing parents
24/03/01 11:21:07 INFO MemoryStore: Block broadcast_19 stored as values in memory (estimated size 25.4 KiB, free 433.6 MiB)
24/03/01 11:21:07 INFO MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 12.1 KiB, free 433.5 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Added broadcast_19_piece0 in memory on soumils-mbp:49385 (size: 12.1 KiB, free: 434.2 MiB)
24/03/01 11:21:07 INFO SparkContext: Created broadcast 19 from broadcast at DAGScheduler.scala:1535
24/03/01 11:21:07 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 18 (MapPartitionsRDD[37] at mapToPair at HoodieJavaRDD.java:149) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7))
24/03/01 11:21:07 INFO TaskSchedulerImpl: Adding task set 18.0 with 8 tasks resource profile 0
24/03/01 11:21:07 INFO DAGScheduler: Submitting ShuffleMapStage 19 (MapPartitionsRDD[47] at mapToPair at HoodieJavaRDD.java:149), which has no missing parents
24/03/01 11:21:07 INFO TaskSetManager: Starting task 0.0 in stage 18.0 (TID 99) (soumils-mbp, executor driver, partition 0, NODE_LOCAL, 7170 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 1.0 in stage 18.0 (TID 100) (soumils-mbp, executor driver, partition 1, NODE_LOCAL, 7170 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 2.0 in stage 18.0 (TID 101) (soumils-mbp, executor driver, partition 2, NODE_LOCAL, 7170 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 3.0 in stage 18.0 (TID 102) (soumils-mbp, executor driver, partition 3, NODE_LOCAL, 7170 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 4.0 in stage 18.0 (TID 103) (soumils-mbp, executor driver, partition 4, NODE_LOCAL, 7170 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 5.0 in stage 18.0 (TID 104) (soumils-mbp, executor driver, partition 5, NODE_LOCAL, 7170 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 6.0 in stage 18.0 (TID 105) (soumils-mbp, executor driver, partition 6, NODE_LOCAL, 7170 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 7.0 in stage 18.0 (TID 106) (soumils-mbp, executor driver, partition 7, NODE_LOCAL, 7170 bytes)
24/03/01 11:21:07 INFO Executor: Running task 1.0 in stage 18.0 (TID 100)
24/03/01 11:21:07 INFO Executor: Running task 2.0 in stage 18.0 (TID 101)
24/03/01 11:21:07 INFO Executor: Running task 3.0 in stage 18.0 (TID 102)
24/03/01 11:21:07 INFO Executor: Running task 7.0 in stage 18.0 (TID 106)
24/03/01 11:21:07 INFO Executor: Running task 5.0 in stage 18.0 (TID 104)
24/03/01 11:21:07 INFO Executor: Running task 0.0 in stage 18.0 (TID 99)
24/03/01 11:21:07 INFO Executor: Running task 6.0 in stage 18.0 (TID 105)
24/03/01 11:21:07 INFO Executor: Running task 4.0 in stage 18.0 (TID 103)
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (13.9 KiB) non-empty blocks including 8 (13.9 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (14.3 KiB) non-empty blocks including 8 (14.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (13.6 KiB) non-empty blocks including 8 (13.6 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (13.7 KiB) non-empty blocks including 8 (13.7 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (13.8 KiB) non-empty blocks including 8 (13.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (14.0 KiB) non-empty blocks including 8 (14.0 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (13.6 KiB) non-empty blocks including 8 (13.6 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (15.7 KiB) non-empty blocks including 8 (15.7 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO Executor: Finished task 2.0 in stage 18.0 (TID 101). 1842 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 2.0 in stage 18.0 (TID 101) in 14 ms on soumils-mbp (executor driver) (1/8)
24/03/01 11:21:07 INFO Executor: Finished task 3.0 in stage 18.0 (TID 102). 1885 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 7.0 in stage 18.0 (TID 106). 1842 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 5.0 in stage 18.0 (TID 104). 1799 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 3.0 in stage 18.0 (TID 102) in 17 ms on soumils-mbp (executor driver) (2/8)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 7.0 in stage 18.0 (TID 106) in 17 ms on soumils-mbp (executor driver) (3/8)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 5.0 in stage 18.0 (TID 104) in 20 ms on soumils-mbp (executor driver) (4/8)
24/03/01 11:21:07 INFO MemoryStore: Block broadcast_20 stored as values in memory (estimated size 451.8 KiB, free 433.1 MiB)
24/03/01 11:21:07 INFO Executor: Finished task 6.0 in stage 18.0 (TID 105). 1885 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 0.0 in stage 18.0 (TID 99). 1928 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 4.0 in stage 18.0 (TID 103). 1842 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 1.0 in stage 18.0 (TID 100). 1885 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 6.0 in stage 18.0 (TID 105) in 22 ms on soumils-mbp (executor driver) (5/8)
24/03/01 11:21:07 INFO BlockManagerInfo: Removed broadcast_18_piece0 on soumils-mbp:49385 in memory (size: 157.2 KiB, free: 434.4 MiB)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 4.0 in stage 18.0 (TID 103) in 22 ms on soumils-mbp (executor driver) (6/8)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 1.0 in stage 18.0 (TID 100) in 22 ms on soumils-mbp (executor driver) (7/8)
24/03/01 11:21:07 INFO MemoryStore: Block broadcast_20_piece0 stored as bytes in memory (estimated size 158.4 KiB, free 433.5 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Added broadcast_20_piece0 in memory on soumils-mbp:49385 (size: 158.4 KiB, free: 434.2 MiB)
24/03/01 11:21:07 INFO SparkContext: Created broadcast 20 from broadcast at DAGScheduler.scala:1535
24/03/01 11:21:07 INFO TaskSetManager: Finished task 0.0 in stage 18.0 (TID 99) in 23 ms on soumils-mbp (executor driver) (8/8)
24/03/01 11:21:07 INFO TaskSchedulerImpl: Removed TaskSet 18.0, whose tasks have all completed, from pool
24/03/01 11:21:07 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 19 (MapPartitionsRDD[47] at mapToPair at HoodieJavaRDD.java:149) (first 15 tasks are for partitions Vector(0))
24/03/01 11:21:07 INFO TaskSchedulerImpl: Adding task set 19.0 with 1 tasks resource profile 0
24/03/01 11:21:07 INFO TaskSetManager: Starting task 0.0 in stage 19.0 (TID 107) (soumils-mbp, executor driver, partition 0, PROCESS_LOCAL, 7231 bytes)
24/03/01 11:21:07 INFO Executor: Running task 0.0 in stage 19.0 (TID 107)
24/03/01 11:21:07 INFO DAGScheduler: ShuffleMapStage 18 (mapToPair at HoodieJavaRDD.java:149) finished in 0.029 s
24/03/01 11:21:07 INFO DAGScheduler: looking for newly runnable stages
24/03/01 11:21:07 INFO DAGScheduler: running: Set(ShuffleMapStage 19)
24/03/01 11:21:07 INFO DAGScheduler: waiting: Set(ShuffleMapStage 20, ResultStage 21)
24/03/01 11:21:07 INFO DAGScheduler: failed: Set()
24/03/01 11:21:07 INFO Executor: Finished task 0.0 in stage 19.0 (TID 107). 810 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 0.0 in stage 19.0 (TID 107) in 11 ms on soumils-mbp (executor driver) (1/1)
24/03/01 11:21:07 INFO TaskSchedulerImpl: Removed TaskSet 19.0, whose tasks have all completed, from pool
24/03/01 11:21:07 INFO DAGScheduler: ShuffleMapStage 19 (mapToPair at HoodieJavaRDD.java:149) finished in 0.034 s
24/03/01 11:21:07 INFO DAGScheduler: looking for newly runnable stages
24/03/01 11:21:07 INFO DAGScheduler: running: Set()
24/03/01 11:21:07 INFO DAGScheduler: waiting: Set(ShuffleMapStage 20, ResultStage 21)
24/03/01 11:21:07 INFO DAGScheduler: failed: Set()
24/03/01 11:21:07 INFO DAGScheduler: Submitting ShuffleMapStage 20 (MapPartitionsRDD[55] at countByKey at HoodieJavaPairRDD.java:105), which has no missing parents
24/03/01 11:21:07 INFO MemoryStore: Block broadcast_21 stored as values in memory (estimated size 9.9 KiB, free 433.5 MiB)
24/03/01 11:21:07 INFO MemoryStore: Block broadcast_21_piece0 stored as bytes in memory (estimated size 5.1 KiB, free 433.5 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Added broadcast_21_piece0 in memory on soumils-mbp:49385 (size: 5.1 KiB, free: 434.2 MiB)
24/03/01 11:21:07 INFO SparkContext: Created broadcast 21 from broadcast at DAGScheduler.scala:1535
24/03/01 11:21:07 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 20 (MapPartitionsRDD[55] at countByKey at HoodieJavaPairRDD.java:105) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7))
24/03/01 11:21:07 INFO TaskSchedulerImpl: Adding task set 20.0 with 8 tasks resource profile 0
24/03/01 11:21:07 INFO TaskSetManager: Starting task 0.0 in stage 20.0 (TID 108) (soumils-mbp, executor driver, partition 0, PROCESS_LOCAL, 7233 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 1.0 in stage 20.0 (TID 109) (soumils-mbp, executor driver, partition 1, PROCESS_LOCAL, 7233 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 2.0 in stage 20.0 (TID 110) (soumils-mbp, executor driver, partition 2, PROCESS_LOCAL, 7233 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 3.0 in stage 20.0 (TID 111) (soumils-mbp, executor driver, partition 3, PROCESS_LOCAL, 7233 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 4.0 in stage 20.0 (TID 112) (soumils-mbp, executor driver, partition 4, PROCESS_LOCAL, 7233 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 5.0 in stage 20.0 (TID 113) (soumils-mbp, executor driver, partition 5, PROCESS_LOCAL, 7233 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 6.0 in stage 20.0 (TID 114) (soumils-mbp, executor driver, partition 6, PROCESS_LOCAL, 7233 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 7.0 in stage 20.0 (TID 115) (soumils-mbp, executor driver, partition 7, PROCESS_LOCAL, 7233 bytes)
24/03/01 11:21:07 INFO Executor: Running task 3.0 in stage 20.0 (TID 111)
24/03/01 11:21:07 INFO Executor: Running task 2.0 in stage 20.0 (TID 110)
24/03/01 11:21:07 INFO Executor: Running task 0.0 in stage 20.0 (TID 108)
24/03/01 11:21:07 INFO Executor: Running task 4.0 in stage 20.0 (TID 112)
24/03/01 11:21:07 INFO Executor: Running task 5.0 in stage 20.0 (TID 113)
24/03/01 11:21:07 INFO Executor: Running task 1.0 in stage 20.0 (TID 109)
24/03/01 11:21:07 INFO Executor: Running task 7.0 in stage 20.0 (TID 115)
24/03/01 11:21:07 INFO Executor: Running task 6.0 in stage 20.0 (TID 114)
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 1 (12.2 KiB) non-empty blocks including 1 (12.2 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 1 (12.2 KiB) non-empty blocks including 1 (12.2 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 1 (12.2 KiB) non-empty blocks including 1 (12.2 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 1 (12.2 KiB) non-empty blocks including 1 (12.2 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 1 (12.2 KiB) non-empty blocks including 1 (12.2 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 1 (13.5 KiB) non-empty blocks including 1 (13.5 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 1 (12.2 KiB) non-empty blocks including 1 (12.2 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 1 (14.8 KiB) non-empty blocks including 1 (14.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO MemoryStore: Block rdd_53_0 stored as values in memory (estimated size 17.7 KiB, free 433.5 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Added rdd_53_0 in memory on soumils-mbp:49385 (size: 17.7 KiB, free: 434.2 MiB)
24/03/01 11:21:07 INFO MemoryStore: Block rdd_53_3 stored as values in memory (estimated size 17.2 KiB, free 433.5 MiB)
24/03/01 11:21:07 INFO MemoryStore: Block rdd_53_2 stored as values in memory (estimated size 18.1 KiB, free 433.5 MiB)
24/03/01 11:21:07 INFO MemoryStore: Block rdd_53_4 stored as values in memory (estimated size 17.9 KiB, free 433.4 MiB)
24/03/01 11:21:07 INFO MemoryStore: Block rdd_53_6 stored as values in memory (estimated size 17.2 KiB, free 433.4 MiB)
24/03/01 11:21:07 INFO MemoryStore: Block rdd_53_1 stored as values in memory (estimated size 17.7 KiB, free 433.4 MiB)
24/03/01 11:21:07 INFO MemoryStore: Block rdd_53_5 stored as values in memory (estimated size 17.4 KiB, free 433.4 MiB)
24/03/01 11:21:07 INFO MemoryStore: Block rdd_53_7 stored as values in memory (estimated size 20.0 KiB, free 433.4 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Added rdd_53_3 in memory on soumils-mbp:49385 (size: 17.2 KiB, free: 434.2 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Added rdd_53_5 in memory on soumils-mbp:49385 (size: 17.4 KiB, free: 434.1 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Added rdd_53_1 in memory on soumils-mbp:49385 (size: 17.7 KiB, free: 434.1 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Added rdd_53_7 in memory on soumils-mbp:49385 (size: 20.0 KiB, free: 434.1 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Added rdd_53_6 in memory on soumils-mbp:49385 (size: 17.2 KiB, free: 434.1 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Added rdd_53_4 in memory on soumils-mbp:49385 (size: 17.9 KiB, free: 434.1 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Added rdd_53_2 in memory on soumils-mbp:49385 (size: 18.1 KiB, free: 434.1 MiB)
24/03/01 11:21:07 INFO Executor: Finished task 0.0 in stage 20.0 (TID 108). 1842 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 0.0 in stage 20.0 (TID 108) in 27 ms on soumils-mbp (executor driver) (1/8)
24/03/01 11:21:07 INFO Executor: Finished task 3.0 in stage 20.0 (TID 111). 1842 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 3.0 in stage 20.0 (TID 111) in 26 ms on soumils-mbp (executor driver) (2/8)
24/03/01 11:21:07 INFO Executor: Finished task 6.0 in stage 20.0 (TID 114). 1842 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 6.0 in stage 20.0 (TID 114) in 27 ms on soumils-mbp (executor driver) (3/8)
24/03/01 11:21:07 INFO Executor: Finished task 4.0 in stage 20.0 (TID 112). 1842 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 4.0 in stage 20.0 (TID 112) in 28 ms on soumils-mbp (executor driver) (4/8)
24/03/01 11:21:07 INFO Executor: Finished task 2.0 in stage 20.0 (TID 110). 1842 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 2.0 in stage 20.0 (TID 110) in 28 ms on soumils-mbp (executor driver) (5/8)
24/03/01 11:21:07 INFO Executor: Finished task 5.0 in stage 20.0 (TID 113). 1842 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 5.0 in stage 20.0 (TID 113) in 29 ms on soumils-mbp (executor driver) (6/8)
24/03/01 11:21:07 INFO Executor: Finished task 1.0 in stage 20.0 (TID 109). 1842 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 1.0 in stage 20.0 (TID 109) in 29 ms on soumils-mbp (executor driver) (7/8)
24/03/01 11:21:07 INFO Executor: Finished task 7.0 in stage 20.0 (TID 115). 1842 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 7.0 in stage 20.0 (TID 115) in 30 ms on soumils-mbp (executor driver) (8/8)
24/03/01 11:21:07 INFO TaskSchedulerImpl: Removed TaskSet 20.0, whose tasks have all completed, from pool
24/03/01 11:21:07 INFO DAGScheduler: ShuffleMapStage 20 (countByKey at HoodieJavaPairRDD.java:105) finished in 0.033 s
24/03/01 11:21:07 INFO DAGScheduler: looking for newly runnable stages
24/03/01 11:21:07 INFO DAGScheduler: running: Set()
24/03/01 11:21:07 INFO DAGScheduler: waiting: Set(ResultStage 21)
24/03/01 11:21:07 INFO DAGScheduler: failed: Set()
24/03/01 11:21:07 INFO DAGScheduler: Submitting ResultStage 21 (ShuffledRDD[56] at countByKey at HoodieJavaPairRDD.java:105), which has no missing parents
24/03/01 11:21:07 INFO MemoryStore: Block broadcast_22 stored as values in memory (estimated size 5.5 KiB, free 433.4 MiB)
24/03/01 11:21:07 INFO MemoryStore: Block broadcast_22_piece0 stored as bytes in memory (estimated size 3.2 KiB, free 433.4 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Added broadcast_22_piece0 in memory on soumils-mbp:49385 (size: 3.2 KiB, free: 434.1 MiB)
24/03/01 11:21:07 INFO SparkContext: Created broadcast 22 from broadcast at DAGScheduler.scala:1535
24/03/01 11:21:07 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 21 (ShuffledRDD[56] at countByKey at HoodieJavaPairRDD.java:105) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7))
24/03/01 11:21:07 INFO TaskSchedulerImpl: Adding task set 21.0 with 8 tasks resource profile 0
24/03/01 11:21:07 INFO TaskSetManager: Starting task 0.0 in stage 21.0 (TID 116) (soumils-mbp, executor driver, partition 0, NODE_LOCAL, 7181 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 1.0 in stage 21.0 (TID 117) (soumils-mbp, executor driver, partition 1, NODE_LOCAL, 7181 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 2.0 in stage 21.0 (TID 118) (soumils-mbp, executor driver, partition 2, NODE_LOCAL, 7181 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 3.0 in stage 21.0 (TID 119) (soumils-mbp, executor driver, partition 3, NODE_LOCAL, 7181 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 4.0 in stage 21.0 (TID 120) (soumils-mbp, executor driver, partition 4, NODE_LOCAL, 7181 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 5.0 in stage 21.0 (TID 121) (soumils-mbp, executor driver, partition 5, NODE_LOCAL, 7181 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 6.0 in stage 21.0 (TID 122) (soumils-mbp, executor driver, partition 6, NODE_LOCAL, 7181 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 7.0 in stage 21.0 (TID 123) (soumils-mbp, executor driver, partition 7, NODE_LOCAL, 7181 bytes)
24/03/01 11:21:07 INFO Executor: Running task 2.0 in stage 21.0 (TID 118)
24/03/01 11:21:07 INFO Executor: Running task 1.0 in stage 21.0 (TID 117)
24/03/01 11:21:07 INFO Executor: Running task 6.0 in stage 21.0 (TID 122)
24/03/01 11:21:07 INFO Executor: Running task 3.0 in stage 21.0 (TID 119)
24/03/01 11:21:07 INFO Executor: Running task 7.0 in stage 21.0 (TID 123)
24/03/01 11:21:07 INFO Executor: Running task 5.0 in stage 21.0 (TID 121)
24/03/01 11:21:07 INFO Executor: Running task 0.0 in stage 21.0 (TID 116)
24/03/01 11:21:07 INFO Executor: Running task 4.0 in stage 21.0 (TID 120)
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (1072.0 B) non-empty blocks including 8 (1072.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (1012.0 B) non-empty blocks including 8 (1012.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (1425.0 B) non-empty blocks including 8 (1425.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (1566.0 B) non-empty blocks including 8 (1566.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (1410.0 B) non-empty blocks including 8 (1410.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (1338.0 B) non-empty blocks including 8 (1338.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (1032.0 B) non-empty blocks including 8 (1032.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Getting 8 (1745.0 B) non-empty blocks including 8 (1745.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
24/03/01 11:21:07 INFO Executor: Finished task 0.0 in stage 21.0 (TID 116). 1747 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 2.0 in stage 21.0 (TID 118). 1924 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 1.0 in stage 21.0 (TID 117). 1776 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 6.0 in stage 21.0 (TID 122). 2040 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 7.0 in stage 21.0 (TID 123). 1718 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 0.0 in stage 21.0 (TID 116) in 7 ms on soumils-mbp (executor driver) (1/8)
24/03/01 11:21:07 INFO Executor: Finished task 4.0 in stage 21.0 (TID 120). 1863 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 1.0 in stage 21.0 (TID 117) in 8 ms on soumils-mbp (executor driver) (2/8)
24/03/01 11:21:07 INFO Executor: Finished task 3.0 in stage 21.0 (TID 119). 1953 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 5.0 in stage 21.0 (TID 121). 1895 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 7.0 in stage 21.0 (TID 123) in 7 ms on soumils-mbp (executor driver) (3/8)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 6.0 in stage 21.0 (TID 122) in 8 ms on soumils-mbp (executor driver) (4/8)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 5.0 in stage 21.0 (TID 121) in 8 ms on soumils-mbp (executor driver) (5/8)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 2.0 in stage 21.0 (TID 118) in 9 ms on soumils-mbp (executor driver) (6/8)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 3.0 in stage 21.0 (TID 119) in 9 ms on soumils-mbp (executor driver) (7/8)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 4.0 in stage 21.0 (TID 120) in 8 ms on soumils-mbp (executor driver) (8/8)
24/03/01 11:21:07 INFO TaskSchedulerImpl: Removed TaskSet 21.0, whose tasks have all completed, from pool
24/03/01 11:21:07 INFO DAGScheduler: ResultStage 21 (countByKey at HoodieJavaPairRDD.java:105) finished in 0.012 s
24/03/01 11:21:07 INFO DAGScheduler: Job 13 is finished. Cancelling potential speculative or zombie tasks for this job
24/03/01 11:21:07 INFO TaskSchedulerImpl: Killing all running tasks in stage 21: Stage finished
24/03/01 11:21:07 INFO DAGScheduler: Job 13 finished: countByKey at HoodieJavaPairRDD.java:105, took 0.089548 s
24/03/01 11:21:07 INFO UpsertPartitioner: AvgRecordSize => 1024
24/03/01 11:21:07 INFO SparkContext: Starting job: collectAsMap at UpsertPartitioner.java:282
24/03/01 11:21:07 INFO DAGScheduler: Got job 14 (collectAsMap at UpsertPartitioner.java:282) with 56 output partitions
24/03/01 11:21:07 INFO DAGScheduler: Final stage: ResultStage 22 (collectAsMap at UpsertPartitioner.java:282)
24/03/01 11:21:07 INFO DAGScheduler: Parents of final stage: List()
24/03/01 11:21:07 INFO DAGScheduler: Missing parents: List()
24/03/01 11:21:07 INFO DAGScheduler: Submitting ResultStage 22 (MapPartitionsRDD[58] at mapToPair at UpsertPartitioner.java:281), which has no missing parents
24/03/01 11:21:07 INFO MemoryStore: Block broadcast_23 stored as values in memory (estimated size 455.1 KiB, free 432.9 MiB)
24/03/01 11:21:07 INFO MemoryStore: Block broadcast_23_piece0 stored as bytes in memory (estimated size 158.2 KiB, free 432.8 MiB)
24/03/01 11:21:07 INFO BlockManagerInfo: Added broadcast_23_piece0 in memory on soumils-mbp:49385 (size: 158.2 KiB, free: 433.9 MiB)
24/03/01 11:21:07 INFO SparkContext: Created broadcast 23 from broadcast at DAGScheduler.scala:1535
24/03/01 11:21:07 INFO DAGScheduler: Submitting 56 missing tasks from ResultStage 22 (MapPartitionsRDD[58] at mapToPair at UpsertPartitioner.java:281) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
24/03/01 11:21:07 INFO TaskSchedulerImpl: Adding task set 22.0 with 56 tasks resource profile 0
24/03/01 11:21:07 INFO TaskSetManager: Starting task 0.0 in stage 22.0 (TID 124) (soumils-mbp, executor driver, partition 0, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 1.0 in stage 22.0 (TID 125) (soumils-mbp, executor driver, partition 1, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 2.0 in stage 22.0 (TID 126) (soumils-mbp, executor driver, partition 2, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 3.0 in stage 22.0 (TID 127) (soumils-mbp, executor driver, partition 3, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 4.0 in stage 22.0 (TID 128) (soumils-mbp, executor driver, partition 4, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 5.0 in stage 22.0 (TID 129) (soumils-mbp, executor driver, partition 5, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 6.0 in stage 22.0 (TID 130) (soumils-mbp, executor driver, partition 6, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 7.0 in stage 22.0 (TID 131) (soumils-mbp, executor driver, partition 7, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 8.0 in stage 22.0 (TID 132) (soumils-mbp, executor driver, partition 8, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 9.0 in stage 22.0 (TID 133) (soumils-mbp, executor driver, partition 9, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 10.0 in stage 22.0 (TID 134) (soumils-mbp, executor driver, partition 10, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 11.0 in stage 22.0 (TID 135) (soumils-mbp, executor driver, partition 11, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 1.0 in stage 22.0 (TID 125)
24/03/01 11:21:07 INFO Executor: Running task 0.0 in stage 22.0 (TID 124)
24/03/01 11:21:07 INFO Executor: Running task 2.0 in stage 22.0 (TID 126)
24/03/01 11:21:07 INFO Executor: Running task 3.0 in stage 22.0 (TID 127)
24/03/01 11:21:07 INFO Executor: Running task 4.0 in stage 22.0 (TID 128)
24/03/01 11:21:07 INFO Executor: Running task 10.0 in stage 22.0 (TID 134)
24/03/01 11:21:07 INFO Executor: Running task 11.0 in stage 22.0 (TID 135)
24/03/01 11:21:07 INFO Executor: Running task 5.0 in stage 22.0 (TID 129)
24/03/01 11:21:07 INFO Executor: Running task 8.0 in stage 22.0 (TID 132)
24/03/01 11:21:07 INFO Executor: Running task 6.0 in stage 22.0 (TID 130)
24/03/01 11:21:07 INFO Executor: Running task 7.0 in stage 22.0 (TID 131)
24/03/01 11:21:07 INFO Executor: Running task 9.0 in stage 22.0 (TID 133)
24/03/01 11:21:07 INFO Executor: Finished task 4.0 in stage 22.0 (TID 128). 803 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 12.0 in stage 22.0 (TID 136) (soumils-mbp, executor driver, partition 12, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 12.0 in stage 22.0 (TID 136)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 4.0 in stage 22.0 (TID 128) in 52 ms on soumils-mbp (executor driver) (1/56)
24/03/01 11:21:07 INFO Executor: Finished task 3.0 in stage 22.0 (TID 127). 803 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 9.0 in stage 22.0 (TID 133). 803 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 13.0 in stage 22.0 (TID 137) (soumils-mbp, executor driver, partition 13, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 14.0 in stage 22.0 (TID 138) (soumils-mbp, executor driver, partition 14, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 13.0 in stage 22.0 (TID 137)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 9.0 in stage 22.0 (TID 133) in 55 ms on soumils-mbp (executor driver) (2/56)
24/03/01 11:21:07 INFO Executor: Running task 14.0 in stage 22.0 (TID 138)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 3.0 in stage 22.0 (TID 127) in 55 ms on soumils-mbp (executor driver) (3/56)
24/03/01 11:21:07 INFO Executor: Finished task 7.0 in stage 22.0 (TID 131). 803 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 15.0 in stage 22.0 (TID 139) (soumils-mbp, executor driver, partition 15, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Finished task 8.0 in stage 22.0 (TID 132). 803 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 1.0 in stage 22.0 (TID 125). 803 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Running task 15.0 in stage 22.0 (TID 139)
24/03/01 11:21:07 INFO Executor: Finished task 11.0 in stage 22.0 (TID 135). 803 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 0.0 in stage 22.0 (TID 124). 803 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 2.0 in stage 22.0 (TID 126). 803 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 16.0 in stage 22.0 (TID 140) (soumils-mbp, executor driver, partition 16, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Finished task 5.0 in stage 22.0 (TID 129). 803 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Running task 16.0 in stage 22.0 (TID 140)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 8.0 in stage 22.0 (TID 132) in 58 ms on soumils-mbp (executor driver) (4/56)
24/03/01 11:21:07 INFO Executor: Finished task 10.0 in stage 22.0 (TID 134). 803 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 6.0 in stage 22.0 (TID 130). 803 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Finished task 7.0 in stage 22.0 (TID 131) in 58 ms on soumils-mbp (executor driver) (5/56)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 17.0 in stage 22.0 (TID 141) (soumils-mbp, executor driver, partition 17, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 18.0 in stage 22.0 (TID 142) (soumils-mbp, executor driver, partition 18, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 17.0 in stage 22.0 (TID 141)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 11.0 in stage 22.0 (TID 135) in 59 ms on soumils-mbp (executor driver) (6/56)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 1.0 in stage 22.0 (TID 125) in 60 ms on soumils-mbp (executor driver) (7/56)
24/03/01 11:21:07 INFO Executor: Running task 18.0 in stage 22.0 (TID 142)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 19.0 in stage 22.0 (TID 143) (soumils-mbp, executor driver, partition 19, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 19.0 in stage 22.0 (TID 143)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 20.0 in stage 22.0 (TID 144) (soumils-mbp, executor driver, partition 20, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 21.0 in stage 22.0 (TID 145) (soumils-mbp, executor driver, partition 21, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 20.0 in stage 22.0 (TID 144)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 22.0 in stage 22.0 (TID 146) (soumils-mbp, executor driver, partition 22, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 21.0 in stage 22.0 (TID 145)
24/03/01 11:21:07 INFO Executor: Running task 22.0 in stage 22.0 (TID 146)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 23.0 in stage 22.0 (TID 147) (soumils-mbp, executor driver, partition 23, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 0.0 in stage 22.0 (TID 124) in 63 ms on soumils-mbp (executor driver) (8/56)
24/03/01 11:21:07 INFO Executor: Running task 23.0 in stage 22.0 (TID 147)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 6.0 in stage 22.0 (TID 130) in 62 ms on soumils-mbp (executor driver) (9/56)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 5.0 in stage 22.0 (TID 129) in 63 ms on soumils-mbp (executor driver) (10/56)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 10.0 in stage 22.0 (TID 134) in 63 ms on soumils-mbp (executor driver) (11/56)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 2.0 in stage 22.0 (TID 126) in 64 ms on soumils-mbp (executor driver) (12/56)
24/03/01 11:21:07 INFO Executor: Finished task 14.0 in stage 22.0 (TID 138). 889 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 12.0 in stage 22.0 (TID 136). 846 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 24.0 in stage 22.0 (TID 148) (soumils-mbp, executor driver, partition 24, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO BlockManagerInfo: Removed broadcast_20_piece0 on soumils-mbp:49385 in memory (size: 158.4 KiB, free: 434.1 MiB)
24/03/01 11:21:07 INFO Executor: Running task 24.0 in stage 22.0 (TID 148)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 25.0 in stage 22.0 (TID 149) (soumils-mbp, executor driver, partition 25, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 14.0 in stage 22.0 (TID 138) in 44 ms on soumils-mbp (executor driver) (13/56)
24/03/01 11:21:07 INFO Executor: Running task 25.0 in stage 22.0 (TID 149)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 12.0 in stage 22.0 (TID 136) in 48 ms on soumils-mbp (executor driver) (14/56)
24/03/01 11:21:07 INFO BlockManagerInfo: Removed broadcast_19_piece0 on soumils-mbp:49385 in memory (size: 12.1 KiB, free: 434.1 MiB)
24/03/01 11:21:07 INFO Executor: Finished task 16.0 in stage 22.0 (TID 140). 846 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 26.0 in stage 22.0 (TID 150) (soumils-mbp, executor driver, partition 26, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 16.0 in stage 22.0 (TID 140) in 47 ms on soumils-mbp (executor driver) (15/56)
24/03/01 11:21:07 INFO Executor: Running task 26.0 in stage 22.0 (TID 150)
24/03/01 11:21:07 INFO Executor: Finished task 18.0 in stage 22.0 (TID 142). 846 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 27.0 in stage 22.0 (TID 151) (soumils-mbp, executor driver, partition 27, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 18.0 in stage 22.0 (TID 142) in 47 ms on soumils-mbp (executor driver) (16/56)
24/03/01 11:21:07 INFO Executor: Running task 27.0 in stage 22.0 (TID 151)
24/03/01 11:21:07 INFO Executor: Finished task 20.0 in stage 22.0 (TID 144). 846 bytes result sent to driver
24/03/01 11:21:07 INFO BlockManagerInfo: Removed broadcast_21_piece0 on soumils-mbp:49385 in memory (size: 5.1 KiB, free: 434.1 MiB)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 28.0 in stage 22.0 (TID 152) (soumils-mbp, executor driver, partition 28, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 28.0 in stage 22.0 (TID 152)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 20.0 in stage 22.0 (TID 144) in 47 ms on soumils-mbp (executor driver) (17/56)
24/03/01 11:21:07 INFO Executor: Finished task 15.0 in stage 22.0 (TID 139). 846 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 29.0 in stage 22.0 (TID 153) (soumils-mbp, executor driver, partition 29, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 15.0 in stage 22.0 (TID 139) in 57 ms on soumils-mbp (executor driver) (18/56)
24/03/01 11:21:07 INFO Executor: Finished task 13.0 in stage 22.0 (TID 137). 889 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 22.0 in stage 22.0 (TID 146). 846 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Running task 29.0 in stage 22.0 (TID 153)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 30.0 in stage 22.0 (TID 154) (soumils-mbp, executor driver, partition 30, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 31.0 in stage 22.0 (TID 155) (soumils-mbp, executor driver, partition 31, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 31.0 in stage 22.0 (TID 155)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 13.0 in stage 22.0 (TID 137) in 59 ms on soumils-mbp (executor driver) (19/56)
24/03/01 11:21:07 INFO Executor: Running task 30.0 in stage 22.0 (TID 154)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 22.0 in stage 22.0 (TID 146) in 53 ms on soumils-mbp (executor driver) (20/56)
24/03/01 11:21:07 INFO BlockManagerInfo: Removed broadcast_22_piece0 on soumils-mbp:49385 in memory (size: 3.2 KiB, free: 434.1 MiB)
24/03/01 11:21:07 INFO Executor: Finished task 19.0 in stage 22.0 (TID 143). 846 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 32.0 in stage 22.0 (TID 156) (soumils-mbp, executor driver, partition 32, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 19.0 in stage 22.0 (TID 143) in 61 ms on soumils-mbp (executor driver) (21/56)
24/03/01 11:21:07 INFO Executor: Running task 32.0 in stage 22.0 (TID 156)
24/03/01 11:21:07 INFO Executor: Finished task 23.0 in stage 22.0 (TID 147). 846 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 17.0 in stage 22.0 (TID 141). 846 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 33.0 in stage 22.0 (TID 157) (soumils-mbp, executor driver, partition 33, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 23.0 in stage 22.0 (TID 147) in 67 ms on soumils-mbp (executor driver) (22/56)
24/03/01 11:21:07 INFO Executor: Running task 33.0 in stage 22.0 (TID 157)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 34.0 in stage 22.0 (TID 158) (soumils-mbp, executor driver, partition 34, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 17.0 in stage 22.0 (TID 141) in 70 ms on soumils-mbp (executor driver) (23/56)
24/03/01 11:21:07 INFO Executor: Running task 34.0 in stage 22.0 (TID 158)
24/03/01 11:21:07 INFO Executor: Finished task 24.0 in stage 22.0 (TID 148). 803 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 21.0 in stage 22.0 (TID 145). 846 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 35.0 in stage 22.0 (TID 159) (soumils-mbp, executor driver, partition 35, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 35.0 in stage 22.0 (TID 159)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 36.0 in stage 22.0 (TID 160) (soumils-mbp, executor driver, partition 36, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 36.0 in stage 22.0 (TID 160)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 24.0 in stage 22.0 (TID 148) in 38 ms on soumils-mbp (executor driver) (24/56)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 21.0 in stage 22.0 (TID 145) in 74 ms on soumils-mbp (executor driver) (25/56)
24/03/01 11:21:07 INFO Executor: Finished task 25.0 in stage 22.0 (TID 149). 803 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 37.0 in stage 22.0 (TID 161) (soumils-mbp, executor driver, partition 37, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 25.0 in stage 22.0 (TID 149) in 46 ms on soumils-mbp (executor driver) (26/56)
24/03/01 11:21:07 INFO Executor: Running task 37.0 in stage 22.0 (TID 161)
24/03/01 11:21:07 INFO Executor: Finished task 26.0 in stage 22.0 (TID 150). 803 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 38.0 in stage 22.0 (TID 162) (soumils-mbp, executor driver, partition 38, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 38.0 in stage 22.0 (TID 162)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 26.0 in stage 22.0 (TID 150) in 47 ms on soumils-mbp (executor driver) (27/56)
24/03/01 11:21:07 INFO Executor: Finished task 28.0 in stage 22.0 (TID 152). 803 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 39.0 in stage 22.0 (TID 163) (soumils-mbp, executor driver, partition 39, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 28.0 in stage 22.0 (TID 152) in 47 ms on soumils-mbp (executor driver) (28/56)
24/03/01 11:21:07 INFO Executor: Running task 39.0 in stage 22.0 (TID 163)
24/03/01 11:21:07 INFO Executor: Finished task 27.0 in stage 22.0 (TID 151). 803 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 40.0 in stage 22.0 (TID 164) (soumils-mbp, executor driver, partition 40, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 40.0 in stage 22.0 (TID 164)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 27.0 in stage 22.0 (TID 151) in 50 ms on soumils-mbp (executor driver) (29/56)
24/03/01 11:21:07 INFO Executor: Finished task 29.0 in stage 22.0 (TID 153). 803 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 41.0 in stage 22.0 (TID 165) (soumils-mbp, executor driver, partition 41, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 29.0 in stage 22.0 (TID 153) in 45 ms on soumils-mbp (executor driver) (30/56)
24/03/01 11:21:07 INFO Executor: Running task 41.0 in stage 22.0 (TID 165)
24/03/01 11:21:07 INFO Executor: Finished task 30.0 in stage 22.0 (TID 154). 803 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 42.0 in stage 22.0 (TID 166) (soumils-mbp, executor driver, partition 42, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 42.0 in stage 22.0 (TID 166)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 30.0 in stage 22.0 (TID 154) in 45 ms on soumils-mbp (executor driver) (31/56)
24/03/01 11:21:07 INFO Executor: Finished task 31.0 in stage 22.0 (TID 155). 803 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 43.0 in stage 22.0 (TID 167) (soumils-mbp, executor driver, partition 43, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 31.0 in stage 22.0 (TID 155) in 49 ms on soumils-mbp (executor driver) (32/56)
24/03/01 11:21:07 INFO Executor: Running task 43.0 in stage 22.0 (TID 167)
24/03/01 11:21:07 INFO Executor: Finished task 32.0 in stage 22.0 (TID 156). 803 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 44.0 in stage 22.0 (TID 168) (soumils-mbp, executor driver, partition 44, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 32.0 in stage 22.0 (TID 156) in 45 ms on soumils-mbp (executor driver) (33/56)
24/03/01 11:21:07 INFO Executor: Running task 44.0 in stage 22.0 (TID 168)
24/03/01 11:21:07 INFO Executor: Finished task 34.0 in stage 22.0 (TID 158). 803 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 45.0 in stage 22.0 (TID 169) (soumils-mbp, executor driver, partition 45, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 34.0 in stage 22.0 (TID 158) in 45 ms on soumils-mbp (executor driver) (34/56)
24/03/01 11:21:07 INFO Executor: Running task 45.0 in stage 22.0 (TID 169)
24/03/01 11:21:07 INFO Executor: Finished task 33.0 in stage 22.0 (TID 157). 803 bytes result sent to driver
24/03/01 11:21:07 INFO Executor: Finished task 36.0 in stage 22.0 (TID 160). 803 bytes result sent to driver
24/03/01 11:21:07 INFO TaskSetManager: Starting task 46.0 in stage 22.0 (TID 170) (soumils-mbp, executor driver, partition 46, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO Executor: Running task 46.0 in stage 22.0 (TID 170)
24/03/01 11:21:07 INFO TaskSetManager: Starting task 47.0 in stage 22.0 (TID 171) (soumils-mbp, executor driver, partition 47, PROCESS_LOCAL, 7254 bytes)
24/03/01 11:21:07 INFO TaskSetManager: Finished task 33.0 in stage 22.0 (TID 157) in 51 ms on soumils-mbp (executor driver) (35/56)
24/03/01 11:21:07 INFO Executor: Running task 47.0 in stage 22.0 (TID 171)
24/03/01

I do not see delta log files
I can try with --enable-sync isn't that for hive sync @the-other-tim-brown  ?

well after lot of headache I got the java working and got the man and build running
RNING] Usually this is not harmful and you can skip these warnings,
[WARNING] otherwise try to manually exclude artifacts based on
[WARNING] mvn dependency:tree -Ddetail=true and the above output.
[WARNING] See https://maven.apache.org/plugins/maven-shade-plugin/
[INFO] Attaching shaded artifact.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for onetable 0.1.0-SNAPSHOT:
[INFO]
[INFO] onetable ........................................... SUCCESS [  0.208 s]
[INFO] api ................................................ SUCCESS [  2.081 s]
[INFO] hudi-support ....................................... SUCCESS [  0.003 s]
[INFO] hudi-utils ......................................... SUCCESS [  0.712 s]
[INFO] core ............................................... SUCCESS [  2.701 s]
[INFO] utilities .......................................... SUCCESS [01:13 min]
[INFO] hudi-extensions .................................... SUCCESS [ 12.657 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:32 min
[INFO] Finished at: 2024-03-01T19:25:34-05:00
[INFO] ------------------------------------------------------------------------
soumilshah@Soumils-MBP inc

let me try with these jar fingers crossed

@soumilshah1995 change hoodie.onetable.formats=DELTA to hoodie.onetable.formats.to.sync=DELTA. The docs are not matching an update to the parameters in the code.

@the-other-tim-brown  Still getting Error
spark-submit \
    --class org.apache.hudi.utilities.streamer.HoodieStreamer \
    --packages 'org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.0' \
    --properties-file spark-config.properties \
    --master 'local[*]' \
    --executor-memory 1g \
    --jars '/Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/jar/hudi-extensions-0.1.0-SNAPSHOT-bundled.jar' \
     /Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/jar/hudi-utilities-slim-bundle_2.12-0.14.0.jar \
    --table-type COPY_ON_WRITE \
    --target-base-path 'file:///Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/hudi/bronze_orders'  \
    --target-table bronze_orders \
    --op UPSERT \
    --enable-sync \
    --sync-tool-classes 'io.onetable.hudi.sync.OneTableSyncTool' \
    --source-limit 4000000 \
    --source-ordering-field ts \
    --source-class org.apache.hudi.utilities.sources.CsvDFSSource \
    --hoodie-conf 'hoodie.datasource.write.recordkey.field=order_id' \
    --hoodie-conf 'hoodie.datasource.write.partitionpath.field=order_date' \
    --hoodie-conf 'hoodie.datasource.write.precombine.field=ts' \
    --hoodie-conf 'hoodie.streamer.source.dfs.root=file://///Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/sampledata/orders' \
    --hoodie-conf 'hoodie.deltastreamer.csv.header=true' \
    --hoodie-conf 'hoodie.deltastreamer.csv.sep=\t' \
    --hoodie-conf 'hoodie.onetable.formats.to.sync=DELTA'

Error
4/03/02 15:47:34 INFO FileSystemViewManager: Creating in-memory based Table View
24/03/02 15:47:34 WARN StreamSync: SyncTool class io.onetable.hudi.sync.OneTableSyncTool failed with exception
org.apache.hudi.exception.HoodieMetaSyncException: Could not sync using the meta sync class io.onetable.hudi.sync.OneTableSyncTool
	at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:81)
	at org.apache.hudi.utilities.streamer.StreamSync.runMetaSync(StreamSync.java:938)
	at org.apache.hudi.utilities.streamer.StreamSync.writeToSink(StreamSync.java:851)
	at org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:446)
	at org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.ingestOnce(HoodieStreamer.java:840)
	at org.apache.hudi.utilities.ingestion.HoodieIngestionService.startIngestion(HoodieIngestionService.java:72)
	at org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
	at org.apache.hudi.utilities.streamer.HoodieStreamer.sync(HoodieStreamer.java:205)
	at org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:584)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.util.ServiceConfigurationError: io.onetable.spi.sync.TargetClient: io.onetable.hudi.HudiTargetClient Unable to get public no-arg constructor
	at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:582)
	at java.base/java.util.ServiceLoader.getConstructor(ServiceLoader.java:673)
	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1233)
	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265)
	at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300)
	at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385)
	at io.onetable.client.TableFormatClientFactory.createTargetClientForName(TableFormatClientFactory.java:67)
	at io.onetable.client.TableFormatClientFactory.createForFormat(TableFormatClientFactory.java:51)
	at io.onetable.client.OneTableClient.lambda$sync$0(OneTableClient.java:99)
	at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:178)
	at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
	at io.onetable.client.OneTableClient.sync(OneTableClient.java:95)
	at io.onetable.hudi.sync.OneTableSyncTool.syncHoodieTable(OneTableSyncTool.java:80)
	at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:79)
	... 20 more
Caused by: java.lang.NoClassDefFoundError: org/apache/hudi/client/common/HoodieJavaEngineContext
	at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
	at java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137)
	at java.base/java.lang.Class.getConstructor0(Class.java:3342)
	at java.base/java.lang.Class.getConstructor(Class.java:2151)
	at java.base/java.util.ServiceLoader$1.run(ServiceLoader.java:660)
	at java.base/java.util.ServiceLoader$1.run(ServiceLoader.java:657)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.util.ServiceLoader.getConstructor(ServiceLoader.java:668)
	... 38 more
Caused by: java.lang.ClassNotFoundException: org.apache.hudi.client.common.HoodieJavaEngineContext
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
	... 46 more
24/03/02 15:47:34 INFO StreamSync: Shutting down embedded timeline server
24/03/02 15:47:34 INFO EmbeddedTimelineService: Closing Timeline server
24/03/02 15:47:34 INFO TimelineService: Closing Timeline Service
24/03/02 15:47:34 INFO Javalin: Stopping Javalin ...
24/03/02 15:47:34 INFO Javalin: Javalin has stopped
24/03/02 15:47:34 INFO TimelineService: Closed Timeline Service
24/03/02 15:47:34 INFO EmbeddedTimelineService: Closed Timeline server
24/03/02 15:47:34 INFO SparkContext: SparkContext is stopping with exitCode 0.
24/03/02 15:47:34 INFO SparkUI: Stopped Spark web UI at http://soumils-mbp:8090
24/03/02 15:47:34 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
24/03/02 15:47:34 INFO MemoryStore: MemoryStore cleared
24/03/02 15:47:34 INFO BlockManager: BlockManager stopped
24/03/02 15:47:34 INFO BlockManagerMaster: BlockManagerMaster stopped
24/03/02 15:47:34 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
24/03/02 15:47:34 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" org.apache.hudi.exception.HoodieMetaSyncException: Could not sync using the meta sync class io.onetable.hudi.sync.OneTableSyncTool
	at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:81)
	at org.apache.hudi.utilities.streamer.StreamSync.runMetaSync(StreamSync.java:938)
	at org.apache.hudi.utilities.streamer.StreamSync.writeToSink(StreamSync.java:851)
	at org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:446)
	at org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.ingestOnce(HoodieStreamer.java:840)
	at org.apache.hudi.utilities.ingestion.HoodieIngestionService.startIngestion(HoodieIngestionService.java:72)
	at org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
	at org.apache.hudi.utilities.streamer.HoodieStreamer.sync(HoodieStreamer.java:205)
	at org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:584)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.util.ServiceConfigurationError: io.onetable.spi.sync.TargetClient: io.onetable.hudi.HudiTargetClient Unable to get public no-arg constructor
	at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:582)
	at java.base/java.util.ServiceLoader.getConstructor(ServiceLoader.java:673)
	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1233)
	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265)
	at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300)
	at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385)
	at io.onetable.client.TableFormatClientFactory.createTargetClientForName(TableFormatClientFactory.java:67)
	at io.onetable.client.TableFormatClientFactory.createForFormat(TableFormatClientFactory.java:51)
	at io.onetable.client.OneTableClient.lambda$sync$0(OneTableClient.java:99)
	at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:178)
	at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
	at io.onetable.client.OneTableClient.sync(OneTableClient.java:95)
	at io.onetable.hudi.sync.OneTableSyncTool.syncHoodieTable(OneTableSyncTool.java:80)
	at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:79)
	... 20 more
Caused by: java.lang.NoClassDefFoundError: org/apache/hudi/client/common/HoodieJavaEngineContext
	at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
	at java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137)
	at java.base/java.lang.Class.getConstructor0(Class.java:3342)
	at java.base/java.lang.Class.getConstructor(Class.java:2151)
	at java.base/java.util.ServiceLoader$1.run(ServiceLoader.java:660)
	at java.base/java.util.ServiceLoader$1.run(ServiceLoader.java:657)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.util.ServiceLoader.getConstructor(ServiceLoader.java:668)
	... 38 more
Caused by: java.lang.ClassNotFoundException: org.apache.hudi.client.common.HoodieJavaEngineContext
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
	... 46 more
24/03/02 15:47:34 INFO ShutdownHookManager: Shutdown hook called
24/03/02 15:47:34 INFO ShutdownHookManager: Deleting directory /private/var/folders/qq/s_1bjv516pn_mck29cwdwxnm0000gp/T/spark-4590b9e3-daae-4317-9782-c7126533de25
24/03/02 15:47:34 INFO ShutdownHookManager: Deleting directory /private/var/folders/qq/s_1bjv516pn_mck29cwdwxnm0000gp/T/spark-ea3fcde5-0a6c-46e7-aad7-264fd4c4c8f6
soumilshah@Soumils-MBP DeltaStreamer %

would it be best for quick call to resolve it ?

Still facing same error on delta streamer

The issue is that the hudi-java-client is listed as provided in the pom so it is not added to the extensions bundle we're making here in onetable. To get around this issue, you can try adding the jar to the path or rebuilding from this branch: https://github.com/apache/incubator-xtable/pull/367

@the-other-tim-brown
As per your feedback 1 adding jar
Approach 1 (Failed)
    --class org.apache.hudi.utilities.streamer.HoodieStreamer \
    --packages 'org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.0' \
    --properties-file spark-config.properties \
    --master 'local[*]' \
    --executor-memory 1g \
    --jars '/Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/jar/hudi-extensions-0.1.0-beta1.jar,/Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/jar/hudi-java-client-0.14.0.jar' \
     /Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/jar/hudi-utilities-slim-bundle_2.12-0.14.0.jar \
    --table-type COPY_ON_WRITE \
    --target-base-path 'file:///Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/hudi/bronze_orders'  \
    --target-table bronze_orders \
    --op UPSERT \
    --enable-sync \
    --sync-tool-classes 'io.onetable.hudi.sync.OneTableSyncTool' \
    --source-limit 4000000 \
    --source-ordering-field ts \
    --source-class org.apache.hudi.utilities.sources.CsvDFSSource \
    --hoodie-conf 'hoodie.datasource.write.recordkey.field=order_id' \
    --hoodie-conf 'hoodie.datasource.write.partitionpath.field=order_date' \
    --hoodie-conf 'hoodie.datasource.write.precombine.field=ts' \
    --hoodie-conf 'hoodie.streamer.source.dfs.root=file://///Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/sampledata/orders' \
    --hoodie-conf 'hoodie.deltastreamer.csv.header=true' \
    --hoodie-conf 'hoodie.deltastreamer.csv.sep=\t' \
    --hoodie-conf 'hoodie.onetable.formats.to.sync=DELTA' \
    --hoodie-conf 'hoodie.onetable.target.metadata.retention.hr=168'

I did add /Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/jar/hudi-java-client-0.14.0.jar
its throwing same error

4/03/03 08:53:36 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
24/03/03 08:53:36 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" org.apache.hudi.exception.HoodieMetaSyncException: Could not sync using the meta sync class io.onetable.hudi.sync.OneTableSyncTool
	at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:81)
	at org.apache.hudi.utilities.streamer.StreamSync.runMetaSync(StreamSync.java:938)
	at org.apache.hudi.utilities.streamer.StreamSync.writeToSink(StreamSync.java:851)
	at org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:446)
	at org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.ingestOnce(HoodieStreamer.java:840)
	at org.apache.hudi.utilities.ingestion.HoodieIngestionService.startIngestion(HoodieIngestionService.java:72)
	at org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
	at org.apache.hudi.utilities.streamer.HoodieStreamer.sync(HoodieStreamer.java:205)
	at org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:584)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NoClassDefFoundError: io/onetable/client/SourceClientProvider
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:315)
	at org.apache.hudi.common.util.ReflectionUtils.getClass(ReflectionUtils.java:55)
	at org.apache.hudi.common.util.ReflectionUtils.hasConstructor(ReflectionUtils.java:111)
	at org.apache.hudi.common.util.ReflectionUtils.hasConstructor(ReflectionUtils.java:93)
	at org.apache.hudi.sync.common.util.SyncUtilHelpers.instantiateMetaSyncTool(SyncUtilHelpers.java:106)
	at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:78)
	... 20 more
Caused by: java.lang.ClassNotFoundException: io.onetable.client.SourceClientProvider
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
	... 27 more
24/03/03 08:53:36 INFO ShutdownHookManager: Shutdown hook called
24/03/03 08:53:36 INFO ShutdownHookManager: Deleting directory /private/var/folders/qq/s_1bjv516pn_mck29cwdwxnm0000gp/T/spark-2a254b1a-6aa4-462e-a576-e18dbb22bd8f
24/03/03 08:53:36 INFO ShutdownHookManager: Deleting directory /private/var/folders/qq/s_1bjv516pn_mck29cwdwxnm0000gp/T/spark-b60a5a20-0a55-4b29-a919-d1774bf40f71
soumilshah@Soumils-MBP DeltaStreamer %

Here is second approach
Approach 2
(venv) soumilshah@Soumils-MBP incubator-xtable % git branch
* 351-add-client-jars-to-bundle
  main
(venv) soumilshah@Soumils-MBP incubator-xtable %

Build Jar

[WARNING]   - META-INF/maven/org.eclipse.jetty/jetty-http/pom.properties
[WARNING]   - META-INF/maven/org.eclipse.jetty/jetty-http/pom.xml
[WARNING] maven-shade-plugin has detected that some class files are
[WARNING] present in two or more JARs. When this happens, only one
[WARNING] single version of the class is copied to the uber jar.
[WARNING] Usually this is not harmful and you can skip these warnings,
[WARNING] otherwise try to manually exclude artifacts based on
[WARNING] mvn dependency:tree -Ddetail=true and the above output.
[WARNING] See https://maven.apache.org/plugins/maven-shade-plugin/
[INFO] Attaching shaded artifact.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for onetable 0.1.0-SNAPSHOT:
[INFO]
[INFO] onetable ........................................... SUCCESS [  0.192 s]
[INFO] api ................................................ SUCCESS [  1.627 s]
[INFO] hudi-support ....................................... SUCCESS [  0.002 s]
[INFO] hudi-utils ......................................... SUCCESS [  0.659 s]
[INFO] core ............................................... SUCCESS [  2.514 s]
[INFO] utilities .......................................... SUCCESS [ 45.646 s]
[INFO] hudi-extensions .................................... SUCCESS [ 12.791 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:03 min
[INFO] Finished at: 2024-03-03T09:01:04-05:00
[INFO] ------------------------------------------------------------------------
(venv) soumilshah@Soumils-MBP incubator-xtable %

New Jar Spark Submit

spark-submit \
    --class org.apache.hudi.utilities.streamer.HoodieStreamer \
    --packages 'org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.0' \
    --properties-file spark-config.properties \
    --master 'local[*]' \
    --executor-memory 1g \
    --jars '/Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/new_jars/hudi-extensions-0.1.0-SNAPSHOT.jar' \
     /Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/jar/hudi-utilities-slim-bundle_2.12-0.14.0.jar \
    --table-type COPY_ON_WRITE \
    --target-base-path 'file:///Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/hudi/bronze_orders'  \
    --target-table bronze_orders \
    --op UPSERT \
    --enable-sync \
    --sync-tool-classes 'io.onetable.hudi.sync.OneTableSyncTool' \
    --source-limit 4000000 \
    --source-ordering-field ts \
    --source-class org.apache.hudi.utilities.sources.CsvDFSSource \
    --hoodie-conf 'hoodie.datasource.write.recordkey.field=order_id' \
    --hoodie-conf 'hoodie.datasource.write.partitionpath.field=order_date' \
    --hoodie-conf 'hoodie.datasource.write.precombine.field=ts' \
    --hoodie-conf 'hoodie.streamer.source.dfs.root=file://///Users/soumilshah/IdeaProjects/SparkProject/DeltaStreamer/sampledata/orders' \
    --hoodie-conf 'hoodie.deltastreamer.csv.header=true' \
    --hoodie-conf 'hoodie.deltastreamer.csv.sep=\t' \
    --hoodie-conf 'hoodie.onetable.formats.to.sync=DELTA' \
    --hoodie-conf 'hoodie.onetable.target.metadata.retention.hr=168'

Error
03/03 09:03:12 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
24/03/03 09:03:12 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" org.apache.hudi.exception.HoodieMetaSyncException: Could not sync using the meta sync class io.onetable.hudi.sync.OneTableSyncTool
	at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:81)
	at org.apache.hudi.utilities.streamer.StreamSync.runMetaSync(StreamSync.java:938)
	at org.apache.hudi.utilities.streamer.StreamSync.writeToSink(StreamSync.java:851)
	at org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:446)
	at org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.ingestOnce(HoodieStreamer.java:840)
	at org.apache.hudi.utilities.ingestion.HoodieIngestionService.startIngestion(HoodieIngestionService.java:72)
	at org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
	at org.apache.hudi.utilities.streamer.HoodieStreamer.sync(HoodieStreamer.java:205)
	at org.apache.hudi.utilities.streamer.HoodieStreamer.main(HoodieStreamer.java:584)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1020)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NoClassDefFoundError: io/onetable/client/SourceClientProvider
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:315)
	at org.apache.hudi.common.util.ReflectionUtils.getClass(ReflectionUtils.java:55)
	at org.apache.hudi.common.util.ReflectionUtils.hasConstructor(ReflectionUtils.java:111)
	at org.apache.hudi.common.util.ReflectionUtils.hasConstructor(ReflectionUtils.java:93)
	at org.apache.hudi.sync.common.util.SyncUtilHelpers.instantiateMetaSyncTool(SyncUtilHelpers.java:106)
	at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:78)
	... 20 more
Caused by: java.lang.ClassNotFoundException: io.onetable.client.SourceClientProvider
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
	... 27 more
24/03/03 09:03:12 INFO ShutdownHookManager: Shutdown hook called
24/03/03 09:03:12 INFO ShutdownHookManager: Deleting directory /private/var/folders/qq/s_1bjv516pn_mck29cwdwxnm0000gp/T/spark-3215b36d-219b-4a88-9223-0be0c3eafe0f
24/03/03 09:03:12 INFO ShutdownHookManager: Deleting directory /private/var/folders/qq/s_1bjv516pn_mck29cwdwxnm0000gp/T/spark-21e211d4-437a-4f83-abf6-8472ad7984da