I20221022 05:35:21.387948 4154418 ctrl_server.cpp:40] CtrlServer listening on <IP_ADDRESS>:33839
I20221022 05:35:21.388311 4154418 rank_info_bootstrap_server.cpp:47] RankInfoBootstrapServer listening on <IP_ADDRESS>:12345
I20221022 05:35:21.963438 4154418 epoll_comm_network.cpp:63] CommNet:Epoll listening on <IP_ADDRESS>:45327
W20221022 05:35:31.132243 4154418 env_global_objects_scope.cpp:285] Skip init RDMA because RDMA is not compiled!
I20221022 05:38:13.956075 4154418 version.cpp:22] OneFlow git version: v0.8.1-377-gff736fbee0
I20221022 05:38:13.956125 4154418 cuda_device_manager_factory.cpp:63] CUDA runtime version: 11.2
I20221022 05:38:13.956153 4154418 cuda_device_manager_factory.cpp:72] cuDNN version: 8.5.0
I20221022 05:38:13.956161 4154418 cuda_device_manager_factory.cpp:85] NCCL version: 2.12.10
I20221022 05:40:32.678282 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_0-InsertPinnedIdentityOpPass
I20221022 05:41:46.822155 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_0-InsertPinnedIdentityOpPass
I20221022 05:41:46.822685 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 InsertPinnedIdentityOpPass","mem_rss":"4641.000000 MB","mem_vm":"56691.000000 MB","time_cost":"74143 milliseconds"}
I20221022 05:41:46.822794 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_1-EliminateDeadNodesPass
I20221022 05:43:04.938809 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_1-EliminateDeadNodesPass
I20221022 05:43:04.940133 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 EliminateDeadNodesPass","mem_rss":"4671.000000 MB","mem_vm":"56722.000000 MB","time_cost":"78116 milliseconds"}
I20221022 05:43:04.940243 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_2-NormalizationExponentialAverageAutoTickPass
I20221022 05:43:04.940302 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_2-NormalizationExponentialAverageAutoTickPass
I20221022 05:43:04.940371 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 NormalizationExponentialAverageAutoTickPass","mem_rss":"4671.000000 MB","mem_vm":"56722.000000 MB","time_cost":"0 milliseconds"}
I20221022 05:43:04.940397 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_3-AutoMixedPrecision
I20221022 05:44:23.309800 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_3-AutoMixedPrecision
I20221022 05:44:23.311022 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AutoMixedPrecision","mem_rss":"4672.000000 MB","mem_vm":"56722.000000 MB","time_cost":"78369 milliseconds"}
I20221022 05:44:23.311136 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_4-PruneAmpWhiteIdentityOpPass
I20221022 05:45:48.394618 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_4-PruneAmpWhiteIdentityOpPass
I20221022 05:45:48.396054 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 PruneAmpWhiteIdentityOpPass","mem_rss":"4681.000000 MB","mem_vm":"56731.000000 MB","time_cost":"85083 milliseconds"}
I20221022 05:45:48.396167 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_5-OptimizerPlacementOptimizationPass
I20221022 05:47:13.555724 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_5-OptimizerPlacementOptimizationPass
I20221022 05:47:13.556977 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 OptimizerPlacementOptimizationPass","mem_rss":"4681.000000 MB","mem_vm":"56731.000000 MB","time_cost":"85159 milliseconds"}
I20221022 05:47:13.557104 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_6-FuseAddToOutputPass
I20221022 05:48:35.749186 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_6-FuseAddToOutputPass
I20221022 05:48:35.750627 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseAddToOutputPass","mem_rss":"4699.000000 MB","mem_vm":"56749.000000 MB","time_cost":"82192 milliseconds"}
I20221022 05:48:35.750733 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_7-DynamicLossScaleSchedulePass
I20221022 05:50:00.436586 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_7-DynamicLossScaleSchedulePass
I20221022 05:50:00.438042 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 DynamicLossScaleSchedulePass","mem_rss":"4699.000000 MB","mem_vm":"56749.000000 MB","time_cost":"84685 milliseconds"}
I20221022 05:50:00.438153 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_8-AutoTrainStep
I20221022 05:51:25.262576 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_8-AutoTrainStep
I20221022 05:51:25.264007 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AutoTrainStep","mem_rss":"4699.000000 MB","mem_vm":"56749.000000 MB","time_cost":"84824 milliseconds"}
I20221022 05:51:25.264115 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_9-AutoLearningRate
I20221022 05:52:49.971565 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_9-AutoLearningRate
I20221022 05:52:49.974534 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AutoLearningRate","mem_rss":"4699.000000 MB","mem_vm":"56749.000000 MB","time_cost":"84707 milliseconds"}
I20221022 05:52:49.974646 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_10-QuantAwareTraining
I20221022 05:52:49.974682 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_10-QuantAwareTraining
I20221022 05:52:49.975986 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 QuantAwareTraining","mem_rss":"4699.000000 MB","mem_vm":"56749.000000 MB","time_cost":"0 milliseconds"}
I20221022 05:52:49.976032 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_11-GenerateOptimizerOpConfs
I20221022 05:54:18.324594 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_11-GenerateOptimizerOpConfs
I20221022 05:54:18.326074 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 GenerateOptimizerOpConfs","mem_rss":"4699.000000 MB","mem_vm":"56749.000000 MB","time_cost":"88348 milliseconds"}
I20221022 05:54:18.326180 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_12-PrunePinnedIdentityOpPass
I20221022 05:56:38.821277 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_12-PrunePinnedIdentityOpPass
I20221022 05:56:38.822690 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 PrunePinnedIdentityOpPass","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"140495 milliseconds"}
I20221022 05:56:38.822783 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_13-ReplaceEmbeddingOps
I20221022 05:56:38.826603 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_13-ReplaceEmbeddingOps
I20221022 05:56:38.827806 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 ReplaceEmbeddingOps","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"3 milliseconds"}
I20221022 05:56:38.827850 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_14-SequentialOneEmbeddingOpsPass
I20221022 05:56:38.827872 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_14-SequentialOneEmbeddingOpsPass
I20221022 05:56:38.829057 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 SequentialOneEmbeddingOpsPass","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"0 milliseconds"}
I20221022 05:56:38.829097 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_15-FuseEmbeddingShuffleInteractionPass
I20221022 05:56:38.829113 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_15-FuseEmbeddingShuffleInteractionPass
I20221022 05:56:38.830181 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseEmbeddingShuffleInteractionPass","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"0 milliseconds"}
I20221022 05:56:38.830219 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_16-FuseBCEReduceMeanFwBwPass
I20221022 05:56:38.830230 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_16-FuseBCEReduceMeanFwBwPass
I20221022 05:56:38.831312 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseBCEReduceMeanFwBwPass","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"0 milliseconds"}
I20221022 05:56:38.831348 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_17-AddSspVariableProxy
I20221022 05:56:38.831384 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_17-AddSspVariableProxy
I20221022 05:56:38.832442 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AddSspVariableProxy","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"0 milliseconds"}
I20221022 05:56:38.832479 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_18-CheckpointingPass
I20221022 05:58:54.473637 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_18-CheckpointingPass
I20221022 05:58:54.475041 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 CheckpointingPass","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"135641 milliseconds"}
I20221022 05:58:54.475134 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_19-CudnnFusedNormalizationAddReluPass
I20221022 05:58:54.480680 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_19-CudnnFusedNormalizationAddReluPass
I20221022 05:58:54.481894 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 CudnnFusedNormalizationAddReluPass","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"5 milliseconds"}
I20221022 05:58:54.481945 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_20-PruneCastToStaticShapeOpsPass
I20221022 06:01:11.140111 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_20-PruneCastToStaticShapeOpsPass
I20221022 06:01:11.141501 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 PruneCastToStaticShapeOpsPass","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"136658 milliseconds"}
I20221022 06:01:11.141592 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_21-FuseAddToOutputPass1
I20221022 06:03:23.708871 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_21-FuseAddToOutputPass1
I20221022 06:03:23.710299 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseAddToOutputPass","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"132567 milliseconds"}
I20221022 06:03:23.710394 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_22-FuseConsecutiveAddPass
I20221022 06:03:23.711709 4154418 time_util.h:97] [count log]{"loc":"start","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"0 milliseconds"}
I20221022 06:05:34.476672 4154418 time_util.h:97] [count log]{"loc":"create op graph","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"130765 milliseconds"}
I20221022 06:05:34.524835 4154418 time_util.h:97] [count log]{"loc":"create job builder","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"48 milliseconds"}
I20221022 06:05:35.078070 4154418 time_util.h:97] [count log]{"loc":"fuse_add_apply","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"552 milliseconds"}
I20221022 06:05:35.744024 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_22-FuseConsecutiveAddPass
I20221022 06:05:35.745489 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseConsecutiveAddPass","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"132033 milliseconds"}
I20221022 06:05:35.745585 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_23-IndexedSlicesOptimizerRewritePass
I20221022 06:05:35.745621 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_23-IndexedSlicesOptimizerRewritePass
I20221022 06:05:35.746891 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 IndexedSlicesOptimizerRewritePass","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"0 milliseconds"}
I20221022 06:05:35.746944 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_24-SplitSparseSoftmaxCrossEntropyOpPass
I20221022 06:07:47.625555 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_24-SplitSparseSoftmaxCrossEntropyOpPass
I20221022 06:07:47.628652 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 SplitSparseSoftmaxCrossEntropyOpPass","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"131878 milliseconds"}
I20221022 06:07:47.628754 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_25-DoParallelCastBeforeWideningTypeCast
I20221022 06:09:59.819835 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_25-DoParallelCastBeforeWideningTypeCast
I20221022 06:09:59.821261 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 DoParallelCastBeforeWideningTypeCast","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"132191 milliseconds"}
I20221022 06:09:59.821353 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_26-FuseCastScalePass
I20221022 06:12:11.193187 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_26-FuseCastScalePass
I20221022 06:12:11.194516 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseCastScalePass","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"131371 milliseconds"}
I20221022 06:12:11.194612 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_27-PruneParallelCastOpsPass
I20221022 06:14:19.711030 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_27-PruneParallelCastOpsPass
I20221022 06:14:19.712443 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 PruneParallelCastOpsPass","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"128516 milliseconds"}
I20221022 06:14:19.712538 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_28-FuseUpdateOpsPass
I20221022 06:16:02.211665 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_28-FuseUpdateOpsPass
I20221022 06:16:02.213161 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseUpdateOpsPass","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"102499 milliseconds"}
I20221022 06:16:02.213255 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_29-FuseModelUpdateCastOpsPass
I20221022 06:16:02.213294 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_29-FuseModelUpdateCastOpsPass
I20221022 06:16:02.214627 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseModelUpdateCastOpsPass","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"0 milliseconds"}
I20221022 06:16:02.214679 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_30-MultiTensorModelUpdatePass
I20221022 06:16:02.214713 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_30-MultiTensorModelUpdatePass
I20221022 06:16:02.215942 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 MultiTensorModelUpdatePass","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"0 milliseconds"}
I20221022 06:16:02.215982 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_31-FixPipelineStageIdPass
I20221022 06:16:02.215996 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_31-FixPipelineStageIdPass
I20221022 06:16:02.217185 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FixPipelineStageIdPass","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"0 milliseconds"}
I20221022 06:16:02.217223 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_32-PipelineBufferPass
I20221022 06:16:02.217236 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_32-PipelineBufferPass
I20221022 06:16:02.218402 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 PipelineBufferPass","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"0 milliseconds"}
I20221022 06:16:02.218443 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_33-AutoParallelPass
I20221022 06:16:02.218456 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_33-AutoParallelPass
I20221022 06:16:02.219581 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AutoParallelPass","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"0 milliseconds"}
I20221022 06:16:02.219619 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_34-DumpVariableInfoPass
I20221022 06:16:02.219638 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_34-DumpVariableInfoPass
I20221022 06:16:02.220770 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 DumpVariableInfoPass","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"0 milliseconds"}
I20221022 06:16:02.220808 4154418 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_35-DumpBlobParallelConfPass
I20221022 06:17:24.803721 4154418 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_35-DumpBlobParallelConfPass
I20221022 06:17:24.805183 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 DumpBlobParallelConfPass","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"82583 milliseconds"}
I20221022 06:17:24.840914 4154418 time_util.h:95] [count log]{"loc":"[GraphCompile]GraphBase_0 OptimizationLogicalGraph","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"2212 seconds"}
I20221022 06:18:25.645573 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 GroupBoxingByDstParallel","mem_rss":"4879.000000 MB","mem_vm":"56928.000000 MB","time_cost":"16 seconds"}
I20221022 06:19:12.214596 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 BoxingWithMiddleNodes","mem_rss":"4879.000000 MB","mem_vm":"56928.000000 MB","time_cost":"46 seconds"}
I20221022 06:19:23.773802 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 SetCtrlInOpName4VariableOp","mem_rss":"4879.000000 MB","mem_vm":"56928.000000 MB","time_cost":"11 seconds"}
I20221022 06:19:38.333357 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AutoPrependTick","mem_rss":"4879.000000 MB","mem_vm":"56928.000000 MB","time_cost":"14 seconds"}
I20221022 06:19:49.823163 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AddTickForTimeShape","mem_rss":"4879.000000 MB","mem_vm":"56928.000000 MB","time_cost":"11 seconds"}
I20221022 06:20:04.942091 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 MultiClientAutoSourceAndSinkTick","mem_rss":"4898.000000 MB","mem_vm":"56946.000000 MB","time_cost":"15 seconds"}
I20221022 06:20:16.437093 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 MultiClientAutoInterfaceCriticalSectionTick","mem_rss":"4898.000000 MB","mem_vm":"56946.000000 MB","time_cost":"11 seconds"}
I20221022 06:20:16.439081 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 SystemOpFillJobNamePass","mem_rss":"4898.000000 MB","mem_vm":"56946.000000 MB","time_cost":"0 seconds"}
I20221022 06:20:28.202147 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 DumpBlobParallelConfPass","mem_rss":"4898.000000 MB","mem_vm":"56946.000000 MB","time_cost":"11 seconds"}
I20221022 06:20:49.055743 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 InsertNcclLogicalOpPass","mem_rss":"4947.000000 MB","mem_vm":"56996.000000 MB","time_cost":"20 seconds"}
I20221022 06:21:02.174326 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 DumpBlobParallelConfPass","mem_rss":"4947.000000 MB","mem_vm":"56996.000000 MB","time_cost":"13 seconds"}
I20221022 06:21:15.132742 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 CheckOpGraph","mem_rss":"4947.000000 MB","mem_vm":"56996.000000 MB","time_cost":"12 seconds"}
I20221022 06:21:15.134864 4154418 time_util.h:95] [count log]{"loc":"[GraphCompile]GraphBase_0 CompleteJob","mem_rss":"4947.000000 MB","mem_vm":"56996.000000 MB","time_cost":"207 seconds"}
I20221022 06:21:27.789683 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 NewOpGraph","mem_rss":"4947.000000 MB","mem_vm":"56996.000000 MB","time_cost":"12 seconds"}
I20221022 06:23:02.981364 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 BuildTaskGraph","mem_rss":"5393.000000 MB","mem_vm":"57441.000000 MB","time_cost":"95 seconds"}
I20221022 06:23:30.783481 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AddTaskToPlan","mem_rss":"6013.000000 MB","mem_vm":"62050.000000 MB","time_cost":"27 seconds"}
I20221022 06:23:35.383213 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 InferMemShare","mem_rss":"6025.000000 MB","mem_vm":"62594.000000 MB","time_cost":"4 seconds"}
I20221022 06:23:43.919066 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 GenMemBlockAndChunk","mem_rss":"6032.000000 MB","mem_vm":"62082.000000 MB","time_cost":"7 seconds"}
I20221022 06:23:43.920292 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 LogPlan","mem_rss":"6032.000000 MB","mem_vm":"62082.000000 MB","time_cost":"0 seconds"}
I20221022 06:23:44.194475 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 GenRegisterHint","mem_rss":"6032.000000 MB","mem_vm":"62082.000000 MB","time_cost":"0 seconds"}
I20221022 06:23:44.200961 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 GenCollectiveBoxingPlan","mem_rss":"6032.000000 MB","mem_vm":"62082.000000 MB","time_cost":"0 seconds"}
I20221022 06:23:44.376152 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 DumpCtrlRegstInfoToPlan","mem_rss":"6032.000000 MB","mem_vm":"62082.000000 MB","time_cost":"0 seconds"}
I20221022 06:23:44.860862 4154418 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 0, Device: 0 needs to allocate [ 11852.6 MiB ] device memory.
   In general, Chunk id: 0  memory is [ 4510 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.6049 MiB ] with mem_block_num = 9405
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 06:23:44.887748 4154418 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 1, Device: 1 needs to allocate [ 11852.6 MiB ] device memory.
   In general, Chunk id: 4  memory is [ 4510 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.606 MiB ] with mem_block_num = 9411
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 06:23:44.914260 4154418 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 2, Device: 2 needs to allocate [ 11852.6 MiB ] device memory.
   In general, Chunk id: 2  memory is [ 4510 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.606 MiB ] with mem_block_num = 9411
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 06:23:44.940851 4154418 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 3, Device: 3 needs to allocate [ 11852.6 MiB ] device memory.
   In general, Chunk id: 1  memory is [ 4510 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.606 MiB ] with mem_block_num = 9411
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 06:23:44.967319 4154418 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 4, Device: 4 needs to allocate [ 11854.9 MiB ] device memory.
   In general, Chunk id: 3  memory is [ 4512.23 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.6075 MiB ] with mem_block_num = 9418
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 06:23:44.993511 4154418 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 5, Device: 5 needs to allocate [ 11852.7 MiB ] device memory.
   In general, Chunk id: 6  memory is [ 4510 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.6075 MiB ] with mem_block_num = 9418
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 06:23:45.019721 4154418 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 6, Device: 6 needs to allocate [ 11852.7 MiB ] device memory.
   In general, Chunk id: 5  memory is [ 4510 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.6075 MiB ] with mem_block_num = 9418
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 06:23:45.046115 4154418 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 7, Device: 7 needs to allocate [ 11852.7 MiB ] device memory.
   In general, Chunk id: 7  memory is [ 4510 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.6075 MiB ] with mem_block_num = 9418
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 06:23:45.309185 4154418 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 GenMemAndLightPlanLog","mem_rss":"6032.000000 MB","mem_vm":"62082.000000 MB","time_cost":"0 seconds"}
I20221022 06:23:45.309792 4154418 time_util.h:95] [count log]{"loc":"[GraphCompile]GraphBase_0 CompilePlan","mem_rss":"6032.000000 MB","mem_vm":"62082.000000 MB","time_cost":"150 seconds"}
I20221022 06:23:56.358386 4154418 time_util.h:95] [count log]{"loc":"[GraphCompile]GraphBase_0 SyncPlan","mem_rss":"6032.000000 MB","mem_vm":"62082.000000 MB","time_cost":"11 seconds"}
F20221022 06:24:10.314756 4154418 memory_allocator.cpp:50] out of memory
  File "/home/xiezipeng/oneflow/oneflow/core/memory/memory_allocator.cpp", line 50, in Allocate
    device->Alloc(options, &ptr, size)
Error Type: oneflow.ErrorProto.out_of_memory_error

05:40:32 ~ 06:17:24 = 37 分钟 = 2220s
OptimizationLogicalGraph","mem_rss":"4797.000000 MB","mem_vm":"56847.000000 MB","time_cost":"2212 seconds"
这时间有点奇怪，和base差异很大。https://github.com/Oneflow-Inc/libai/issues/407#issuecomment-1284914484
@xiezipeng-ML  再使用提供给 idea 的 oneflow 版本跑下看看：
python3 -m pip install --pre oneflow -f https://staging.oneflow.info/branch/release/compile_cost_cnt/cu112

version: 0.8.1+cu112.git.63a3b287
git_commit: 63a3b287
cmake_build_type: Release
rdma: True
mlir: True

Log file created at: 2022/10/22 10:55:22
Running on machine: oneflow-28
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20221022 10:55:22.566063 14175 env_global_objects_scope.cpp:167] Using rpc backend: gRPC
I20221022 10:55:22.566787 14175 ctrl_server.cpp:40] CtrlServer listening on <IP_ADDRESS>:36307
I20221022 10:55:22.566893 14175 rank_info_bootstrap_server.cpp:47] RankInfoBootstrapServer listening on <IP_ADDRESS>:12345
I20221022 10:55:23.123452 14175 epoll_comm_network.cpp:63] CommNet:Epoll listening on <IP_ADDRESS>:45343
I20221022 10:55:32.340703 14175 ibverbs_comm_network.cpp:147] Using IB device mlx5_0 port 1 gid index 0
I20221022 10:55:39.492198 14175 ibverbs_comm_network.cpp:177] Connected to peer 7
I20221022 10:55:39.553431 14175 ibverbs_comm_network.cpp:177] Connected to peer 2
I20221022 10:55:39.609606 14175 ibverbs_comm_network.cpp:177] Connected to peer 1
I20221022 10:55:39.667016 14175 ibverbs_comm_network.cpp:177] Connected to peer 3
I20221022 10:55:39.728710 14175 ibverbs_comm_network.cpp:177] Connected to peer 4
I20221022 10:55:39.784607 14175 ibverbs_comm_network.cpp:177] Connected to peer 5
I20221022 10:55:39.833755 14175 ibverbs_comm_network.cpp:177] Connected to peer 6
I20221022 10:57:58.425228 14175 version.cpp:22] OneFlow git version: 63a3b287
I20221022 10:57:58.425261 14175 cuda_device_manager_factory.cpp:63] CUDA runtime version: 11.2
I20221022 10:57:58.425271 14175 cuda_device_manager_factory.cpp:72] cuDNN version: 8.1.1
I20221022 10:57:58.425278 14175 cuda_device_manager_factory.cpp:85] NCCL version: 2.12.10
I20221022 10:57:59.410562 14175 ibverbs_comm_network.cpp:147] Using IB device mlx5_0 port 1 gid index 0
I20221022 10:57:59.584656 14175 ibverbs_comm_network.cpp:177] Connected to peer 7
I20221022 10:57:59.585238 14175 ibverbs_comm_network.cpp:177] Connected to peer 2
I20221022 10:57:59.585676 14175 ibverbs_comm_network.cpp:177] Connected to peer 1
I20221022 10:57:59.586221 14175 ibverbs_comm_network.cpp:177] Connected to peer 3
I20221022 10:57:59.587119 14175 ibverbs_comm_network.cpp:177] Connected to peer 4
I20221022 10:57:59.587698 14175 ibverbs_comm_network.cpp:177] Connected to peer 5
I20221022 10:57:59.588161 14175 ibverbs_comm_network.cpp:177] Connected to peer 6
I20221022 10:58:23.431185 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_0-InsertPinnedIdentityOpPass
I20221022 10:58:31.888835 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_0-InsertPinnedIdentityOpPass
I20221022 10:58:31.889170 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 InsertPinnedIdentityOpPass","mem_rss":"4497.000000 MB","mem_vm":"57407.000000 MB","time_cost":"8 seconds"}
I20221022 10:58:31.889187 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_1-EliminateDeadNodesPass
I20221022 10:58:40.873653 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_1-EliminateDeadNodesPass
I20221022 10:58:40.874619 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 EliminateDeadNodesPass","mem_rss":"4518.000000 MB","mem_vm":"57428.000000 MB","time_cost":"8 seconds"}
I20221022 10:58:40.874642 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_2-NormalizationExponentialAverageAutoTickPass
I20221022 10:58:40.874658 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_2-NormalizationExponentialAverageAutoTickPass
I20221022 10:58:40.874689 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 NormalizationExponentialAverageAutoTickPass","mem_rss":"4518.000000 MB","mem_vm":"57428.000000 MB","time_cost":"0 seconds"}
I20221022 10:58:40.874707 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_3-AutoMixedPrecision
I20221022 10:58:49.803017 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_3-AutoMixedPrecision
I20221022 10:58:49.804092 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AutoMixedPrecision","mem_rss":"4518.000000 MB","mem_vm":"57428.000000 MB","time_cost":"8 seconds"}
I20221022 10:58:49.804131 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_4-PruneAmpWhiteIdentityOpPass
I20221022 10:58:59.530297 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_4-PruneAmpWhiteIdentityOpPass
I20221022 10:58:59.531471 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 PruneAmpWhiteIdentityOpPass","mem_rss":"4525.000000 MB","mem_vm":"57435.000000 MB","time_cost":"9 seconds"}
I20221022 10:58:59.531494 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_5-OptimizerPlacementOptimizationPass
I20221022 10:59:09.231950 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_5-OptimizerPlacementOptimizationPass
I20221022 10:59:09.232913 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 OptimizerPlacementOptimizationPass","mem_rss":"4525.000000 MB","mem_vm":"57435.000000 MB","time_cost":"9 seconds"}
I20221022 10:59:09.232935 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_6-FuseAddToOutputPass
I20221022 10:59:18.671962 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_6-FuseAddToOutputPass
I20221022 10:59:18.673094 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseAddToOutputPass","mem_rss":"4540.000000 MB","mem_vm":"57450.000000 MB","time_cost":"9 seconds"}
I20221022 10:59:18.673116 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_7-IRRoundTripBeforeAD
I20221022 10:59:18.673132 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_7-IRRoundTripBeforeAD
I20221022 10:59:18.674122 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 IRRoundTripBeforeAD","mem_rss":"4540.000000 MB","mem_vm":"57450.000000 MB","time_cost":"0 seconds"}
I20221022 10:59:18.674139 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_8-DynamicLossScaleSchedulePass
I20221022 10:59:28.306999 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_8-DynamicLossScaleSchedulePass
I20221022 10:59:28.308109 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 DynamicLossScaleSchedulePass","mem_rss":"4540.000000 MB","mem_vm":"57450.000000 MB","time_cost":"9 seconds"}
I20221022 10:59:28.308131 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_9-AutoTrainStep
I20221022 10:59:37.948289 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_9-AutoTrainStep
I20221022 10:59:37.949409 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AutoTrainStep","mem_rss":"4540.000000 MB","mem_vm":"57450.000000 MB","time_cost":"9 seconds"}
I20221022 10:59:37.949430 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_10-AutoLearningRate
I20221022 10:59:47.565222 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_10-AutoLearningRate
I20221022 10:59:47.566335 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AutoLearningRate","mem_rss":"4540.000000 MB","mem_vm":"57450.000000 MB","time_cost":"9 seconds"}
I20221022 10:59:47.566360 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_11-QuantAwareTraining
I20221022 10:59:47.566371 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_11-QuantAwareTraining
I20221022 10:59:47.567209 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 QuantAwareTraining","mem_rss":"4540.000000 MB","mem_vm":"57450.000000 MB","time_cost":"0 seconds"}
I20221022 10:59:47.567224 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_12-GenerateOptimizerOpConfs
I20221022 10:59:57.560886 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_12-GenerateOptimizerOpConfs
I20221022 10:59:57.562186 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 GenerateOptimizerOpConfs","mem_rss":"4540.000000 MB","mem_vm":"57450.000000 MB","time_cost":"9 seconds"}
I20221022 10:59:57.562211 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_13-PrunePinnedIdentityOpPass
I20221022 11:02:16.803112 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_13-PrunePinnedIdentityOpPass
I20221022 11:02:16.803378 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 PrunePinnedIdentityOpPass","mem_rss":"4616.000000 MB","mem_vm":"57526.000000 MB","time_cost":"139 seconds"}
I20221022 11:02:16.803396 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_14-ReplaceEmbeddingOps
I20221022 11:02:32.237046 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_14-ReplaceEmbeddingOps
I20221022 11:02:32.238175 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 ReplaceEmbeddingOps","mem_rss":"4616.000000 MB","mem_vm":"57526.000000 MB","time_cost":"15 seconds"}
I20221022 11:02:32.238198 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_15-SequentialOneEmbeddingOpsPass
I20221022 11:02:32.238214 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_15-SequentialOneEmbeddingOpsPass
I20221022 11:02:32.239285 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 SequentialOneEmbeddingOpsPass","mem_rss":"4616.000000 MB","mem_vm":"57526.000000 MB","time_cost":"0 seconds"}
I20221022 11:02:32.239301 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_16-FuseEmbeddingShuffleInteractionPass
I20221022 11:02:32.239310 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_16-FuseEmbeddingShuffleInteractionPass
I20221022 11:02:32.240469 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseEmbeddingShuffleInteractionPass","mem_rss":"4616.000000 MB","mem_vm":"57526.000000 MB","time_cost":"0 seconds"}
I20221022 11:02:32.240484 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_17-FuseBCEReduceMeanFwBwPass
I20221022 11:02:32.240491 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_17-FuseBCEReduceMeanFwBwPass
I20221022 11:02:32.241539 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseBCEReduceMeanFwBwPass","mem_rss":"4616.000000 MB","mem_vm":"57526.000000 MB","time_cost":"0 seconds"}
I20221022 11:02:32.241554 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_18-AddSspVariableProxy
I20221022 11:02:32.241571 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_18-AddSspVariableProxy
I20221022 11:02:32.242605 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AddSspVariableProxy","mem_rss":"4616.000000 MB","mem_vm":"57526.000000 MB","time_cost":"0 seconds"}
I20221022 11:02:32.242619 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_19-CheckpointingPass
I20221022 11:02:47.658746 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_19-CheckpointingPass
I20221022 11:02:47.659873 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 CheckpointingPass","mem_rss":"4616.000000 MB","mem_vm":"57526.000000 MB","time_cost":"15 seconds"}
I20221022 11:02:47.659895 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_20-CudnnFusedNormalizationAddReluPass
I20221022 11:02:47.664316 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_20-CudnnFusedNormalizationAddReluPass
I20221022 11:02:47.665417 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 CudnnFusedNormalizationAddReluPass","mem_rss":"4616.000000 MB","mem_vm":"57526.000000 MB","time_cost":"0 seconds"}
I20221022 11:02:47.665449 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_21-PruneCastToStaticShapeOpsPass
I20221022 11:03:03.287129 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_21-PruneCastToStaticShapeOpsPass
I20221022 11:03:03.288264 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 PruneCastToStaticShapeOpsPass","mem_rss":"4616.000000 MB","mem_vm":"57526.000000 MB","time_cost":"15 seconds"}
I20221022 11:03:03.288287 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_22-IRRoundTrip
I20221022 11:03:03.288307 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_22-IRRoundTrip
I20221022 11:03:03.289397 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 IRRoundTrip","mem_rss":"4616.000000 MB","mem_vm":"57526.000000 MB","time_cost":"0 seconds"}
I20221022 11:03:03.289413 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_23-FuseAddToOutputPass1
I20221022 11:03:18.397606 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_23-FuseAddToOutputPass1
I20221022 11:03:18.398777 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseAddToOutputPass","mem_rss":"4619.000000 MB","mem_vm":"57529.000000 MB","time_cost":"15 seconds"}
I20221022 11:03:18.398799 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_24-FuseConsecutiveAddPass
I20221022 11:04:48.886399 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_24-FuseConsecutiveAddPass
I20221022 11:04:48.887552 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseConsecutiveAddPass","mem_rss":"4619.000000 MB","mem_vm":"57529.000000 MB","time_cost":"90 seconds"}
I20221022 11:04:48.887573 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_25-IndexedSlicesOptimizerRewritePass
I20221022 11:04:48.887588 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_25-IndexedSlicesOptimizerRewritePass
I20221022 11:04:48.888661 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 IndexedSlicesOptimizerRewritePass","mem_rss":"4619.000000 MB","mem_vm":"57529.000000 MB","time_cost":"0 seconds"}
I20221022 11:04:48.888677 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_26-SplitSparseSoftmaxCrossEntropyOpPass
I20221022 11:05:04.100203 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_26-SplitSparseSoftmaxCrossEntropyOpPass
I20221022 11:05:04.101336 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 SplitSparseSoftmaxCrossEntropyOpPass","mem_rss":"4619.000000 MB","mem_vm":"57529.000000 MB","time_cost":"15 seconds"}
I20221022 11:05:04.101358 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_27-DoParallelCastBeforeWideningTypeCast
I20221022 11:05:19.041433 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_27-DoParallelCastBeforeWideningTypeCast
I20221022 11:05:19.042604 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 DoParallelCastBeforeWideningTypeCast","mem_rss":"4619.000000 MB","mem_vm":"57529.000000 MB","time_cost":"14 seconds"}
I20221022 11:05:19.042626 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_28-FuseCastScalePass
I20221022 11:05:34.012495 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_28-FuseCastScalePass
I20221022 11:05:34.013909 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseCastScalePass","mem_rss":"4619.000000 MB","mem_vm":"57529.000000 MB","time_cost":"14 seconds"}
I20221022 11:05:34.013933 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_29-PruneParallelCastOpsPass
I20221022 11:05:48.619318 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_29-PruneParallelCastOpsPass
I20221022 11:05:48.620549 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 PruneParallelCastOpsPass","mem_rss":"4619.000000 MB","mem_vm":"57529.000000 MB","time_cost":"14 seconds"}
I20221022 11:05:48.620573 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_30-FuseUpdateOpsPass
I20221022 11:06:00.374001 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_30-FuseUpdateOpsPass
I20221022 11:06:00.375268 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseUpdateOpsPass","mem_rss":"4619.000000 MB","mem_vm":"57529.000000 MB","time_cost":"11 seconds"}
I20221022 11:06:00.375291 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_31-FuseModelUpdateCastOpsPass
I20221022 11:06:00.375303 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_31-FuseModelUpdateCastOpsPass
I20221022 11:06:00.375629 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseModelUpdateCastOpsPass","mem_rss":"4619.000000 MB","mem_vm":"57529.000000 MB","time_cost":"0 seconds"}
I20221022 11:06:00.375639 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_32-MultiTensorModelUpdatePass
I20221022 11:06:00.375645 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_32-MultiTensorModelUpdatePass
I20221022 11:06:00.376771 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 MultiTensorModelUpdatePass","mem_rss":"4619.000000 MB","mem_vm":"57529.000000 MB","time_cost":"0 seconds"}
I20221022 11:06:00.376785 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_33-FixPipelineStageIdPass
I20221022 11:06:09.759965 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_33-FixPipelineStageIdPass
I20221022 11:06:09.761188 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FixPipelineStageIdPass","mem_rss":"4619.000000 MB","mem_vm":"57529.000000 MB","time_cost":"9 seconds"}
I20221022 11:06:09.761210 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_34-PipelineBufferPass
I20221022 11:06:19.146199 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_34-PipelineBufferPass
I20221022 11:06:19.148458 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 PipelineBufferPass","mem_rss":"4619.000000 MB","mem_vm":"57529.000000 MB","time_cost":"9 seconds"}
I20221022 11:06:19.148483 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_35-AutoParallelPass
I20221022 11:06:19.148499 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_35-AutoParallelPass
I20221022 11:06:19.150429 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AutoParallelPass","mem_rss":"4619.000000 MB","mem_vm":"57529.000000 MB","time_cost":"0 seconds"}
I20221022 11:06:19.150446 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_36-DumpVariableInfoPass
I20221022 11:06:19.150455 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_36-DumpVariableInfoPass
I20221022 11:06:19.151705 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 DumpVariableInfoPass","mem_rss":"4619.000000 MB","mem_vm":"57529.000000 MB","time_cost":"0 seconds"}
I20221022 11:06:19.151717 14175 job_build_and_infer_ctx.cpp:943] GraphBase_0 start compiling with pass pass_cnt_37-DumpBlobParallelConfPass
I20221022 11:06:28.569283 14175 job_build_and_infer_ctx.cpp:957] GraphBase_0 finish compiling with pass pass_cnt_37-DumpBlobParallelConfPass
I20221022 11:06:28.570505 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 DumpBlobParallelConfPass","mem_rss":"4619.000000 MB","mem_vm":"57529.000000 MB","time_cost":"9 seconds"}
I20221022 11:06:28.596138 14175 time_util.h:95] [count log]{"loc":"[GraphCompile]GraphBase_0 OptimizationLogicalGraph","mem_rss":"4619.000000 MB","mem_vm":"57529.000000 MB","time_cost":"485 seconds"}
I20221022 11:06:38.123991 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 GroupBoxingByDstParallel","mem_rss":"4714.000000 MB","mem_vm":"57624.000000 MB","time_cost":"2 seconds"}
I20221022 11:06:50.423911 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 BoxingWithMiddleNodes","mem_rss":"4714.000000 MB","mem_vm":"57624.000000 MB","time_cost":"12 seconds"}
I20221022 11:06:51.981482 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 SetCtrlInOpName4VariableOp","mem_rss":"4714.000000 MB","mem_vm":"57624.000000 MB","time_cost":"1 seconds"}
I20221022 11:06:53.710587 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AutoPrependTick","mem_rss":"4714.000000 MB","mem_vm":"57624.000000 MB","time_cost":"1 seconds"}
I20221022 11:06:55.253793 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AddTickForTimeShape","mem_rss":"4714.000000 MB","mem_vm":"57624.000000 MB","time_cost":"1 seconds"}
I20221022 11:06:57.123100 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 MultiClientAutoSourceAndSinkTick","mem_rss":"4722.000000 MB","mem_vm":"57631.000000 MB","time_cost":"1 seconds"}
I20221022 11:06:58.677875 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 MultiClientAutoInterfaceCriticalSectionTick","mem_rss":"4722.000000 MB","mem_vm":"57631.000000 MB","time_cost":"1 seconds"}
I20221022 11:06:58.679165 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 SystemOpFillJobNamePass","mem_rss":"4722.000000 MB","mem_vm":"57631.000000 MB","time_cost":"0 seconds"}
I20221022 11:07:00.303792 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 DumpBlobParallelConfPass","mem_rss":"4722.000000 MB","mem_vm":"57631.000000 MB","time_cost":"1 seconds"}
I20221022 11:07:03.175424 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 InsertNcclLogicalOpPass","mem_rss":"4764.000000 MB","mem_vm":"57673.000000 MB","time_cost":"2 seconds"}
I20221022 11:07:05.096524 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 DumpBlobParallelConfPass","mem_rss":"4764.000000 MB","mem_vm":"57673.000000 MB","time_cost":"1 seconds"}
I20221022 11:07:06.988651 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 CheckOpGraph","mem_rss":"4764.000000 MB","mem_vm":"57673.000000 MB","time_cost":"1 seconds"}
I20221022 11:07:06.989764 14175 time_util.h:95] [count log]{"loc":"[GraphCompile]GraphBase_0 CompleteJob","mem_rss":"4764.000000 MB","mem_vm":"57673.000000 MB","time_cost":"33 seconds"}
I20221022 11:07:08.469779 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 NewOpGraph","mem_rss":"4764.000000 MB","mem_vm":"57673.000000 MB","time_cost":"1 seconds"}
I20221022 11:07:15.863741 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 BuildTaskGraph","mem_rss":"5171.000000 MB","mem_vm":"58079.000000 MB","time_cost":"7 seconds"}
I20221022 11:07:19.546250 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AddTaskToPlan","mem_rss":"5687.000000 MB","mem_vm":"62689.000000 MB","time_cost":"3 seconds"}
I20221022 11:07:20.458809 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 InferMemShare","mem_rss":"5699.000000 MB","mem_vm":"63233.000000 MB","time_cost":"0 seconds"}
I20221022 11:07:21.753882 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 GenMemBlockAndChunk","mem_rss":"5705.000000 MB","mem_vm":"62721.000000 MB","time_cost":"0 seconds"}
I20221022 11:07:21.754942 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 LogPlan","mem_rss":"5705.000000 MB","mem_vm":"62721.000000 MB","time_cost":"0 seconds"}
I20221022 11:07:21.824506 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 GenRegisterHint","mem_rss":"5705.000000 MB","mem_vm":"62721.000000 MB","time_cost":"0 seconds"}
I20221022 11:07:21.826962 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 GenCollectiveBoxingPlan","mem_rss":"5705.000000 MB","mem_vm":"62721.000000 MB","time_cost":"0 seconds"}
I20221022 11:07:21.912904 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 DumpCtrlRegstInfoToPlan","mem_rss":"5705.000000 MB","mem_vm":"62721.000000 MB","time_cost":"0 seconds"}
I20221022 11:07:22.057842 14175 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 0, Device: 0 needs to allocate [ 11854.9 MiB ] device memory.
   In general, Chunk id: 6  memory is [ 4512.24 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.6049 MiB ] with mem_block_num = 9406
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 11:07:22.059667 14175 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 1, Device: 1 needs to allocate [ 11854.9 MiB ] device memory.
   In general, Chunk id: 3  memory is [ 4512.24 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.606 MiB ] with mem_block_num = 9412
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 11:07:22.061463 14175 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 2, Device: 2 needs to allocate [ 11854.9 MiB ] device memory.
   In general, Chunk id: 4  memory is [ 4512.24 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.606 MiB ] with mem_block_num = 9412
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 11:07:22.063234 14175 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 3, Device: 3 needs to allocate [ 11852.7 MiB ] device memory.
   In general, Chunk id: 1  memory is [ 4510.03 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.606 MiB ] with mem_block_num = 9412
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 11:07:22.065021 14175 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 4, Device: 4 needs to allocate [ 11854.9 MiB ] device memory.
   In general, Chunk id: 0  memory is [ 4512.24 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.6075 MiB ] with mem_block_num = 9419
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 11:07:22.066795 14175 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 5, Device: 5 needs to allocate [ 11852.7 MiB ] device memory.
   In general, Chunk id: 7  memory is [ 4510.03 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.6075 MiB ] with mem_block_num = 9419
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 11:07:22.068562 14175 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 6, Device: 6 needs to allocate [ 11852.7 MiB ] device memory.
   In general, Chunk id: 2  memory is [ 4510.03 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.6075 MiB ] with mem_block_num = 9419
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 11:07:22.070333 14175 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 7, Device: 7 needs to allocate [ 11852.7 MiB ] device memory.
   In general, Chunk id: 5  memory is [ 4510.03 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.6075 MiB ] with mem_block_num = 9419
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 11:07:22.257691 14175 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 GenMemAndLightPlanLog","mem_rss":"5705.000000 MB","mem_vm":"62721.000000 MB","time_cost":"0 seconds"}
I20221022 11:07:22.258664 14175 time_util.h:95] [count log]{"loc":"[GraphCompile]GraphBase_0 CompilePlan","mem_rss":"5705.000000 MB","mem_vm":"62721.000000 MB","time_cost":"15 seconds"}
I20221022 11:07:24.616750 14175 time_util.h:95] [count log]{"loc":"[GraphCompile]GraphBase_0 SyncPlan","mem_rss":"5705.000000 MB","mem_vm":"62721.000000 MB","time_cost":"2 seconds"}
F20221022 11:07:29.913981 14175 memory_allocator.cpp:50] out of memory
  File "/home/ci-user/runners/release/_work/oneflow/oneflow/oneflow/core/memory/memory_allocator.cpp", line 50, in Allocate
    device->Alloc(options, &ptr, size)
Error Type: oneflow.ErrorProto.out_of_memory_error

@strint

195.47112141002435 compile graph time.......................................
version: 0.8.1+cu112.git.ff736fbee0
git_commit: ff736fbee0
cmake_build_type: Release
rdma: False
mlir: False

Log file created at: 2022/10/22 13:09:15
Running on machine: oneflow-28
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20221022 13:09:15.897207 240087 env_global_objects_scope.cpp:167] Using rpc backend: gRPC
I20221022 13:09:15.897891 240087 ctrl_server.cpp:40] CtrlServer listening on <IP_ADDRESS>:43899
I20221022 13:09:15.897990 240087 rank_info_bootstrap_server.cpp:47] RankInfoBootstrapServer listening on <IP_ADDRESS>:12345
I20221022 13:09:16.427913 240087 epoll_comm_network.cpp:63] CommNet:Epoll listening on <IP_ADDRESS>:41569
W20221022 13:09:26.467613 240087 env_global_objects_scope.cpp:285] Skip init RDMA because RDMA is not compiled!
I20221022 13:11:45.522619 240087 version.cpp:22] OneFlow git version: v0.8.1-377-gff736fbee0-snapshot
I20221022 13:11:45.522656 240087 cuda_device_manager_factory.cpp:63] CUDA runtime version: 11.2
I20221022 13:11:45.522679 240087 cuda_device_manager_factory.cpp:72] cuDNN version: 8.5.0
I20221022 13:11:45.522683 240087 cuda_device_manager_factory.cpp:85] NCCL version: 2.12.10
I20221022 13:12:03.942937 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_0-InsertPinnedIdentityOpPass
I20221022 13:12:10.412919 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_0-InsertPinnedIdentityOpPass
I20221022 13:12:10.413372 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 InsertPinnedIdentityOpPass","mem_rss":"4329.000000 MB","mem_vm":"56348.000000 MB","time_cost":"6470 milliseconds"}
I20221022 13:12:10.413403 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_1-EliminateDeadNodesPass
I20221022 13:12:17.357093 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_1-EliminateDeadNodesPass
I20221022 13:12:17.358330 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 EliminateDeadNodesPass","mem_rss":"4360.000000 MB","mem_vm":"56379.000000 MB","time_cost":"6943 milliseconds"}
I20221022 13:12:17.358357 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_2-NormalizationExponentialAverageAutoTickPass
I20221022 13:12:17.358379 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_2-NormalizationExponentialAverageAutoTickPass
I20221022 13:12:17.358420 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 NormalizationExponentialAverageAutoTickPass","mem_rss":"4360.000000 MB","mem_vm":"56379.000000 MB","time_cost":"0 milliseconds"}
I20221022 13:12:17.358429 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_3-AutoMixedPrecision
I20221022 13:12:24.232499 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_3-AutoMixedPrecision
I20221022 13:12:24.233603 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AutoMixedPrecision","mem_rss":"4360.000000 MB","mem_vm":"56379.000000 MB","time_cost":"6874 milliseconds"}
I20221022 13:12:24.233630 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_4-PruneAmpWhiteIdentityOpPass
I20221022 13:12:31.839972 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_4-PruneAmpWhiteIdentityOpPass
I20221022 13:12:31.841379 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 PruneAmpWhiteIdentityOpPass","mem_rss":"4370.000000 MB","mem_vm":"56389.000000 MB","time_cost":"7606 milliseconds"}
I20221022 13:12:31.841421 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_5-OptimizerPlacementOptimizationPass
I20221022 13:12:39.271486 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_5-OptimizerPlacementOptimizationPass
I20221022 13:12:39.272691 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 OptimizerPlacementOptimizationPass","mem_rss":"4370.000000 MB","mem_vm":"56389.000000 MB","time_cost":"7430 milliseconds"}
I20221022 13:12:39.272716 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_6-FuseAddToOutputPass
I20221022 13:12:46.663730 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_6-FuseAddToOutputPass
I20221022 13:12:46.665120 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseAddToOutputPass","mem_rss":"4388.000000 MB","mem_vm":"56407.000000 MB","time_cost":"7391 milliseconds"}
I20221022 13:12:46.665144 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_7-DynamicLossScaleSchedulePass
I20221022 13:12:54.140133 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_7-DynamicLossScaleSchedulePass
I20221022 13:12:54.141510 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 DynamicLossScaleSchedulePass","mem_rss":"4388.000000 MB","mem_vm":"56407.000000 MB","time_cost":"7475 milliseconds"}
I20221022 13:12:54.141535 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_8-AutoTrainStep
I20221022 13:13:01.567118 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_8-AutoTrainStep
I20221022 13:13:01.568488 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AutoTrainStep","mem_rss":"4388.000000 MB","mem_vm":"56407.000000 MB","time_cost":"7425 milliseconds"}
I20221022 13:13:01.568511 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_9-AutoLearningRate
I20221022 13:13:08.981353 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_9-AutoLearningRate
I20221022 13:13:08.982764 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AutoLearningRate","mem_rss":"4388.000000 MB","mem_vm":"56407.000000 MB","time_cost":"7412 milliseconds"}
I20221022 13:13:08.982787 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_10-QuantAwareTraining
I20221022 13:13:08.982797 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_10-QuantAwareTraining
I20221022 13:13:08.983955 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 QuantAwareTraining","mem_rss":"4388.000000 MB","mem_vm":"56407.000000 MB","time_cost":"0 milliseconds"}
I20221022 13:13:08.983969 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_11-GenerateOptimizerOpConfs
I20221022 13:13:16.783442 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_11-GenerateOptimizerOpConfs
I20221022 13:13:16.784693 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 GenerateOptimizerOpConfs","mem_rss":"4388.000000 MB","mem_vm":"56407.000000 MB","time_cost":"7799 milliseconds"}
I20221022 13:13:16.784718 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_12-PrunePinnedIdentityOpPass
I20221022 13:13:29.295590 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_12-PrunePinnedIdentityOpPass
I20221022 13:13:29.296972 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 PrunePinnedIdentityOpPass","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"12510 milliseconds"}
I20221022 13:13:29.296999 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_13-ReplaceEmbeddingOps
I20221022 13:13:29.300660 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_13-ReplaceEmbeddingOps
I20221022 13:13:29.301877 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 ReplaceEmbeddingOps","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"3 milliseconds"}
I20221022 13:13:29.301894 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_14-SequentialOneEmbeddingOpsPass
I20221022 13:13:29.301905 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_14-SequentialOneEmbeddingOpsPass
I20221022 13:13:29.303092 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 SequentialOneEmbeddingOpsPass","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"0 milliseconds"}
I20221022 13:13:29.303125 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_15-FuseEmbeddingShuffleInteractionPass
I20221022 13:13:29.303133 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_15-FuseEmbeddingShuffleInteractionPass
I20221022 13:13:29.304189 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseEmbeddingShuffleInteractionPass","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"0 milliseconds"}
I20221022 13:13:29.304200 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_16-FuseBCEReduceMeanFwBwPass
I20221022 13:13:29.304208 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_16-FuseBCEReduceMeanFwBwPass
I20221022 13:13:29.305286 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseBCEReduceMeanFwBwPass","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"0 milliseconds"}
I20221022 13:13:29.305299 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_17-AddSspVariableProxy
I20221022 13:13:29.305318 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_17-AddSspVariableProxy
I20221022 13:13:29.306389 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AddSspVariableProxy","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"0 milliseconds"}
I20221022 13:13:29.306402 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_18-CheckpointingPass
I20221022 13:13:41.173058 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_18-CheckpointingPass
I20221022 13:13:41.174474 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 CheckpointingPass","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"11866 milliseconds"}
I20221022 13:13:41.174499 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_19-CudnnFusedNormalizationAddReluPass
I20221022 13:13:41.179014 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_19-CudnnFusedNormalizationAddReluPass
I20221022 13:13:41.180243 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 CudnnFusedNormalizationAddReluPass","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"4 milliseconds"}
I20221022 13:13:41.180259 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_20-PruneCastToStaticShapeOpsPass
I20221022 13:13:53.324599 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_20-PruneCastToStaticShapeOpsPass
I20221022 13:13:53.325943 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 PruneCastToStaticShapeOpsPass","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"12144 milliseconds"}
I20221022 13:13:53.325969 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_21-FuseAddToOutputPass1
I20221022 13:14:05.108726 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_21-FuseAddToOutputPass1
I20221022 13:14:05.110198 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseAddToOutputPass","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"11782 milliseconds"}
I20221022 13:14:05.110222 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_22-FuseConsecutiveAddPass
I20221022 13:14:05.111447 240087 time_util.h:97] [count log]{"loc":"start","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"0 milliseconds"}
I20221022 13:14:16.325372 240087 time_util.h:97] [count log]{"loc":"create op graph","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"11214 milliseconds"}
I20221022 13:14:16.354195 240087 time_util.h:97] [count log]{"loc":"create job builder","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"28 milliseconds"}
I20221022 13:14:16.613657 240087 time_util.h:97] [count log]{"loc":"fuse_add_apply","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"259 milliseconds"}
I20221022 13:14:16.887930 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_22-FuseConsecutiveAddPass
I20221022 13:14:16.889369 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseConsecutiveAddPass","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"11777 milliseconds"}
I20221022 13:14:16.889408 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_23-IndexedSlicesOptimizerRewritePass
I20221022 13:14:16.889432 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_23-IndexedSlicesOptimizerRewritePass
I20221022 13:14:16.890709 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 IndexedSlicesOptimizerRewritePass","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"0 milliseconds"}
I20221022 13:14:16.890723 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_24-SplitSparseSoftmaxCrossEntropyOpPass
I20221022 13:14:28.530187 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_24-SplitSparseSoftmaxCrossEntropyOpPass
I20221022 13:14:28.531649 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 SplitSparseSoftmaxCrossEntropyOpPass","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"11639 milliseconds"}
I20221022 13:14:28.531674 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_25-DoParallelCastBeforeWideningTypeCast
I20221022 13:14:40.150116 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_25-DoParallelCastBeforeWideningTypeCast
I20221022 13:14:40.151437 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 DoParallelCastBeforeWideningTypeCast","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"11618 milliseconds"}
I20221022 13:14:40.151458 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_26-FuseCastScalePass
I20221022 13:14:51.797700 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_26-FuseCastScalePass
I20221022 13:14:51.798893 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseCastScalePass","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"11646 milliseconds"}
I20221022 13:14:51.798918 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_27-PruneParallelCastOpsPass
I20221022 13:15:03.215318 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_27-PruneParallelCastOpsPass
I20221022 13:15:03.216629 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 PruneParallelCastOpsPass","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"11416 milliseconds"}
I20221022 13:15:03.216652 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_28-FuseUpdateOpsPass
I20221022 13:15:12.452742 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_28-FuseUpdateOpsPass
I20221022 13:15:12.456284 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseUpdateOpsPass","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"9236 milliseconds"}
I20221022 13:15:12.456310 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_29-FuseModelUpdateCastOpsPass
I20221022 13:15:12.456326 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_29-FuseModelUpdateCastOpsPass
I20221022 13:15:12.459229 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FuseModelUpdateCastOpsPass","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"0 milliseconds"}
I20221022 13:15:12.459259 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_30-MultiTensorModelUpdatePass
I20221022 13:15:12.459266 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_30-MultiTensorModelUpdatePass
I20221022 13:15:12.461501 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 MultiTensorModelUpdatePass","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"0 milliseconds"}
I20221022 13:15:12.461516 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_31-FixPipelineStageIdPass
I20221022 13:15:12.461524 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_31-FixPipelineStageIdPass
I20221022 13:15:12.463618 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 FixPipelineStageIdPass","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"0 milliseconds"}
I20221022 13:15:12.463635 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_32-PipelineBufferPass
I20221022 13:15:12.463645 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_32-PipelineBufferPass
I20221022 13:15:12.465061 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 PipelineBufferPass","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"0 milliseconds"}
I20221022 13:15:12.465075 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_33-AutoParallelPass
I20221022 13:15:12.465081 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_33-AutoParallelPass
I20221022 13:15:12.466367 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AutoParallelPass","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"0 milliseconds"}
I20221022 13:15:12.466380 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_34-DumpVariableInfoPass
I20221022 13:15:12.466388 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_34-DumpVariableInfoPass
I20221022 13:15:12.467504 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 DumpVariableInfoPass","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"0 milliseconds"}
I20221022 13:15:12.467515 240087 job_build_and_infer_ctx.cpp:944] GraphBase_0 start compiling with pass pass_cnt_35-DumpBlobParallelConfPass
I20221022 13:15:19.872992 240087 job_build_and_infer_ctx.cpp:958] GraphBase_0 finish compiling with pass pass_cnt_35-DumpBlobParallelConfPass
I20221022 13:15:19.874497 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 DumpBlobParallelConfPass","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"7405 milliseconds"}
I20221022 13:15:19.899092 240087 time_util.h:95] [count log]{"loc":"[GraphCompile]GraphBase_0 OptimizationLogicalGraph","mem_rss":"4486.000000 MB","mem_vm":"56505.000000 MB","time_cost":"195 seconds"}
I20221022 13:15:24.927664 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 GroupBoxingByDstParallel","mem_rss":"4567.000000 MB","mem_vm":"56586.000000 MB","time_cost":"1 seconds"}
I20221022 13:15:32.856587 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 BoxingWithMiddleNodes","mem_rss":"4567.000000 MB","mem_vm":"56586.000000 MB","time_cost":"7 seconds"}
I20221022 13:15:34.249616 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 SetCtrlInOpName4VariableOp","mem_rss":"4567.000000 MB","mem_vm":"56586.000000 MB","time_cost":"1 seconds"}
I20221022 13:15:35.775238 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AutoPrependTick","mem_rss":"4567.000000 MB","mem_vm":"56586.000000 MB","time_cost":"1 seconds"}
I20221022 13:15:37.156741 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AddTickForTimeShape","mem_rss":"4567.000000 MB","mem_vm":"56586.000000 MB","time_cost":"1 seconds"}
I20221022 13:15:38.828123 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 MultiClientAutoSourceAndSinkTick","mem_rss":"4586.000000 MB","mem_vm":"56604.000000 MB","time_cost":"1 seconds"}
I20221022 13:15:40.219985 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 MultiClientAutoInterfaceCriticalSectionTick","mem_rss":"4586.000000 MB","mem_vm":"56604.000000 MB","time_cost":"1 seconds"}
I20221022 13:15:40.221357 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 SystemOpFillJobNamePass","mem_rss":"4586.000000 MB","mem_vm":"56604.000000 MB","time_cost":"0 seconds"}
I20221022 13:15:41.672730 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 DumpBlobParallelConfPass","mem_rss":"4586.000000 MB","mem_vm":"56604.000000 MB","time_cost":"1 seconds"}
I20221022 13:15:44.102623 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 InsertNcclLogicalOpPass","mem_rss":"4635.000000 MB","mem_vm":"56654.000000 MB","time_cost":"2 seconds"}
I20221022 13:15:45.754177 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 DumpBlobParallelConfPass","mem_rss":"4635.000000 MB","mem_vm":"56654.000000 MB","time_cost":"1 seconds"}
I20221022 13:15:47.406502 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 CheckOpGraph","mem_rss":"4635.000000 MB","mem_vm":"56654.000000 MB","time_cost":"1 seconds"}
I20221022 13:15:47.407749 240087 time_util.h:95] [count log]{"loc":"[GraphCompile]GraphBase_0 CompleteJob","mem_rss":"4635.000000 MB","mem_vm":"56654.000000 MB","time_cost":"26 seconds"}
I20221022 13:15:48.659965 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 NewOpGraph","mem_rss":"4635.000000 MB","mem_vm":"56654.000000 MB","time_cost":"1 seconds"}
I20221022 13:15:55.909827 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 BuildTaskGraph","mem_rss":"5082.000000 MB","mem_vm":"57098.000000 MB","time_cost":"7 seconds"}
I20221022 13:16:00.287573 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 AddTaskToPlan","mem_rss":"5700.000000 MB","mem_vm":"61708.000000 MB","time_cost":"4 seconds"}
I20221022 13:16:01.043056 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 InferMemShare","mem_rss":"5712.000000 MB","mem_vm":"62252.000000 MB","time_cost":"0 seconds"}
I20221022 13:16:02.033635 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 GenMemBlockAndChunk","mem_rss":"5720.000000 MB","mem_vm":"61740.000000 MB","time_cost":"0 seconds"}
I20221022 13:16:02.034662 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 LogPlan","mem_rss":"5720.000000 MB","mem_vm":"61740.000000 MB","time_cost":"0 seconds"}
I20221022 13:16:02.113368 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 GenRegisterHint","mem_rss":"5720.000000 MB","mem_vm":"61740.000000 MB","time_cost":"0 seconds"}
I20221022 13:16:02.115805 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 GenCollectiveBoxingPlan","mem_rss":"5720.000000 MB","mem_vm":"61740.000000 MB","time_cost":"0 seconds"}
I20221022 13:16:02.185315 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 DumpCtrlRegstInfoToPlan","mem_rss":"5720.000000 MB","mem_vm":"61740.000000 MB","time_cost":"0 seconds"}
I20221022 13:16:02.321313 240087 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 0, Device: 0 needs to allocate [ 11854.9 MiB ] device memory.
   In general, Chunk id: 0  memory is [ 4512.23 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.6049 MiB ] with mem_block_num = 9405
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 13:16:02.322976 240087 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 1, Device: 1 needs to allocate [ 11854.9 MiB ] device memory.
   In general, Chunk id: 7  memory is [ 4512.23 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.606 MiB ] with mem_block_num = 9411
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 13:16:02.324580 240087 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 2, Device: 2 needs to allocate [ 11854.9 MiB ] device memory.
   In general, Chunk id: 3  memory is [ 4512.23 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.606 MiB ] with mem_block_num = 9411
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 13:16:02.326141 240087 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 3, Device: 3 needs to allocate [ 11854.9 MiB ] device memory.
   In general, Chunk id: 1  memory is [ 4512.23 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.606 MiB ] with mem_block_num = 9411
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 13:16:02.327700 240087 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 4, Device: 4 needs to allocate [ 11852.7 MiB ] device memory.
   In general, Chunk id: 2  memory is [ 4510.03 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.6075 MiB ] with mem_block_num = 9418
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 13:16:02.329223 240087 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 5, Device: 5 needs to allocate [ 11852.7 MiB ] device memory.
   In general, Chunk id: 4  memory is [ 4510.03 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.6075 MiB ] with mem_block_num = 9418
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 13:16:02.330744 240087 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 6, Device: 6 needs to allocate [ 11852.7 MiB ] device memory.
   In general, Chunk id: 6  memory is [ 4510.03 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.6075 MiB ] with mem_block_num = 9418
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 13:16:02.332269 240087 plan_util.cpp:965]
 Graph name GraphBase_0 in Rank: 7, Device: 7 needs to allocate [ 11852.7 MiB ] device memory.
   In general, Chunk id: 5  memory is [ 4510.03 MiB ] with mem_block_num = 1
        Unreused memory not eager var is  [ 12.6075 MiB ] with mem_block_num = 9418
        Eager Variable Tensor total memory is [ 7330.05 MiB ] with mem_block_num = 2616
I20221022 13:16:02.476676 240087 time_util.h:97] [count log]{"loc":"[GraphCompile]GraphBase_0 GenMemAndLightPlanLog","mem_rss":"5720.000000 MB","mem_vm":"61740.000000 MB","time_cost":"0 seconds"}
I20221022 13:16:02.477138 240087 time_util.h:95] [count log]{"loc":"[GraphCompile]GraphBase_0 CompilePlan","mem_rss":"5720.000000 MB","mem_vm":"61740.000000 MB","time_cost":"15 seconds"}
I20221022 13:16:04.704389 240087 time_util.h:95] [count log]{"loc":"[GraphCompile]GraphBase_0 SyncPlan","mem_rss":"5720.000000 MB","mem_vm":"61740.000000 MB","time_cost":"2 seconds"}
F20221022 13:16:09.865818 240087 memory_allocator.cpp:50] out of memory
  File "/home/xiezipeng/oneflow/oneflow/core/memory/memory_allocator.cpp", line 50, in Allocate
    device->Alloc(options, &ptr, size)
Error Type: oneflow.ErrorProto.out_of_memory_error

@strint 啸宇 这里重新编译测了一遍，应该正常了

看graph的编译时间是快了2.5倍
compile graph time：195.47112141002435 （https://github.com/Oneflow-Inc/oneflow/pull/9281）
compile graph time<PHONE_NUMBER>2699587 （nightly）

之前 pass 的编译时间：

graph 编译开销 15:14:04~15:33:37，总时间为 19.5

现在是 9.1 分钟。减少 53%，符合预期。
I20221028 15:12:11.334821 2360039 time_util.h:95] [count log]{"loc":"[GraphCompile]GraphBase_0 OptimizationLogicalGraph","mem_rss":"21991.000000 MB","mem_vm":"354360.000000 MB","time_cost":"546 seconds"}


File Locksmith unclear localization

Microsoft PowerToys version
0.64.0
Utility with translation issue
File Locksmith
🌐 Language affected
Italian
❌ Actual phrase(s)
Fabbro File
✔ Expected phrase(s)
File Bloccati (or something similar)
ℹ Why is the current translation wrong
In italian the word "Fabbro" is more related to a blacksmith than the person who repair the locks

👍 "Fabbro file" localization is not misleading, is totally unsense! 😁

https://microsoft.visualstudio.com/OS/_workitems/edit/42623503 /loc

This should be fixed in a newer build such as .70.1, can you please verify? /needinfo

Ciao @crutkas 👋
Fixed? No, I wouldn't say that. The new name assigned is "Blocco file", in english much similar to "Locking file" and not "File locked" as one would expect from a correct italian translation.
The correct choice would be the one proposed by @westart (then "File bloccati") at the beginning of the discussion, or keep the original name of the tool and then return to Locksmith, kind of like what happens with FancyZones, PowerRename or PowerToys Run, leaving only the explanation in Italian, which is already correct and needs no further editing.


Github deprecated Ubuntu-18.04 in CI builds.

Search before asking

[X] I had searched in the issues and found no similar issues.

Motivation
See GitHub Actions: The Ubuntu 18.04 Actions runner image is being deprecated and will be removed by 12/1/22
Solution
So, maybe we can:

start to migrate on next version of Ubuntu base images (20.04)
add Ubuntu 22.04 as second images into matrix

Are you willing to submit a PR?

[X] I'm willing to submit a PR!

Also, tcl will be upgraded to 8.6 (default in Ubuntu 20.04 and above)

I have tested it in private fork, all builds and tests passed OK. Try to add Ubuntu 22.04 into matrix

I also noticed the problem, feel free to open a PR!

At now, a default ubuntu-latest images changed to 20.04, so cpplint not working now. In cooperation with @Torvig working to fix them

I noticed that there has been an issue proposing to replace cpplint with clang-*, raised by @PragmaTwice .  If you have time, you can try using clang-* by the way to solve this problem.

You can try to modify the current flags used in cpplint here: https://github.com/apache/incubator-kvrocks/blob/unstable/x.py#L119-L123

@torwig could we doing here? I will prepare a PR with updated Ubuntu, but cpp* is on your side

@aleksraiden Sure, I'll help you.

All task are done, except use Ubuntu 22, maybe this is too much for us, ci perform too long in this way.
Some troubles with tcl test on macos-11, try to rebuild.
Awesome @torwig fix all files, currently we are still used cpplint/check (v1.6.1) not clang.


Fix deprecated createVariable call by using collection node

Solved:

"Calling createVariable with a collection id is deprecated. Please pass the collection node instead."
"Calling setBoundVariable with a variable id is deprecated. Please pass the variable node instead."

https://github.com/figma/plugin-typings/issues/276

Thanks for this! We'll be including this change in a more comprehensive PR: https://github.com/figma/plugin-typings/pull/277


[Flowbroker] Add a search field on the geofence node

Today, users have to find the area where the polygon will be drawn by dragging and zooming the map. A search field for looking up map areas would make it much easier for users to locate the desired location.

We don't have plans to implement it, so this issue will be closed.


chore: add codeowners rule for readme.md

Hey folks
I had an offline discussion with @alequetzalli if should feel comfortable with that change. And she said YES.
The basic idea is to be more efficient in merging changes to README, especially simple PRs like adding contributors to the readme. Readme is only project documentation for contributors, not something that lands on the website, so do not always have to be signed off by Alejandra.
Please also remember that even though Alejandra's approval for readme changes will no longer be mandatory, just do not overuse it. If there is a PR that changes some content or adds new content, just be a nice citizen and ping Alejandra and give her a few days to react 🙏🏼 😄

@derberg does this PR also make a rule that every change in README.md, apart from @alequetzalli, all other CODEOWNERS also have to give their review approvals?

@akshatnema, this PR still keeps @alequetzalli as a mandatory approver for MD changes, except the Readme, where any maintainer can approve changes (still, only one approval is enough)

@fmvilas @mcturco @akshatnema all good?


rpdb2 warnings

Originally on 2012-04-25
Currently, if one enable CFG_DEVEL_SITE but doesn't have rpdb2 package installed, a set of warnings are printed on the screen per every invenio executable call:

Traceback (most recent call last):
  File "/usr/lib64/python2.6/site-packages/invenio/remote_debugger.py", line 94, in <module>
    globals()[name] = __import__(path, globals(), locals(), path.split('.'))
ImportError: No module named pydev.pydevd_file_utils
Error in remote_debugger, import of the pydev.pydevd_file_utils failedTraceback (most recent call last):
  File "/usr/lib64/python2.6/site-packages/invenio/remote_debugger.py", line 94, in <module>
    globals()[name] = __import__(path, globals(), locals(), path.split('.'))
ImportError: No module named pydev.pydevd
Error in remote_debugger, import of the pydev.pydevd failedTraceback (most recent call last):
  File "/usr/lib64/python2.6/site-packages/invenio/remote_debugger.py", line 96, in <module>
    globals()[name] = __import__(path)
ImportError: No module named rpdb2

On the other hand the rpdb2 is optional hence these warnings should either not appear or at least prettified.

@kaplun do you know if anybody is using this feature since we have werkzeug in Invenio 2.0?

+1 for removing. I don't think it's relevant anymore with werkzeug, ipdb and the development server.

... and for people using Invenio v1.x, there is winpdb in requirements-extras.txt, so if people are debugging, chances are they have it installed already.  I'm therefore closing this issue.


Is it possible to convert tesseract traineddata to other format?

I'm wondering if it's possible to convert tesseract traineddata to other format so that it can be used by Core ML?

That might be possible, but requires some work. Do you want to try it?
A conversion of the LSTM models between OCRopus and Tesseract would also be nice.

A conversion of the LSTM models between OCRopus and Tesseract would also be nice.

The models themselves has different net spec (1D vs 2D).
Of course, you can use the same dataset to train both of these apps.

Not so familiar with this. Any hint? What should I learn to do the conversion?

See also: https://github.com/mittagessen/kraken/issues/152


[Bug]:  APPSMITH_INSTANCE_NAME is not escaped when being stored in docker.env. Can result in entrypoint.sh failure.

Is there an existing issue for this?

[X] I have searched the existing issues

Description
I changed the name of my appsmith instance in the web interface to something with a space in it. For example "XYZ Group".  When starting the container, the entrypoint.sh bash script is fails due to space in the string with the following error.
Initialize .env file
Load environment configuration
/appsmith-stacks/configuration/docker.env: line 92: Group: command not found

Expected behaviour is that container is able to start without error.
Resolution was to edit docker.env
Steps To Reproduce

Deploy appsmith
Login as supervisor and go to admin settings.
Change instance name to "XYZ Group".
Press "save and restart".
After restart, stop the docker container.  docker stop appsmith
Start container docker start appsmith

Public Sample App
No response
Version
Self-Hosted ee - 1.7.4

This seems like a duplicate of https://github.com/appsmithorg/appsmith/issues/14659 - or at least a specific example of the wider problem with spaces in the .env file.

This seems like a duplicate of https://github.com/appsmithorg/appsmith/issues/13335 - or at least a specific example of the wider problem with spaces in the .env file.

@tomharvey thanks for pointing this out! I'll close this in favor of #13335


explicit order for Panamanian communities

I've explicit set order to the default value 0 for the YMUP Chapter, 10 for the official forum, and 5 for the local Telegram group. This is based on my experience, after having spent 4½ years in this country.

The YouthMappers chapter will still rank higher than the other sources because it is tiny, and small communities are ranked higher than big ones.
Aside from that, the rest of the rankings will work the way you expect 👍

The YouthMappers chapter will still rank higher than the other sources because it is tiny, and small communities are ranked higher than big ones.

I'm sorry, but I need to insist on this, but I don't think the https://openstreetmap.community/ is informing correctly, by ranking this (and other) YouthMappers Chapter higher than the local official openstreetmap.org forum.
ranking a group higher because it's smaller … I don't follow the logic.  then by absurd I could put myself, a single individual, as top of the list? :wink: ?
in my opinion you need some other sort of statistics for ranking communities, like responsiveness, level of activity, as a group and as its members, visibility of membership, maybe whether you can expect to write to them in English, I don't know, but something objective.
so I have two points:

rank objectively active community higher
rank official OSM community spaces higher

ranking a group higher because it's smaller … I don't follow the logic. then by absurd I could put myself, a single individual, as top of the list? 😉 ?

Yes if you can convince me to add your single community-of-one to this index, it will indeed rank highest and I don't consider this "absurd".  I think you should at least invite a few some people over for drinks and talk about maps.  We encourage this, and communities often start with a single motivated individual. 🍻
It's really hard for me to see why you don't like seeing the local YouthMappers chapter ranked higher than the Panama forum in this list of 18 things.  They exist only in a little bubble around Panama City, and there are plenty of other things on the list.  Maybe you should reach out to them and see how they're doing?


Renter download

Ok, there's more than just download going on here...
There's a race condition with the storageObligation locks :I
Seems like you need to grab a host lock before you check the storageObligation lock map.

oh, you definitely do, that's my fault. Do you want to fix it, or should I? Make sure that unlock is called correctly, and that you apply it to all 3 functions that grab a storage obligation lock (formcontract, revisecontract, and download)


add a Random User Generator

Random User Generator will generate random user with name, email and phone no

hey @iamrahulmahato i would like to work on this


Can not complete group tasks

Some of the group tasks can be completed even if I join groups manually.
Stamstat.us shows Steam servers are fine so It could be the script?

Steam Community may not be completely offline, but it's not 100% operational either.  Steamstat.us isn't giving the full picture.
There's something funky going on with Steam Community that prevents Gleam.io from seeing that you've joined a group.  Sometimes when this happens, it's impossible for you to join or leave a group.


Options to remove problems from extension

Give an option to remove monitoring a problem from the popup.
Also give an option in the popup to remove all monitored problems from the extension.

Clear Active and Clear Complete buttons provided in the bottom of the popup.

Modifications to the Clear functionality:
Clear Active clears all the problems that are currently active. However, it should delete all the active sessions of all the problems.
Similarly, Clear Complete should clear all the complete sessions of all problems. Now, in Clear Complete case this won't matter, as a problem is complete only after it is active i.e. there cannot be an active session for a problem if there is an active session of the same problem. But for the sake of maintaining accuracy, just remove the Complete sessions of the problem, and not the problem itself, when the Clear Complete button is called.

Completed in commit 60873166e816745a65ce05b0344172b098b9e944


[#163592114] Remove Elasticsearch 5.x from billing

What
As Elasticsearch 5.x is now EOL and no tenants are using it, it can be removed from the billing, so it is no longer visible in the calculator.
How to review

Code review
Ensure no tenants are running ES5 instances (instructions are available on https://github.com/alphagov/paas-cf/pull/1779)
Who can review

Not @tnwhitwell or @AP-Hunt

paas-billing requires that plans exist in order to consolidate bills each month, and to reconsolidate old bills should be the need arise, and paas-admin reads the pricing plans via the API to build the calculator.
In the past, we've deprecated Compose plans but not removed them from billing. Instead, the plans were blacklisted in paas-admin. For ES 5.x we're also going to leave the plans in place for the time being, so that we don't break billing.
Closing this PR, and raising a story to address how to deprecate plans properly in the future.