Low accuracy when training on waymo dataset

Hi, I add the recent update on the waymo dataset to the previous version of OpenPCDet (maybe June2020 version), and train second/PartA2 on the waymo dataset. I do not modify the preprocess script and model cfg for waymo.
However, the accuracy is much lower. For second, the final accuracy is as follows:
742 2020-11-24 08:00:32,763   INFO  Evaluation done.*
743 2020-11-24 09:47:11,516   INFO  *************** EPOCH 0 EVALUATION *****************
744 2020-11-24 09:48:05,627   INFO  *************** Performance of EPOCH 0 *****************
745 2020-11-24 09:48:05,627   INFO  Generate label finished(sec_per_example: 0.0068 second).
746 2020-11-24 09:48:05,628   INFO  recall_roi_0.3: 0.000000 (0 / 296561)
747 2020-11-24 09:48:05,628   INFO  recall_rcnn_0.3: 0.689605 (204510 / 296561)
748 2020-11-24 09:48:05,628   INFO  recall_roi_0.5: 0.000000 (0 / 296561)
749 2020-11-24 09:48:05,628   INFO  recall_rcnn_0.5: 0.574320 (170321 / 296561)
750 2020-11-24 09:48:05,628   INFO  recall_roi_0.7: 0.000000 (0 / 296561)
751 2020-11-24 09:48:05,628   INFO  recall_rcnn_0.7: 0.316451 (93847 / 296561)
752 2020-11-24 09:48:05,637   INFO  Average predicted number of objects(7998 samples): 100.557
753 2020-11-24 09:51:25,703   INFO
754 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/AP: 0.3911
755 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/APH: 0.3855
756 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP: 0.3620
757 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH: 0.3568
758 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/AP: 0.3631
759 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/APH: 0.2795
760 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/AP: 0.3308
761 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/APH: 0.2545
762 OBJECT_TYPE_TYPE_SIGN_LEVEL_1/AP: 0.0000
763 OBJECT_TYPE_TYPE_SIGN_LEVEL_1/APH: 0.0000
764 OBJECT_TYPE_TYPE_SIGN_LEVEL_2/AP: 0.0000
765 OBJECT_TYPE_TYPE_SIGN_LEVEL_2/APH: 0.0000
766 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/AP: 0.2761
767 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/APH: 0.2636
768 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/AP: 0.2757
769 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/APH: 0.2632
I'd like to ask what may cause this problem.
Thank you very much.

I trained 80 epoches, as the previous version does not load the "NUM_EPOCHS" parameter in the cfg

What's your model for evaluation? It seems you used the epoch 0 for evaulation.

What's your model for evaluation? It seems you used the epoch 0 for evaulation.

I used the last epoch to evaluate, which is the online evaluation during the training process. This is just a typo error.

And I do not use "merge_all_iters_to_one_epoch".

Let me check whether there is a bug in the dataloader.

Let me check whether there is a bug in the dataloader.

Thank you.

Currently I didn't find any obvious bug, mayby you could try to train with the latest codes.
I will also try to train a model.

Currently I didn't find any obvious bug, mayby you could try to train with the latest codes.
I will also try to train a model.

Okay

I got much better results than your reported performance, hence it should not be a problem of the dataloader.
I will try more models since the released dataloader is a little bit different with me since my dataloader is an old version.

I just donwload the updated version and try to train it. However, out-of-memory issue happened. I do not know if the updated version need much more memory for the waymo dataset.

No, I could train SECOND with the given config on GPU 1080Ti with 11G memory.

I use the latest version. My training will always be stuck here：
2020-11-25 15:59:13,004   INFO  Start training data1/OpenPCDet-master20201117/tools/cfgs/waymo_models/second(test_in_shell)
epochs:   0%|                                                                                | 0/30 [00:00<?, ?it/s]

It seems that the problem is caused by the multi-gpu training. When I use the single gpu, the training is OK. This problem does not happen when I use the old version. I will check it.

https://github.com/open-mmlab/OpenPCDet/blob/594962844729353842207b9e3c50f8b2434484f9/tools/train_utils/train_utils.py#L76
When using two gpus for training, the code is stack here. When using the old version in the same device, I think that maybe the dataloader cause the problem. Is the updated dataloader much different from the old one?

When I set the num_worker=0, multi-gpu training is ok. Such problem does not exsit in the previous version. I will look for the cause of this situation.

@LZDSJTU , I do find some problems in the evaluation codes, please see https://github.com/open-mmlab/OpenPCDet/pull/384.
But the data and training codes are fine.
You could follow the instructions to re-create the validation infos and re-evaluate your models.

OK, I will modify the evaluation code. I think that the latest version has some problem in dataloader. I run different versions in the same environment with multi-gpus, the previous version is OK but the current version is deadlock. I haven't found the reason.

Have you tried to run with other datasets like KITTI?

Yep, the similar problem still exsits for KITTI. The difference is that using KITTI may not meet deadlock. It just gets stack for a long time and start running. However, if using the previous version, it is very fast to start running.

I do find some problems in the evaluation codes, please see #384.
But the data and training codes are fine.
You could follow the instructions to re-create the validation infos and re-evaluate your models.

Saw this message just in time minutes after my script started to create them.
I had to re-download WOD (I used to work with v1.0 instead of v1.2).
I will try to reproduce your results on WOD next week.

@LZDSJTU , I do find some problems in the evaluation codes, please see #384.
But the data and training codes are fine.
You could follow the instructions to re-create the validation infos and re-evaluate your models.

Hi, I re-preprocess the waymo val set and evaluate the accuracy. Actually, the accuracy increases but is still much lower than yours. I trained it with the old pcdet with the waymo cfg.
I have not trained on the latest version as the multi-gpu training has some problem in my environment before. I found that If I insert the following sentence in eval_utils.py, the multi-gpu training will be okay:
from pcdet.datasets.kitti.kitti_object_eval_python.eval import print_str
I cannot understand the reason now as the "print_str" is not used in eval_utils.py and my code is not stuck here. It got stuck at the beginning of training.

@sshaoshuai @MartinHahner
I train the second model using the latest version, and I have re-created the val info for the waymo dataset.
I follow the instruction to train the model for 30 epochs. The final accuracy is still much lower:
5 2020-12-01 03:47:44,740   INFO  *************** EPOCH 30 EVALUATION *****************
366 2020-12-01 03:48:32,127   INFO  *************** Performance of EPOCH 30 *****************
367 2020-12-01 03:48:32,128   INFO  Generate label finished(sec_per_example: 0.0059 second).
368 2020-12-01 03:48:32,128   INFO  recall_roi_0.3: 0.000000
369 2020-12-01 03:48:32,128   INFO  recall_rcnn_0.3: 0.668910
370 2020-12-01 03:48:32,128   INFO  recall_roi_0.5: 0.000000
371 2020-12-01 03:48:32,128   INFO  recall_rcnn_0.5: 0.552218
372 2020-12-01 03:48:32,128   INFO  recall_roi_0.7: 0.000000
373 2020-12-01 03:48:32,128   INFO  recall_rcnn_0.7: 0.297340
374 2020-12-01 03:48:32,133   INFO  Average predicted number of objects(7998 samples): 101.049
375 2020-12-01 03:51:04,877   INFO
376 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/AP: 0.4284
377 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/APH: 0.4213
378 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP: 0.3733
379 OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH: 0.3670
380 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/AP: 0.4020
381 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/APH: 0.3015
382 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/AP: 0.3456
383 OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/APH: 0.2589
384 OBJECT_TYPE_TYPE_SIGN_LEVEL_1/AP: 0.0000
385 OBJECT_TYPE_TYPE_SIGN_LEVEL_1/APH: 0.0000
386 OBJECT_TYPE_TYPE_SIGN_LEVEL_2/AP: 0.0000
387 OBJECT_TYPE_TYPE_SIGN_LEVEL_2/APH: 0.0000
388 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/AP: 0.2885
389 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/APH: 0.2714
390 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/AP: 0.2791
391 OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/APH: 0.2626
The result is very similar with that using the previous PCDet version which adds the waymo cfg (after re-creating the val info).

Currently I only re-create the val info for the waymo dataset. I do not know whether the train info needs to be re-created as the gt_sample needs to the parameter 'num_points_in_gt' to filter the database.
But my accuracy is very different from the your provided, so I am not sure whether the gap is caused by this reason.

This is my training log for SECOND model.
2020-11-30 19:51:53,592   INFO  Start logging
2020-11-30 19:51:53,592   INFO  CUDA_VISIBLE_DEVICES=ALL
2020-11-30 19:51:53,592   INFO  total_batch_size: 32
2020-11-30 19:51:53,592   INFO  cfg_file         /data1/OpenPCDet-master_new/tools/cfgs/waymo_models/second.yaml
2020-11-30 19:51:53,592   INFO  batch_size       4
2020-11-30 19:51:53,592   INFO  epochs           30
2020-11-30 19:51:53,592   INFO  workers          8
2020-11-30 19:51:53,592   INFO  extra_tag        20201130_second
2020-11-30 19:51:53,592   INFO  ckpt             None
2020-11-30 19:51:53,592   INFO  pretrained_model None
2020-11-30 19:51:53,592   INFO  launcher         pytorch
2020-11-30 19:51:53,592   INFO  tcp_port         18865
2020-11-30 19:51:53,592   INFO  sync_bn          False
2020-11-30 19:51:53,592   INFO  fix_random_seed  False
2020-11-30 19:51:53,593   INFO  ckpt_save_interval 5
2020-11-30 19:51:53,593   INFO  local_rank       0
2020-11-30 19:51:53,593   INFO  max_ckpt_save_num 30
2020-11-30 19:51:53,593   INFO  merge_all_iters_to_one_epoch False
2020-11-30 19:51:53,593   INFO  set_cfgs         None
2020-11-30 19:51:53,593   INFO  max_waiting_mins 0
2020-11-30 19:51:53,593   INFO  start_epoch      0
2020-11-30 19:51:53,593   INFO  save_to_file     False
2020-11-30 19:51:53,593   INFO  cfg.ROOT_DIR: /data1/OpenPCDet-master_new
2020-11-30 19:51:53,593   INFO  cfg.LOCAL_RANK: 0
2020-11-30 19:51:53,593   INFO  cfg.CLASS_NAMES: ['Vehicle', 'Pedestrian', 'Cyclist']
2020-11-30 19:51:53,593   INFO
cfg.DATA_CONFIG = edict()
2020-11-30 19:51:53,593   INFO  cfg.DATA_CONFIG.DATASET: WaymoDataset
2020-11-30 19:51:53,593   INFO  cfg.DATA_CONFIG.DATA_PATH: ../data/waymo
2020-11-30 19:51:53,593   INFO  cfg.DATA_CONFIG.PROCESSED_DATA_TAG: waymo_processed_data
2020-11-30 19:51:53,593   INFO  cfg.DATA_CONFIG.POINT_CLOUD_RANGE: [-75.2, -75.2, -2, 75.2, 75.2, 4]
2020-11-30 19:51:53,593   INFO
cfg.DATA_CONFIG.DATA_SPLIT = edict()
2020-11-30 19:51:53,593   INFO  cfg.DATA_CONFIG.DATA_SPLIT.train: train
2020-11-30 19:51:53,593   INFO  cfg.DATA_CONFIG.DATA_SPLIT.test: val
2020-11-30 19:51:53,593   INFO
cfg.DATA_CONFIG.SAMPLED_INTERVAL = edict()
2020-11-30 19:51:53,593   INFO  cfg.DATA_CONFIG.SAMPLED_INTERVAL.train: 5
2020-11-30 19:51:53,593   INFO  cfg.DATA_CONFIG.SAMPLED_INTERVAL.test: 5
2020-11-30 19:51:53,593   INFO
cfg.DATA_CONFIG.DATA_AUGMENTOR = edict()
2020-11-30 19:51:53,593   INFO  cfg.DATA_CONFIG.DATA_AUGMENTOR.DISABLE_AUG_LIST: ['placeholder']
2020-11-30 19:51:53,594   INFO  cfg.DATA_CONFIG.DATA_AUGMENTOR.AUG_CONFIG_LIST: [{'NAME': 'gt_sampling', 'USE_ROAD_PLANE': False, 'DB_INFO_PATH': ['pcdet_waymo_dbinfos_train_sampled_10.pkl'], 'PREPARE': {'filter_by_min_points': ['Vehicle:5', 'Pedestrian:5', 'Cyclist:5'], 'filter_by_difficulty': [-1]}, 'SAMPLE_GROUPS': ['Vehicle:15', 'Pedestrian:10', 'Cyclist:10'], 'NUM_POINT_FEATURES': 5, 'REMOVE_EXTRA_WIDTH': [0.0, 0.0, 0.0], 'LIMIT_WHOLE_SCENE': True}, {'NAME': 'random_world_flip', 'ALONG_AXIS_LIST': ['x', 'y']}, {'NAME': 'random_world_rotation', 'WORLD_ROT_ANGLE': [-0.78539816, 0.78539816]}, {'NAME': 'random_world_scaling', 'WORLD_SCALE_RANGE': [0.95, 1.05]}]
2020-11-30 19:51:53,594   INFO
cfg.DATA_CONFIG.POINT_FEATURE_ENCODING = edict()
2020-11-30 19:51:53,594   INFO  cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.encoding_type: absolute_coordinates_encoding
2020-11-30 19:51:53,594   INFO  cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.used_feature_list: ['x', 'y', 'z', 'intensity', 'elongation']
2020-11-30 19:51:53,594   INFO  cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.src_feature_list: ['x', 'y', 'z', 'intensity', 'elongation']
2020-11-30 19:51:53,594   INFO  cfg.DATA_CONFIG.DATA_PROCESSOR: [{'NAME': 'mask_points_and_boxes_outside_range', 'REMOVE_OUTSIDE_BOXES': True}, {'NAME': 'shuffle_points', 'SHUFFLE_ENABLED': {'train': True, 'test': True}}, {'NAME': 'transform_points_to_voxels', 'VOXEL_SIZE': [0.1, 0.1, 0.15], 'MAX_POINTS_PER_VOXEL': 5, 'MAX_NUMBER_OF_VOXELS': {'train': 80000, 'test': 90000}}]
2020-11-30 19:51:53,594   INFO  cfg.DATA_CONFIG.BASE_CONFIG: cfgs/dataset_configs/waymo_dataset.yaml
2020-11-30 19:51:53,594   INFO
cfg.MODEL = edict()
2020-11-30 19:51:53,594   INFO  cfg.MODEL.NAME: SECONDNet
2020-11-30 19:51:53,594   INFO
cfg.MODEL.VFE = edict()
2020-11-30 19:51:53,594   INFO  cfg.MODEL.VFE.NAME: MeanVFE
2020-11-30 19:51:53,595   INFO
cfg.MODEL.BACKBONE_3D = edict()
2020-11-30 19:51:53,595   INFO  cfg.MODEL.BACKBONE_3D.NAME: VoxelBackBone8x
2020-11-30 19:51:53,595   INFO
cfg.MODEL.MAP_TO_BEV = edict()
2020-11-30 19:51:53,595   INFO  cfg.MODEL.MAP_TO_BEV.NAME: HeightCompression
2020-11-30 19:51:53,595   INFO  cfg.MODEL.MAP_TO_BEV.NUM_BEV_FEATURES: 256
2020-11-30 19:51:53,595   INFO
cfg.MODEL.BACKBONE_2D = edict()
2020-11-30 19:51:53,595   INFO  cfg.MODEL.BACKBONE_2D.NAME: BaseBEVBackbone
2020-11-30 19:51:53,595   INFO  cfg.MODEL.BACKBONE_2D.LAYER_NUMS: [5, 5]
2020-11-30 19:51:53,595   INFO  cfg.MODEL.BACKBONE_2D.LAYER_STRIDES: [1, 2]
2020-11-30 19:51:53,595   INFO  cfg.MODEL.BACKBONE_2D.NUM_FILTERS: [128, 256]
2020-11-30 19:51:53,595   INFO  cfg.MODEL.BACKBONE_2D.UPSAMPLE_STRIDES: [1, 2]
2020-11-30 19:51:53,595   INFO  cfg.MODEL.BACKBONE_2D.NUM_UPSAMPLE_FILTERS: [256, 256]
2020-11-30 19:51:53,595   INFO
cfg.MODEL.DENSE_HEAD = edict()
2020-11-30 19:51:53,595   INFO  cfg.MODEL.DENSE_HEAD.NAME: AnchorHeadSingle
2020-11-30 19:51:53,595   INFO  cfg.MODEL.DENSE_HEAD.CLASS_AGNOSTIC: False
2020-11-30 19:51:53,595   INFO  cfg.MODEL.DENSE_HEAD.USE_DIRECTION_CLASSIFIER: True
2020-11-30 19:51:53,595   INFO  cfg.MODEL.DENSE_HEAD.DIR_OFFSET: 0.78539
2020-11-30 19:51:53,595   INFO  cfg.MODEL.DENSE_HEAD.DIR_LIMIT_OFFSET: 0.0
2020-11-30 19:51:53,595   INFO  cfg.MODEL.DENSE_HEAD.NUM_DIR_BINS: 2
2020-11-30 19:51:53,595   INFO  cfg.MODEL.DENSE_HEAD.ANCHOR_GENERATOR_CONFIG: [{'class_name': 'Vehicle', 'anchor_sizes': [[4.7, 2.1, 1.7]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [0], 'align_center': False, 'feature_map_stride': 8, 'matched_threshold': 0.55, 'unmatched_threshold': 0.4}, {'class_name': 'Pedestrian', 'anchor_sizes': [[0.91, 0.86, 1.73]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [0], 'align_center': False, 'feature_map_stride': 8, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}, {'class_name': 'Cyclist', 'anchor_sizes': [[1.78, 0.84, 1.78]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [0], 'align_center': False, 'feature_map_stride': 8, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}]
2020-11-30 19:51:53,595   INFO
cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG = edict()
2020-11-30 19:51:53,595   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NAME: AxisAlignedTargetAssigner
2020-11-30 19:51:53,596   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.POS_FRACTION: -1.0
2020-11-30 19:51:53,596   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.SAMPLE_SIZE: 512
2020-11-30 19:51:53,596   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NORM_BY_NUM_EXAMPLES: False
2020-11-30 19:51:53,596   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.MATCH_HEIGHT: False
2020-11-30 19:51:53,596   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.BOX_CODER: ResidualCoder
2020-11-30 19:51:53,596   INFO
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG = edict()
2020-11-30 19:51:53,596   INFO
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict()
2020-11-30 19:51:53,596   INFO  cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.cls_weight: 1.0
2020-11-30 19:51:53,596   INFO  cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.loc_weight: 2.0
2020-11-30 19:51:53,596   INFO  cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.dir_weight: 0.2
2020-11-30 19:51:53,596   INFO  cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
2020-11-30 19:51:53,596   INFO
cfg.MODEL.POST_PROCESSING = edict()
2020-11-30 19:51:53,596   INFO  cfg.MODEL.POST_PROCESSING.RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
2020-11-30 19:51:53,596   INFO  cfg.MODEL.POST_PROCESSING.SCORE_THRESH: 0.1
2020-11-30 19:51:53,596   INFO  cfg.MODEL.POST_PROCESSING.OUTPUT_RAW_SCORE: False
2020-11-30 19:51:53,596   INFO  cfg.MODEL.POST_PROCESSING.EVAL_METRIC: waymo
2020-11-30 19:51:53,596   INFO
cfg.MODEL.POST_PROCESSING.NMS_CONFIG = edict()
2020-11-30 19:51:53,596   INFO  cfg.MODEL.POST_PROCESSING.NMS_CONFIG.MULTI_CLASSES_NMS: False
2020-11-30 19:51:53,597   INFO  cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_TYPE: nms_gpu
2020-11-30 19:51:53,597   INFO  cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_THRESH: 0.01
2020-11-30 19:51:53,597   INFO  cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_PRE_MAXSIZE: 4096
2020-11-30 19:51:53,597   INFO  cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_POST_MAXSIZE: 500
2020-11-30 19:51:53,597   INFO
cfg.OPTIMIZATION = edict()
2020-11-30 19:51:53,597   INFO  cfg.OPTIMIZATION.BATCH_SIZE_PER_GPU: 4
2020-11-30 19:51:53,597   INFO  cfg.OPTIMIZATION.NUM_EPOCHS: 30
2020-11-30 19:51:53,597   INFO  cfg.OPTIMIZATION.OPTIMIZER: adam_onecycle
2020-11-30 19:51:53,597   INFO  cfg.OPTIMIZATION.LR: 0.003
2020-11-30 19:51:53,597   INFO  cfg.OPTIMIZATION.WEIGHT_DECAY: 0.01
2020-11-30 19:51:53,597   INFO  cfg.OPTIMIZATION.MOMENTUM: 0.9
2020-11-30 19:51:53,597   INFO  cfg.OPTIMIZATION.MOMS: [0.95, 0.85]
2020-11-30 19:51:53,597   INFO  cfg.OPTIMIZATION.PCT_START: 0.4
2020-11-30 19:51:53,597   INFO  cfg.OPTIMIZATION.DIV_FACTOR: 10
2020-11-30 19:51:53,597   INFO  cfg.OPTIMIZATION.DECAY_STEP_LIST: [35, 45]
2020-11-30 19:51:53,597   INFO  cfg.OPTIMIZATION.LR_DECAY: 0.1
2020-11-30 19:51:53,597   INFO  cfg.OPTIMIZATION.LR_CLIP: 1e-07
2020-11-30 19:51:53,597   INFO  cfg.OPTIMIZATION.LR_WARMUP: False
2020-11-30 19:51:53,598   INFO  cfg.OPTIMIZATION.WARMUP_EPOCH: 1
2020-11-30 19:51:53,598   INFO  cfg.OPTIMIZATION.GRAD_NORM_CLIP: 10
2020-11-30 19:51:53,598   INFO  cfg.TAG: second
2020-11-30 19:51:53,598   INFO  cfg.EXP_GROUP_PATH: data1/OpenPCDet-master_new/tools/cfgs/waymo_models
2020-11-30 19:51:56,666   INFO  Database filter by min points Vehicle: 477761 => 310377
2020-11-30 19:51:56,697   INFO  Database filter by min points Pedestrian: 222675 => 136965
2020-11-30 19:51:56,700   INFO  Database filter by min points Cyclist: 5344 => 3717
2020-11-30 19:51:56,794   INFO  Database filter by difficulty Vehicle: 310377 => 310377
2020-11-30 19:51:56,836   INFO  Database filter by difficulty Pedestrian: 136965 => 136965
2020-11-30 19:51:56,837   INFO  Database filter by difficulty Cyclist: 3717 => 3717
2020-11-30 19:51:57,209   INFO  Loading Waymo dataset
2020-11-30 19:52:07,651   INFO  Total skipped info 0
2020-11-30 19:52:07,651   INFO  Total samples for Waymo dataset: 158081
2020-11-30 19:52:07,659   INFO  Total sampled samples for Waymo dataset: 31617
2020-11-30 19:52:19,787   INFO  DistributedDataParallel(
(module): SECONDNet(
(vfe): MeanVFE()
(backbone_3d): VoxelBackBone8x(
(conv_input): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(conv1): SparseSequential(
(0): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
)
(conv2): SparseSequential(
(0): SparseSequential(
(0): SparseConv3d()
(1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(1): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(2): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
)
(conv3): SparseSequential(
(0): SparseSequential(
(0): SparseConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(1): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(2): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
)
(conv4): SparseSequential(
(0): SparseSequential(
(0): SparseConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(1): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(2): SparseSequential(
(0): SubMConv3d()
(1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
)
(conv_out): SparseSequential(
(0): SparseConv3d()
(1): BatchNorm1d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
)
(map_to_bev_module): HeightCompression()
(pfe): None
(backbone_2d): BaseBEVBackbone(
(blocks): ModuleList(
(0): Sequential(
(0): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)
(1): Conv2d(256, 128, kernel_size=(3, 3), stride=(1, 1), bias=False)
(2): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(3): ReLU()
(4): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(5): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(6): ReLU()
(7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(8): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(9): ReLU()
(10): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(11): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(12): ReLU()
(13): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(14): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(15): ReLU()
(16): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(17): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(18): ReLU()
)
(1): Sequential(
(0): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)
(1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), bias=False)
(2): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(3): ReLU()
(4): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(5): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(6): ReLU()
(7): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(8): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(9): ReLU()
(10): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(11): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(12): ReLU()
(13): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(14): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(15): ReLU()
(16): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(17): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(18): ReLU()
)
)
(deblocks): ModuleList(
(0): Sequential(
(0): ConvTranspose2d(128, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(1): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
(1): Sequential(
(0): ConvTranspose2d(256, 256, kernel_size=(2, 2), stride=(2, 2), bias=False)
(1): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True)
(2): ReLU()
)
)
)
(dense_head): AnchorHeadSingle(
(cls_loss_func): SigmoidFocalClassificationLoss()
(reg_loss_func): WeightedSmoothL1Loss()
(dir_loss_func): WeightedCrossEntropyLoss()
(conv_cls): Conv2d(512, 18, kernel_size=(1, 1), stride=(1, 1))
(conv_box): Conv2d(512, 42, kernel_size=(1, 1), stride=(1, 1))
(conv_dir_cls): Conv2d(512, 12, kernel_size=(1, 1), stride=(1, 1))
)
(point_head): None
(roi_head): None
)
)
2020-11-30 19:52:19,789   INFO  Start training data1/OpenPCDet-master_new/tools/cfgs/waymo_models/second(20201130_second)
2020-12-01 03:39:10,303   INFO  End training data1/OpenPCDet-master_new/tools/cfgs/waymo_models/second(20201130_second)
2020-12-01 03:39:10,304   INFO  Start evaluation data1/OpenPCDet-master_new/tools/cfgs/waymo_models/second(20201130_second)
2020-12-01 03:39:10,480   INFO  Loading Waymo dataset
2020-12-01 03:39:13,718   INFO  Total skipped info 0
2020-12-01 03:39:13,719   INFO  Total samples for Waymo dataset: 39987
2020-12-01 03:39:13,720   INFO  Total sampled samples for Waymo dataset: 7998
2020-12-01 03:39:13,978   INFO  ==> Loading parameters from checkpoint /data1/OpenPCDet-master_new/output/data1/OpenPCDet-master_new/tools/cfgs/waymo_models/second/20201130_second/ckpt/checkpoint_epoch_20.pth to CPU
2020-12-01 03:39:14,051   INFO  ==> Checkpoint trained from version: pcdet+0.3.0+0000000
2020-12-01 03:39:14,182   INFO  ==> Done (loaded 163/163)
2020-12-01 03:39:14,184   INFO  *************** EPOCH 20 EVALUATION *****************
2020-12-01 03:40:00,176   INFO  *************** Performance of EPOCH 20 *****************
2020-12-01 03:40:00,176   INFO  Generate label finished(sec_per_example: 0.0057 second).
2020-12-01 03:40:00,176   INFO  recall_roi_0.3: 0.000000
2020-12-01 03:40:00,176   INFO  recall_rcnn_0.3: 0.646826
2020-12-01 03:40:00,176   INFO  recall_roi_0.5: 0.000000
2020-12-01 03:40:00,176   INFO  recall_rcnn_0.5: 0.527019
2020-12-01 03:40:00,176   INFO  recall_roi_0.7: 0.000000
2020-12-01 03:40:00,176   INFO  recall_rcnn_0.7: 0.265897
2020-12-01 03:40:00,182   INFO  Average predicted number of objects(7998 samples): 105.472
2020-12-01 03:44:04,417   INFO
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/AP: 0.3743
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/APH: 0.3658
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP: 0.3248
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH: 0.3173
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/AP: 0.3563
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/APH: 0.2631
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/AP: 0.3046
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/APH: 0.2246
OBJECT_TYPE_TYPE_SIGN_LEVEL_1/AP: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_1/APH: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_2/AP: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_2/APH: 0.0000
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/AP: 0.2470
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/APH: 0.2303
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/AP: 0.2379
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/APH: 0.2219
2020-12-01 03:44:04,418   INFO  Result is save to /data1/OpenPCDet-master_new/output/data1/OpenPCDet-master_new/tools/cfgs/waymo_models/second/20201130_second/eval/eval_with_train/epoch_20/val
2020-12-01 03:44:04,419   INFO  Evaluation done.*
2020-12-01 03:44:04,425   INFO  Epoch 20 has been evaluated
2020-12-01 03:44:04,427   INFO  ==> Loading parameters from checkpoint /data1/OpenPCDet-master_new/output/data1/OpenPCDet-master_new/tools/cfgs/waymo_models/second/20201130_second/ckpt/checkpoint_epoch_25.pth to CPU
2020-12-01 03:44:04,483   INFO  ==> Checkpoint trained from version: pcdet+0.3.0+0000000
2020-12-01 03:44:04,609   INFO  ==> Done (loaded 163/163)
2020-12-01 03:44:04,611   INFO  *************** EPOCH 25 EVALUATION *****************
2020-12-01 03:44:51,699   INFO  *************** Performance of EPOCH 25 *****************
2020-12-01 03:44:51,700   INFO  Generate label finished(sec_per_example: 0.0059 second).
2020-12-01 03:44:51,701   INFO  recall_roi_0.3: 0.000000
2020-12-01 03:44:51,701   INFO  recall_rcnn_0.3: 0.664714
2020-12-01 03:44:51,701   INFO  recall_roi_0.5: 0.000000
2020-12-01 03:44:51,701   INFO  recall_rcnn_0.5: 0.543964
2020-12-01 03:44:51,701   INFO  recall_roi_0.7: 0.000000
2020-12-01 03:44:51,701   INFO  recall_rcnn_0.7: 0.284320
2020-12-01 03:44:51,710   INFO  Average predicted number of objects(7998 samples): 100.893
2020-12-01 03:47:44,519   INFO
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/AP: 0.4097
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/APH: 0.4028
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP: 0.3567
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH: 0.3505
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/AP: 0.3829
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/APH: 0.2832
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/AP: 0.3247
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/APH: 0.2400
OBJECT_TYPE_TYPE_SIGN_LEVEL_1/AP: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_1/APH: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_2/AP: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_2/APH: 0.0000
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/AP: 0.2680
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/APH: 0.2518
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/AP: 0.2581
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/APH: 0.2425
2020-12-01 03:47:44,520   INFO  Result is save to /data1/OpenPCDet-master_new/output/data1/OpenPCDet-master_new/tools/cfgs/waymo_models/second/20201130_second/eval/eval_with_train/epoch_25/val
2020-12-01 03:47:44,520   INFO  Evaluation done.*
2020-12-01 03:47:44,553   INFO  Epoch 25 has been evaluated
2020-12-01 03:47:44,554   INFO  ==> Loading parameters from checkpoint /data1/OpenPCDet-master_new/output/data1/OpenPCDet-master_new/tools/cfgs/waymo_models/second/20201130_second/ckpt/checkpoint_epoch_30.pth to CPU
2020-12-01 03:47:44,604   INFO  ==> Checkpoint trained from version: pcdet+0.3.0+0000000
2020-12-01 03:47:44,737   INFO  ==> Done (loaded 163/163)
2020-12-01 03:47:44,740   INFO  *************** EPOCH 30 EVALUATION *****************
2020-12-01 03:48:32,127   INFO  *************** Performance of EPOCH 30 *****************
2020-12-01 03:48:32,128   INFO  Generate label finished(sec_per_example: 0.0059 second).
2020-12-01 03:48:32,128   INFO  recall_roi_0.3: 0.000000
2020-12-01 03:48:32,128   INFO  recall_rcnn_0.3: 0.668910
2020-12-01 03:48:32,128   INFO  recall_roi_0.5: 0.000000
2020-12-01 03:48:32,128   INFO  recall_rcnn_0.5: 0.552218
2020-12-01 03:48:32,128   INFO  recall_roi_0.7: 0.000000
2020-12-01 03:48:32,128   INFO  recall_rcnn_0.7: 0.297340
2020-12-01 03:48:32,133   INFO  Average predicted number of objects(7998 samples): 101.049
2020-12-01 03:51:04,877   INFO
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/AP: 0.4284
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/APH: 0.4213
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP: 0.3733
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH: 0.3670
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/AP: 0.4020
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/APH: 0.3015
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/AP: 0.3456
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/APH: 0.2589
OBJECT_TYPE_TYPE_SIGN_LEVEL_1/AP: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_1/APH: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_2/AP: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_2/APH: 0.0000
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/AP: 0.2885
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/APH: 0.2714
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/AP: 0.2791
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/APH: 0.2626
2020-12-01 03:51:04,878   INFO  Result is save to /data1/OpenPCDet-master_new/output/data1/OpenPCDet-master_new/tools/cfgs/waymo_models/second/20201130_second/eval/eval_with_train/epoch_30/val
2020-12-01 03:51:04,878   INFO  Evaluation done.*
2020-12-01 03:51:04,910   INFO  Epoch 30 has been evaluated
2020-12-01 03:51:34,941   INFO  End evaluation data1/OpenPCDet-master_new/tools/cfgs/waymo_models/second(20201130_second)

It seems that your data has some problems, the number of gt_database filter should be similar with the following:
 129 2020-11-24 15:55:00,019   INFO  Database filter by min points Vehicle: 477761 => 392931
 130 2020-11-24 15:55:00,064   INFO  Database filter by min points Pedestrian: 222675 => 175422
 131 2020-11-24 15:55:00,069   INFO  Database filter by min points Cyclist: 5344 => 4638
 132 2020-11-24 15:55:00,220   INFO  Database filter by difficulty Vehicle: 392931 => 392931
 133 2020-11-24 15:55:00,286   INFO  Database filter by difficulty Pedestrian: 175422 => 175422
 134 2020-11-24 15:55:00,288   INFO  Database filter by difficulty Cyclist: 4638 => 4638
 135 2020-11-24 15:55:00,670   INFO  Loading Waymo dataset
 136 2020-11-24 15:55:17,695   INFO  Total skipped info 0
 137 2020-11-24 15:55:17,696   INFO  Total samples for Waymo dataset: 158081
 138 2020-11-24 15:55:17,702   INFO  Total sampled samples for Waymo dataset: 31617

It may be caused by your enviroment of tensorflow and waymo_open_dataset.
Please try to re-create a new enviroment with the following command (You could change the version if you find there are some problems of your current version):
# tf 2.1.0.
pip install waymo-open-dataset-tf-2-1-0==1.2.0
# tf 2.0.0
# pip install waymo-open-dataset-tf-2-0-0==1.2.0
# tf 1.15.0
# pip install waymo-open-dataset-tf-1-15-0==1.2.0

It seems that your data has some problems, the number of gt_database filter should be similar with the following:
 129 2020-11-24 15:55:00,019   INFO  Database filter by min points Vehicle: 477761 => 392931
 130 2020-11-24 15:55:00,064   INFO  Database filter by min points Pedestrian: 222675 => 175422
 131 2020-11-24 15:55:00,069   INFO  Database filter by min points Cyclist: 5344 => 4638
 132 2020-11-24 15:55:00,220   INFO  Database filter by difficulty Vehicle: 392931 => 392931
 133 2020-11-24 15:55:00,286   INFO  Database filter by difficulty Pedestrian: 175422 => 175422
 134 2020-11-24 15:55:00,288   INFO  Database filter by difficulty Cyclist: 4638 => 4638
 135 2020-11-24 15:55:00,670   INFO  Loading Waymo dataset
 136 2020-11-24 15:55:17,695   INFO  Total skipped info 0
 137 2020-11-24 15:55:17,696   INFO  Total samples for Waymo dataset: 158081
 138 2020-11-24 15:55:17,702   INFO  Total sampled samples for Waymo dataset: 31617

It may be caused by your enviroment of tensorflow and waymo_open_dataset.
Please try to re-create a new enviroment with the following command (You could change the version if you find there are some problems of your current version):
# tf 2.1.0.
pip install waymo-open-dataset-tf-2-1-0==1.2.0
# tf 2.0.0
# pip install waymo-open-dataset-tf-2-0-0==1.2.0
# tf 1.15.0
# pip install waymo-open-dataset-tf-1-15-0==1.2.0

Yes, using tf 2.0.0 will cause wrong preprocessed results. Using tf2.1.1 seems to be right. I will re-process the waymo dataset and train the model.

Am I getting you right, that the instructions listed here are not correct and one has to use
pip install waymo-open-dataset-tf-2-1-0==1.2.0
instead of
pip3 install waymo-open-dataset-tf-2-0-0==1.2.0?

Not exactly.
Actually I tried to create a new environment and use the following for generating data:
pip3 install waymo-open-dataset-tf-2-0-0==1.2.0

and I could get the results correctly.
You could visualize some generated xxxx.npy files to see whether the point clouds are reasonable. If so, your environment should be fine.
I give a reference here, you could find the following file:
waymo_processed_data/segment-10017090168044687777_6380_000_6400_000_with_camera_labels/0000.npy

and load it with the following line:
A = np.load('0000.npy')
print(A.shape)
print(A.sum())

it should be (171606, 6) and 609530.0.
If these numbers are matched, your data should be fine.

@sshaoshuai
Thx! My numbers match with yours and I used
pip3 install waymo-open-dataset-tf-2-0-0==1.2.0 from the Getting Started page.
My waymo_processed_data is still getting copied to AWS since I had issues generating it there.
Once all necessary data is there, I will try to validate your results as mentioned before.

I followed the readme and tried to reproduce the results. I trained a Part-A^2-Anchor and tested it using the tools/test.py
Before the bugfix:
V 1 : AP / APH | 2 : AP / APH	P 1 : AP / APH | 2 : AP / APH	C 1 : AP / APH | 2 : AP / APH
0.6197 / 0.6150 | 0.5721 / 0.5677	0.5428 / 0.4706 | 0.4981 / 0.4312	0.5856 / 0.5751 | 0.5846 / 0.5741
After the bugfix:
0.7198 / 0.7144 | 0.6447 / 0.6396	0.6348 / 0.5508 | 0.5454 / 0.4722 0.6562 / 0.6444 | 0.6314 / 0.6201
which is very close to the reported result
71.82/71.29 | 64.33/63.82 | 63.15/54.96 | 54.24/47.11 | 65.23/63.92 | 62.61/61.35

Not exactly.
Actually I tried to create a new environment and use the following for generating data:
pip3 install waymo-open-dataset-tf-2-0-0==1.2.0

and I could get the results correctly.
You could visualize some generated xxxx.npy files to see whether the point clouds are reasonable. If so, your environment should be fine.
I give a reference here, you could find the following file:
waymo_processed_data/segment-10017090168044687777_6380_000_6400_000_with_camera_labels/0000.npy

and load it with the following line:
A = np.load('0000.npy')
print(A.shape)
print(A.sum())

The results should be (171606, 6) and 609530.0.
If these numbers are matched, your data should be fine.

I tried all versions of waymo-open-dataset, the A.sum()s are all different. But none of them equal to yours. Do you have any idea what might be the problem?

I followed the readme and tried to reproduce the results. I trained a Part-A^2-Anchor and tested it using the tools/test.py
Before the bugfix:
V 1 : AP / APH | 2 : AP / APH P 1 : AP / APH | 2 : AP / APH C 1 : AP / APH | 2 : AP / APH
0.6197 / 0.6150 | 0.5721 / 0.5677 0.5428 / 0.4706 | 0.4981 / 0.4312 0.5856 / 0.5751 | 0.5846 / 0.5741
After the bugfix:
0.7198 / 0.7144 | 0.6447 / 0.6396 0.6348 / 0.5508 | 0.5454 / 0.4722 0.6562 / 0.6444 | 0.6314 / 0.6201
which is very close to the reported result
71.82/71.29 | 64.33/63.82 | 63.15/54.96 | 54.24/47.11 | 65.23/63.92 | 62.61/61.35

Hi, do you mean this bugfix #384 ?
if I need to re-create the val info for the waymo dataset, whether I only need to keep the line333-349 in the function create_waymo_infos() of waymo_dataset.py? Because train info is too time-consuming, I want to be sure I will not modify it.
https://github.com/open-mmlab/OpenPCDet/blob/master/pcdet/datasets/waymo/waymo_dataset.py#L309
Thanks.

I followed the readme and tried to reproduce the results. I trained a Part-A^2-Anchor and tested it using the tools/test.py
Before the bugfix:
V 1 : AP / APH | 2 : AP / APH P 1 : AP / APH | 2 : AP / APH C 1 : AP / APH | 2 : AP / APH
0.6197 / 0.6150 | 0.5721 / 0.5677 0.5428 / 0.4706 | 0.4981 / 0.4312 0.5856 / 0.5751 | 0.5846 / 0.5741
After the bugfix:
0.7198 / 0.7144 | 0.6447 / 0.6396 0.6348 / 0.5508 | 0.5454 / 0.4722 0.6562 / 0.6444 | 0.6314 / 0.6201
which is very close to the reported result
71.82/71.29 | 64.33/63.82 | 63.15/54.96 | 54.24/47.11 | 65.23/63.92 | 62.61/61.35

Hi, do you mean this bugfix #384 ?
if I need to re-create the val info for the waymo dataset, whether I only need to keep the line333-349 in the function create_waymo_infos() of waymo_dataset.py? Because train info is too time-consuming, I want to be sure I will not modify it.
https://github.com/open-mmlab/OpenPCDet/blob/master/pcdet/datasets/waymo/waymo_dataset.py#L309
Thanks.

I want to confirm (most of) the numbers given in README.md.
Numbers provided by OpenPCDet on the full evaluation set:

Vec_L1
Vec_L2
Ped_L1
Ped_L2
Cyc_L1
Cyc_L2

SECOND
68.03/67.44
59.57/59.04
61.14/50.33
53.00/43.56
54.66/53.31
52.67/51.37

Part-A^2-Anchor
71.82/71.29
64.33/63.82
63.15/54.96
54.24/47.11
65.23/63.92
62.61/61.35

PV-RCNN
74.06/73.38
64.99/64.38
62.66/52.68
53.80/45.14
63.32/61.71
60.72/59.18

My training runs trained and evaluated on 1/5th of the data:

Vec_L1
Vec_L2
Ped_L1
Ped_L2
Cyc_L1
Cyc_L2

SECOND
67.85/67.24
59.35/58.80
57.94/47.85
49.93/41.18
55.54/54.12
53.75/52.38

Part-A^2-Anchor
71.57/71.03
64.08/63.57
63.34/55.08
54.39/47.20
64.93/63.71
62.53/61.36

PV-RCNN
74.36/73.68
65.27/64.67
63.29/53.04
54.36/45.46
64.02/62.07
61.53/59.66

These are the deltas:

Vec_L1
Vec_L2
Ped_L1
Ped_L2
Cyc_L1
Cyc_L2

SECOND
-0.18/-0.20
-0.22/-0.24
-3.20/-2.48
-3.07/-2.38
+0.88/+0.81
+1.08/+1.01

Part-A^2-Anchor
-0.25/-0.26
-0.25/-0.25
+0.19/+0.12
+0.15/+0.09
-0.30/-0.21
-0.08/+0.01

PV-RCNN
+0.30/+0.30
+0.28/+0.31
+0.63/+0.36
+0.56/+0.32
+0.70/+0.36
+0.81/+0.48

The evaluation on the full validation set is still ongoing, but the trend goes in the same direction.

Depicted are two PV-RCNN runs (Step = Epoch).
Grey: trained and evaluated on 1/5th of the data.
Red: trained on 1/5th of the data, evaluation (ongoing) on the full validation set.

I want to confirm (most of) the numbers given in README.md.
Numbers provided by OpenPCDet on the full evaluation set:

Vec_L1
Vec_L2
Ped_L1
Ped_L2
Cyc_L1
Cyc_L2

SECOND
68.03/67.44
59.57/59.04
61.14/50.33
53.00/43.56
54.66/53.31
52.67/51.37

Part-A^2-Anchor
71.82/71.29
64.33/63.82
63.15/54.96
54.24/47.11
65.23/63.92
62.61/61.35

PV-RCNN
74.06/73.38
64.99/64.38
62.66/52.68
53.80/45.14
63.32/61.71
60.72/59.18

My training runs trained and evaluated on 1/5th of the data:

Vec_L1
Vec_L2
Ped_L1
Ped_L2
Cyc_L1
Cyc_L2

SECOND
67.85/67.24
59.35/58.80
57.94/47.85
49.93/41.18
55.54/54.12
53.75/52.38

Part-A^2-Anchor
71.57/71.03
64.08/63.57
63.34/55.08
54.39/47.20
64.93/63.71
62.53/61.36

PV-RCNN
74.36/73.68
65.27/64.67
63.29/53.04
54.36/45.46
64.02/62.07
61.53/59.66

These are the deltas:

Vec_L1
Vec_L2
Ped_L1
Ped_L2
Cyc_L1
Cyc_L2

SECOND
-0.18/-0.20
-0.22/-0.24
-3.20/-2.48
-3.07/-2.38
+0.88/+0.81
+1.08/+1.01

Part-A^2-Anchor
-0.25/-0.26
-0.25/-0.25
+0.19/+0.12
+0.15/+0.09
-0.30/-0.21
-0.08/+0.01

PV-RCNN
+0.30/+0.30
+0.28/+0.31
+0.63/+0.36
+0.56/+0.32
+0.70/+0.36
+0.81/+0.48

The evaluation on the full validation set is still ongoing, but the trend goes in the same direction.

Depicted are two PV-RCNN runs (Step = Epoch).
Grey: trained and evaluated on 1/5th of the data.
Red: trained on 1/5th of the data, evaluation (ongoing) on the full validation set.

This is a comparison between an evaluation done on 1/5th of the validation set (in green)
and an evaluation done on the complete validation set (in blue and pink [continued])
of a Part-A^2-Anchor training run on 1/5th of the training set.
So, in the end, there is almost no difference between the two final numbers.

This is a comparison between an evaluation done on 1/5th of the validation set (in green)
and an evaluation done on the complete validation set (in blue and pink [continued])
of a Part-A^2-Anchor training run on 1/5th of the training set.
So, in the end, there is almost no difference between the two final numbers.

I want to confirm (most of) the numbers given in README.md.
Numbers provided by OpenPCDet on the full evaluation set:
Vec_L1	Vec_L2	Ped_L1	Ped_L2	Cyc_L1	Cyc_L2
SECOND	68.03/67.44	59.57/59.04	61.14/50.33	53.00/43.56	54.66/53.31	52.67/51.37
Part-A^2-Anchor	71.82/71.29	64.33/63.82	63.15/54.96	54.24/47.11	65.23/63.92	62.61/61.35
PV-RCNN	74.06/73.38	64.99/64.38	62.66/52.68	53.80/45.14	63.32/61.71	60.72/59.18
My training runs trained and evaluated on 1/5th of the data:
Vec_L1	Vec_L2	Ped_L1	Ped_L2	Cyc_L1	Cyc_L2
SECOND	67.85/67.24	59.35/58.80	57.94/47.85	49.93/41.18	55.54/54.12 53.75/52.38
Part-A^2-Anchor	71.57/71.03	64.08/63.57	63.34/55.08 54.39/47.20	64.93/63.71	62.53/61.36
PV-RCNN 74.36/73.68 65.27/64.67 63.29/53.04 54.36/45.46 64.02/62.07 61.53/59.66
These are the deltas:
Vec_L1	Vec_L2	Ped_L1	Ped_L2	Cyc_L1	Cyc_L2
SECOND	-0.18/-0.20	-0.22/-0.24	-3.20/-2.48	-3.07/-2.38	+0.88/+0.81 +1.08/+1.01
Part-A^2-Anchor	-0.25/-0.26	-0.25/-0.25	+0.19/+0.12 +0.15/+0.09	-0.30/-0.21	-0.08/+0.01
PV-RCNN +0.30/+0.30 +0.28/+0.31 +0.63/+0.36 +0.56/+0.32 +0.70/+0.36 +0.81/+0.48
The evaluation on the full validation set is still ongoing, but the trend goes in the same direction.

Depicted are two PV-RCNN runs (Step = Epoch).
Grey: trained and evaluated on 1/5th of the data.
Red: trained on 1/5th of the data, evaluation (ongoing) on the full validation set.
@MartinHahner
I trained the SECOND with the default configuration and test the model on the whole validation, and I got the similar results with yours. The numbers of PED L1/L2 are lower than the numbers given in README.md about 3%. Did you find out the problem? Thanks.

If you trained SECOND once and you observe this, I think it is within the "normal" variance of different training runs.
If you let's say trained SECOND five times, and every time it is by 3% lower than the reported numbers, then I don't know why.

I want to confirm (most of) the numbers given in README.md.
Numbers provided by OpenPCDet on the full evaluation set:
Vec_L1 	Vec_L2 	Ped_L1 	Ped_L2 	Cyc_L1 	Cyc_L2
SECOND 	68.03/67.44 	59.57/59.04 	61.14/50.33 	53.00/43.56 	54.66/53.31 	52.67/51.37
Part-A^2-Anchor 	71.82/71.29 	64.33/63.82 	63.15/54.96 	54.24/47.11 	65.23/63.92 	62.61/61.35
PV-RCNN 	74.06/73.38 	64.99/64.38 	62.66/52.68 	53.80/45.14 	63.32/61.71 	60.72/59.18
My training runs trained and evaluated on 1/5th of the data:
Vec_L1 	Vec_L2 	Ped_L1 	Ped_L2 	Cyc_L1 	Cyc_L2
SECOND 	67.85/67.24 	59.35/58.80 	57.94/47.85 	49.93/41.18 	55.54/54.12 53.75/52.38
Part-A^2-Anchor 	71.57/71.03 	64.08/63.57 	63.34/55.08 54.39/47.20 	64.93/63.71 	62.53/61.36
PV-RCNN 74.36/73.68 65.27/64.67 63.29/53.04 54.36/45.46 64.02/62.07 61.53/59.66
These are the deltas:
Vec_L1 	Vec_L2 	Ped_L1 	Ped_L2 	Cyc_L1 	Cyc_L2
SECOND 	-0.18/-0.20 	-0.22/-0.24 	-3.20/-2.48 	-3.07/-2.38 	+0.88/+0.81 +1.08/+1.01
Part-A^2-Anchor 	-0.25/-0.26 	-0.25/-0.25 	+0.19/+0.12 +0.15/+0.09 	-0.30/-0.21 	-0.08/+0.01
PV-RCNN +0.30/+0.30 +0.28/+0.31 +0.63/+0.36 +0.56/+0.32 +0.70/+0.36 +0.81/+0.48
The evaluation on the full validation set is still ongoing, but the trend goes in the same direction.

Depicted are two PV-RCNN runs (Step = Epoch).
Grey: trained and evaluated on 1/5th of the data.
Red: trained on 1/5th of the data, evaluation (ongoing) on the full validation set.

Thanks for sharing the reproduce results. How many GPUs are you using to reproduce the pv_rcnn results? Thanks.

I think I used 4 or 8 GPUs. This shouldn't matter though, it would only have an impact on the training time.

Thanks for your reply! Since I am trying to reproduce pv-rcnn results on 4 P6000 GPUs, but I am getting ~0.61 at epoch 24, which is way below what you get. I am wondering what the problem could be. Are you using the default evaluation code w/o anything modification? https://github.com/open-mmlab/OpenPCDet/blob/0642cf06d0fd84f50cc4c6c01ea28edbc72ea810/pcdet/datasets/waymo/waymo_eval.py#L239 indicates the dist_threshold to be 1000, so I am not sure whether you change it to be 30/50 to be consistent with the Waymo official metric. Thank you!

Nope, I did not change anything.

Thanks for your information!

training pv-rcnn on waymo dataset:
/home/xuan/code/pytorch/OpenPCDet/pcdet/datasets/augmentor/augmentor_utils.py:76: RuntimeWarning: overflow encountered in multiply03:41,  2.71s/it, total_it=89722]
points[:, :3] *= noise_scale
epochs:  90%|██████████████████████████████████████████████████████████████████████████████▎        | 27/30 [70:45:38<7:52:40, 9453.55s/it, loss=1.66, lr=0.000627]/home/xuan/code/pytorch/OpenPCDet/pcdet/datasets/augmentor/augmentor_utils.py:76: RuntimeWarning: overflow encountered in multiply19:00,  2.68s/it, total_it=93601]
points[:, :3] *= noise_scale
epochs:  93%|█████████████████████████████████████████████████████████████████████████████████▏     | 28/30 [73:22:38<5:15:11, 9455.94s/it, loss=1.56, lr=0.000274]/home/xuan/code/pytorch/OpenPCDet/pcdet/datasets/augmentor/augmentor_utils.py:76: RuntimeWarning: overflow encountered in multiply20:07,  2.69s/it, total_it=97040]
points[:, :3] *= noise_scale
epochs:  93%|█████████████████████████████████████████████████████████████████████████████████▏     | 28/30 [74:20:43<5:15:11, 9455.94s/it, loss=2.27, lr=0.000178]/home/xuan/code/pytorch/OpenPCDet/pcdet/datasets/augmentor/augmentor_utils.py:76: RuntimeWarning: overflow encountered in multiply21:23,  2.66s/it, total_it=98332]
points[:, :3] *= noise_scale
train: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 3454/3454 [2:35:10<00:00,  2.70s/it, total_it=103620]
epochs: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [78:20:16<00:00, 9400.56s/it, loss=2.79, lr=1e-7]
2021-03-30 17:18:14,818   INFO  End training waymo_models/pv_rcnn(default)
epoch 30:
2021-03-30 19:22:49,337   INFO  recall_roi_0.3: 0.613171
2021-03-30 19:22:49,337   INFO  recall_rcnn_0.3: 0.613844
2021-03-30 19:22:49,337   INFO  recall_roi_0.5: 0.550942
2021-03-30 19:22:49,337   INFO  recall_rcnn_0.5: 0.562655
2021-03-30 19:22:49,337   INFO  recall_roi_0.7: 0.308685
2021-03-30 19:22:49,337   INFO  recall_rcnn_0.7: 0.365962
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/AP: 0.5073
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/APH: 0.4977
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP: 0.4482
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH: 0.4395
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/AP: 0.4421
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/APH: 0.3412
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/AP: 0.3704
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/APH: 0.2857
OBJECT_TYPE_TYPE_SIGN_LEVEL_1/AP: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_1/APH: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_2/AP: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_2/APH: 0.0000
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/AP: 0.3261
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/APH: 0.3033
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/AP: 0.3154
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/APH: 0.2933

Hi, excuse me，waymo_processed_data_v0_5_0/segment-10017090168044687777_6380_000_6400_000_with_camera_labels/0000.npy generated by waymo-open-dataset-tf-2-5-0 in the latest version v0.5.2 is still the same as before?
However,  load it with the following line:
A = np.load('0000.npy')
print(A.shape)
print(A.sum())

I get the results are (183680, 6) and 586412.5. Is there something wrong with my generation?
Is this the reason why I can't fully reproduce the accuracy of PV_RCNN and other algorithms on Waymo dataset? Three or four points lower on average.
I hope you can get a reply and the problem can be solved. Thank you!
I get pv_rcnn :
2022-03-13 23:23:58,300   INFO  *************** EPOCH 30 EVALUATION *****************
2022-03-13 23:52:17,842   INFO  *************** Performance of EPOCH 30 *****************
2022-03-13 23:52:17,858   INFO  Generate label finished(sec_per_example: 0.0425 second).
2022-03-13 23:52:17,858   INFO  recall_roi_0.3: 0.804661
2022-03-13 23:52:17,858   INFO  recall_rcnn_0.3: 0.805084
2022-03-13 23:52:17,858   INFO  recall_roi_0.5: 0.741911
2022-03-13 23:52:17,858   INFO  recall_rcnn_0.5: 0.751780
2022-03-13 23:52:17,859   INFO  recall_roi_0.7: 0.486838
2022-03-13 23:52:17,859   INFO  recall_rcnn_0.7: 0.533652
2022-03-13 23:52:17,884   INFO  Average predicted number of objects(39987 samples): 88.996
2022-03-14 00:03:08,822   INFO
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/AP: 0.7297
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/APH: 0.7227
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP: 0.6441
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH: 0.6377
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/AP: 0.6967
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/APH: 0.5787
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/AP: 0.6053
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/APH: 0.5013
OBJECT_TYPE_TYPE_SIGN_LEVEL_1/AP: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_1/APH: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_2/AP: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_2/APH: 0.0000
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/AP: 0.6462
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/APH: 0.6244
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/AP: 0.6222
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/APH: 0.6013

pv_rcnn_center:
2022-03-19 12:57:09,953   INFO  **********************Start evaluation waymo_models/pv_rcnn_with_centerhead_rpn(default)**********************
2022-03-19 12:57:09,955   INFO  Loading Waymo dataset
2022-03-19 12:57:22,970   INFO  Total skipped info 0
2022-03-19 12:57:22,971   INFO  Total samples for Waymo dataset: 39987
2022-03-19 12:57:22,973   INFO  ==> Loading parameters from checkpoint ./OpenPCDet/output/waymo_models/pv_rcnn_with_centerhead_rpn/default/ckpt/checkpoint_epoch_30.pth to CPU
2022-03-19 12:57:23,098   INFO  ==> Checkpoint trained from version: pcdet+0.5.2+a9c66fe
2022-03-19 12:57:23,154   INFO  ==> Done (loaded 437/437)
2022-03-19 12:57:23,158   INFO  *************** EPOCH 30 EVALUATION *****************
2022-03-19 13:28:13,964   INFO  *************** Performance of EPOCH 30 *****************
2022-03-19 13:28:14,026   INFO  Generate label finished(sec_per_example: 0.0463 second).
2022-03-19 13:28:14,026   INFO  recall_roi_0.3: 0.834954
2022-03-19 13:28:14,026   INFO  recall_rcnn_0.3: 0.835461
2022-03-19 13:28:14,027   INFO  recall_roi_0.5: 0.762491
2022-03-19 13:28:14,027   INFO  recall_rcnn_0.5: 0.772247
2022-03-19 13:28:14,027   INFO  recall_roi_0.7: 0.473409
2022-03-19 13:28:14,027   INFO  recall_rcnn_0.7: 0.532824
2022-03-19 13:28:14,058   INFO  Average predicted number of objects(39987 samples): 86.808
2022-03-19 13:43:13,291   INFO
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/AP: 0.7393
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1/APH: 0.7338
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/AP: 0.6538
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2/APH: 0.6488
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/AP: 0.7248
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1/APH: 0.6550
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/AP: 0.6359
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2/APH: 0.5732
OBJECT_TYPE_TYPE_SIGN_LEVEL_1/AP: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_1/APH: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_2/AP: 0.0000
OBJECT_TYPE_TYPE_SIGN_LEVEL_2/APH: 0.0000
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/AP: 0.6900
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1/APH: 0.6762
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/AP: 0.6646
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2/APH: 0.6513

@r-sy ,
Hi, the data has been updated after version 0.5 and your current data state is consistent with me.
So you may check other reasons to match the performance.

Ok，thank you very much！---- Replied Message ----FromShaoshuai @.>Date03/19/2022 20:22 @.><EMAIL_ADDRESS>[open-mmlab/OpenPCDet] Low accuracy when training on waymo dataset (#370)
@r-sy ,
Hi, the data has been updated after version 0.5 and your current data state is consistent with me.
So you may check other reasons to match the performance.
—Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you were mentioned.Message ID: @.>
[
{
@.": "http://schema.org",
@.": "EmailMessage",
"potentialAction": {
@.": "ViewAction",
"target": "https://github.com/open-mmlab/OpenPCDet/issues/370#issuecomment-1073000369",
"url": "https://github.com/open-mmlab/OpenPCDet/issues/370#issuecomment-1073000369",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
@.***": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]

@r-sy , Hi, the data has been updated after version 0.5 and your current data state is consistent with me. So you may check other reasons to match the performance.
@sshaoshuai
Hi, I also use the new version 0.5 to generate the waymo dataset(waymo version 1.0) with waymo-open-dataset-tf-2-5-0-1.4.1. But I do not obtain the same result.
I think the problem is the numpy version. The waymo-open-dataset-tf-2-5-0 need numpy==1.19.5.  I statisfy this version, but occur error: "ImportError: numpy.core.multiarray failed to import". And this error comes from "import SharedArray" . Based on this error, I update numpy version to 1.20, this error does not occur.  After I trained Voxel-RCNN and val, another error occur:"can not convert a symbolic Tensor(strided_slice:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a Numpy call. which is not supported" . I return to the origninal numpy version==1.19.5 and try to delete these code about "SharedArray", the code could run but all the result are zero.(Start the waymo evaluation... Number:(pd, 1322657)VS. (gt,0)) also generates. Can you give me some advice on these problems? Thank u very much!!!

@sshaoshuai
Hi, I also use the new version 0.5 to generate the waymo dataset(waymo version 1.0) with waymo-open-dataset-tf-2-5-0-1.4.1. But I do not obtain the same result.
I think the problem is the numpy version. The waymo-open-dataset-tf-2-5-0 need numpy==1.19.5.  I statisfy this version, but occur error: "ImportError: numpy.core.multiarray failed to import". And this error comes from "import SharedArray" . Based on this error, I update numpy version to 1.20, this error does not occur.  After I trained Voxel-RCNN and val, another error occur:"can not convert a symbolic Tensor(strided_slice:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a Numpy call. which is not supported" . I return to the origninal numpy version==1.19.5 and try to delete these code about "SharedArray", the code could run but all the result are zero.(Start the waymo evaluation... Number:(pd, 1322657)VS. (gt,0)) also generates. Can you give me some advice on these problems? Thank u very much!!!

@r-sy Hi, I also use waymo-open-dataset-tf-2-5-0 and the numpy version is 1.19.5. But I do not run code nomally and generate this error:"ImportError: numpy.core.multiarray failed to import". Can u tell me your numpy version or other suggestions? Thank u very much!!!

@r-sy Hi, I also use waymo-open-dataset-tf-2-5-0 and the numpy version is 1.19.5. But I do not run code nomally and generate this error:"ImportError: numpy.core.multiarray failed to import". Can u tell me your numpy version or other suggestions? Thank u very much!!!

You can try to install SharedArray==3.1.0.

Thanks!!! It works well!
------------------ 原始邮件 ------------------
发件人:                                                                                "open-mmlab/OpenPCDet"                                                                                @.>;
发送时间: 2022年5月6日(星期五) 凌晨0:19
@.>;
<EMAIL_ADDRESS>主题: Re: [open-mmlab/OpenPCDet] Low accuracy when training on waymo dataset (#370)
@r-sy Hi, I also use waymo-open-dataset-tf-2-5-0 and the numpy version is 1.19.5. But I do not run code nomally and generate this error:"ImportError: numpy.core.multiarray failed to import". Can u tell me your numpy version or other suggestions? Thank u very much!!!
You can try to install SharedArray==3.1.0.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: @.***>

Hi, I also have this problem when I use waymo v1.0 for object detection(Start the waymo evaluation... Number:(pd, 1322657)VS. (gt,0))，i also use the environment，Can you tell me how you finally solved it?Thank u very much!!!

@r-sy Hi, I also use waymo-open-dataset-tf-2-5-0 and the numpy version is 1.19.5. But I do not run code nomally and generate this error:"ImportError: numpy.core.multiarray failed to import". Can u tell me your numpy version or other suggestions? Thank u very much!!!

You can try to install SharedArray==3.1.0.

I have the same issue with jwma0725, and install SharedArray==3.1.0 work!!!!!!!


Upgrade codemirror

Includes #929 and #938
Needed to make a new PR because the history got all weird after #942.
I also snuck in a fix for #939.
As far as styling goes, it looks like yaml files look a bit different. The newer version of CodeMirror is applying a class that it didn't previously, which is affecting the custom theme we're using. The other languages I looked at seemed to match up, so I'll try to find time to look at this but don't consider it to be a show-stopper.

Old on left, new on right

I've also updated our test libraries and tied test libraries with the ones in node_modules instead of hard-coded ones. Also some clean-up of test code. Merging this as soon as it passes CI tests.


Add system dell/xps/15-9500

Added dell xps 15-9500, with Intel and Nvidia prime configurations. Not sure this is the correct way to provide a default thermald-conf for this, but the chip isn't currently supported by default.

Thanks for all the help @Mic92!

@bahaynes do you know what kernel version you were using with Nvidia? I was having some trouble with 20.09 on this, perhaps because of version issues:

Linux 5.4 might just be too old??
Linux 5.9 was not working because https://bugs.gentoo.org/753629 prevented the module from loading properly.


Libgdx app not running on pi4

Please ensure you have given all the following requested information in your report.
Issue details
I am trying to run my libgdx app that I made on my rpi4. It used to run fine until suddenly it didn't. The jar works fine on my Windows PC and as stated before, it used to run on the pi. But then it started crashing before even opening the window. No code was changed and I created a new fresh project template and that also did not run.
Version of libGDX and/or relevant dependencies
v1.10.1 SNAPSHOT (and other versions supported with lwjgl3
Stacktrace
Exception in thread "main" java.lang.NullPointerException
	at org.lwjgl.system.Checks.check(Checks.java:188)
	at org.lwjgl.glfw.GLFW.nglfwGetMonitorWorkarea(GLFW.java:1432)
	at org.lwjgl.glfw.GLFW.glfwGetMonitorWorkarea(GLFW.java:1463)
	at com.badlogic.gdx.backends.lwjgl3.Lwjgl3Application.createGlfwWindow(Lwjgl3Application.java:547)
	at com.badlogic.gdx.backends.lwjgl3.Lwjgl3Application.createWindow(Lwjgl3Application.java:462)
	at com.badlogic.gdx.backends.lwjgl3.Lwjgl3Application.createWindow(Lwjgl3Application.java:448)
	at com.badlogic.gdx.backends.lwjgl3.Lwjgl3Application.<init>(Lwjgl3Application.java:162)
	at pibot.lib.desktop.DesktopLauncher.main(DesktopLauncher.java:18)

Please select the affected platforms

[x] Linux

It seems like glfwGetPrimaryMonitor is returning NULL somehow.
I'm not aware of any changes that would have happened within lwjgl3 or glfw to cause this.


Merge lora weights

I have:
LoRA weights stored at "lit_model_lora_finetuned.pth"
Base model stored at "checkpoints/meta-llama/Llama-2-7b-chat-hf"
Could you please help explain how to merge the lora weights with the base model to produce the desired combined model? Thank you very much for your help!

@zl-liu There is currently no such utility. One could be made though, it's relatively easy. You would basically run everything up to this line here: https://github.com/Lightning-AI/lit-gpt/blob/fc4c14d89e62799eeae162548b6aec24f1ab901a/generate/lora.py#L114C1-L114C1
and then torch.save(filepath, model.state_dict()) to save the merged weights.
Here is a sketch:
model = GPT(config)
with lazy_load(checkpoint_path) as checkpoint, lazy_load(lora_path) as lora_checkpoint:
    checkpoint.update(lora_checkpoint.get("model", lora_checkpoint))
    model.load_state_dict(checkpoint)
merge_lora_weights(model)

# drop lora params from checkpoint
state_dict = {k: v for k, v in model.state_dict() if "lora" not in key}
# save the merged checkpoint
torch.save("lora_merged_weights", state_dict)

@awaelchli Thank you! I will try to do it based on your suggestions. If it works I will share the code here.

@zl-liu I actually made this PR to lit-llama awhile ago, I just spotted this repo and will focus on contributing here instead now. You can check it out: https://github.com/Lightning-AI/lit-llama/pull/310

The code was a sketch. You can just add the missing .items() when iterating over the state dict :)

Added in #495


Hello！How to see the inference speed of the model in UI?

Like:  xxx tokens / sec.

We currently only save that in the history.json when using --save_dir=foo for some folder foo.  The code could be modified to show in UI (we used to do for summarize), but now for summarize it's under verbose flag only: https://github.com/h2oai/h2ogpt/blob/858664b4f3e6e1343da95b5be77525ba01ea903d/src/gpt_langchain.py#L3169
I don't plan to add to UI myself anytime soon.

Thanks!


Add TransformTimeSeriesGenerator

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (942c674) to head (a78a5bf).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #449   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files          131       132    +1
  Lines         3973      3991   +18
  Branches       277       277
=========================================
+ Hits          3973      3991   +18

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Add Philips Hue action type

This adds Philips hue support using the phue library.
It would be nice to configure the hue bridge in the global configuration file, but I wasn't able to find a nice way to pass this into the actions.py class. As such I've forced the user to set it for each actor request.
Example use where the hostname for my Hue bridge is philips-hue.
actor.add_keyword(_('change to purple'), ChangeLightColour(say, "philips-hue", "Lounge Lamp", "FFFFbe"))

I signed it!

Thanks. Please squash the changes and make sure travis is happy (follow Details link).

Squashed and fixed travis. Also use if bridge: as suggested.

Thanks for the PR, @bazwilliams! If you can use rgbxy from PyPI, it's just some style changes required.

@drigz What style changes do I need to make?
I tried the rgbxy lib from PyPI as I'd rather install the dependency, but it failed to install. I'll have another look tonight to see what I can do.

You should see the comments on the GitHub pull request.
Also, now #48 is merged, you can add phue and rgbxy to requirements.txt. If rgbxy is broken, maybe you could get in touch with the author? If they're not responding, you'll have to fork their project. Unfortunately, we can't bring the code into our repo.

Thanks for clarifying and will get the requirements.txt file updated, worst case I'll fork publish a new pypi lib for rgbxy.

FYI, I missed clicking Submit on the review, but I have now.

Codecov Report

Merging #33 into master will decrease coverage by 23.43%.
The diff coverage is 25%.

@@             Coverage Diff             @@
##           master      #33       +/-   ##
===========================================
- Coverage    79.2%   55.77%   -23.44%
===========================================
  Files           5        6        +1
  Lines         202      355      +153
===========================================
+ Hits          160      198       +38
- Misses         42      157      +115

Impacted Files
Coverage Δ

src/action.py
50.43% <25%> (-6.71%)
:arrow_down:

src/triggers/__init__.py
24.8% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4169573...d2d87fb. Read the comment docs.

I added a test to improve the coverage, but it hasn't been recognised by codecov. Any idea why not?

@ensonic you were right! The coverage never ran due to that pep8 warning that was added by mistake. Fixing that and including the suggested .coverage file shows the coverage I was expecting!
Thanks for helping out! I've rebased the PR.

Thanks!


Why I cannot save model?

raise RuntimeError(
RuntimeError:
Some tensors share memory, this will lead to duplicate memory on disk and potential differences when loading them again: [{'0.auto_model.shared.weight', '0.auto_model.encoder.embed_tokens.weight'}].
A potential way to correctly save your model is to use save_model.
More information at https://huggingface.co/docs/safetensors/torch_shared_tensors

RuntimeError:
Some tensors share memory, this will lead to duplicate memory on disk and potential differences when loading them again: [{'0.auto_model.encoder.embed_tokens.weight', '0.auto_model.shared.weight'}].
A potential way to correctly save your model is to use save_model.
More information at https://huggingface.co/docs/safetensors/torch_shared_tensors
       Could you please help me to resolve this?

Hi, Thanks a lot for your interest in the INSTRUCTOR model!
Could you provide a short script for me to reproduce the error?

I am getting the same error! I don't know how to solve this @hongjin-su I hope you would help me in this:
Traceback (most recent call last):
File "/ClusterLLM/perspective/2_finetune/finetune.py", line 617, in
main()
File "ClusterLLM/perspective/2_finetune/finetune.py", line 598, in main
train_result = trainer.train(resume_from_checkpoint=checkpoint)
File ".conda/envs/696ds/lib/python3.9/site-packages/transformers/trainer.py", line 1624, in train
return inner_training_loop(
File ".conda/envs/696ds/lib/python3.9/site-packages/transformers/trainer.py", line 2029, in _inner_training_loop
self._maybe_log_save_evaluate(tr_loss, grad_norm, model, trial, epoch, ignore_keys_for_eval)
File ".conda/envs/696ds/lib/python3.9/site-packages/transformers/trainer.py", line 2423, in _maybe_log_save_evaluate
self._save_checkpoint(model, trial, metrics=metrics)
File "conda/envs/696ds/lib/python3.9/site-packages/transformers/trainer.py", line 2499, in _save_checkpoint
self.save_model(staging_output_dir, _internal_call=True)
File ".conda/envs/696ds/lib/python3.9/site-packages/transformers/trainer.py", line 3016, in save_model
self._save(output_dir)
File ".conda/envs/696ds/lib/python3.9/site-packages/transformers/trainer.py", line 3083, in _save
safetensors.torch.save_file(
File ".conda/envs/696ds/lib/python3.9/site-packages/safetensors/torch.py", line 281, in save_file
serialize_file(_flatten(tensors), filename, metadata=metadata)
File ".conda/envs/696ds/lib/python3.9/site-packages/safetensors/torch.py", line 477, in _flatten
raise RuntimeError(
RuntimeError:
Some tensors share memory, this will lead to duplicate memory on disk and potential differences when loading them again: [{'0.auto_model.shared.weight', '0.auto_model.encoder.embed_tokens.weight'}].
A potential way to correctly save your model is to use save_model.
More information at https://huggingface.co/docs/safetensors/torch_shared_tensors

I have same issue
'''
File "/home/gaya/group2/instructor-embedding-1/train.py", line 586, in main
trainer.train(resume_from_checkpoint=checkpoint)
File "/home/gaya/anaconda3/envs/group2-re/lib/python3.10/site-packages/transformers/trainer.py", line 2123, in train
return inner_training_loop(
File "/home/gaya/anaconda3/envs/group2-re/lib/python3.10/site-packages/transformers/trainer.py", line 2548, in _inner_training_loop
self._maybe_log_save_evaluate(tr_loss, grad_norm, model, trial, epoch, ignore_keys_for_eval)
File "/home/gaya/anaconda3/envs/group2-re/lib/python3.10/site-packages/transformers/trainer.py", line 3007, in _maybe_log_save_evaluate
self._save_checkpoint(model, trial, metrics=metrics)
File "/home/gaya/anaconda3/envs/group2-re/lib/python3.10/site-packages/transformers/trainer.py", line 3097, in _save_checkpoint
self.save_model(output_dir, _internal_call=True)
File "/home/gaya/anaconda3/envs/group2-re/lib/python3.10/site-packages/transformers/trainer.py", line 3730, in save_model
self._save(output_dir)
File "/home/gaya/anaconda3/envs/group2-re/lib/python3.10/site-packages/transformers/trainer.py", line 3828, in _save
safetensors.torch.save_file(
File "/home/gaya/anaconda3/envs/group2-re/lib/python3.10/site-packages/safetensors/torch.py", line 286, in save_file
serialize_file(_flatten(tensors), filename, metadata=metadata)
File "/home/gaya/anaconda3/envs/group2-re/lib/python3.10/site-packages/safetensors/torch.py", line 488, in _flatten
raise RuntimeError(
RuntimeError:
Some tensors share memory, this will lead to duplicate memory on disk and potential differences when loading them again: [{'0.auto_model.shared.weight', '0.auto_model.encoder.embed_tokens.weight'}].
A potential way to correctly save your model is to use save_model.
More information at https://huggingface.co/docs/safetensors/torch_shared_tensors
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 107/107 [01:25<00:00,  1.26it/s]
(group2-re) (base) gaya@aicoss-PowerEdge-T640:~/group2/instructor-embedding-1$ CUDA_VISIBLE_DEVICES=0 python3 train.py --model_name_or_path sentence-transformers/gtr-t5-base --output_dir output --cache_dir cache --max_source_length 512 --num_train_epochs 1 --save_steps 500 --cl_temperature 0.1 --warmup_ratio 0.1 --learning_rate 2e-5 --overwrite_output_dir
one batch in task 303 is skipped
one batch in task 304 is skipped
one batch in task 305 is skipped
one batch in task 306 is skipped
one batch in task 307 is skipped
one batch in task 309 is skipped
one batch in task 310 is skipped
one batch in task 311 is skipped
one batch in task 312 is skipped
one batch in task 313 is skipped
one batch in task 314 is skipped
one batch in task 315 is skipped
one batch in task 316 is skipped
one batch in task 317 is skipped
one batch in task 318 is skipped
one batch in task 319 is skipped
one batch in task 320 is skipped
one batch in task 322 is skipped
one batch in task 323 is skipped
one batch in task 324 is skipped
one batch in task 325 is skipped
one batch in task 326 is skipped
one batch in task 328 is skipped
one batch in task 329 is skipped
one batch in task 1 is skipped
There are 856 pairs to train in total.
real_name_or_path:sentence-transformers/gtr-t5-base, model_args.cache_dir: cache
Using Hugging Face!!
Running tokenizer on train dataset: 100%|████████████████████████████████████████████████████████████████| 856/856 [00:02<00:00, 413.62 examples/s]
/home/gaya/group2/instructor-embedding-1/train.py:571: FutureWarning: tokenizer is deprecated and will be removed in version 5.0.0 for InstructorTrainer.__init__. Use processing_class instead.
trainer = InstructorTrainer(
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 107/107 [01:18<00:00,  1.43it/s]Traceback (most recent call last):
File "/home/gaya/group2/instructor-embedding-1/train.py", line 602, in
main()
File "/home/gaya/group2/instructor-embedding-1/train.py", line 586, in main
trainer.train(resume_from_checkpoint=checkpoint)
File "/home/gaya/anaconda3/envs/group2-re/lib/python3.10/site-packages/transformers/trainer.py", line 2123, in train
return inner_training_loop(
File "/home/gaya/anaconda3/envs/group2-re/lib/python3.10/site-packages/transformers/trainer.py", line 2548, in _inner_training_loop
self._maybe_log_save_evaluate(tr_loss, grad_norm, model, trial, epoch, ignore_keys_for_eval)
File "/home/gaya/anaconda3/envs/group2-re/lib/python3.10/site-packages/transformers/trainer.py", line 3007, in _maybe_log_save_evaluate
self._save_checkpoint(model, trial, metrics=metrics)
File "/home/gaya/anaconda3/envs/group2-re/lib/python3.10/site-packages/transformers/trainer.py", line 3097, in _save_checkpoint
self.save_model(output_dir, _internal_call=True)
File "/home/gaya/anaconda3/envs/group2-re/lib/python3.10/site-packages/transformers/trainer.py", line 3730, in save_model
self._save(output_dir)
File "/home/gaya/anaconda3/envs/group2-re/lib/python3.10/site-packages/transformers/trainer.py", line 3828, in _save
safetensors.torch.save_file(
File "/home/gaya/anaconda3/envs/group2-re/lib/python3.10/site-packages/safetensors/torch.py", line 286, in save_file
serialize_file(_flatten(tensors), filename, metadata=metadata)
File "/home/gaya/anaconda3/envs/group2-re/lib/python3.10/site-packages/safetensors/torch.py", line 488, in _flatten
raise RuntimeError(
RuntimeError:
Some tensors share memory, this will lead to duplicate memory on disk and potential differences when loading them again: [{'0.auto_model.shared.weight', '0.auto_model.encoder.embed_tokens.weight'}].
A potential way to correctly save your model is to use save_model.
More information at https://huggingface.co/docs/safetensors/torch_shared_tensors
'''


go/workerpool: support backoff on failures

Adds support for backoff on failures to the workerpool module.
Use the backoff in the storage sync protocol to fix the case where currently if storage sync requests start failing (e.g. due to a network error), the storage worker will crazily retry and fill up logs / cpu.
TODO:

[x] minor todo's in code
[x] add some unit tests to the workerpool module
[x] test that this fixes the mentioned storage sync issue

Before fix (with an artificial instant failure):
cat node.log | grep "error calling getdiff" | grep "11:22:03" | wc -l
18202

after:
cat node.log | grep "error calling getdiff" | grep "11:50:5" | wc -l
23

From ~18k calls per second, to ~2 per second.

Fixed in: https://github.com/oasisprotocol/oasis-core/pull/5341


Build pipeline

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: bartmalagon sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.


Ijemmy/add integ tests recommendations

Issue #, if available: N/A
Description of changes:
Adds integration tests to the recommendations service (for local devevelopment)
This service contains feature that depends on AWS Personalize campaigns. By default, we do not have this setup when developing locally.  (Building the campaigns will take 1-2 hours). So the test scope includes only  features that works by default, without campaigns
Description of testing performed to validate your changes (required if pull request includes CloudFormation or source code changes):
This change does not touch source code. It only adds integration tests to the recommendations service. You can run them by going to recommendations/tests/integ and run
# In terminal #1
docker-compose up --build swagger-ui recommendations products xray offers

# In terminal #2
RECOMMENDATIONS_API_URL="http://localhost:8005" ptw test_*.py

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

We'll wait until the product API test has been merged first. Then we'll create this PR on the top of that.


TST: add osx to travis matrix

Attempt to add OSX entry to travis test matrix.

Thanks for working on this @andyfaff!
If you have to iterate a bunch of times on this, probably best to comment out all the other build matrix entries.

I've been working on this on my own fork with several iterations already.
This one should build and tests pass. However, the log seems really big.
Not sure if it's line #169, change from |& to 2>&1 |.
However, when I try to build and test master on my own Mac I see the
failure in #7590.
On 15 July 2017 at 13:51, Ralf Gommers<EMAIL_ADDRESS>wrote:

Thanks for working on this @andyfaff https://github.com/andyfaff!
If you have to iterate a bunch of times on this, probably best to comment
out all the other build matrix entries.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/scipy/scipy/pull/7607#issuecomment-315507166, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAq51m-mcGuHtz5IQn1oz1G-DptVN3ngks5sODdEgaJpZM4OY57W
.

--

Dr. Andrew Nelson

Looks like it'll pass. Not sure if cloning from https://github.com/matthew-brett/multibuild.git is the best place to get multibuild from. Should I be using https://github.com/matthew-brett/multibuild/tree/e6ebbfa42588c26579eff99c50e84326dd9f6b3e ? That's the commit numpy-wheels seems to use. There don't appear to be any releases of multibuild. @matthew-brett

Not sure if cloning from https://github.com/matthew-brett/multibuild.git is the best place to get multibuild from.

I saw an answer from @matthew-brett on another PR somewhere - master branch of multibuild should be OK.

So this should be good to go then? The only thing I thought of was to split the build steps into separate linux and osx scripts, but I prefer to see everything in one place.

Yep, just reviewing the build logs, looking if I can hit the green button

All LGTM, merging. Thanks @andyfaff !

However, the log seems really big. Not sure if it's line 169, change from |& to 2>&1 |.

The log looks fine to me - it's usually over 10k lines so you have to look at the raw log.

Was USE_SDIST=1 set on purpose for the osx build?
It spams the travis log quite a lot.

Note also that |& is synonym for 2>&1 |

Finally, is there a way to get ccache enabled also on the osx side for clang? If it's enabled, it will greatly speed up the build due to the various c++ modules...

Note also that |& is shorthand for 2>&1 |

I had to change that because the shorthand doesn't work on OSX. Had to have several tries to do that one.

Was USE_SDIST=1 set on purpose for the osx build?

I used it to start with to see how the build was happening on travis, but I guess it's not necessary, because the build will still happen in the runtests.py. Feel free to remove.

ok thanks, |& may be a bash-ism. gh-7640 tries to add ccache...

Apparently caching is possible. I didn't do that, I was just happy to get the thing working.
I can start looking into this if it's going to speed things up?

I have a the above caching PR open, and we can hope it works easily...

I think |& is a blow in from zsh, or something like that.


Doesn't pass the last test

Challenge Factorialize a Number has an issue.
User Agent is: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36.
Please describe how to reproduce this issue, and include links to screenshots if possible.
My code:

function factorialize(num) {
  var numArray = [];
  for(var i=0;i<num;i++){
    numArray[i]=i+1;
  }

  return numArray.reduce(function(previousValue,currentValue){
    return previousValue*currentValue;
  });
}

factorialize(5);

This doesn't pass the last test. I understand why, but why is the last test expecting a value of 1? If 0 is factorialized it should be 0 should it not?

0 factorial is 1. So what last test is expecting is true value; not some mistake or bug.

@JCollison thanks for the issue. As @Jaskaranbir has pointed out, 0! is equal to 1. At some point in the curriculum, we want students to develop some critical thinking skills along the way so that is why we didn't explicitly tell you in the instructions.
A quick Google search may land you at this page https://www.zero-factorial.com/whatis.html where it shows you 0! equal to 1.
Also, here's a fun video on what zero factorial is https://www.youtube.com/watch?v=Mfk_L4Nx2ZI.
There's no issue with the challenge so I'll be closing this issue. Happy coding!


ICU: Upgrade to version 68.1 on non-Darwin platforms

Linux is currently using ICU 65.1, upgrade to the latest release version.
Any regressions are usually caught by the tests in swift-corelibs-foundation in the *Formatter classes.

@swift-ci test

cc @compnerd @drexin @shahmishal

Is there something specific prompting the update?  I kinda worry about the OSS version being too far apart from the Darwin behaviour.


Slicer crashes at specific planes

Hi!
I'm creating various 2D planes out of a render by slicing a scene and I noticed a bizarre behavior.
Slicing the scene at very specific planes makes the software stall. It does not raise any error but it never completes running.
After trying in a notebook it makes the kernel die.
after a bit of trial and error, I narrowed it down to an example code that on my machine reproduces this behavior consistently. Specifically, it does not like the plane 11147 but it does like 11155, a few micrometers behind. This is just an example, but it happens randomly with very few planes, but it is still annoying when slicing a set of planes obtained with np.linspace since you never know whether the planes you want to slice will work or not.
I solved it by just changing the plane a bit, but I figured maybe it's worth investigating since it does not even throw an error.
import bgheatmaps as bgh
from bgheatmaps.slicer import Slicer

heatmap = bgh.heatmap(
    {'VISp':10,'fiber tracts':5},      # Dummy data
    position=5000,
    orientation="frontal",
    thickness=100,
    title="bugReport",
    cmap='Blues',
    vmin=0,
    vmax=10,
    format="2D",
)
heatmap.scene.close()

blessedPlane = 11155
cursedPlane = 11147

print(f"Slicing blessed plane...")
s = Slicer(blessedPlane, "frontal", 100, heatmap.scene.root)
projected, _ = s.get_structures_slice_coords(
    heatmap.regions_meshes, heatmap.scene.root
)
print("Done")

print(f"Slicing cursed plane...")
s = Slicer(cursedPlane, "frontal", 100, heatmap.scene.root)
projected, _ = s.get_structures_slice_coords(
    heatmap.regions_meshes, heatmap.scene.root
)
print("Done")

I'm on:

Vedo 2021.0.5
Brainrender <IP_ADDRESS>
bgheatmap 0.0.3

In the previous code, the structure 'fiber tracts' seems to be the one responsible for this behavior.
I also sliced my collection of areas (about 250 areas) and, if I specifically remove fiber tracts from the structures set I don't get this behavior anymore. This of course is not a comprehensive test, since I didn't test all the possible planes, but it seems that this is the structure that is causing the most problems.
I could not notice anything in particular about these planes, but by debugging the script I noticed that it stalls not at the instantiation of the Slicer object, but when the function get_structures_slice_coords() is called, so it must be something inside that.

I think these issues are relevant:
https://github.com/marcomusy/vedo/issues/696
https://discourse.vtk.org/t/vtkintersectionpolydatafilter-crashing/9428

I managed to make it work reliably by not using bg-heatmaps.Slicer and backporting a binding function to vtx from vedo 2023.4.6.
Do you want me to open a PR so that we can continue discussing it over there?

Hi @LeonardoLupori, I think this should be fixed now following #21. The slightly updated example below doesn't cause any errors for me. Would you mind testing main sometime? A new release will be out soon.
import brainglobe_heatmap as bgh
from brainglobe_heatmap.slicer import Slicer

heatmap = bgh.heatmap(
    {'VISp':10,'fiber tracts':5},      # Dummy data
    position=5000,
    orientation="frontal",
    thickness=100,
    title="bugReport",
    cmap='Blues',
    vmin=0,
    vmax=10,
    format="2D",
)

blessedPlane = 11155
cursedPlane = 11147

print(f"Slicing blessed plane...")
s = Slicer(blessedPlane, "frontal", 100, heatmap.scene.root)
projected, _ = s.get_structures_slice_coords(
    heatmap.regions_meshes, heatmap.scene.root
)
print("Done")

print(f"Slicing cursed plane...")
s = Slicer(cursedPlane, "frontal", 100, heatmap.scene.root)
projected, _ = s.get_structures_slice_coords(
    heatmap.regions_meshes, heatmap.scene.root
)
print("Done")


Add hplip and enable remote printing

Enable printer sharing and allow printing from internet in Server settings by default
Add a startup message
Add hplip package which provides support for numerous popular hp printers

closes #1

According to the documentation :

The Listen directive is similar to the Port directive but allows you to restrict access to specific interfaces or networks

Let me change it back actually. I think using different directives together can cause issues.

I can confirm it works with iOS, macOS, Android and Windows

@willswire would you happen to know if the order of the Listen directives matter?
using
Listen 80
Listen 631

on Android in detects the printer but cannot get it's information or send print job while swapping to
Listen 631
Listen 80

works immediately. I'm not sure if this is related to my device or the case for all

@rahul-thakoor not sure, although I would say let's just go with the second ordering as it works.  Also, I am getting a merge conflict for the Dockerfile - could you reformat and commit that file so I can merge this PR online?

I'm keeping the original ordering since it breaks other platforms.
Let's see if others experience this issue with Android devices then we can tackle it. Thanks a lot Will

Thank you for the changes!


ZHA Log Errors in 0.50.2

I added the ZHA component while on 0.47.1 and was receiving numerous log errors. After researching issues and forum postings it appeared the errors were going to fixed in release 0.50 with an update to bellows.  I updated to 0.50.2 this week and I am still receiving the errors below. They seem to occur within 10 minutes with multiple errors occurring within a minute.
I currently only have one zigbee device attached at the moment - MCT-340E door/window sensor. I should mention that the sensor appears to be working correctly and reporting open/close and temp, it's just that I receive numerous errors in the log.
HUSBZB-1 Stick
HA Release 0.50.2
python 3.4.2
component ZHA
expected - no errors in log
2017-08-04 09:46:16 WARNING (MainThread) [bellows.zigbee.application] Unexpected message send notification
2017-08-04 09:46:17 WARNING (MainThread) [bellows.zigbee.zcl] Data remains after deserializing ZCL frame
2017-08-04 09:46:17 WARNING (MainThread) [bellows.zigbee.endpoint] [0xbe0e:1] Message on unknown cluster 0x0019
2017-08-04 09:46:20 WARNING (MainThread) [bellows.zigbee.zcl] Data remains after deserializing ZCL frame
2017-08-04 09:46:20 WARNING (MainThread) [bellows.zigbee.endpoint] [0xbe0e:1] Message on unknown cluster 0x0019
2017-08-04 09:46:23 WARNING (MainThread) [bellows.zigbee.zcl] Data remains after deserializing ZCL frame
2017-08-04 09:46:23 WARNING (MainThread) [bellows.zigbee.endpoint] [0xbe0e:1] Message on unknown cluster 0x0019
2017-08-04 09:46:26 WARNING (MainThread) [bellows.zigbee.zcl] Data remains after deserializing ZCL frame
2017-08-04 09:46:26 WARNING (MainThread) [bellows.zigbee.endpoint] [0xbe0e:1] Message on unknown cluster 0x0019
2017-08-04 09:46:29 WARNING (MainThread) [bellows.zigbee.zcl] Data remains after deserializing ZCL frame
2017-08-04 09:46:29 WARNING (MainThread) [bellows.zigbee.endpoint] [0xbe0e:1] Message on unknown cluster 0x0019
2017-08-04 09:56:13 WARNING (MainThread) [bellows.zigbee.application] Unexpected message send notification
2017-08-04 09:56:15 WARNING (MainThread) [bellows.zigbee.zcl] Data remains after deserializing ZCL frame
2017-08-04 09:56:15 WARNING (MainThread) [bellows.zigbee.endpoint] [0xbe0e:1] Message on unknown cluster 0x0019
2017-08-04 09:56:18 WARNING (MainThread) [bellows.zigbee.zcl] Data remains after deserializing ZCL frame
2017-08-04 09:56:18 WARNING (MainThread) [bellows.zigbee.endpoint] [0xbe0e:1] Message on unknown cluster 0x0019
2017-08-04 09:56:21 WARNING (MainThread) [bellows.zigbee.zcl] Data remains after deserializing ZCL frame
2017-08-04 09:56:21 WARNING (MainThread) [bellows.zigbee.endpoint] [0xbe0e:1] Message on unknown cluster 0x0019
2017-08-04 09:56:24 WARNING (MainThread) [bellows.zigbee.zcl] Data remains after deserializing ZCL frame
2017-08-04 09:56:24 WARNING (MainThread) [bellows.zigbee.endpoint] [0xbe0e:1] Message on unknown cluster 0x0019

OK!
Figured it out. The Nortek USB Zwave/Zigbee stick is working now.
zwave:
usb_path: /dev/ttyUSB0
zha:
usb_path: /dev/ttyUSB1
database_path: zigbee.db
It would not find the db using the full path. This seems to have worked and is much simplified. Thank you to all!

This is as of today in version 0.50.2

I see the same warnings in my log, as well as the below repeatedly. I'm guessing that the device is reporting information on clusters that HA doesn't yet support? Will the ZHA component continue to be updated to support additional clusters?
2017-08-04 16:05:09 WARNING (MainThread) [bellows.zigbee.zcl] [0xdf53:1:0x0019] No handler for cluster command 1
2017-08-04 16:06:09 WARNING (MainThread) [bellows.zigbee.zcl] [0xf480:1:0x0500] No handler for cluster command 0

Cluster 0x0019 is over the air firmware upgrades. bellows trunk has fixes to substantially reduce the noise ... I'll push up a new version and update HA.

Is there any component information on ZHA on home-assistant.io?

With 0.51 the messages have changed slightly, but the logspam is still there. For all three of my contacts I see this block show up periodically:
2017-08-28 13:03:35 WARNING (MainThread) [bellows.zigbee.application] Unexpected message send notification
2017-08-28 13:03:36 WARNING (MainThread) [bellows.zigbee.zcl] Data remains after deserializing ZCL frame
2017-08-28 13:03:36 WARNING (MainThread) [bellows.zigbee.zcl] [0xf9cd:1:0x0019] No handler for cluster command 1
2017-08-28 13:03:39 WARNING (MainThread) [bellows.zigbee.zcl] Data remains after deserializing ZCL frame
2017-08-28 13:03:39 WARNING (MainThread) [bellows.zigbee.zcl] [0xf9cd:1:0x0019] No handler for cluster command 1
2017-08-28 13:03:42 WARNING (MainThread) [bellows.zigbee.zcl] Data remains after deserializing ZCL frame
2017-08-28 13:03:42 WARNING (MainThread) [bellows.zigbee.zcl] [0xf9cd:1:0x0019] No handler for cluster command 1
2017-08-28 13:03:45 WARNING (MainThread) [bellows.zigbee.zcl] Data remains after deserializing ZCL frame
2017-08-28 13:03:45 WARNING (MainThread) [bellows.zigbee.zcl] [0xf9cd:1:0x0019] No handler for cluster command 1
2017-08-28 13:03:48 WARNING (MainThread) [bellows.zigbee.zcl] Data remains after deserializing ZCL frame
2017-08-28 13:03:48 WARNING (MainThread) [bellows.zigbee.zcl] [0xf9cd:1:0x0019] No handler for cluster command 1

Is there a way for Bellows/Hass to tell the device "I do not implement cluster 0x19" so hopefully it'll stop trying, or at the very least not retry four times, and save a little battery?

This is likely a separate issue, but I also get
2017-08-28 13:21:17 WARNING (MainThread) [bellows.zigbee.zcl] [0xf9cd:1:0x0500] No handler for cluster command 0

Whenever a contact opens/closes, followed by the usual state change message. Odd, because isn't that the update notify command?
Should that be in a separate bug?
Strangely enough, I haven't gotten any warnings originating with my Cree bulbs. Maybe they just don't talk much.

@rcloran I am on 0.52.0 with the same errors after attaching several Sengled Element bulbs.
 WARNING (MainThread) [bellows.zigbee.application] Unexpected message send notification
2017-08-28 13:03:36 WARNING (MainThread) [bellows.zigbee.zcl] Data remains after deserializing ZCL frame
2017-08-28 13:03:36 WARNING (MainThread) [bellows.zigbee.zcl] [0xf9cd:1:0x0019] No handler for cluster command 1

And the state of the bulb is never updated in HA if the light is turned on or off at the device.

@rcloran hi I'm on HA 0.54 and also seeing these errors. I have a few Lowe's Iris Motion Sensors and a couple Centralite 3305-S Motion Sensor (SmartThings) paired.
2017-09-24 07:06:33 WARNING (MainThread) [bellows.zigbee.zcl] Data remains after deserializing ZCL frame
2017-09-24 07:06:33 WARNING (MainThread) [bellows.zigbee.zcl] [0x64d2:1:0x0019] No handler for cluster command 1
Thank you for your efforts to bring ZigBee to HA.  It is working good for me except for these errors.

I have osram lightify RBGW bulb and 4 cree connected bulb and 2 sengled buld.. The HA log is full of bellows error. Please see below.    Resetting 2017-11-08 10:00:43 ERROR (MainThread) [homeassistant.core] Timer got out of sync.  Resetting 2017-11-08 12:03:52 ERROR (MainThread) [aiohttp.access] Error in logging Traceback (most recent call last):   File "/usr/lib/python3.6/site-packages/aiohttp/helpers.py", line 500, in log     for key, value in fmt_info:   File "/usr/lib/python3.6/site-packages/aiohttp/helpers.py", line 483, in <genexpr>     return ((key, method(args)) for key, method in self._methods)   File "/usr/lib/python3.6/site-packages/aiohttp/helpers.py", line 436, in _format_a     peername = args[3].get_extra_info('peername')   File "/usr/lib/python3.6/asyncio/sslproto.py", line 306, in get_extra_info     return self._ssl_protocol._get_extra_info(name, default)   File "/usr/lib/python3.6/asyncio/sslproto.py", line 547, in _get_extra_info     return self._transport.get_extra_info(name, default) AttributeError: 'NoneType' object has no attribute 'get_extra_info' 2017-11-08 12:03:57 WARNING (MainThread) [bellows.zigbee.application] Unexpected message send notification 2017-11-08 12:05:21 WARNING (MainThread) [bellows.zigbee.application] Unexpected message send notification 2017-11-08 12:06:04 WARNING (MainThread) [bellows.zigbee.application] Unexpected message send notification 2017-11-08 12:15:24 WARNING (MainThread) [bellows.zigbee.application] Unexpected message send notification 2017-11-08 12:16:06 WARNING (MainThread) [bellows.zigbee.application] Unexpected message send notification 2017-11-08 12:20:28 ERROR (MainThread) [homeassistant.core] Error doing job: Fatal error on transport TCPTransport (error status in uv_stream_t.read callback) TimeoutError: [Errno 110] Operation timed out 2017-11-08 12:25:26 WARNING (MainThread) [bellows.zigbee.application] Unexpected message send notification 2017-11-08 12:26:09 WARNING (MainThread) [bellows.zigbee.application] Unexpected message send notification 2017-11-08 12:26:48 WARNING (MainThread) [bellows.zigbee.application] Unexpected message send notification 2017-11-08 12:27:21 WARNING (MainThread) [bellows.zigbee.application] Unexpected message send notification 2017-11-08 12:35:29 WARNING (MainThread) [bellows.zigbee.application] Unexpected message send notification 2017-11-08 12:36:11 WARNING (MainThread) [bellows.zigbee.application] Unexpected message send notification 2017-11-08 12:39:33 WARNING (MainThread) [bellows.zigbee.application] Unexpected message send notification 2017-11-08 12:41:51 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved Traceback (most recent call last):   File "/usr/lib/python3.6/asyncio/tasks.py", line 179, in _step     result = coro.send(None)   File "/usr/lib/python3.6/site-packages/homeassistant/core.py", line 1038, in _event_to_service_call     yield from service_handler.func(service_call)   File "/usr/lib/python3.6/site-packages/homeassistant/components/light/__init__.py", line 282, in async_handle_light_service     yield from light.async_turn_off(**params)   File "/usr/lib/python3.6/site-packages/homeassistant/components/light/zha.py", line 118, in async_turn_off     yield from self._endpoint.on_off.off()   File "/usr/lib/python3.6/site-packages/bellows/zigbee/application.py", line 286, in request     v = yield from asyncio.wait_for(reply_fut, timeout)   File "/usr/lib/python3.6/asyncio/tasks.py", line 356, in wait_for     raise futures.TimeoutError() concurrent.futures._base.TimeoutError 2017-11-08 12:42:21 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved Traceback (most recent call last):   File "/usr/lib/python3.6/asyncio/tasks.py", line 181, in _step     result = coro.throw(exc)   File "/usr/lib/python3.6/site-packages/homeassistant/core.py", line 1038, in _event_to_service_call     yield from service_handler.func(service_call)   File "/usr/lib/python3.6/site-packages/homeassistant/components/light/__init__.py", line 282, in async_handle_light_service     yield from light.async_turn_off(**params)   File "/usr/lib/python3.6/site-packages/homeassistant/components/light/zha.py", line 118, in async_turn_off     yield from self._endpoint.on_off.off()   File "/usr/lib/python3.6/site-packages/bellows/zigbee/application.py", line 284, in request     v = yield from send_fut   File "/usr/lib/python3.6/asyncio/futures.py", line 331, in __iter__     yield self  # This tells Task to wait for completion.


State how we selected assembly parameters

For the assembly, how were these parameters picked, and is there any evaluation of sensitivity or specificity?

Related to #31


[Service Bus] [Event Hubs] Update access control related docs

For more context refer to #5239

@ramya-rao-a Is there anything else blocking the PR?