[TOPI] Improve memory layout inside GPU NMS kernel

Motivation
Currently, our NMS API expects input and output data to be in a format [batch, num_anchors, 6], where 6 is for 4 bbox coordinates, score and class id. This format is highly inefficient for class-aware NMS, because to check if two bboxes are the same class, we need to load other 5 values. More discussion in https://github.com/apache/tvm/pull/7154#issuecomment-750376423 and other comments in that PR.
This PR improved the memory layout of buffers used in the NMS kernel. Specifically, it unpacks [batch, num_anchors, 6] input into one [batch, num_anchors, 4] buffer for bboxes, and two [batch, num_anchors] buffers for scores and class ids.
Speed up
This change is expected to give a good speed up for the cases when the number valid boxes is large and there are class ids involved. For example, although the change in https://github.com/apache/tvm/pull/7154 brings only modest speed up, combined with this PR the speed up becomes good, as shown below:
PyTorch NMS workload from FasterRCNN / MaskRCNN (4500 boxes):

Current main : 5.78 milli sec
After rewrite in #7154 to restore class aware NMS: 5.23 milli sec
With memory layout improvement in this PR and the rewrite in #7154: 4.51 milli sec

Other consideration
For other cases, it is likely that there would be no speed up, but it should be no worse. For example, it doesn't improve NMS workload from Gluon SSD. When return_indices=False, which I think only applies to MXNet, there is additional concatenation of three buffers at the end. For Gluon SSD, this concat takes 40 micro sec on GTX 1070 ti, while the main NMS kernel takes 200 micro sec. If people think this additional concat is expensive, we really need to rethink our NMS API: The same concat is already happenning in all other frontends, because all frameworks other than MXNet gives NMS inputs in an unpacked form and concat is necessary to satisfy our NMS API.
please review @mbrookhart @Laurawly @vinx13 @trevor-m

Even after this, I think we will still need a loop over classes for ONNX and TF, since ONNX explicitly and TF implicitly need max_output_boxes_per_class, while this op even with class id will return max_output_boxes for all classes.

Even after this, I think we will still need a loop over classes for ONNX and TF, since ONNX explicitly and TF implicitly need max_output_boxes_per_class, while this op even with class id will return max_output_boxes for all classes.

I don't quite follow, maybe you are missing something?
First, this PR doesn't change our NMS API, it only changes the buffer layouts used internally.
Second, the final concat is only required for MXNet, which uses return_indices=False. Our NMS returns something completely different depending on return_indices flag : If True, it returns a big output tensor packed with bboxes, scores and class ids, with invalid entries indicated by -1.
(The valid entries are supposed to move to the top, if  invalid_to_bottom flag is True. But our GPU NMS kernel ignores this argument and the output is not reordered. This is another difference with CPU implementation, I think this is a bug)
https://github.com/apache/tvm/blob/4e8cc4fc26e931e38017d198d29f45cba04f5a60/python/tvm/topi/cuda/nms.py#L762-L763
If return_indices=True, which applies to TF, ONNX, and PyTorch, we only return survived box indices, so there is no need to concat bboxes, scores, and class ids. For this case, there is zero additional overhead, it is just that concat before NMS done by frontends are now completely pointless.

I don't quite follow, maybe you are missing something?
First, this PR doesn't change our NMS API, it only changes the buffer layouts used internally.
Second, the final concat is only required for MXNet, which uses return_indices=False. Our NMS returns something completely different depending on return_indices flag : If True, it returns a big output tensor packed with bboxes, scores and class ids, with invalid entries indicated by -1.
(The valid entries are supposed to move to the top, if  invalid_to_bottom flag is True. But our GPU NMS kernel ignores this argument and the output is not reordered. This is another difference with CPU implementation, I think this is a bug)
https://github.com/apache/tvm/blob/4e8cc4fc26e931e38017d198d29f45cba04f5a60/python/tvm/topi/cuda/nms.py#L762-L763
If return_indices=True, which applies to TF, ONNX, and PyTorch, we only return survived box indices, so there is no need to concat bboxes, scores, and class ids. For this case, there is zero additional overhead, it is just that concat before NMS done by frontends are now completely pointless.

Yes, ideally I want to update our NMS to be more closer to TF/ONNX/PyTorch, and let MXNet frontend handle split and concat, rather than the other way around (what we have now). Current API is over complicated due to the need to support both styles. If we can assume that return_indices is always True, we can clean up our API a lot. For example, invalid_to_bottom argument only makes sense for MXNet. We don't need coord_start, score_index, and id_index arguments, if inputs are only unpacked.
Supporting max_output_boxes_per_class needs change in the implementation as well. We need to count the number of survived boxes per class. But that's the only change I think, it is definitely doable without writing another kernel.

Yes, ideally I want to update our NMS to be more closer to TF/ONNX/PyTorch, and let MXNet frontend handle split and concat, rather than the other way around (what we have now). Current API is over complicated due to the need to support both styles. If we can assume that return_indices is always True, we can clean up our API a lot. For example, invalid_to_bottom argument only makes sense for MXNet. We don't need coord_start, score_index, and id_index arguments, if inputs are only unpacked.
Supporting max_output_boxes_per_class needs change in the implementation as well. We need to count the number of survived boxes per class. But that's the only change I think, it is definitely doable without writing another kernel.

Thanks @mbrookhart

Thanks @mbrookhart


post-start script of rep fail for a single node cloud foundry deployment

I was trying to deploy all cf & diego jobs in a single node, and got an error saying that post-start script of rep faill.
bosh task log:
Getting deployment properties from director...
Unable to get properties list from director, trying without it...
Cannot get current deployment information from director, possibly a new deployment

Deploying
---------

Director task 512
  Started preparing deployment > Preparing deployment. Done (00:00:01)

  Started preparing package compilation > Finding packages to compile. Done (00:00:00)

  Started creating missing vms
  Started creating missing vms > postgres_z1/0 (06ad2a5a-3763-402f-a9f2-dabe574e5ff0)
  Started creating missing vms > cf_z1/0 (d9ae1dd2-9557-481d-a898-8f7473149444)
     Done creating missing vms > postgres_z1/0 (06ad2a5a-3763-402f-a9f2-dabe574e5ff0) (00:02:26)
     Done creating missing vms > cf_z1/0 (d9ae1dd2-9557-481d-a898-8f7473149444) (00:02:49)
     Done creating missing vms (00:02:49)

  Started updating job postgres_z1 > postgres_z1/0 (06ad2a5a-3763-402f-a9f2-dabe574e5ff0) (canary). Done (00:01:25)
  Started updating job cf_z1 > cf_z1/0 (d9ae1dd2-9557-481d-a898-8f7473149444) (canary). Failed: Action Failed get_task: Task 4e432186-719b-407f-618f-43a9db41aa96 result: 1 of 4 post-start scripts failed. Failed Jobs: rep. Successful Jobs: bbs, uaa, cloud_controller_ng. (00:16:23)

Error 450001: Action Failed get_task: Task 4e432186-719b-407f-618f-43a9db41aa96 result: 1 of 4 post-start scripts failed. Failed Jobs: rep. Successful Jobs: bbs, uaa, cloud_controller_ng.

Task 512 error

monit summary indicated that all jobs are running:
The Monit daemon 5.2.5 uptime: 18m

Process 'consul_agent'              running
Process 'nats'                      running
Process 'nats_stream_forwarder'     running
Process 'metron_agent'              running
Process 'etcd'                      running
Process 'etcd_metrics_server'       running
Process 'rpc_nfsd'                  running
Process 'rpc_mountd'                running
Process 'cloud_controller_ng'       running
Process 'cloud_controller_worker_local_1' running
Process 'cloud_controller_worker_local_2' running
Process 'nginx_cc'                  running
Process 'cloud_controller_worker_1' running
Process 'cloud_controller_clock'    running
File 'nfs_mounter'                  accessible
Process 'route_registrar'           running
Process 'consul_template'           running
File 'haproxy_config'               accessible
Process 'haproxy'                   running
Process 'doppler'                   running
Process 'loggregator_trafficcontroller' running
Process 'uaa'                       running
Process 'gorouter'                  running
Process 'statsd-injector'           running
Process 'syslog_drain_binder'       running
Process 'bbs'                       running
Process 'auctioneer'                running
Process 'converger'                 running
Process 'rep'                       running
Process 'garden'                    running
Process 'stager'                    running
Process 'nsync_listener'            running
Process 'nsync_bulker'              running
Process 'tps_listener'              running
Process 'tps_watcher'               running
Process 'cc_uploader'               running
Process 'ssh_proxy'                 running
Process 'file_server'               running
Process 'route_emitter'             running
System 'system_localhost'           running

But from post-start log of rep, it failed for "timout out pinging rep".
Thu Jul 14 06:26:18 UTC 2016: Pinging rep...
Thu Jul 14 06:26:19 UTC 2016: Attempt 1 failed. Trying again...
Thu Jul 14 06:26:20 UTC 2016: Attempt 2 failed. Trying again...
Thu Jul 14 06:26:21 UTC 2016: Attempt 3 failed. Trying again...
Thu Jul 14 06:26:22 UTC 2016: Attempt 4 failed. Trying again...
...
Thu Jul 14 06:31:14 UTC 2016: Attempt 291 failed. Trying again...
Thu Jul 14 06:31:15 UTC 2016: Attempt 292 failed. Trying again...
Thu Jul 14 06:31:16 UTC 2016: Attempt 293 failed. Trying again...
Thu Jul 14 06:31:17 UTC 2016: Attempt 294 failed. Trying again...
Thu Jul 14 06:31:18 UTC 2016: Timed out pinging rep. Failing post-start.

From /var/vcap/monit/monit.log, I can see rep has restarted several times.
[UTC Jul 14 06:13:48] info     : monit: generated unique Monit id da648ecabcdb851f427aa42e45db1b20 and stored to '/root/.monit.id'
[UTC Jul 14 06:13:48] info     : Starting monit daemon with http interface at [<IP_ADDRESS>:2822]
[UTC Jul 14 06:13:48] info     : Starting monit HTTP server at [<IP_ADDRESS>:2822]
[UTC Jul 14 06:13:48] info     : monit HTTP server started
[UTC Jul 14 06:13:48] info     : 'system_localhost' Monit started
[UTC Jul 14 06:19:46] info     : Reinitializing monit daemon
[UTC Jul 14 06:19:46] info     : Awakened by the SIGHUP signal
[UTC Jul 14 06:19:46] info     : Reinitializing monit - Control file '/var/vcap/bosh/etc/monitrc'
[UTC Jul 14 06:19:46] info     : Shutting down monit HTTP server
[UTC Jul 14 06:19:46] info     : monit HTTP server stopped
[UTC Jul 14 06:19:46] warning  : /var/vcap/bosh/etc/monitrc:8: Warning: include files not found '/var/vcap/monit/job/*.monitrc'
[UTC Jul 14 06:19:46] info     : Starting monit HTTP server at [<IP_ADDRESS>:2822]
[UTC Jul 14 06:19:46] info     : monit HTTP server started
[UTC Jul 14 06:19:46] info     : 'system_localhost' Monit reloaded
[UTC Jul 14 06:22:33] info     : Reinitializing monit daemon
[UTC Jul 14 06:22:33] info     : Awakened by the SIGHUP signal
[UTC Jul 14 06:22:33] info     : Reinitializing monit - Control file '/var/vcap/bosh/etc/monitrc'
[UTC Jul 14 06:22:33] info     : Shutting down monit HTTP server
[UTC Jul 14 06:22:33] info     : monit HTTP server stopped
[UTC Jul 14 06:22:33] info     : Starting monit HTTP server at [<IP_ADDRESS>:2822]
[UTC Jul 14 06:22:33] info     : monit HTTP server started
[UTC Jul 14 06:22:33] info     : 'system_localhost' Monit reloaded
[UTC Jul 14 06:22:33] error    : 'consul_agent' process is not running
[UTC Jul 14 06:22:33] info     : 'consul_agent' trying to restart
[UTC Jul 14 06:22:33] info     : 'consul_agent' start: /var/vcap/jobs/consul_agent/bin/agent_ctl
[UTC Jul 14 06:22:38] info     : start service 'route_emitter' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'file_server' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'ssh_proxy' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'cc_uploader' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'tps_watcher' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'tps_listener' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'nsync_bulker' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'nsync_listener' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'stager' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'garden' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'rep' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'converger' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'auctioneer' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'bbs' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'syslog_drain_binder' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'statsd-injector' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'gorouter' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'uaa' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'loggregator_trafficcontroller' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'doppler' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'haproxy' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'haproxy_config' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'consul_template' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'route_registrar' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'nfs_mounter' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'cloud_controller_clock' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'cloud_controller_worker_1' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'nginx_cc' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'cloud_controller_worker_local_2' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'cloud_controller_worker_local_1' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'cloud_controller_ng' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'rpc_mountd' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'rpc_nfsd' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'etcd_metrics_server' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'etcd' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'metron_agent' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'nats_stream_forwarder' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'nats' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:38] info     : start service 'consul_agent' on user request
[UTC Jul 14 06:22:38] info     : monit daemon at 2036 awakened
[UTC Jul 14 06:22:40] info     : 'consul_template' start: /var/vcap/jobs/haproxy/bin/consul_template_ctl
[UTC Jul 14 06:22:41] info     : 'nats' start: /var/vcap/jobs/nats/bin/nats_ctl
[UTC Jul 14 06:22:42] info     : 'nats_stream_forwarder' start: /var/vcap/jobs/nats_stream_forwarder/bin/nats_stream_forwarder_ctl
[UTC Jul 14 06:22:43] info     : 'nats' start action done
[UTC Jul 14 06:22:43] info     : 'nats_stream_forwarder' start action done
[UTC Jul 14 06:22:43] info     : 'metron_agent' start: /var/vcap/jobs/metron_agent/bin/metron_agent_ctl
[UTC Jul 14 06:22:44] info     : 'metron_agent' start action done
[UTC Jul 14 06:22:44] info     : 'etcd' start: /usr/bin/timeout
[UTC Jul 14 06:22:52] info     : 'etcd' start action done
[UTC Jul 14 06:22:53] info     : 'etcd_metrics_server' start: /var/vcap/jobs/etcd_metrics_server/bin/etcd_metrics_server_ctl
[UTC Jul 14 06:22:54] info     : 'etcd_metrics_server' start action done
[UTC Jul 14 06:22:54] info     : 'rpc_nfsd' start: /var/vcap/jobs/debian_nfs_server/bin/rpc_nfsd_ctl
[UTC Jul 14 06:23:00] info     : 'rpc_mountd' start: /var/vcap/jobs/debian_nfs_server/bin/rpc_mountd_ctl
[UTC Jul 14 06:23:01] info     : 'rpc_nfsd' start action done
[UTC Jul 14 06:23:01] info     : 'rpc_mountd' start action done
[UTC Jul 14 06:23:01] info     : 'cloud_controller_ng' start: /var/vcap/jobs/cloud_controller_ng/bin/cloud_controller_ng_ctl
[UTC Jul 14 06:23:31] error    : 'cloud_controller_ng' failed to start
[UTC Jul 14 06:23:31] info     : 'nginx_cc' start: /var/vcap/jobs/cloud_controller_ng/bin/nginx_ctl
[UTC Jul 14 06:23:32] info     : 'cloud_controller_ng' start action done
[UTC Jul 14 06:23:32] info     : 'cloud_controller_worker_local_1' start: /var/vcap/jobs/cloud_controller_ng/bin/cloud_controller_worker_ctl
[UTC Jul 14 06:23:33] info     : 'cloud_controller_worker_local_1' start action done
[UTC Jul 14 06:23:33] info     : 'cloud_controller_worker_local_2' start: /var/vcap/jobs/cloud_controller_ng/bin/cloud_controller_worker_ctl
[UTC Jul 14 06:23:34] info     : 'cloud_controller_worker_local_2' start action done
[UTC Jul 14 06:23:34] info     : 'nginx_cc' start action done
[UTC Jul 14 06:23:34] info     : 'cloud_controller_worker_1' start: /var/vcap/jobs/cloud_controller_worker/bin/cloud_controller_worker_ctl
[UTC Jul 14 06:23:35] info     : 'cloud_controller_worker_1' start action done
[UTC Jul 14 06:23:35] info     : 'cloud_controller_clock' start: /var/vcap/jobs/cloud_controller_clock/bin/cloud_controller_clock_ctl
[UTC Jul 14 06:23:36] info     : 'cloud_controller_clock' start action done
[UTC Jul 14 06:23:36] info     : 'nfs_mounter' start: /var/vcap/jobs/nfs_mounter/bin/nfs_mounter_ctl
[UTC Jul 14 06:23:36] info     : 'nfs_mounter' start action done
[UTC Jul 14 06:23:36] info     : 'route_registrar' start: /var/vcap/jobs/route_registrar/bin/route_registrar_ctl
[UTC Jul 14 06:23:37] info     : 'route_registrar' start action done
[UTC Jul 14 06:23:38] info     : 'consul_template' start action done
[UTC Jul 14 06:23:38] info     : 'haproxy' start: /var/vcap/jobs/haproxy/bin/haproxy_ctl
[UTC Jul 14 06:23:39] info     : 'haproxy' start action done
[UTC Jul 14 06:23:39] info     : 'doppler' start: /var/vcap/jobs/doppler/bin/doppler_ctl
[UTC Jul 14 06:23:40] info     : 'doppler' start action done
[UTC Jul 14 06:23:40] info     : 'loggregator_trafficcontroller' start: /var/vcap/jobs/loggregator_trafficcontroller/bin/loggregator_trafficcontroller_ctl
[UTC Jul 14 06:23:41] info     : 'loggregator_trafficcontroller' start action done
[UTC Jul 14 06:23:41] info     : 'uaa' start: /var/vcap/jobs/uaa/bin/uaa_ctl
[UTC Jul 14 06:23:42] info     : 'uaa' start action done
[UTC Jul 14 06:23:42] info     : 'gorouter' start: /var/vcap/jobs/gorouter/bin/gorouter_ctl
[UTC Jul 14 06:24:07] info     : 'gorouter' start action done
[UTC Jul 14 06:24:07] info     : 'statsd-injector' start: /var/vcap/jobs/statsd-injector/bin/statsd-injector-ctl
[UTC Jul 14 06:24:08] info     : 'statsd-injector' start action done
[UTC Jul 14 06:24:08] info     : 'syslog_drain_binder' start: /var/vcap/jobs/syslog_drain_binder/bin/syslog_drain_binder_ctl
[UTC Jul 14 06:24:09] info     : 'syslog_drain_binder' start action done
[UTC Jul 14 06:24:09] info     : 'bbs' start: /var/vcap/jobs/bbs/bin/bbs_ctl
[UTC Jul 14 06:24:10] info     : 'bbs' start action done
[UTC Jul 14 06:24:10] info     : 'auctioneer' start: /var/vcap/jobs/auctioneer/bin/auctioneer_ctl
[UTC Jul 14 06:24:11] info     : 'auctioneer' start action done
[UTC Jul 14 06:24:11] info     : 'converger' start: /var/vcap/jobs/converger/bin/converger_ctl
[UTC Jul 14 06:24:12] info     : 'converger' start action done
[UTC Jul 14 06:24:12] info     : 'garden' start: /var/vcap/jobs/garden/bin/garden_ctl
[UTC Jul 14 06:24:14] info     : 'rep' start: /var/vcap/jobs/rep/bin/rep_ctl
[UTC Jul 14 06:24:44] error    : 'rep' failed to start
[UTC Jul 14 06:24:44] info     : 'garden' start action done
[UTC Jul 14 06:24:44] info     : 'stager' start: /var/vcap/jobs/stager/bin/stager_ctl
[UTC Jul 14 06:24:45] info     : 'stager' start action done
[UTC Jul 14 06:24:45] info     : 'nsync_listener' start: /var/vcap/jobs/nsync/bin/nsync_listener_ctl
[UTC Jul 14 06:24:46] info     : 'nsync_listener' start action done
[UTC Jul 14 06:24:46] info     : 'nsync_bulker' start: /var/vcap/jobs/nsync/bin/nsync_bulker_ctl
[UTC Jul 14 06:24:47] info     : 'nsync_bulker' start action done
[UTC Jul 14 06:24:47] info     : 'tps_listener' start: /var/vcap/jobs/tps/bin/tps_listener_ctl
[UTC Jul 14 06:24:48] info     : 'tps_listener' start action done
[UTC Jul 14 06:24:48] info     : 'tps_watcher' start: /var/vcap/jobs/tps/bin/tps_watcher_ctl
[UTC Jul 14 06:24:49] info     : 'tps_watcher' start action done
[UTC Jul 14 06:24:49] info     : 'cc_uploader' start: /var/vcap/jobs/cc_uploader/bin/cc_uploader_ctl
[UTC Jul 14 06:24:50] info     : 'cc_uploader' start action done
[UTC Jul 14 06:24:50] info     : 'ssh_proxy' start: /var/vcap/jobs/ssh_proxy/bin/ssh_proxy_ctl
[UTC Jul 14 06:24:51] info     : 'ssh_proxy' start action done
[UTC Jul 14 06:24:52] info     : 'file_server' start: /var/vcap/jobs/file_server/bin/file_server_ctl
[UTC Jul 14 06:24:53] info     : 'file_server' start action done
[UTC Jul 14 06:24:53] info     : 'route_emitter' start: /var/vcap/jobs/route_emitter/bin/route_emitter_ctl
[UTC Jul 14 06:24:54] info     : 'route_emitter' start action done
[UTC Jul 14 06:24:54] info     : 'haproxy_config' start action done
[UTC Jul 14 06:24:54] info     : 'rep' start action done
[UTC Jul 14 06:24:54] info     : Awakened by User defined signal 1
[UTC Jul 14 06:24:54] info     : 'consul_agent' start action done
[UTC Jul 14 06:24:54] info     : 'consul_agent' process is running with pid 11565
[UTC Jul 14 06:24:54] error    : 'cloud_controller_ng' process is not running
[UTC Jul 14 06:24:54] info     : 'cloud_controller_ng' trying to restart
[UTC Jul 14 06:24:54] info     : 'nginx_cc' stop: /var/vcap/jobs/cloud_controller_ng/bin/nginx_ctl
[UTC Jul 14 06:24:55] info     : 'cloud_controller_ng' start: /var/vcap/jobs/cloud_controller_ng/bin/cloud_controller_ng_ctl
[UTC Jul 14 06:25:03] info     : 'cloud_controller_ng' started
[UTC Jul 14 06:25:03] info     : 'nginx_cc' start: /var/vcap/jobs/cloud_controller_ng/bin/nginx_ctl
[UTC Jul 14 06:25:16] error    : 'haproxy_config' timestamp was changed for /var/vcap/jobs/haproxy/config/haproxy.conf
[UTC Jul 14 06:25:16] info     : 'haproxy_config' exec: /var/vcap/jobs/haproxy/bin/haproxy_ctl
[UTC Jul 14 06:25:16] error    : 'rep' process is not running
[UTC Jul 14 06:25:16] info     : 'rep' trying to restart
[UTC Jul 14 06:25:16] info     : 'rep' start: /var/vcap/jobs/rep/bin/rep_ctl
[UTC Jul 14 06:25:18] info     : 'rep' started
[UTC Jul 14 06:25:28] info     : 'cloud_controller_ng' process is running with pid 14525
[UTC Jul 14 06:25:28] error    : 'haproxy' process PID changed from 13002 to 14800
[UTC Jul 14 06:25:28] info     : 'haproxy_config' timestamp was not changed for /var/vcap/jobs/haproxy/config/haproxy.conf
[UTC Jul 14 06:25:28] info     : 'rep' process is running with pid 14780
[UTC Jul 14 06:25:38] info     : 'haproxy' process PID has not changed since last cycle
[UTC Jul 14 06:32:09] error    : 'haproxy_config' timestamp was changed for /var/vcap/jobs/haproxy/config/haproxy.conf
[UTC Jul 14 06:32:09] info     : 'haproxy_config' exec: /var/vcap/jobs/haproxy/bin/haproxy_ctl
[UTC Jul 14 06:32:19] error    : 'haproxy' process PID changed from 14800 to 18242
[UTC Jul 14 06:32:19] info     : 'haproxy_config' timestamp was not changed for /var/vcap/jobs/haproxy/config/haproxy.conf
[UTC Jul 14 06:32:29] info     : 'haproxy' process PID has not changed since last cycle
[UTC Jul 14 06:35:20] error    : 'rep' process is not running
[UTC Jul 14 06:35:20] info     : 'rep' trying to restart
[UTC Jul 14 06:35:20] info     : 'rep' start: /var/vcap/jobs/rep/bin/rep_ctl
[UTC Jul 14 06:35:31] info     : 'rep' process is running with pid 19090

The result indicates that rep is not available before the post-start timeout (300 seconds).  So I tried to set time out of rep post-start as 800, then I got my deployment pass.
Not sure why rep need to restart, so my question is: is it reasonable to just change the timeout value of post-start for a single node deployment? If yes, I think a parameter such as diego.rep.post_start_timeout_in_seconds can be added to rep job spec; if not, I believe there is some other issues that I was not aware of, so could you please guide me how to debug the issue.

Thanks for the report, @gossion. Could you please provide logs from the rep and garden-linux jobs, both from the failed start-up attempt with the 300-second timeout and the successful start-up with the 800-second timeout? They should be located in /var/vcap/sys/log/rep/rep.stdout.log and /var/vcap/sys/log/garden/garden.stdout.log, respectively. The Diego team and I would be interested to understand why the rep and/or garden-linux took so long to become healthy. In general, I'd like to avoid adding another configuration setting on the release if it's not truly necessary.
Best,
Eric, CF Runtime Diego PM

Thank you, @ematpl. I have uploaded rep log and garden log to my personal repo, you can also find the manifest yaml file and all job logs  in the same repo.
And please ignore my words about changing timeout value, because I just found that it still failed at post-start sometimes after changing timeout value to 800 seconds. sorry. So we need to find out the reason why rep can't become healthy.

Hi @gossion, sorry about the delay. It seems like it's just taking a while to spin up a container, maybe because the VM is overloaded. Have you tried increasing the size of the VM? What does the resource utilization of the VM look like when you're seeing this?

Thank you, @jenspinney . You are really helpful, it deployed successfully by increasing the size of the VM.
My original VM size is Standard_D1 in Azure, now with Standard_D2 I get the deployment passed.
Thank you very much.


Simplify StaticRouter render().

I don't think getLocation() needs to be called twice per render.

Looks like @mjackson beat you to the punch. https://github.com/ReactTraining/react-router/commit/5550cc344c863b2aca255fadf146c96a289149ad

Thanks for the PR, @BradEstey. You definitely beat me to the punch here, but I hadn't seen the PR. I was just doing some general cleaning up in the code today.
I'll do my best to look out for existing PRs before doing a bunch of refactoring of my own :)


How to return the execution back to the caller

In my case go-prompt is just a part of application and I would like to do something like this:
func main() {
  myPrompt := prompt.New()
  myPrompt.Run()

// some other logic, executed once prompt returns
}

Any hints on how to do that are welcome!

if you haven't found an answer for this yet, try looking at the exit checker option. I've done worked with this in my module at https://gitlab.com/kbickmore/go-cmd

Hey @ksbickmore,
I changed the application architecture, so that I did not need to do what I mentioned in the OP. Thanks for the reply and the pointer, though.


Add tooling to only build diffs

Add get-diffs.py that filters a matrix.json on the dirs parameter that have diffs (also allows specifying --dirs or returning --all)
Convert build-latest to use matrix.json

TODO:

Filter matrix.json and matrix-preview.json based on diffs (in the GHA)
create / improve docstrings / etc.
Decide whether to "only test diffs & only build diffs" or "test all and only build diffs"

My recommendation: since there are no interdependencies and you don't want to gate PRs on unrelated failures, I think testing only diffs is appropriate.


Set task_type when starting a worker pool (#64)

make queue fields public

Set task_type when starting a worker pool

make fields private again

LGTM


Adding a new game type package

Hi!
I'm trying to make a campaign using Valkyrie for Dark Souls the Board Game.
I know this is not a game based upon scenarios, however I would like to make a campaign that would add exploration, lore and meaningful character progression to the board game. I've already been developing the campaign for a few weeks, so I began to think about using Valkyrie to actually play the scenario.
I have messaged NPBruce and asked him about this. He said that it is possible, I just need to make a content pack for MoM (easier) or a package system for the new game type (harder, but better). I don't need any advanced features, only the basic things, such as tiles, tokens and NPC dialogue. However, I would like to avoid using MoM editor, as there is no way of disabling investigator selection or Mythos Phase. That's why I would prefer option number 2, creating a new game package.
I thought it would be good to write about it here, as maybe somebody has already done something like this or knows how to do this.
I would really appreciate any help :)

To clarify:
There is no reason not to use the editor.  Investigator/Hero selection is a part of the game type and manually creating scenarios will not avoid this.  For now you can just ignore it and get started.
All game assets can be created easily enough with a content pack that can build on top of MoM/D2E.  As I am not familiar with Dark Souls.  I don't know what would be most appropriate settings.  For reference it goes something like this:

Game
D2E
MoM
DS

Layout
Grid
Freeform
?

Activation
Alternating
Investigators then Monsters
?

Hero Selection
Yes
Yes
No?

Hero Selection during Game
Yes
No
?

Monsters Grouped
Yes
No
?

Items Selectable
No
Yes
?

Starting Items
No
Yes
?

There are also things like what fonts to use.

I'll try to explain what I would like to do in the Dark Souls Scenario
Layout: Tiles in DS are all the same size and shape (square) and are layed in a grid. On the tiles themselves there are nodes, which can be used to show where exactly on a tile something is. Here's an example of how it looks like:
https://static1.squarespace.com/static/56728f72a128e6b1e548ec55/t/589dfe099de4bba912b5f78f/1486749226351/DS-Boards-AnorLondo.png
Activation: In the game the turns are Player1->All Monsters->Player2->All Monsters etc., but I don't think this needs to be in the scenario. The beaviour of the monsters is shown on certain cards, allowing players to make strategic decisions as they know what the enemy will do. I don't think I want to change that so I don't think that the phases should be in the app, as I can think of no way I (or anybody else making a campaign for DS) would use it
Hero Selection: Yes. There are 10 classes and which classes were chosen impacts some small parts of the story and the starting items.
Hero Selection during Game: No
Monsters: In DS, the enemies appear during encounters. Each encounter is a separate tile, and ends with either the players or the monsters dead. That's why there is no need of keeping track of enemies in the app. However, every time a player enters a new room the app should show a description of the place, and, in most cases, the placement and types of enemies in that room. I think the best way of doing this would be sort of like when the players go into a new room in MoM: There is some text and the camera moves to show where you place which tokens (or in case of DS, also monsters).
Items: Just like in the original Dark Souls I want to tell the story through NPC dialogue and item descriptions. The first can be done through interactable tokens, but the second one is a bit more tricky. Basically every time the players find an item it appears in the Item menu, where it can be inspected to read it's desription. I have been thinking to do the same thing with enemies with a second menu to give the players more information about the scenario's story. The reason I think it has to be done differently from MoM is that in MoM the players only have a few items, while in DS this ammount can come close to or even over 50.
One more thing: is it possible to show only a part of the map at a time? One box of Dark Souls doesn't have enough tiles to make all of the scenario at once, so I divided it into areas. I think it will help the players a lot if they only see the tiles that should be on the table right now, so they are not confused about this.
I hope I described what I would like to do well. I tried to think not only about my campaign, but also what could be used by other people making a DS scenario in the future.
Thank you a lot for helping me with this:)

Almost all of this can be done without changing Valkyrie.
Try putting this in valkyrie_Data\StreamingAssets\content\MoM:
ds.zip
You should be able to turn it on and off in content selection and can edit the files to add whatever content you want.
Grid:
The snapping for MoM is a little odd because of the token/board size comparisons, so you might have issues with the editor snap function, but otherwise this should work just fine.  You can manually edit the position in your scenario ini files for now also.
Activation:
Sounds like one room would be one 'investigator phase' and you can progress to the next stage when the room is cleared.  I can't change the investigator phase text easily at the moment, but that can come as a future feature.  No monster activations will be used - this is easy just don't use spawns.
Hero selection:
You can add the classes as 'investigators'  as above some work would be required on my end to rename this.
Monsters:
OK so these will just be tokens, no problems there.
Items:
This can be done currently, but as you say the UI is not designed for 50 items.  As a board game adaptation my first reaction is that this is too much anyway, and should be limited.  When I do a new game type it wouldn't be hard to change the item selection UI, but coming up with a good mobile device interface for this might not be easy anyway.
Showing part of the map:
No problem, you can and and remove tiles whenever you want (in events).  Also note that a scenario in Valkyrie can be linked to other scenarios to form a campaign.  This can be easier to develop and manage.  Also it means that updating the campaign (getting a new version) will take effect when you change scenario, which can be important for bug fixes in long campaigns (though it means you have to maintain compatability on scenario changes).
Before starting on anything my end I would like to see how this goes, as I am sure there will be additional requirements.  I see not reason you can't do all of this now, just that some of the labelling will be wrong.

I did what you told me to (putting it in MoM folder, but the tiles don't appear. I looked through the tiles, tried to look at how the other content packs did it and change the files according to it, but the tiles still don't appear in the editor.
Btw. I enebled the content pack through "select content"

Sorry, typo (TileSide not Tile).
ds.zip


THIS PR IS FOR DEBUGGING Add node fix with new elk infra

Motivation
this PR is for debugging new add_node fix with the new ELK infra

bors try


Kernels 2021-03-31

Motivation for this change
Updates
Things done

[X] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[ ] NixOS
[ ] macOS
[X] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[X] Ensured that relevant documentation is up to date
[X] Fits CONTRIBUTING.md.

Waiting for hardened patch set

@ofborg test kernel-latest kernel-lts latestKernel.login kernel-latest-ath-user-regd latestKernel.hardened

@ofborg test kernel-latest kernel-lts latestKernel.login kernel-latest-ath-user-regd latestKernel.hardened

@ofborg test kernel-latest kernel-lts latestKernel.login latestKernel.hardened

5.10.27-hardened1 seems to work fine in my two vmware NixOS master machines.

Backport in 20.09 in b103839d00505bbaa9f9beff26e3d0f6a3abf6df...e67e6a0e8a269a2b4d2ecbaec779f05dece6e089


Better push service demo

Since you can't send data to GCM, we need to lookup the message on the push GET.
ref Chrome Data
Topic queues
Create a series of queues, one for each topic, that contains the messages.
Putting a message into one of these queues should trigger a push message to all the subscribers of a topic.
The messages might have a data structure like this:
{
  topic-name: 'The Topic Name',
  topic-tag: 'the-topic-tag',
  pushed: false,
  timestamp: <epochtime-number>
  payload: {
    icon: '/public/images/some-icon.png'
    title: 'The message title'
    body: 'The message body, a little more verbosity'
    tag: 'the-message-tag'
  }
}

Retrieving the message
We can send the subscriptionId and timestamp as a parameter to the push GET.
The subscriptionId can then be used to lookup the topics subscribed to, and from there, the timestamp can be used to get the latest message in a push-message-by-topics queue. Pretty crude, but that seems to be all that is available at the moment.
Demo
Make the demo message a separate parameter. If that exists, just get the canned demo message.
The push.create method will become a 'demo only' method. This won't be used in any real application.

I'm suspending this until some real way of identifying a message comes along, hoping for data payloads in Chrome.


Create fclive.nl

Email domain for students of Friesland College (frieslandcollege.nl)

Merged into https://github.com/mapbox/swot


OAuth2 support

Ebert has finished reviewing this Pull Request and has found:

2 possible new issues (including those that may have been commented here).

You can see more details about this review at https://ebertapp.io/github/Nebo15/annon.api/pulls/229.

Changes Unknown when pulling ef46bd979b8f81eb370c446d9586569ff170b15f on gmile:oauth2-support into ** on Nebo15:master**.

Changes Unknown when pulling ef46bd979b8f81eb370c446d9586569ff170b15f on gmile:oauth2-support into ** on Nebo15:master**.


[Frontend] Add global selection color & "text" attribute to buttons

[Frontend]

Add ::selection psuedoclass to globals, so when you select a piece of text, it looks like this instead of the default

Add "text" attribute to buttons, so now you can add text like this:  <Button colour={"blue"} text={"testing testing 1 2 3"} />

Which will return something like this:

^
|
Yeah I committed to the wrong branch. I reverted it, and you can ignore that.

I like the selection colour (although we'll need to wait for @TheModdedChicken's approval on that), but why add a required property to buttons when text can already be passed in the children prop?

The selection color seems good to me 👍

Interesting suggestions, but please make sure to update the stories for the changed component.
Also, I know it's not technically necessary, but to keep it consistent with all the other hex colours, could you please change the #fff to a #ffffff.

Ok, cooking up a commit right now.

why add a required property to buttons when text can already be passed in the children prop

Just seems a bit easier and more clear to me.

Side note, what needs work most? I'd like to help out as much as possible but it feels like I'm blindly trying to figure out what needs work

There's a list of the components/pages that need to be built in React on the initial release project. If you want to work on the backend, there isn't as much of a clear to-do list set out, but you can look at the Figma designs to get an idea of what the requirements would be.

Alrighty, added stories and changed it to #ffffff

How about we make Buttons require either children or text but not both? That way you don't need to pass empty strings to them when you want (for example) a grey button with an SVG icon inside it. I'll happily make the change

Alright, makes sense, but what if I just made the "text" optional? Would that solve the problem?

Alright, how's that?

Looks good to me!

Hey @lerichardson, would you be interested in joining our core team of contributors? You seem very keen

Sure!

If you have Discord, I'll add you to a channel we use for discussing things.

DM it to me, Litbelb#4112


feat(iam): inline policies as part of iam role

Signed-off-by: Ivan K<EMAIL_ADDRESS>
Description of your changes

Follow up https://github.com/crossplane-contrib/provider-aws/pull/938
Fixes #
I have:

[ ] Read and followed Crossplane's contribution process.
[ ] Run make reviewable test to ensure this PR is ready for review.

How has this code been tested

In this PR only an example is part of the commit ? Can you check the controller code ? ;)

I will change the code too. Just testing it.

I'l change the code too. Just trying to understand how to build and test it on live cluster

Will close this PR in favour https://github.com/crossplane-contrib/provider-aws/pull/938, as it looks like the PR owner is still working on it


Support for ARMv7

This modern 32bit arch can still be considered quite widely used in old Raspberry Pi and other cheap development boards.

@roshkhatri Can you take a look?


57 improve trace interface

Added Trace interface again. Note: Requires Trace Beans to be "prototype".

Thanks, Padi!  I think is now a much better solution than we had before.
I'll merge it in dev.

Cheers Mate.
Great. Thank you.
LG
Padi
From: gianlucafrei<EMAIL_ADDRESS>Sent: Sonntag, 28. Februar 2021 17:03
To: gianlucafrei/Application-Gateway<EMAIL_ADDRESS>Cc: Padi Steger (OWASP Switzerland<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [gianlucafrei/Application-Gateway] 57 improve trace interface (#58)
Thanks, Padi! I think is now a much better solution than we had before.
I'll merge it in dev.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub https://github.com/gianlucafrei/Application-Gateway/pull/58#issuecomment-787474785 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ASQQG6KSDWQYJ4MC5MRP32LTBJSKZANCNFSM4YK6HZUQ .


OCPBUGS-23306: Add annotation to CSI driver Pod preventing eviction from the cluster-autoscaler

/lgtm

/cherry-pick release-4.15

/cherry-pick release-4.14


Update README.md

Why:

What's being changed:

Check off the following:

[ ] All of the tests are passing.
[ ] I have reviewed my changes in staging.
[ ] For content changes, I have reviewed the localization checklist
[ ] For content changes, I have reviewed the Content style guide for GitHub Docs.

Thanks @vicky-hemnani! We don't use periods at the end of section headers, so I'm going to close this out. Next time, please fill out the pull request template in the top comment in the pull request, to help us better understand your changes.


Add tmux support

This commit adds tmux support simply by adding a elseif branch to detect the tmux filetype.
I guess this is only useful for editing tmux.conf.
Please take a look if this is good for this plugin.

File type supports are always welcome. Thanks for the contribution.


[Bug]: Python BatchElements uses minimum or arbitrary element timestamps for output timestamps with GlobalWindow specialization

What happened?
The python sdk BatchElements specializes the global window with _GlobalWindowsBatchingDoFn
This builds up batches and outputs them in two possible places:

during element processing if the batch is deemed complete, this uses the current element's timestamp for the output
during finish_bundle if there was an unfinished batch, this uses GlobalWindows.windowed_value which uses the minimum timestamp

It seems like this should be made to be consistent by either:

always outputtting with maximum timestamp. This matches the behavior of the non-global window case where the window end timestamp is used.
keep track of the minimum timestamp of elements in the input batch and use that for the output timestamp

Issue Priority
Priority: 2
Issue Component
Component: sdk-py-core

over to @TheNeuralBit to take a look or triage.

Interesting. I'm not sure which of these is the oversight, perhaps @robertwb can comment as the original author.
Looking at the code: https://github.com/apache/beam/blob/c7f64264451af12ff6c7c0ef4bc95fd7ce0f5418/sdks/python/apache_beam/transforms/util.py#L539-L552
In the process case we just yield the batch - likely this was just written ignoring the timestamp, but it turns out that Beam actually attaches the current element's timestamp in the OutputProcessor.
In the finish_bundle case we wrap in a GlobalWindows.windowed_value - likely this was just to satisfy some type check (static or in output processing), but it turns out we end up using min_timestamp().
As @scwhittle noted, the per window version always uses the current window's max timestamp: https://github.com/apache/beam/blob/c7f64264451af12ff6c7c0ef4bc95fd7ce0f5418/sdks/python/apache_beam/transforms/util.py#L584-L585
https://github.com/apache/beam/blob/c7f64264451af12ff6c7c0ef4bc95fd7ce0f5418/sdks/python/apache_beam/transforms/util.py#L602-L603
I took a look at what the GroupIntoBatches implementations do here, but AFAICT they're not opinionated either, and rely on the SDK worker to decide what timestamp to use: https://github.com/apache/beam/blob/c7f64264451af12ff6c7c0ef4bc95fd7ce0f5418/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupIntoBatches.java#L545
Presumably this is typically the most recent element's timestamp, when it completes a batch. I'm not sure about the case when a timer fires though.

It might be worth bringing this to the dev list to see if we have, or should establish, a convention for handling timestamps in transforms that produce batches of elements.


[Aliases] Aliased column name stored in wizard

Context
Some wizards have a column dropdown where users can select a column to apply style by. It's been observed that when the default chosen column is aliased, this alias is stored as a column name in the wizard, generating errors.
STR

Create a dataset subject to using a "Category" wizard
Make sure the first column (default for the wizard) is aliased
Choose a "Category" wizard for the dataset style

Expected result
The default column is the aliased column, and everything works fine.
Actual restul
The default column is the aliased column, but the value passed on is that one instead of the original column name.
SOW
Changes should be made so that original name is passed on from a dropdown selector even when the aliased value is shown.
Extra
This is the original report, containing some clues as to where the issue is located:
This issue occurs when a column with an alias is the default value for the form which picks columns for the category wizard.

As per [FormSchema#_fillCoulmns](https://github.com/bloomberg/cartodb/blob/blp_dev/lib/assets/javascripts/cartodb/models/wizard.js#L106), the `value` member within `FormSchema`, which indicates the default selection of that form field if nothing is previously selected, will be the alias of the first column in extra (if an alias exists).

This `value` will ultimately be used to populate the form, and will in turn be saved into a `WizardProperties` object.  This happens through the following code path when activated the categories widget after setting:
* [CartoCSSWizard#_onWizardClick](https://github.com/bloomberg/cartodb/blob/blp_dev/lib/assets/javascripts/cartodb/table/menu_modules/carto_wizard.js#L50)
* [WizardProperties#active](https://github.com/bloomberg/cartodb/blob/blp_dev/lib/assets/javascripts/cartodb/models/wizard.js#L447)
  * See [WizardProperties#defaultStyleForType](https://github.com/bloomberg/cartodb/blob/blp_dev/lib/assets/javascripts/cartodb/models/wizard.js#L391) for how `geomForm` is populated with the `property` field which is ultimately used to fill the form.

Note that an aliased column must appear first in the sorted `extra` list for the issue to occur.  If the category dropdown column is subsequently changed by the user, it appears the alias is overwritten and saved correctly.

This should correct the issue.  Please test before pushing to Bloomberg. I tried to test all possible use cases but may have missed one.  My testing showed no alias names in the wizard properties and the wizards them selves were showing alias names where applicable but using original names.

Thanks.
I have updated the branch with some new stuff from blp_dev and couldn't reproduce the problem even before this patch. Did you reproduce it on source before preparing the fix?


Reported a bug

Describe the bug
I got a warning that many columns  have mixed types and Specify dtype option on import or set low_memory=False.
The solution of setting low_memory = False wont solve the problem .
I have even attached the screenshot , u can see unnamed 3, 4 ,5 6....1262.
May be the problem is with the dataset or i dont know .. please look after this issue .
Expected behaviour
Based on the notebook in this repo , it should have print like this
data_credits columns:  Index(['movie_id', 'title', 'cast', 'crew'], dtype='object')
but i got like the below
data_credits columns:  Index(['movie_id', 'title', 'cast', 'Unnamed: 3', 'Unnamed: 4', 'Unnamed: 5',
'Unnamed: 6', 'Unnamed: 7', 'Unnamed: 8', 'Unnamed: 9',
...
'Unnamed: 1255', 'Unnamed: 1256', 'Unnamed: 1257', 'Unnamed: 1258',
'Unnamed: 1259', 'Unnamed: 1260', 'Unnamed: 1261', 'Unnamed: 1262',
'Unnamed: 1263', 'Unnamed: 1264'],
dtype='object', length=1265)
Desktop

OS: [e.g. Windows]
Browser [e.g. chrome]
Version [e.g. 10]

Additional context
If this is my mistake please specify the error i have made or please provide solution

@Abhilash1781 check Stack Overflow to find out why u r having these issues


Fix a message template to accept vim's default errorformat.

Hi
This PR fix a message template to accept vim's default error format.
I'm a vimmer, and use its quickfix feature.
This feature requires that error message formats of some tools(like pydocstyle) are in errorformat settings of vim.
It's default settings support %f:%l:%m, but current format of pydocstyle is %f:%l %m(=lack of : after line number).
Of course, we can change vim's errorformat settings, but I'm happy if I use default settings.
FYI: pycodestyle's format is in default errorformat settings.
https://github.com/PyCQA/pycodestyle/blob/master/pycodestyle.py#L107
Could you consider this change? Thanks!
=========
Check

[ ] Add unit tests and integration tests where applicable.
I'm sorry, but I can't find the best place to put tests checking error message format. Could you help it?

[ ] Add a line to the release notes (docs/release_notes.rst) under "Current Development Version".
Make sure to include the PR number after you open and get one.

@sigmavirus24
You are right. I'll try to create a config about message format.
Anyway, I close this PR.
Thank you for your comment!


shadow ioeventfd: add demo

I already merged this into multiple-shadow-ioeventfd but I should have waited until the destination branch had been merged into master and then merge.


Fix Testnet Network Identifier

As I see, "networkIdentifier":"15f0dacc1060e91818224a94286b13aa04279c640bd5d6f193182031d133df7c" for Testnet.
https://lisknode1-test.adamant.im/api/node/info
Is mainnet networkIdentifier correct?

Mainnet network ID will probably need update after the Mainnet migration :)


fix(firebase_messaging): Replace print() call to not log messages in release builds

Description
This PR removes print() to not write error logs in release builds of apps using Firebase Messaging plugin.
The replacement uses log() function which won't print anything into console in release builds and, additionally, allows to pass error as parameter.
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]).
This will ensure a smooth and quick review process. Updating the pubspec.yaml and changelogs is not required.

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] My PR includes unit or integration tests for all changed/updated/fixed behaviors (See Contributor Guide).
[x] All existing and new tests are passing.
[x] I updated/added relevant documentation (doc comments with ///).
[x] The analyzer (melos run analyze) does not report any problems on my PR.
[x] I read and followed the Flutter Style Guide.
[x] I signed the CLA.
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Does your PR require plugin users to manually update their apps to accommodate your change?

[ ] Yes, this is a breaking change.
[x] No, this is not a breaking change.

After internal discussion, we've decided not to proceed with this change. We're of the mind that this constitutes a breaking change, but more significantly, we believe that this log ought to print out in production so folks can diagnose problems.


Does it make sense to see similiar performance with/without 'es.nodes.client.only' set to 'true'  in spark? this related to #373

Hi @costin,
Thank you again for #373. The fix works great.
We have been trying to benchmark the performance with and without es.nodes.client.only set to true. Here comes the ES cluster settings:

There is around 27GB and 2,685,643 documents in this cluster. All of them have http enabled. We then started to use 1 spark worker node to dump entire data out with the following code:
https://gist.github.com/wingchen/24345b5dabf212a19ee6
We set/unset conf.set("es.nodes.client.only", "true") at line 11 to see the difference.
It turns out that the performance is not much different from each other.
With es.nodes.client.only set to true:
Elapsed time:<PHONE_NUMBER>18ns
Elapsed time:<PHONE_NUMBER>46ns
Elapsed time:<PHONE_NUMBER>22ns
Elapsed time:<PHONE_NUMBER>20ns
Elapsed time:<PHONE_NUMBER>28ns
Elapsed time:<PHONE_NUMBER>87ns
Elapsed time:<PHONE_NUMBER>27ns
Elapsed time:<PHONE_NUMBER>47ns

Without es.nodes.client.only set to true:
Elapsed time:<PHONE_NUMBER>49ns
Elapsed time:<PHONE_NUMBER>99ns
Elapsed time:<PHONE_NUMBER>15ns
Elapsed time:<PHONE_NUMBER>09ns
Elapsed time:<PHONE_NUMBER>15ns
Elapsed time:<PHONE_NUMBER>33ns
Elapsed time:<PHONE_NUMBER>96ns
Elapsed time:<PHONE_NUMBER>30ns

We tried this for couple more times with different ways to loop through all the docs and got similar result.
Does this make sense to you? Did we do anything wrong?

On such a small cluster and with this data size I would not expect the difference too be too large. ES can work just fine through one or multiple nodes; however as your cluster grows and the number of data nodes >> client nodes, especially for large indices you are likely to see the difference.
Note that in your case you are using only 1 thread so it's unlikely to reap the benefits of using data nodes - with 1 worker you basically serialize all access to ES; if you were to use 2 or more than the data node access could be paralelized and thus differentiate from the client one.

@costin Thank you for the response.
I am adding 2 more ES data nodes into the cluster. I will put the findings in this thread too.
Although we are only using 1 spark worker, it has 4 concurrent threads querying data from ES. This part should be fine though.

updating..
When the data nodes goes to 4:
Queries directly to nodes:
Elapsed time:<PHONE_NUMBER>77ns
Elapsed time:<PHONE_NUMBER>19ns
Elapsed time:<PHONE_NUMBER>57ns
Elapsed time:<PHONE_NUMBER>43ns
Elapsed time:<PHONE_NUMBER>60ns
Elapsed time:<PHONE_NUMBER>82ns
Elapsed time:<PHONE_NUMBER>35ns
Elapsed time:<PHONE_NUMBER>37ns
Queries to client:
Elapsed time:<PHONE_NUMBER>65ns
Elapsed time:<PHONE_NUMBER>28ns
Elapsed time:<PHONE_NUMBER>78ns
Elapsed time:<PHONE_NUMBER>18ns
Elapsed time:<PHONE_NUMBER>92ns
Elapsed time:<PHONE_NUMBER>47ns
Elapsed time:<PHONE_NUMBER>29ns
Elapsed time:<PHONE_NUMBER>03ns
When data nodes goes to 6:
Queries directly to nodes:
Elapsed time:<PHONE_NUMBER>49ns
Elapsed time:<PHONE_NUMBER>20ns
Elapsed time:<PHONE_NUMBER>17ns
Elapsed time:<PHONE_NUMBER>52ns
Elapsed time:<PHONE_NUMBER>07ns
Elapsed time:<PHONE_NUMBER>33ns
Elapsed time:<PHONE_NUMBER>40ns
Elapsed time:<PHONE_NUMBER>95ns
Queries to client:
Elapsed time:<PHONE_NUMBER>10ns
Elapsed time:<PHONE_NUMBER>26ns
Elapsed time:<PHONE_NUMBER>12ns
Elapsed time:<PHONE_NUMBER>34ns
Elapsed time:<PHONE_NUMBER>21ns
Elapsed time:<PHONE_NUMBER>29ns
Elapsed time:<PHONE_NUMBER>81ns
Elapsed time:<PHONE_NUMBER>44ns
@costin Like you said, the performance difference will gradually unfold as the number of data nodes grows. Also, I am increasing spark worker count to see what will happen there too.

@costin
I increased spark workers number to 4 (12 cores) and still see similar results (it's way faster now):
Queries directly to nodes:
Elapsed time:<PHONE_NUMBER>46ns
Elapsed time:<PHONE_NUMBER>78ns
Elapsed time:<PHONE_NUMBER>38ns
Elapsed time:<PHONE_NUMBER>76ns
Elapsed time:<PHONE_NUMBER>25ns
Elapsed time:<PHONE_NUMBER>53ns
Elapsed time:<PHONE_NUMBER>04ns
Elapsed time:<PHONE_NUMBER>74ns
Queries to client:
Elapsed time:<PHONE_NUMBER>77ns
Elapsed time:<PHONE_NUMBER>85ns
Elapsed time:<PHONE_NUMBER>46ns
Elapsed time:<PHONE_NUMBER>38ns
Elapsed time:<PHONE_NUMBER>45ns
Elapsed time:<PHONE_NUMBER>66ns
Elapsed time:<PHONE_NUMBER>33ns
Elapsed time:<PHONE_NUMBER>32ns
The numbers are too odd, so I went into newrelic to monitor the traffic of two different configurations: client only and otherwise.
It turns out that there is no significant amount of traffic going through client at all during the whole time:

However, the data nodes are transmitting data without any problem:

(Please note that I ran client-only and otherwise interchangeably during the 6 times of execution.)
In this case, it seems that no matter I set es.nodes.client.only or not, data is passing out through data nodes, not client. Can you confirm it?
The ES cluster configuration is here: https://gist.github.com/wingchen/fb332414d0e1e5a3b844 Please let me know if any of the configuration is preventing client nodes configuration from working.
Thanks

@wingchen Repicking up this thread.
By using the client nodes, the connector is simply re-routed; the client nodes forwards the request internally to the data node. So the traffic will occur on that node - and in case of queries, the results is simply returned.
Now if all the data is being returned and the connection doesn't happen through the client nodes than this indicates an issue.
The easiest ways to double check this is:

disable HTTP on data nodes and only leave it on client nodes. The connector won't be able to connect to the data nodes, even if it wants to.

2 turn on logging on the rest package and monitor the connections made. Additionally, log the traffic and what the IPs used.
Do note that if you a running client nodes on the same machines as the data nodes, the same network card and potentially IP will be used (but on a different port). And that's to be expected however it might be interpreted incorrectly when monitoring.

Closing this out as inactive


Add the GSI of Japan map tiles plugin

What's this?
This PR suggests you to add a new map tiles plugin which allows us to use The Geospatial Information Authority of Japan (GSI)'s map tiles as a base map layer of IITC.
They provide us with comprehensive and accurate contour maps of Japan. Their maps are considered highly reliable by professional users, outdoor enthusiasts and so forth. I believe this plugin is really useful for us when playing Ingress in rural and/or mountainous regions of Japan.
GSI's terms of use
According to the GSI's website, their terms of use are compatible with Creative Commons Attribution 4.0 International Public License.

The Terms of Use are compatible with the Creative Commons Attribution License 4.0 (hereinafter referred to as the CC License). This means that Content based on the Terms of Use may be used under the CC License in lieu of the Terms of Use.

Therefore, I think that it's legal to use their map tiles in IITC with the proper attribution.
Screenshot

Sorry, I withdraw this for now for some personal reasons.


fix: fix oci sum missing

1. Does this PR affect any open issues?(Y/N) and add issue references (e.g. "fix #123", "re #123".):

[ ] N
[ ] Y

2. What is the scope of this PR (e.g. component or file name):

3. Provide a description of the PR(e.g. more details, effects, motivations or doc link):

[ ] Affects user behaviors
[ ] Contains syntax changes
[ ] Contains variable changes
[ ] Contains experimental features
[ ] Performance regression: Consumes more CPU
[ ] Performance regression: Consumes more Memory
[ ] Other

4. Are there any breaking changes?(Y/N) and describe the breaking changes(e.g. more details, motivations or doc link):

[ ] N
[ ] Y

5. Are there test cases for these changes?(Y/N) select and add more details, references or doc links:

[ ] Unit test
[ ] Integration test
[ ] Benchmark (add benchmark stats below)
[ ] Manual test (add detailed scripts or steps below)
[ ] Other

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

2 of 3 (66.67%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.05%) to 43.332%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pkg/client/client.go
2
3
66.67%

Totals

Change from base Build<PHONE_NUMBER>:
0.05%

Covered Lines:
2466

Relevant Lines:
5691

💛  - Coveralls


Fix Application error: a client-side exception through change the input handler condition in <TradeClipboard />

https://user-images.githubusercontent.com/74899114/212817429-5b6cea74-4ca3-4629-92f4-6c7c7d856598.mov

There was a build error that made it to master, if you update from master, the previews should build

@jonator Hi, jonator. But I haven't any permission for contributions. So I can't push the commit to a remote branch.
Could you tell me how I can contribute to this repo?


Script Execution Issues

hi @andrefreitas ,
thanks for your script anyway.
kindly need your suggestion.
im quite struggling with your script.
i try by execute :
python matternag.py --url http://chat.abc.io/abc/channels/trial- --hostalias server1 --notificationtype Problem --serviceoutput "Service Apache Down!"
the last line output are :
`

`
since our mattermost server are working well, is there anything else i need to do ?
do i need to put my username and password inside the script ?
looking forward for your information.
many thanks
cheers,
henry

Hi @henryudha,
You only need to give the URL generated by Mattermost when adding an Incoming Webhook (it's an URL with a token).
Let's run a checklist (they work great). Please answer y/n for the following points:

Did you create an Incoming Webhook following the instructions at http://docs.mattermost.com/developer/webhooks-incoming.html#enabling-incoming-webhooks ?

Did you tested the generated URL with a token with the following command?
 ./mattermost.py --url [URL]  --hostalias "$HOSTNAME$" --notificationtype "$NOTIFICATIONTYPE$" --servicedesc "$SERVICEDESC$" --servicestate "$SERVICESTATE$" --serviceoutput "$SERVICEOUTPUT$"

hi @andrefreitas
thanks for your reply,

No. i just create the webhook url.

then retry to execute with the webhook url, and its work.

you just awesome mate!
once again thanks for your help and reply
cheers,
henry

@henryudha you are welcome, glad to see I could help :)


~/.nerves disappears when Docker container restarts

~/.nerves contains archives of Nerves toolchains and nerves_system_br.
It will disappear when Docker container restarts or rebuilds. As a result, archives should be always downloaded on mix deps.get.

Same situation occurs for ~/.ssh/id_rsa. We can resolve this situation by creating it on current (development) directory and making symbolic link to /root/.ssh on the Docker container.

791c15d on #2 may fixes this issue.
mix deps.get at first will take a while to unzip archives to ./nerves/artifacts due to poor performance about bindings between host and container filesystems. However I think it is acceptable compared to the usual wait when running mix deps.get.

Same situation occurs for ~/.ssh/id_rsa. We can resolve this situation by creating it on current (development) directory and making symbolic link to /root/.ssh on the Docker container.
mkdir .ssh
ssh-keygen -t rsa -f .ssh/id_rsa
ln -s `pwd`/.ssh/id_rsa* ~/.ssh/

ce64a39 on #2 creates symbolic link from /workspaces/nerves-devcontainer/.ssh/id_rsa* on host to /root/.ssh on container.
Users are required to run ssh-keygen -t rsa -N "" -f .ssh/id_rsa at first to generate keys.

fixed by merging #2


#0: Rebase the tracy submodule

Ticket
Problem description
Rebasing to the latest on actual tracy module
Checklist

[ ] Post commit CI passes
[ ] Device Regression
[ ] T3K profiler

Is this PR still alive?


Workaround solution. Too easy. Instead of using splice and slice it is easier to substitute with shift()

Challenge Slasher Flick has an issue.
User Agent is: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.81 Safari/537.36.
Please describe how to reproduce this issue, and include links to screenshots if possible.
removed solution by mod

@leetboy
Please do not post your solution on here. GitHub Issues are for reporting bugs on the website only.
Happy Coding!


Next.js 12 - target config deprecated

Getting a warning when building with next.js 12:
warn  - The `target` config is deprecated and will be removed in a future version.
See more info here https://nextjs.org/docs/messages/deprecated-target-config

Those docs suggest refactoring to use output file tracing (don't know much about it but sounds cool!): https://nextjs.org/docs/advanced-features/output-file-tracing

We are transition this repo to https://github.com/sst/v2 and are closing this issue for now. If it is still relevant please reopen it there - sorry for the trouble


PDOException thrown whenever SIGALARM occurs during a query

+## PHP Driver version or file name
+## SQL Server version
SQL2016
+## Client operating system
CentOS 7
+## PHP version
PHP 7.2.11
+## Microsoft ODBC Driver version
5.3.0
+## Problem description
When you use PCNTL to protect execution time of a worker like with Laravel's Job Workers and a signal arrives (for instance SIGALRM) queries currently running will throw PDOExceptions:
PHP Fatal error:  Uncaught PDOException: SQLSTATE[08S01]: [Microsoft][ODBC Driver 17 for SQL Server]TCP Provider: Error code 0x2714 in test2.php:14

This prevents the alarm handler from actually being called.
+## Expected behavior and actual behavior
I think I understand why this throws an exception. The whole call is ending up in an inconsistent state as you don't know what will happen in the signal handler function. It doesn't make sense to think the current query will successfully finish even if the signal handler doesn't do anything bad and yields control quickly enough.
I'm not entirely sure what should happen besides that I would expect the signal handler to be called; perhaps before the exception is thrown.
+## Repro code or steps to reproduce
Minimized example, obviously not real life code :)
$pdo = new PDO('sqlsrv:Server=;Database=', 'user', 'password');
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);

pcntl_async_signals(true);
pcntl_signal( SIGALRM, function () { echo "I should shutdown"; });
pcntl_alarm(2);

while(1) {
        $testresult = $pdo->query("SELECT * FROM users;");
        foreach($testresult as $r) {}
}

@mathieuk Thank you for this info - we will test out the repro you provided and get backto you shortly.

Ok we can reproduce this behaviour. Do you see something similar happening with sqlsrv?

It yields similar, yet subtly different, behaviour:
<?php

$serverName = '';
$connectionOptions = [];

//Establishes the connection
$conn = sqlsrv_connect($serverName, $connectionOptions);

pcntl_async_signals(true);
pcntl_signal( SIGALRM, function () { echo "I should shutdown"; });
pcntl_alarm(2);

$errors = 0;

while(1) {
        echo "=> Starting query\n";
        $testresult = sqlsrv_query($conn,"SELECT * FROM users;");
        echo "\t - Query done\n";

        if (!$testresult) {
                $errorMsgs = sqlsrv_errors(SQLSRV_ERR_ALL);

                foreach ($errorMsgs as $error) {
                        echo "\n", $error['message'], "\n";
                }

                if ($errors++ > 5) exit;

                continue;
        }

        echo "\t - Fetching data\n";
        $rowCount = 0;

        while ($r = sqlsrv_fetch_array($testresult)) { /* echo "\t - Got row #$rowCount\n"; */ $rowCount++; }

        echo "<= Query complete\n";
}

Results in a few variations of:
/* ... X succesfull calls snipped for brevity ... */

=> Starting query
	 - Query done
	 - Fetching data
<= Query complete
=> Starting query
I should shutdown	 - Query done

[Microsoft][ODBC Driver 17 for SQL Server]TCP Provider: Error code 0x2714

[Microsoft][ODBC Driver 17 for SQL Server]Communication link failure
=> Starting query
	 - Query done

[Microsoft][ODBC Driver 17 for SQL Server]The connection is broken and recovery is not possible. The connection is marked by the client driver as unrecoverable. No attempt was made to restore the connection.
=> Starting query
	 - Query done

[Microsoft][ODBC Driver 17 for SQL Server]The connection is broken and recovery is not possible. The connection is marked by the client driver as unrecoverable. No attempt was made to restore the connection.
=> Starting query
	 - Query done

[Microsoft][ODBC Driver 17 for SQL Server]The connection is broken and recovery is not possible. The connection is marked by the client driver as unrecoverable. No attempt was made to restore the connection.
=> Starting query
	 - Query done

A few remarks/observations:

This only seems to occur whenever SIGALARM arrives when we're in the query execution. Haven't seen anything (yet) indicating similar behaviour during data fetching.

With SQLSRV the signal handler seems to always be called. I haven't seen errors happening before the signal handler was called and I always got the output specified in the signal handler. I've tried > 20 times and I did see that happening in the same amount of tries with PDO. SQLSRV doesn't throw exceptions so the behaviour I'm seeing with PDO may well be caused by the way pcntl + exceptions in PHP work. Perhaps the extension can yield to the signal handler when it's interrupted, before throwing the exception? This may be of interest: https://github.com/php/php-src/pull/2902

The initial error messages aren't always the same. Sometimes it's Error code 0x2714; other times it's SMux Provider: Physical connection is not usable [xFFFFFFFF].

I understand the query not completing successfully but I didn't expect the connection to fail completely here.

Just to be complete, there are instances where the signal arrives and is handled succesfully. Presumably we're not doing any queries at that time :)

HTH

@mathieuk please check your unixodbc version odbcinst -j. I can't seem to reproduce this with unixodbc 2.3.6

@mathieuk as you have pointed out, whenever SIGALRM arrives, the connection is broken, as indicated by SQLSTATE 08S01 as well as all the error messages you have listed in the sqlsrv example.
When it fails to connect, the query fails to execute, so an exception is thrown. As you said, there is no exception after the signal handler is done and yields control. This is apparent when we keep trying to run the same php script again and again.
Maybe it will help us understand the problem if you can provide more details?

The first thing is about the signal handler. When SIGALRM comes in the connection is often immediately broken and an exception is thrown before the alarm handler can be called. Even if the exception is handled, the alarm handler is never called. This is confusing for the surrounding code (yay semi-concurrency).
Things would be clearer if the signal handler would still be called; before the exception.
Then the second thing is that the connection is completely terminated from the second SIGALRM comes in. That strikes me as odd; it seems to me that yes, the query may be interrupted and that might be unrecoverable for that query but it seems odd that the connection is completely broken at that point. Would it be possible to recover and issue new queries on the connection?

I see your points @mathieuk
We will investigate the causes of broken connections and get back to you on this.

@mathieuk For what it's worth, I have experienced the same symptoms (random exceptions of 0x2714) when using Laravel's Horizon (Job Worker) stack. I cannot confirm it is specifically SIGALRM (or any other signal), but something is definitely at odds with the PDO SQLSrv connection inside a Laravel Worker.
If I run the same code outside a worker, it works as expected and without any TCP errors. This is running with 5.5.0-preview+11606, unixodbc/xenial,now 2.3.1-4.1, php 7.2.

@mathieuk and @ralphschindler
After some investigation, disabling MARS seems to have reduced the likelihood of broken connections. To disable MARS, which is true by default, you need to set it to false, as shown below:
$pdo = new PDO('sqlsrv:Server=yourServer; Database=yourDB; MultipleActiveResultSets=false', 'yourUser', 'yourPassword');

@mathieuk: we are still investigating, and please shed some lights on how to reproduce this issue. If I see "I should shutdown" when running in a command terminal, I leave it running and launch another terminal to run the same php script. Yet @david-puglielli did it by canceling (CTRL+C) and re-ran the script in the same terminal window.

@yitam unfortunately, adding that setting does not reduce the likelihood of receiving SQLSTATE[08S01]: [Microsoft][ODBC Driver 17 for SQL Server]TCP Provider: Error code 0x2714. if you have any further fixes, I am more than willing to try. Thanks!

@ralphschindler when I tested with ODBC driver 17.2 the problem frequently occurred indeed but with an internal build (post CTP preview) it does seem to have improved. In the meantime, your patience is appreciated while odbc team is looking into this.

@yitam
I've updated my repo script to hopefully make things a bit clearer. It's attached below (test-pdo.php). When you run it it will eventually run into the error situation, every time you run the script.
The repro script simulates a typical worker daemon like you'd see in Laravel.
test-pdo.php
<?php

$pdo = new PDO('sqlsrv:Server=;Database=', '', '');
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);

function handleAlarm() {
        echo "Received SIGALRM!";
        pcntl_alarm(2);
}

pcntl_async_signals(true);
pcntl_signal( SIGALRM, 'handleAlarm');
pcntl_alarm(2);

while(1) {
        $testresult = $pdo->query("SELECT * FROM users;");
        foreach($testresult as $r) {}
        echo '.';
}

FWIW, it differs in that a worker in Laravel would set the alarm at the beginning of each loop instead of whenever the alarm went off. That way the last alarm would be replaced with a new one so it doesnt go off unnecessarily. In my case, I -want- it to go off unnecessarily as I want to see the effect on queries.
When I run this I see two different outcomes:
$ php test-pdo.php
..........................................................................................................................................................................................................................................................................Received SIGALRM!.........................................................................................................................................................................................................................................................................Received SIGALRM!..................................................................................................................................................................................................................................................................PHP Fatal error:  Uncaught PDOException: SQLSTATE[08S01]: [Microsoft][ODBC Driver 17 for SQL Server]TCP Provider: Error code 0x2714 in test-pdo.php:16
Stack trace:
#0 test-pdo.php(16): PDO->query('SELECT * FROM u...')
#1 {main}
  thrown in test-pdo.php on line 16

In this case the alarm was successfully handled 2 times (read: we were not executing the query at that time) and then the third time it was executing the query and the exception was thrown. Note how the message "Received SIGALRM!" is absent that time.
This is the worst case as now I wasn't able to clean-up after something went wrong. This can result in 'run-away' workers in the case of Laravel workers. In my case it also messes up some job housekeeping that I have to do.
If I run the script a few times I'll also run into this variant:
$ php test-pdo.php
.........................................................................................................................................................................................................................................................................Received SIGALRM!PHP Fatal error:  Uncaught PDOException: SQLSTATE[08S01]: [Microsoft][ODBC Driver 17 for SQL Server]Session Provider: Physical connection is not usable [xFFFFFFFF].  in test-pdo.php:17
Stack trace:
#0 {main}

Next PDOException: SQLSTATE[08S01]: [Microsoft][ODBC Driver 17 for SQL Server]Session Provider: Physical connection is not usable [xFFFFFFFF].  in test-pdo.php:17
Stack trace:
#0 {main}

Next PDOException: SQLSTATE[08S01]: [Microsoft][ODBC Driver 17 for SQL Server]Session Provider: Physical connection is not usable [xFFFFFFFF].  in test-pdo.php:17
Stack trace:
#0 {main}

Next PDOException: SQLSTATE[08S01]: [Microsoft][ODBC Driver 17 for SQL Server]Session Provider: Physical connection is not usable [xFFFFFFFF].  in test-pdo.php:17
Stack trace:
#0 {main}

Next PDOException: SQLSTATE[08S01]: [Microsoft][ODBC Driver 17 for SQL Server]Session Provider: Physical connection is not usable [xFFFFFFFF].  in test-pdo.php:17
Stack trace:
#0 {main}

Next  in test-pdo.php on line 17

In this case the signal handler WAS called but then the connection was completely broken.
Does this help?

Like @ralphschindler, I did not see any difference in behaviour when using MultipleActiveResultSets=false.

Thanks @mathieuk for a more detailed repro scenario. It certainly helps with the investigation.

I did not see any difference in behaviour when using MultipleActiveResultSets=false.

I should have mentioned that this makes a small difference but only with the internal ODBC build. Thanks for trying nonetheless.

@yitam do you feel comfortable making a PR to the php project, so shall I to usher this along?

hi @ralphschindler thanks for your reminder. Yes maybe I should make a pull request.

👍 - let me know if you need any assistance :)

Thanks @ralphschindler ! Done, please check PR 3717

@ralphschindler

let me know if you need any assistance :)

Yes, can you think of a test case without using our PDO driver?

@yitam I've tried the change shown in https://github.com/php/php-src/pull/3717 and I have the basis for a test that you could use.
For this comment I'm using http://man7.org/linux/man-pages/man7/signal.7.html as a reference.
So I think what you're saying is that SIGALRM is triggering a EINTR return value for a read() or write() (or somesuch) call in the ODBC driver which causes it to return an error which in turn causes PDO to throw an exception. By enabling SA_RESTART that read() call would be restarted right after the signal handler returns and no bad things happen.
It seemed to me that flock() was one of the easier 'primitives' to test this with without this extension. What I'm seeing is that indeed: without your change the behaviour is always as if argument 3 'restart_syscalls' to pcntl_signal() is false. This means that the function call is interrupted, my signal handler is called, and then the function call returns with an error. In the case of flock() that's just a returned false value.
With your change, the function call is interrupted but it appears my signal handler is never called. This means that, while there were no exceptions from PDO_SQLSRV, other functionality may now be broken. If the restarted read() in the ODBC driver never returns (because of a hung connection?) my process will be hang.
That behaviour seems to come from pcntl actually queuing signal handler functions to be called by pcntl_signal_dispatch which might be called by the pcntl_interrupt_function (which overrides the global zend_interrupt_function) which became available in PHP 7.1. The actual signal handler would then be called when control is handed back to the Zend VM: it'd detect an interrupt was requested (EG(zend_interrupt)==1) and it'd call the zend_interrupt_function (pcntl_interrupt_function) which calls pcntl_signal_dispatch which calls the actual signal handler etc etc.
However, with SA_RESTART enabled, as soon as the signal handler returns (that is: when ext/pcntl/pcntl.c pcntl_signal_handler returns) the syscall that was interrupted is restarted automatically. That is: before PHP has had the chance to yield to the Zend VM so that it could call the actual signal handler. It then just hangs in flock() forever.
So now we have two issues. One might say the PCTNL way of calling signal handlers is flawed as they get called outside of the actual signal handler rendering SA_RESTART pretty useless? OTOH one might say the ODBC driver should just handle EINTR properly (retrying for EINTR instead of erroring out).
What do you think ?

Thanks @mathieuk for your detailed test scenario and script. I haven't had time to try it out yet, but as you can see in my earlier reply, there wasn't any issue using signal / alarm in the native ODBC program.

What I'm seeing is that indeed: without your change the behaviour is always as if argument 3 'restart_syscalls' to pcntl_signal() is false.

Yes that is exactly the case. My patch was merely an attempt to show that PCTNL way of calling signal handlers is flawed. The third argument does not make any difference with SIGALRM. For this reason we quoted the pgsql fix to justify why we should consider the following:
Use SA_RESTART for all signals, including SIGALRM, in which it said (which you might have already noticed):
Nowadays, allowing it to interrupt kernel calls doesn't seem like a very good idea, since its
use for statement_timeout means SIGALRM could occur anyplace in the code, and there are
far too many call sites where we aren't prepared to deal with EINTR failures.

According to the above, I was tempted to make a drastic change like this (which I haven't tested):
-	if (signo == SIGALRM || (! restart)) {
+	if (! restart) {
 #ifdef SA_INTERRUPT
 		act.sa_flags |= SA_INTERRUPT; /* SunOS */
 #endif

That being said, to make the 3rd argument 'restart_syscalls' to pcntl_signal() false by default, that is, changing the API, might break others' existing scripts.
As soon as I have time I will try your test case and/or continue some more testing.

I also tried with async_signals=false and using declare(ticks=1) to see if using the tick function made any difference, but it doesn't. This is basically happening practically 'outside' of PHP's control.
The bigger fix indeed seems more appropriate but I totally understand your hesitation. I think it should be less of an issue when the argument restart_syscalls to pcntl_signal() is changed to be default false. That seems like an API breaking change, but it is in fact keeping the expected behaviour for existing scripts (that are not providing restart_syscalls=true) in tact.
That leaves us with PCNTL potentially not being able to (timely) call the userland signal handler if the primitive used is taking a long time or just blocks forever when SA_RESTART is enabled. I'm not sure what options there are for changing/improving that.
I think we should either update your bug report to include the above; or maybe report a new one regarding the things I described above?

hi @mathieuk

Here is the flock-based test script I used:

I checked your script and not exactly sure what you're trying to do:

without the alarm, it's just an infinite while loop?
with the alarm, it locks the text file and exits right away.

I modified your script slightly as follows,
if ($wantAlarm) {
        echo "Yes, alarm please\n";
        pcntl_async_signals(true);
        pcntl_signal( SIGALRM, function () { echo "ALARM! "; pcntl_alarm(1); });
        pcntl_alarm(3);
}

$fp = fopen('lock.txt', 'r');
echo "Locking" . PHP_EOL;

$result = flock($fp, LOCK_EX);
var_dump($result);

while(true) { sleep(1); }

And I got the following output:
Yes, alarm please
Locking
bool(true)
ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ALARM! ^C

Makes no difference whether I made the new change or not:
        act.sa_flags = 0;
#ifdef HAVE_STRUCT_SIGINFO_T
        act.sa_flags |= SA_SIGINFO;
#endif
        if (! restart) {
        // if (signo == SIGALRM && (! restart)) {
#ifdef SA_INTERRUPT
                act.sa_flags |= SA_INTERRUPT; /* SunOS */
#endif
        } else {

@yitam you had to run it twice for that to work. I've created a new, single, script that tests 3 scenarios. It builds on the fact that the signal handler actually doesn't get called with SA_RESTART and flock(). You can find it here: https://gist.github.com/mathieuk/9495b76da54ead09297deb62915d4f46
Output is as follows:
// PHP without @yitam's patch:
$ php fork.php
Runnning 'Without SA_RESTART, without alarm, interrupted flock fails' .... SUCCESS
Runnning 'Without SA_RESTART, with alarm, signal handler gets called' .... SUCCESS
Runnning 'With SA_RESTART: interrupted flock fails to call signal handler and gets terminated' .... FAILED! Got exitcode 3 (EXITCODE_ALARM_HANDLED) instead of 0 (EXITCODE_PARENT_INTERVEINED)

// PHP with @yitam's patch:
$ ~/php-7.2.13/sapi/cli/php fork.php
Runnning 'Without SA_RESTART, without alarm, interrupted flock fails' .... SUCCESS
Runnning 'Without SA_RESTART, with alarm, signal handler gets called' .... SUCCESS
Runnning 'With SA_RESTART: interrupted flock fails to call signal handler and gets terminated' .... SUCCESS

@yitam judging by the comments of Nikic Popov it appears that PCNTL won't be able to work in such a way that SA_RESTART will be actually useful in my initial stated problem. With the SA_RESTART flag I cannot trust on my signal handler being called. In that situation not using SA_RESTART would actually be better as I can atleast catch the exception, reconnect and do my housekeeping then. It's not pretty but it works.
It seems that while PGSQL (like you referenced) uses SA_RESTART MySQL actually fully implemented EINTR. Would it be possible to post handling EINTR as a feature request with the ODBC team?

Tested the fix (see PHP issue 3742 for the detailed discussions) in PHP 7.4.0RC4 today.
No more exception thrown from ODBC driver if the optional argument $restart_syscalls is set to TRUE. That is, change
pcntl_signal(SIGALRM, 'handleAlarm');
to
pcntl_signal(SIGALRM, 'handleAlarm', true);
Thus, I'm closing this issue now.


[Bug] Typo in the repo description

🐛 Bug
I think there's a typo in the repo description?

Libra’s mission is to enable a simple global currency and financial infrastructure that empowers billions of people with billions.

Hi. I can confirm and reproduce this bug. I recommend fix by destroying the entire system and starting over.

We're here to empower billions of people :)


Mac keeps asking for sudo password after uninstalling the utility

What is the issue? (required)
My mac keeps asking for sudo password even after I uninstalled the app. I know it's asking the password because of Battery since I've opened Activity Monitor and I checked which files were open in the bash process that shows the sudo password prompt. It shows "/usr/local/bin/battery" but the file has been removed.
What exactly did you do to produce the issue? (required)
I uninstalled it through the command "battery uninstall". It didn't work so I tried reinstalling it and uninstalling it through brew. Didn't work either.

Can you try running: pkill -f "/usr/local/bin/battery.*"?

Thank you, it worked

Thanks for confirming. I added an auto-kill to the uninstall script, so it should not happen again.


EMNIST dataset error

When I want to set the dataset 'EMNIST' for federated learning, I found that RuntimeError: ECONNRESET: connection reset by peer.
What I changed is just the dataset name and log dir location.
Thanks .
os
MAC book m1pro
python version
3.9.10

The torch.distributed.rpc module only supports the Linux operating system, so you must ensure that you run the relevant code on the Linux operating system, otherwise an error will be reported (see GitHub issues: torch.distributed.rpc).

Haha, that's really weird.


feat(derive): refactor out online providers

feat(derive): refactor out online providers
feat(derive): refactor out online providers

#569  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @refcell and the rest of your teammates on  Graphite


Loading the modified ucs on a big-ip system fails

Describe the bug
Running the command "load sys ucs CPE-7-20-2022_modified.ucs no-license"
Produces error: "2022 Jul 20 09:58:33 ip-10-1-1-12.us-west-2.compute.internal load_config_files[13026]: "/usr/bin/tmsh -n -g -a load sys config partitions all " - failed. -- 0107134a:3: File object by name (/Common/DoD_CAs_12_7_2021) is missing.
WARNING: There were one or more errors detected during installation.
Check the error messages and take the proper actions if needed.
ERROR: UCS installation failed.
Operation aborted.
Unexpected Error: UCS loading process failed."
To Reproduce
Steps to reproduce the behavior:
On step 4 on the target system produces the above error.
Expected behavior
The ucs should load.
Screenshots
No Virtual servers are loaded because the ucs restore fails.

Desktop (please complete the following information):

OS: Centos 7 for docker, Windows 10 for ssh ucs restore
Browser Firefox
Version 102.0.1

Additional context
I have the .ucs if you want to test it.

Internal: SOLUTIONS-7296

fixed with v1.0.3


rocksdb: add enableLiburing option

Description of changes
Rocksdb added support for io-uring in v6.7.0. This PR adds an enableLiburing option to the package to enable this.
In the process, I discovered that the liburing package is broken in static stdenv, and outputs both static and dynamic libraries unconditionally. This causes rocksdb to link against the static liburing when built in a dynamic stdenv. In conduwuit, we are using an overlay to add liburing support to rocksdb and fix the static library issues with liburing, but would like to be able to just use upstream nixpkgs.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

@Benjamin-L can you rebase this branch?

Please fix the conflict.

@PedroHLC Rebased onto current staging.

db863ae6a444 liburing: don't emit extra static libraries

I'm not opposed in principle, but this breaks ceph build (verified by testing build with that single change reverted).  Example log: https://hydra.nixos.org/build/265092568/nixlog/2/tail

db863ae liburing: don't emit extra static libraries

I'm not opposed in principle, but this breaks ceph build (verified by testing build with that single change reverted). Example log: https://hydra.nixos.org/build/265092568/nixlog/2/tail

Hummm, that smells like something to be fixed in Ceph and not here. It looks like it's using either the static with the shared library stdenv's output or the inverse...

In principle, static linkage isn't all-or-nothing situation.


Set up bumpversion

Enables us to use bumpversion minor to lower the risk of missing a version var. :relaxed:

Current coverage is 99.22%

Merging #582 into master will not affect coverage as of 4bb9c08

@@            master    #582   diff @@
======================================
  Files           12      12
  Stmts          514     514
  Branches         0       0
  Methods          0       0
======================================
  Hit            510     510
  Partial          0       0
  Missed           4       4

Review entire Coverage Diff as of 4bb9c08

Uncovered Suggestions

+0.39% via ...ookiecutter/utils.py#27...28
+0.39% via ...okiecutter/config.py#19...20

Powered by Codecov. Updated on successful CI builds.


difference between my manual tests and "basic tests" with analogous input

project: ft_printf, basic tests
my ft_printf behaves the same as original in this case
0179 (unsigned int)
ft_printf("%u", -1);

(   20) -->18446744073709551615<--
(   10) -->4294967295<--

Hello
can you confirm that using this file to compile your ft_printf within a program and using the following arguments (see below), you obtain the expected behavior?
./a.out u -1


Now the list of import types in --help is generated from the plugins.

Now the list of import formats shown in --help is generated from the list of plugins rather than being hard-coded.

Merged manually - thanks!