Errors when trying to pruning a YOLOv5 model, and use ModelSpeedup.speedup_model() to export to one model pt.

Environment:

NNI version:2.1
NNI mode (local|remote|pai):local
Client OS:
Server OS (for remote mode only):
Python version:3.7.5
PyTorch/TensorFlow version:PyTorch 1.7.1
Is conda/virtualenv/venv used?:no
Is running in Docker?:no

When I tried to use prunner do the yolov5 model compression (FPGMPruner used), the mask.pt and model.pt were generated.
Then, trying to use the ModelSpeedup.speedup_model() to export the two pt( mask.pt and model.pt ) in one model pt.
Q1. error log shows that the 'Concat' node not support?
log:
  File "detect_nni.py", line 447, in <module>
    detect()
  File "detect_nni.py", line 279, in detect
    m_speedup.speedup_model()
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/nni/compression/pytorch/speedup/compressor.py", line 183, in speedup_model
    self.infer_modules_masks()
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/nni/compression/pytorch/speedup/compressor.py", line 140, in infer_modules_masks
    self.infer_module_mask(module_name, None, mask=mask)
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/nni/compression/pytorch/speedup/compressor.py", line 124, in infer_module_mask
    self.infer_module_mask(_module_name, module_name, in_shape=output_cmask)
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/nni/compression/pytorch/speedup/compressor.py", line 124, in infer_module_mask
    self.infer_module_mask(_module_name, module_name, in_shape=output_cmask)
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/nni/compression/pytorch/speedup/compressor.py", line 124, in infer_module_mask
    self.infer_module_mask(_module_name, module_name, in_shape=output_cmask)
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/nni/compression/pytorch/speedup/compressor.py", line 92, in infer_module_mask
    .format(m_type, module_name))
RuntimeError: Has not supported infering output shape from input shape for module/function: `Concat`, model.16

Q2. I have tried to modify the source code in nni, to deal with the 'Concat' as 'atten::cat', then  errorlog showed that 'Upsample' node not support, which I tried deal as 'MaxPool2d' node. At last, some error showed in the 'view' node fuction ('view_inshape'), seems that it only support  output shape size as 2.(assert len(shape['out_shape']) == 2) But in the 'Detect' fuction in yolov5, the tensor were viewed as 3,5 or 6 dimentions.(https://github.com/ultralytics/yolov5/blob/238583b7d5c19029920d56c417c406c829569c75/models/yolo.py#L24).I am wondering is there any better way to fix this?
Above all, I am trying to find a solution to export the two pt( mask.pt and model.pt ) in one model pt, is there any better way?
Many thanks for your answering～

Q1:
In yolov5, you should add Concat, upsample and SiLU to infer_from_inshape and replace_module.
In Cancat node, you should modify the graph_utils.py and compressor.py
Q2:
I recommend that you pruning the yolov5 without "Detect layer".
After you producing the speedup model, you add Detect layer after the  speedup model.

@ichejun
Q1:
In yolov5, you should add Concat, upsample and SiLU to infer_from_inshape and replace_module.
In Cancat node, you should modify the graph_utils.py and compressor.py
Q2:
I recommend that you pruning the yolov5 without "Detect layer".
After you producing the speedup model, you should add Detect layer after the speedup model.

Thanks for your reply. I am going to modify the files as soon as possible.
As for the Q2, I am wondering if there is any recomended way to ‘pruning the yolov5 without "Detect layer"’？Could you please provide some suggestions?
Thank you very much!

Hi, @ping-Huang
After I completed the modification of 6 files and adjusted the prune configuration of "Detect layer", I was able to quantify the yolov5 model successfully.Thanks!
For related modifications, please refer to https://github.com/ichejun/nni/commits/master. The 6 commits on "Commits on Apr 25, 2021" are adjustments to the corresponding 6 files.
However, after the adjustment, I encountered 3 questions and troubles:
Q1. After completing the adjustment and modification of the above 6 files, although the yolov5 model can be successfully compressed and speed_up. But when I try to run the test/ut/sdk/test_pruners.py unit test from nni, I get an error, the error log is as follows,
/usr/local/lib/python3.7/site-packages/scipy/fft/__init__.py:97: DeprecationWarning: The module numpy.dual is deprecated.  Instead of using dual, use the functions directly from numpy or scipy.
  from numpy.dual import register_func
/usr/local/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
  return f(*args, **kwds)
/usr/local/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
  return f(*args, **kwds)
/usr/local/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
  return f(*args, **kwds)
/usr/local/lib/python3.7/site-packages/scipy/special/orthogonal.py:81: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  from numpy import (exp, inf, pi, sqrt, floor, sin, cos, around, int,
/usr/local/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
  return f(*args, **kwds)
..EE
======================================================================
ERROR: test_pruners (__main__.PrunerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_pruners.py", line 294, in test_pruners
    pruners_test(bias=True)
  File "test_pruners.py", line 235, in pruners_test
    pruner.compress()
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/nni/algorithms/compression/pytorch/pruning/auto_compress_pruner.py", line 225, in compress
    m_speedup = ModelSpeedup(self._model_to_prune, self._dummy_input, masks_file, device)
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/nni/compression/pytorch/speedup/compressor.py", line 38, in __init__
    self.torch_graph = build_module_graph(model, dummy_input)
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/nni/common/graph_utils.py", line 24, in build_module_graph
    return TorchModuleGraph(model, dummy_input)
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/nni/common/graph_utils.py", line 271, in __init__
    super().__init__(model, dummy_input, traced_model)
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/nni/common/graph_utils.py", line 66, in __init__
    self._trace(model, dummy_input)
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/nni/common/graph_utils.py", line 75, in _trace
    print(model.model[-1].export)
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 779, in __getattr__
    type(self).__name__, name))
torch.nn.modules.module.ModuleAttributeError: 'Model' object has no attribute 'model'

======================================================================
ERROR: test_pruners_no_bias (__main__.PrunerTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_pruners.py", line 297, in test_pruners_no_bias
    pruners_test(bias=False)
  File "test_pruners.py", line 235, in pruners_test
    pruner.compress()
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/nni/algorithms/compression/pytorch/pruning/auto_compress_pruner.py", line 225, in compress
    m_speedup = ModelSpeedup(self._model_to_prune, self._dummy_input, masks_file, device)
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/nni/compression/pytorch/speedup/compressor.py", line 38, in __init__
    self.torch_graph = build_module_graph(model, dummy_input)
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/nni/common/graph_utils.py", line 24, in build_module_graph
    return TorchModuleGraph(model, dummy_input)
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/nni/common/graph_utils.py", line 271, in __init__
    super().__init__(model, dummy_input, traced_model)
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/nni/common/graph_utils.py", line 66, in __init__
    self._trace(model, dummy_input)
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/nni/common/graph_utils.py", line 75, in _trace
    print(model.model[-1].export)
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 779, in __getattr__
    type(self).__name__, name))
torch.nn.modules.module.ModuleAttributeError: 'Model' object has no attribute 'model'

----------------------------------------------------------------------
Ran 4 tests in 439.166s

FAILED (errors=2)

I think some bug has been introduced by my modification(), could you please take a look at my modifications. Is there anything wrong?
The modified logs were:
https://github.com/ichejun/nni/commit/82a22263c55185620e47b62f3a2a18575c6eb06d
https://github.com/ichejun/nni/commit/b07c76fac3ff0a45cbad3f69232ea2583af245a1
https://github.com/ichejun/nni/commit/761f6b353a58a3c4f864e91eb8bc507ee5148104
https://github.com/ichejun/nni/commit/6e8b03126b0b5498fc9abdf35e014a242c30f4b5
https://github.com/ichejun/nni/commit/4420f352f497131b7e2326b852ede6645d5b005d
https://github.com/ichejun/nni/commit/3fc20a1095137199af2d1965e29f357efe7ece0b
Q2.About pruning the yolov5 without "Detect layer". My current approach is to put the convolution starting from the 3 detection heads as the exclude parameter in the prune configuration parameters:
    config_list = [{
        'sparsity': 0.03,
        'op_types': ['Conv2d']
    }, {
        'op_names':['model.24.m.0','model.24.m.1','model.24.m.2'],
        'exclude': True
    }]

It works! But I am wondering if there is a more general method or suggestion, otherwise, I have to manually find the relevant convolution of the detection head every time, and manually write the configuration to try?
Q3. I encountered a problem when I tried another pruner algorithm, namely AGPPruner.
The default configuration can run through, but when I try to add the exclude configuration, like Q2. There is a problem with configuration parsing.
The code:
    config_list = [{
        'initial_sparsity': 0.,
        'final_sparsity': 0.05,
        'start_epoch': 0,
        'end_epoch': 10,
        'frequency': 1,
        'op_types': ['Conv2d']
    }, {
        'op_names':['model.24.m.0','model.24.m.1','model.24.m.2'],
        'exclude': True
    }]
    pruner = AGPPruner(model, config_list, optimizer, pruning_algorithm='fpgm')
    pruner.compress()

The error log:
Traceback (most recent call last):
  File "train_nni_up_agp.py", line 980, in <module>
    train(hyp, opt, device, tb_writer)
  File "train_nni_up_agp.py", line 537, in train
    pruner = AGPPruner(model, config_list, optimizer, pruning_algorithm='fpgm')
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/nni/algorithms/compression/pytorch/pruning/agp.py", line 44, in __init__
    super().__init__(model, config_list, optimizer)
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/nni/compression/pytorch/compressor.py", line 322, in __init__
    super().__init__(model, config_list, optimizer)
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/nni/compression/pytorch/compressor.py", line 44, in __init__
    self.validate_config(model, config_list)
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/nni/algorithms/compression/pytorch/pruning/agp.py", line 70, in validate_config
    schema.validate(config_list)
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/nni/compression/pytorch/utils/config_validation.py", line 53, in validate
    self.compressor_schema.validate(data)
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/schema.py", line 357, in validate
    return type(data)(o.validate(d) for d in data)
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/schema.py", line 357, in <genexpr>
    return type(data)(o.validate(d) for d in data)
  File "/home/admin/work_dir/.local/lib/python3.7/site-packages/schema.py", line 167, in validate
    [self._error.format(data) if self._error else None] + errors,
schema.SchemaError: Or(And({'initial_sparsity': And(<class 'float'>, <function AGPPruner.validate_config.<locals>.<lambda> at 0x7f987df74b90>), 'final_sparsity': And(<class 'float'>, <function AGPPruner.validate_config.<locals>.<lambda> at 0x7f987df74c20>), 'start_epoch': And(<class 'int'>, <function AGPPruner.validate_config.<locals>.<lambda> at 0x7f987df74cb0>), 'end_epoch': And(<class 'int'>, <function AGPPruner.validate_config.<locals>.<lambda> at 0x7f987df74d40>), 'frequency': And(<class 'int'>, <function AGPPruner.validate_config.<locals>.<lambda> at 0x7f987df74dd0>), Optional('op_types'): And([<class 'str'>], <function CompressorSchema._modify_schema.<locals>.<lambda> at 0x7f987df74e60>), Optional('op_names'): And([<class 'str'>], <function CompressorSchema._modify_schema.<locals>.<lambda> at 0x7f987df74ef0>)}, <function CompressorSchema._modify_schema.<locals>.<lambda> at 0x7f987df74f80>)) did not validate {'op_names': ['model.24.m.0', 'model.24.m.1', 'model.24.m.2'], 'exclude': True}
Missing keys: 'end_epoch', 'final_sparsity', 'frequency', 'initial_sparsity', 'start_epoch'

Besides the parsing problem, can AGPPruner with pruning_algorithm, like 'fpgm', work in dependency_aware  mode? It seems that there is no corresponding api interface?
Thanks a lot!

Hi, @ichejun
Q1: The second parameter of ModelSpeedup() should be the original model with training weights.
model_speedup.py
Q2: If you don't skip the Detect layer or exclude the exclude the parameters, you may encounter two problems.
First, the filter size of last three layer may be the (class_number+5)*anchor_number. If you don't exclude these layers, the pruning model output may not fit the class number of original model.
Second, pruning yolov5 with Detect layer will cause the node difference between pruning model and original model in  ModelSpeedup() step.
Q3: In our previous experiment, some pruners do not implement the dependency_aware.
We only successfully speedup model of yolov5 using L1Filter Pruner.

@ping-Huang Thank you very much for your prompt reply, it helps a lot. As for questions 1 and 2, I will refer to your suggestions and try as soon as possible. For the question 3 about agp pruner, I am going to raise an issue to follow up. Thanks~

@ping-Huang Thank you very much for your prompt reply, it helps a lot. As for questions 1 and 2, I will refer to your suggestions and try as soon as possible. For the question 3 about agp pruner, I am going to raise an issue to follow up. Thanks~

Hi @ichejun i want to do same work ( speed up model with nni ) .. did u get success.. Can u share some early steps to put me step in . In my case i m working on yolov3 spp.. and will soon start on v5 .. any suggestions..

Hi @sharoseali , as for yolo v5, you may follow https://github.com/microsoft/nni/issues/3548#issuecomment-826269418. The comments for Q1 and Q2 may help in some one-shot pruner. I have not tried yolov3 spp yet...:)

@ping-Huang Thank you very much for your prompt reply, it helps a lot. As for questions 1 and 2, I will refer to your suggestions and try as soon as possible. For the question 3 about agp pruner, I am going to raise an issue to follow up. Thanks~

Hi @ichejun i want to do same work ( speed up model with nni ) .. did u get success.. Can u share some early steps to put me step in . In my case i m working on yolov3 spp.. and will soon start on v5 .. any suggestions..

Hi @sharoseali , as for yolo v5, you may follow #3548 (comment). The comments for Q1 and Q2 may help in some one-shot pruner. I have not tried yolov3 spp yet...:)

@ichejun For Q2 where i have to make changes , that you wrote and for pruning what exactely parameters and py files I have to run
Q2.About pruning the yolov5 without "Detect layer". My current approach is to put the convolution starting from the 3 detection heads as the exclude parameter in the prune configuration parameters: these ones below ?
     config_list = [{         'sparsity': 0.03,         'op_types': ['Conv2d']     },      {         'op_names':['model.24.m.0','model.24.m.1','model.24.m.2'],         'exclude': True     }]


in readme file section content you It should be at the top of the page

@GAbdulWahab what do you mean? I cannot understand the issue.

content should be in the top


Added Utils plugin

Added Utils plugin (yeah @jadb, we seperated the plugin ;)), and changed CakeManager's description.

@jadb: added description about components and behaviors


my starter-kit [is not an issue]

Hello,
I just published my own starter-kit with nebular theme v13 and angular v17.
Here the github link:
https://github.com/matzrm/nebular-dashboard
Here the demo:
https://nebular.progel.net

cool add RTL and some important component to make it awesome
a did generate it using microfrontend check
https://github.com/khalifa005/e-proc

Hi @khalifa005,
I just updated it to angular 18 and nebular 14.
https://github.com/matzrm/nebular-dashboard


Hello Yoru Choi

Yorosiku Onegai simas~

Thank you dude, if you don't mind, give me a star🫡

Poor Choi! You're beggin me a star!

Good Job Bro! I clicked the star


Fix RBAC for configmap query to be for etcd as well

This was in the KDD-only section, but actually needs to be present even
for etcd mode.

[ ] Tests
[ ] Documentation
[ ] Release note

None required

Deploy preview for calico ready!
Built with commit 34e679096d355b80d222fb0f70ea078a518f43ee
https://deploy-preview-3263--calico.netlify.com


feature: start fnf right away before stdin is completely read

I don't mind writing my own scripts and setting up aliases with fnf, but one
thing I do miss from fzf is the instant processing of stdin with new lines being
added as the last command is executing.
like the following:
find --a --really --long --query | fnf

fnf waits for find to completely finish to take in results, and that delay is
annoying. Any chance we could get a feature to run as the stream is coming in?

Hi @angelacastaneda, and thanks for this suggestion. I'll take a look at it as soon as I can.

How difficult is it to implement input streaming in fnf?

Hi @apprehensions . Can't tell, cause I haven't tried. I will as soon as I have some spare time.


added github profile

added Nishant-Jangid and url of github profile
Related issue
What changes does this PR do?
in this PR I have added my name and Github profile link
Screenshots (If any)
Questions

this PR is still open. will it be merged?

this PR is still open. will it be merged?

Done✅


  Then users can't run the next command.
I think it's better to remind users to open a new terminal.

https://kubernetes.io/docs/tutorials/hello-minikube/#enable-addons
kubectl get pod,svc -n kube-system
the namespace of tutorials should be default ?

Related Doc -  https://kubernetes.io/docs/tutorials/hello-minikube/#create-a-deployment
/language en

Hi @Jun10ng, Thanks for raising this issue, we can introduce an instruction here,

Katacoda environment only: At the top of the terminal pane, click the plus sign, and then click Select open a new terminal.

like it has been mentioned before here

Katacoda environment only: At the top of the terminal pane, click the plus sign, and then click Select port to view on Host 1.

/close
Won't fix because we're removing Katacoda.


Switch IsValveMod back to use a EnablerHook

The KHook version did not hook the Function correctly.

@KittenPopo


Add EL6 instructions for Let's Encrypt

SOFTWARE-3733

I dunno. The other big change is using certbot-auto everywhere instead of certbot, so I'm worried that intermingling the instructions will result in admins copy-pasting from the wrong box.


What is the correct way to HTML Symbols?

Thank you so much for this library, and sorry for bothering, I don't seem to find a way to do this

I will display ←

I will display ←

I will display ←

when I use DOM.text "←" it gets sanitized as &amp#8592;, for react it would be something like:

I will display {←}

but I also don't know how to do {}
Tried with DOM.unsafeCreateDOMComponent "↑" >>=  React.element but it needs an actual component like p and I just want to write "↑" escaped.
What is the correct way to do it?

You should be able to do
R.text "I will display \x2190"

https://try.purescript.org/?gist=852f261fb3aefbf9f41d17c6a2ef4af8

Thank you so much!


[Windows] Fix setting initial non-exclusive window mode.

Fixes https://github.com/godotengine/godot/issues/78530
Fixes https://github.com/godotengine/godot/issues/78723
Fixes https://github.com/godotengine/godot/issues/77378

Also seems to fix #78723 and likely #77378 @NotDaze.

Thanks!

Cherry-picked for 4.1.1.


RNPM integration

https://github.com/rnpm/rnpm

:+1:

had to link in android/settings.gradle manually after rnpm linkage


includes method to check if a class includes a mixin

It would be quite helpful to add a mixins field to classes, adding mixins to this array when include is called, and an includes method on class instances that return true if the class of that object includes the specified mixin.

I would love to see this! I implement on my fork and it comes in very useful. If anyone's interested, I would love to make a PR for it.


Long press params

The long press event now receives the media object as an input parameter.

Thank you @spacegraym3!


Error - com.microsoft.sqlserver.jdbc.SQLServerDriver does not allow create table as select

Hello,
I get error connecting to Azure SQL Server database when using spark connector using DataBricks notebook. DataBricks cluster has the spark connector loaded. Any help appreciated.
Trying to extract data from parquet files in DataBricks and load into Azure SQL servere database.
Following the example mentioned in https://docs.microsoft.com/en-us/sql/connect/spark/connector?view=sql-server-ver15
I have used 2 different options to create SQL Server database

Azure Dedicated SQL Pool (formerly SQL DW)

Synapse workspace dedicated SQL pool

To extract data I use
df =<EMAIL_ADDRESS>I have com.microsoft.azure:spark-mssql-connector:1.0.1 installed on databricks cluster and use the below code to load data into SQL Server database on Azure.
ERROR java.lang.RuntimeException: com.microsoft.sqlserver.jdbc.SQLServerDriver does not allow create table as select.
server_name = "jdbc:sqlserver://esqlserver.database.windows.net,1433"
database_name = "dedpool"
url = server_name + ";" + "databaseName=" + database_name + ";"
table_name = "dbo.spark_test1"
username = "ETL"
password = xxxx
try:
df.write \
.format("com.microsoft.sqlserver.jdbc.SQLServerDriver") \
.mode("append") \ .option("url", url) \
.option("dbtable", table_name) \
.option("user", username) \
.option("password", password) \
.save()
except ValueError as error : print("Connector write failed", error)
Thanks,

format is wrong
.format("com.microsoft.sqlserver.jdbc.spark")

Closing as resolved. Thanks @B4PJS


Fix to respect CLA ping timeperiod.

Updated PingNotication call in CLA munger with the correct startDate.

cc @foxish See https://github.com/kubernetes-incubator/node-feature-discovery/pull/20 for details.

I'm not entirely sure that is the fix to the pinger logic. Also, we run the same code in kubernetes/kubernetes and kubernetes/contrib and don't have those issues of repeated pings. Can you show us how you're running your deployment (arguments, logs, etc)?

@foxish I did some more testing. You are right. This doesn't fix the ping frequency. I should have tested more before opening this PR. Sorry for the trouble. If you want, I can open a issue and close this PR.
Following are the details of the deployment:
Args (I used --period=30s for testing. Rest of the args are same as the vanilla deployment.yaml):
command:
        - /mungegithub
        - --dry-run=false
        - --stderrthreshold=INFO
        - --http-cache-dir=$(HTTP_CACHE_DIR)
        - --organization=$(ORGANIZATION)
        - --project=$(PROJECT)
        - --pr-mungers=$(PR_MUNGERS)
        - --state=$(STATE)
        - --token-file=$(TOKEN_FILE)
        - --required-retest-contexts=$(REQUIRED_RETEST_CONTEXTS)
        - --required-contexts=$(REQUIRED_CONTEXTS)
        - --nonblocking-jenkins-jobs=$(NONBLOCKING_JENKINS_JOBS)
        - --jenkins-jobs=$(JENKINS_JOBS)
        - --presubmit-jobs=$(PRESUBMIT_JOBS)
        - --weak-stable-jobs=$(WEAK_STABLE_JOBS)
        - --do-not-merge-milestones=$(DO_NOT_MERGE_MILESTONES)
        - --admin-port=$(ADMIN_PORT)
        - --gcs-bucket=$(GCS_BUCKET)
        - --gcs-logs-dir=$(GCS_LOGS_DIR)
        - --pull-logs-dir=$(PULL_LOGS_DIR)
        - --pull-key=$(PULL_KEY)
        - --path-label-config=$(PATH_LABEL_CONFIG)
        - --period=30s
        - --block-path-config=$(BLOCK_PATH_CONFIG)
        - --repo-dir=$(REPO_DIR)
        - --test-owners-csv=$(TEST_OWNERS_CSV)
        - --number-of-old-test-results=$(NUMBER_OF_OLD_TEST_RESULTS)
        - --fixes-issue-reassign=$(FIXES_ISSUE_REASSIGN)
        - --blunderbuss-reassign=$(BLUNDERBUSS_REASSIGN)
        - --generated-files-config=$(GENERATED_FILES_CONFIG)
        - --chart-url=$(CHART_URL)
        - --history-url=$(HISTORY_URL)
        - --enable-md-yaml=$(ENABLE_MD_YAML)
        - --label-file=$(LABEL_FILE)
        - --triager-url=$(TRIAGER_URL)
        - --cla-status-context=$(CLA_STATUS_CONTEXT)

Configmap:
apiVersion: v1
kind: ConfigMap
metadata:
  name: node-feature-discovery-sq-config
data:
  # basic config options.
  config.http-cache-dir: /cache/httpcache
  config.organization: kubernetes-incubator
  config.project: node-feature-discovery
  config.pr-mungers: blunderbuss,cla,lgtm-after-commit,submit-queue,needs-rebase,lgtm-handler
  config.state: open
  config.token-file: /etc/secret-volume/token
  gitrepo.repo-dir: /gitrepos
  gitrepo.enable-md-yaml: "false"

  # test-options feature options.
  test-options.required-retest-contexts: "\"\""

  # submit-queue options.
  submit-queue.required-contexts: "continuous-integration/travis-ci/pr"
  submit-queue.nonblocking-jenkins-jobs: "\"\""
  submit-queue.jenkins-jobs: "\"\""
  submit-queue.presubmit-jobs: "\"\""
  submit-queue.weak-stable-jobs: "\"\""
  submit-queue.do-not-merge-milestones: "\"\""
  submit-queue.admin-port: "9999"
  submit-queue.chart-url: ""
  submit-queue.history-url: ""

  # options used by the GCS feature.
  gcs.gcs-bucket: ""
  gcs.gcs-logs-dir: ""
  gcs.pull-logs-dir: ""
  gcs.pull-key: ""

  # munger specific options.
  path-label.path-label-config: ""
  block-path.block-path-config: ""
  flake-manager.test-owners-csv: ""
  old-test-getter.number-of-old-test-results: "0"
  assign-fixes.fixes-issue-reassign: "false"
  blunderbuss.blunderbuss-reassign: "false"
  size.generated-files-config: ""
  check-labels.label-file: "/gitrepos/node-feature-discovery/labels.yaml"
  issue-categorizer.triager-url: ""
  cla.cla-status-context: "cla/linuxfoundation"

I1020 00:53:07.178157       1 mungegithub.go:100] {
  "Org": "kubernetes-incubator",
  "Project": "node-feature-discovery",
  "State": "open",
  "Labels": [],
  "TokenFile": "/etc/secret-volume/token",
  "Address": ":8080",
  "WWWRoot": "www",
  "HTTPCacheDir": "/cache/httpcache",
  "HTTPCacheSize": 1000,
  "MinPRNumber": 0,
  "MaxPRNumber":<PHONE_NUMBER>854775807,
  "DryRun": false,
  "BaseWaitTime": 0,
  "MinIssueNumber": 0,
  "PRMungersList": [
    "blunderbuss",
    "lgtm-after-commit",
    "submit-queue",
    "needs-rebase",
    "lgtm-handler",
    "cla"
  ],
  "IssueReportsList": [],
  "Once": false,
  "Period":<PHONE_NUMBER>0,
  "StateMachineEnabled": false,
  "Aliases": null,
  "Repos": null,
  "GCSInfo": null,
  "TestOptions": null
}
I1020 00:53:07.178240       1 github.go:228] Made 0 API calls since the last Reset 0.000000 calls/sec
I1020 00:53:07.178346       1 features.go:55] Initializing feature: aliases
I1020 00:53:07.178360       1 features.go:55] Initializing feature: gitrepos
I1020 00:53:29.805781       1 repo-updates.go:187] Loaded config from /gitrepos/node-feature-discovery:31ec89474a5059212613f64920baea596930d954
I1020 00:53:29.805814       1 features.go:55] Initializing feature: google-cloud-storage
I1020 00:53:29.805820       1 features.go:55] Initializing feature: test-options
I1020 00:53:29.805841       1 mungers.go:92] {
  "BlunderbussReassign": false
}
I1020 00:53:29.805845       1 mungers.go:93] Initialized munger: blunderbuss
I1020 00:53:29.805862       1 mungers.go:92] {}
I1020 00:53:29.805870       1 mungers.go:93] Initialized munger: lgtm-after-commit
I1020 00:53:29.806039       1 mungers.go:92] {
  "BlockingJobNames": [],
  "NonBlockingJobNames": [],
  "PresubmitJobNames": [],
  "WeakStableJobNames": [],
  "FakeE2E": false,
  "Committers": "",
  "RequiredStatusContexts": [
    "continuous-integration/travis-ci/pr"
  ],
  "DoNotMergeMilestones": [],
  "RequiredRetestContexts": [],
  "RetestBody": "@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]",
  "Metadata": {
    "ProjectName": "Node-Feature-Discovery",
    "ChartUrl": "",
    "HistoryUrl": "",
    "RepoPullUrl": "https://github.com/kubernetes-incubator/node-feature-discovery/pulls/"
  },
  "AdminPort": 9999
}
I1020 00:53:29.806052       1 mungers.go:93] Initialized munger: submit-queue
I1020 00:53:29.806061       1 mungers.go:92] {}
I1020 00:53:29.806064       1 mungers.go:93] Initialized munger: needs-rebase
I1020 00:53:29.806074       1 mungers.go:92] {}
I1020 00:53:29.806080       1 mungers.go:93] Initialized munger: lgtm-handler
I1020 00:53:29.806093       1 mungers.go:92] {
  "CLAStatusContext": "cla/linuxfoundation"
}
I1020 00:53:29.806099       1 mungers.go:93] Initialized munger: cla
I1020 00:53:29.806103       1 mungegithub.go:63] Running mungers
I1020 00:53:31.106615       1 repo-updates.go:187] Loaded config from /gitrepos/node-feature-discovery:31ec89474a5059212613f64920baea596930d954
I1020 00:53:33.042487       1 github.go:1635] For 20: supposedly there weren't more comments, but we asked anyway and found 39 more.
I1020 00:53:33.497563       1 github.go:1015] For 20: supposedly there weren't more events, but we asked anyway and found 100 more.
I1020 00:53:33.651705       1 github.go:1015] For 20: supposedly there weren't more events, but we asked anyway and found 94 more.
I1020 00:53:35.633354       1 github.go:1661] Commenting in 23: "[CLA-PING] @danielscottt\n\nThanks for your pull request.  It looks like this may be your first contribution to a CNCF open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).\n\n:memo: **Please visit <https://identity.linuxfoundation.org/projects/cncf> to sign.**\n\nOnce you've signed, please reply here (e.g. \"I signed it!\") and we'll verify.  Thanks.\n\n---\n\n- If you've already signed a CLA, it's possible we don't have your GitHub username or you're "
I1020 00:53:35.840099       1 github.go:228] Made 28 API calls since the last Reset 0.976912 calls/sec
I1020 00:53:35.840875       1 mungegithub.go:85] Sleeping for 23.965231806s
I1020 00:53:59.865677       1 mungegithub.go:63] Running mungers
I1020 00:54:01.305072       1 repo-updates.go:187] Loaded config from /gitrepos/node-feature-discovery:31ec89474a5059212613f64920baea596930d954
I1020 00:54:02.734924       1 github.go:1635] For 20: supposedly there weren't more comments, but we asked anyway and found 39 more.
I1020 00:54:03.102034       1 github.go:1015] For 20: supposedly there weren't more events, but we asked anyway and found 100 more.
I1020 00:54:03.400995       1 github.go:1015] For 20: supposedly there weren't more events, but we asked anyway and found 94 more.
I1020 00:54:05.311030       1 github.go:1661] Commenting in 23: "[CLA-PING] @danielscottt\n\nThanks for your pull request.  It looks like this may be your first contribution to a CNCF open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).\n\n:memo: **Please visit <https://identity.linuxfoundation.org/projects/cncf> to sign.**\n\nOnce you've signed, please reply here (e.g. \"I signed it!\") and we'll verify.  Thanks.\n\n---\n\n- If you've already signed a CLA, it's possible we don't have your GitHub username or you're "
I1020 00:54:05.573394       1 github.go:228] Made 28 API calls since the last Reset 0.941730 calls/sec
I1020 00:54:05.573490       1 mungegithub.go:85] Sleeping for 24.292178352s

@balajismaniam I think I know why this is happening. it's because within the notification code, it is looking for the bot to have authored the cla-ping (https://github.com/kubernetes/contrib/blob/b70447aa59ea14468f4cd349760e45b6a0a9b15d/mungegithub/mungers/matchers/comment/interactions.go#L91) but the bot is currently hardcoded to k8s-merge-bot. If you added your bot's name there, and made a new PR with the added matcher, I'd be glad to get that merged in.

@foxish Thanks for working with me on this and opening the issue. I will submit a new PR with the appropriate changes. Closing this one.

@foxish It seems like the bot name is hard-coded in couple more places and it affects the lgtm_handler and flake_manager mungers. See the following:
https://github.com/kubernetes/contrib/blob/master/mungegithub/mungers/mungerutil/util.go#L29
https://github.com/kubernetes/contrib/blob/master/mungegithub/mungers/mungerutil/util.go#L112
https://github.com/kubernetes/contrib/blob/master/mungegithub/mungers/lgtm_handler.go#L137
https://github.com/kubernetes/contrib/blob/master/mungegithub/mungers/sync/issue-sync.go#L30
https://github.com/kubernetes/contrib/blob/master/mungegithub/mungers/flake-manager.go#L331
I will be using the lgtm_handler munger and as such I don't think it is a good idea to upstream the changes for making it work for the nfd-merge-bot. For now, I will run a forked version of mungegithub on our incubator repo. Let me know your thoughts on this.
Parameterizing the bot name is better. Let me know if I can help with it.

@foxish Do we still want to fix that?


strtol() in calculator::toRPN can't support 64bit integers on MinGW, Windows

Hi, cparse is work pretty well on Linux. But when I compile it by MinGW on Windows, the high 32 bits of numbers is allways 0.
It seems long int strtol(...) function return 32-bit integer, rather than 64-bit integer on MinGW/Windows, because of the data model of Windows is LLP.
related code:
      // If the token is a number, add it to the output queue.
      int64_t _int;
      _int = strtol(expr, &nextChar, 10);

Maybe the strtoll() is better?

Hello @thelxz, sorry for taking so long to answer and thank you for reporting this. This seems like a good idea, I will try to create a MinGW environment and test if this fixes the problem.
If it doesn't I will look for another solution.

Hello @thelxz, I was unable to reproduce the problem on my environment so I can't be sure my last push really fixed the problem. I only replaced strtol() for strtoll() as you suggested.
Can you confirm if this fixes the problem?

It works! This problem has been fixed.
My fault, I should have provided the failed test case at first. The test code:
  TokenMap vars;
  calculator c1("17587891077120+1");  //  0xFFF_0000_0000 + 1
  std::cout << c1.eval(vars) << std::endl;

Output with strtol():
2.14748e+009

Output with strtoll():
1.75879e+013

That's good! Thanks for the test code, I will add a test for it so this doesn't happen again.


tests broken since fixing proper export for fetch

tests are broken since 7c66f9ce822a3a06fe01eb583a30b4c087c596c2
(after adding module.exports = fetch;)
adding this fixes it
fetch.Headers = Headers;
fetch.Request = Request;
fetch.Response = Response;
fetch.FetchError = FetchError;

EDIT: I'm not sure if this is the solution pkg maintainers would like or do tests need to be rewritten?

c167190c6ee21234ba41bd8430700d4720bf64ce this was fixed by reverting the changes made
I'm closing this for now since the tests are working again


Error in REM when using 'norm' for time

##### Load packages
packages_to_install <- c("purrr", "furrr", "magrittr", "dplyr",
                         "tibble", "data.table", "survival", "tidyverse",
                         "devtools", "reprex", "styler", "igraph", "RColorBrewer",
                         "visNetwork", "relevent", "sna")

for (pkg in packages_to_install) {
  if (!require(pkg, character.only = TRUE)) {
    # If not, install the package
    install.packages(pkg)
  }
}
#> Loading required package: purrr
#> Loading required package: furrr
#> Loading required package: future
#> Loading required package: magrittr
#>
#> Attaching package: 'magrittr'
#> The following object is masked from 'package:purrr':
#>
#>     set_names
#> Loading required package: dplyr
#>
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#>
#>     filter, lag
#> The following objects are masked from 'package:base':
#>
#>     intersect, setdiff, setequal, union
#> Loading required package: tibble
#> Loading required package: data.table
#>
#> Attaching package: 'data.table'
#> The following objects are masked from 'package:dplyr':
#>
#>     between, first, last
#> The following object is masked from 'package:purrr':
#>
#>     transpose
#> Loading required package: survival
#>
#> Attaching package: 'survival'
#> The following object is masked from 'package:future':
#>
#>     cluster
#> Loading required package: tidyverse
#> Loading required package: devtools
#> Loading required package: usethis
#> Loading required package: reprex
#> Loading required package: styler
#> Loading required package: igraph
#>
#> Attaching package: 'igraph'
#> The following objects are masked from 'package:lubridate':
#>
#>     %--%, union
#> The following object is masked from 'package:tidyr':
#>
#>     crossing
#> The following object is masked from 'package:tibble':
#>
#>     as_data_frame
#> The following objects are masked from 'package:dplyr':
#>
#>     as_data_frame, groups, union
#> The following objects are masked from 'package:future':
#>
#>     %->%, %<-%
#> The following objects are masked from 'package:purrr':
#>
#>     compose, simplify
#> The following objects are masked from 'package:stats':
#>
#>     decompose, spectrum
#> The following object is masked from 'package:base':
#>
#>     union
#> Loading required package: RColorBrewer
#> Loading required package: visNetwork
#> Loading required package: relevent
#> Loading required package: trust
#> Loading required package: sna
#> Loading required package: statnet.common
#>
#> Attaching package: 'statnet.common'
#> The following objects are masked from 'package:base':
#>
#>     attr, order
#> Loading required package: network
#>
#> 'network' 1.18.2 (2023-12-04), part of the Statnet Project
#> * 'news(package="network")' for changes since last version
#> * 'citation("network")' for citation information
#> * 'https://statnet.org' for help, support, and other information
#>
#> Attaching package: 'network'
#> The following objects are masked from 'package:igraph':
#>
#>     %c%, %s%, add.edges, add.vertices, delete.edges, delete.vertices,
#>     get.edge.attribute, get.edges, get.vertex.attribute, is.bipartite,
#>     is.directed, list.edge.attributes, list.vertex.attributes,
#>     set.edge.attribute, set.vertex.attribute
#> sna: Tools for Social Network Analysis
#> Version 2.7-2 created on 2023-12-05.
#> copyright (c) 2005, Carter T. Butts, University of California-Irvine
#>  For citation information, type citation("sna").
#>  Type help(package="sna") to get started.
#>
#> Attaching package: 'sna'
#> The following objects are masked from 'package:igraph':
#>
#>     betweenness, bonpow, closeness, components, degree, dyad.census,
#>     evcent, hierarchy, is.connected, neighborhood, triad.census
#> Loading required package: coda
#> relevent: Relational Event Models
#> Version 1.2-1 created on 2023-01-24.
#> copyright (c) 2007, Carter T. Butts, University of California-
#> Irvine
#>  For citation information, type citation("relevent").
#>  Type help(package="relevent") to get started.

library(purrr, warn.conflicts = FALSE)    # for functional programming
library(furrr, warn.conflicts = FALSE)    # for functional futures
library(magrittr, warn.conflicts = FALSE) # for pipes
library(dplyr, warn.conflicts = FALSE)    # for data manipulation
library(tibble, warn.conflicts = FALSE)     # for tibbles
library(data.table, warn.conflicts = FALSE)
library(survival, warn.conflicts = FALSE) # for REM analysis
library(tidyverse, warn.conflicts = FALSE)
library(tidyr, warn.conflicts = FALSE)
library(reprex, warn.conflicts = FALSE)   # reproducible code
library(styler, warn.conflicts = FALSE)
library(igraph, warn.conflicts = FALSE)
library(RColorBrewer, warn.conflicts = FALSE)
library(visNetwork, warn.conflicts = FALSE)
library(relevent, warn.conflicts = FALSE)
library(sna, warn.conflicts = FALSE)

devtools::install_github("TilburgNetworkGroup/remify")
#> Using GitHub PAT from the git credential store.
#> Skipping install of 'remify' from a github remote, the SHA1 (fbd8e53e) has not changed since last install.
#>   Use `force = TRUE` to force installation
devtools::install_github("gerkovink/mice@match_conditional_current")
#> Using GitHub PAT from the git credential store.
#> Skipping install of 'mice' from a github remote, the SHA1 (776d92b4) has not changed since last install.
#>   Use `force = TRUE` to force installation
devtools::install_github("TilburgNetworkGroup/remstats")
#> Using GitHub PAT from the git credential store.
#> Skipping install of 'remstats' from a github remote, the SHA1 (1fd890e1) has not changed since last install.
#>   Use `force = TRUE` to force installation

library(mice, warn.conflicts = FALSE)     # for imputation and amputation
library(remstats, warn.conflicts = FALSE) # for REM statistics
library(remify, warn.conflicts = FALSE)   # for converting

##### Load data
con <- url("https://github.com/mshafieek/ADS-Missing-data-social-network/raw/main/literature_%20REM/Tutorial_REM_REH_DATA/UUsummerschool.Rdata")
load(con)
apollo <- PartOfApollo_13 %>%
  rename(
    actor1 = sender,
    actor2 = receiver
  )
rm(Class, PartOfApollo_13, Twitter_data_rem3, WTCPoliceCalls, ClassIntercept,
   ClassIsFemale, ClassIsTeacher, WTCPoliceIsICR, con, pkg)

# Missingness simulations

set.seed(123) # fix seed to realize a sufficient set
apollo <- as.tibble(apollo)
#> Warning: `as.tibble()` was deprecated in tibble 2.0.0.
#> ℹ Please use `as_tibble()` instead.
#> ℹ The signature and semantics have changed, see `?as_tibble`.
#> This warning is displayed once every 8 hours.
#> Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
#> generated.
indic <- sample(1:nrow(apollo), 1500)

remify(apollo[indic, ], model = "tie") %>% dim()
#> Warning: Unknown or uninitialised column: `type`.
#> Warning: Unknown or uninitialised column: `weight`.
#> Warning in remify(apollo[indic, ], model = "tie"):
#> Warning: the `time` variable is not sorted. Sorting will be forced.
#> events actors  dyads
#>   1500     16    240
#### Combine the sufficient set and the incomplete set
make_missing <- function(x, indic){
  sufficient <- x[indic, ]
  miss <- x[-c(indic), ] |>
    # prop is set to .4 to get close to .2 missingness since almost half of the dataset is used for the sufficient set
    ampute(prop = .4,
           mech = "MCAR",
           patterns = matrix(c(1,1,0, # missing in actor 2
                               1,0,1, # missing in actor 1
                               0,1,1, # missing in time
                               0,0,0, # missing in all
                               1,0,0, # missing in actor 1 + 2
                               0,1,0, # missing in time + actor 2
                               0,0,1  # missing in time + actor 1
                               ),
                             nrow=7,  # change to number of missingness patterns above
                             byrow=TRUE)) %>%
    .$amp
  combined <- rbind(sufficient, miss)
  return(combined[order(combined$time), ]) # sort it all like apollo
}

whichcol <- c("", "actor2", "actor1") # Ensure that actor 1 != actor 2 in imputations
names(whichcol) <- colnames(apollo)

## predictor matrix
pred <- make.predictorMatrix(apollo)
#pred[ ,"time"] <- 0 # using time for imputation creates loops
## use the pmm.conditional method
method <- make.method(apollo)
method[1] <- "norm" # imputation of time correct? Something Like Last Obs Carried Forward?
method[c(2,3)] <- "pmm.conditional"
## post squeeze time between min and max time
post <- make.post(apollo)
min(apollo$time)
#> [1] 11849.2
max(apollo$time)
#> [1] 50014.8
post["time"] <- "imp[[j]][, i] <- squeeze(imp[[j]][, i], c(11849.2, 50014.8))"

##### Model-based finite populations
mbased_finite_apollo <-
  furrr::future_map(1:10, ~ {   # start of with 10 simulations for testing
    make_missing(apollo, indic) %>%
      mice(m = 5,
           maxit = 5,
           method = method,
           whichcolumn = whichcol,
           predictorMatrix = pred,
           post = post,
           print = FALSE)
  }, .options = furrr_options(seed = 123))

##### Defining effects for the relational event model
effects <- ~ -1 + reciprocity(scaling = ("std")) + indegreeSender() + outdegreeReceiver()

##### Function to get the statistics of the previously defined effects.
stats_function <- function(data) {
  # remify the data
  reh <- remify::remify(edgelist = data, model = "tie")
  # calculate effect statistics
  statsObject_imp <- remstats(reh = reh, tie_effects = effects)
  # Return the statistics
  return(statsObject_imp)
}

##### Function for making the data compatible with coxph()
prepare_coxph_data <- function(statsObject, apollo) {
  risk_sets <- attr(statsObject, "riskset")
  risk_sets <- risk_sets %>% select(-'id')

  # Get the times
  time <- apollo$time

  # merge riskset with each timepoint
  combined <- merge(risk_sets, time, by = NULL)

  combined <- combined %>% rename("time" = "y")
  combined <- lapply(combined, as.numeric)
  combined <- as.data.frame(combined)

  # Create matrices for subtraction to make a status column for coxph
  combined_matrix <- data.matrix(combined)
  matrix_rows <- nrow(combined)

  repeated_df <- apollo[rep(seq_len(nrow(apollo)), each = 240), ]
  repeated_df <- repeated_df[, c(2,3,1)]
  apollo_matrix <- data.matrix(repeated_df)

  status_matrix <- apollo_matrix - combined_matrix

  # create a status column
  status <- as.integer(rowSums(status_matrix == 0) == ncol(status_matrix))
  status <- as.data.frame(status)

  # Add status to the combined set
  combined <- cbind(combined, status)

  # Extract statistics and add them to the dataframe
  reciprocity <- statsObject[,,1]
  indegreeSender <- statsObject[,,2]
  outdegreeReceiver <- statsObject[,,3]

  combined$reciprocity <- c(reciprocity)
  combined$indegreeSender <- c(indegreeSender)
  combined$outdegreeReceiver <- c(outdegreeReceiver)

  return(combined)
}

###### TRUE ANALYSIS
true.reh <- remify(edgelist = apollo,
                   model = "tie")
#> Warning: Unknown or uninitialised column: `type`.
#> Warning: Unknown or uninitialised column: `weight`.
# calculate stats
stats <- remstats(tie_effects = effects,
                  reh = true.reh)
# use the function to create the correct format of the dataframe
true.cox.set <- prepare_coxph_data(stats, apollo)
# fit cox model
true.cox.fit <- coxph(Surv(time, status) ~ reciprocity + indegreeSender +
                        outdegreeReceiver,
                      data=true.cox.set)
true <- coefficients(true.cox.fit)

true.cox.fit
#> Call:
#> coxph(formula = Surv(time, status) ~ reciprocity + indegreeSender +
#>     outdegreeReceiver, data = true.cox.set)
#>
#>                         coef  exp(coef)   se(coef)      z        p
#> reciprocity        2.357e-02  1.024e+00  1.858e-02  1.269    0.204
#> indegreeSender     4.314e-04  1.000e+00  7.400e-05  5.830 5.55e-09
#> outdegreeReceiver -9.115e-05  9.999e-01  7.437e-05 -1.226    0.220
#>
#> Likelihood ratio test=41.29  on 3 df, p=5.673e-09
#> n= 931680, number of events= 3882

Results1 <-
  mbased_finite_apollo[1:5] %>%
  map(~.x %>% # for every simulation
        complete("all") %>%
        map(~.x %>% # for every imputation
              stats_function() %>% # do stats function
              prepare_coxph_data(apollo = apollo) %$% # prepare cox ph
              coxph(Surv(time, status) ~
                      reciprocity +
                      indegreeSender +
                      outdegreeReceiver)) %>%
        pool(custom.t = ".data$b + .data$b / .data$m") %>% # pool coefficients
        .$pooled %>% # extract table of pooled coefficients
          mutate(true = true, # add true
                 df = m-1,  # correct df
                 riv = Inf, # correct riv
                 std.error = sqrt(t), # standard error
                 statistic = estimate / std.error, # test statistic
                 p.value = 2 * (pt(abs(statistic),
                                   pmax(df, 0.001),
                                   lower.tail = FALSE)), # correct p.value
                 `2.5 %` = estimate - qt(.975, df) * std.error, # lower bound CI
                 `97.5 %` = estimate + qt(.975, df) * std.error, # upper bound CI
                 cov = `2.5 %` < true & true < `97.5 %`, # coverage
                 bias = estimate - true) %>% # bias
          select(term, m, true, estimate, std.error, statistic, p.value,
                 riv, `2.5 %`, `97.5 %`, cov, bias) %>%
          column_to_rownames("term") # `term` as rownames
    )
#> Warning in remify::remify(edgelist = data, model = "tie"):
#> Warning: the `time` variable is not sorted. Sorting will be forced.
#> Error in `map()`:
#> ℹ In index: 1.
#> Caused by error in `map()`:
#> ℹ In index: 1.
#> ℹ With name: 1.
#> Caused by error in `$<-.data.frame`:
#> ! replacement has 913680 rows, data has 931680

Results1
#> Error in eval(expr, envir, enclos): object 'Results1' not found

Created on 2024-05-07 with reprex v2.1.0

I get this error when using the squeeze function in MICE to confine time to fall within the minimum and maximum of time.
Also note that this technique also does not keep the correct order of the time variable.
Any suggestions, i'm stuck..

Thanks, Dominic, for informing us. I couldn't run your code as I'm unable to install remify at the moment. Were there some time points missing sequentially? For example, t1, missing, missing, missing, t2,...? if so, Yyu can add a little bit of noise to ensure the correct order. For the error,  did you ensure that the names in the formula match the columns in the Cox data frame of statistics? @gerkovink, what do you think?

Squeeze is indeed not an option if the current order has to be preserved. The squeeze was proposed when I thought that the order was also unknown. Interpolation or some form thereof is a much better option here

@dominic-comerford is that clear?

Thanks for the responses.
Yes it is clear that squeeze will not work then as the order is crucial.
I just dont see how to add noise in the methods and how that would keep the order intact. The names should be correct as this is still last years statistics. We havent decided yet on the 4th stat.


Another Undefined symbols for architecture x86_64 on MacOS

Required Info

Camera Model
D400

Firmware Version
(Open RealSense Viewer --> Click info)

Operating System & Version
MacOS

Platform
PC/Raspberry Pi/ NVIDIA Jetson / etc..

SDK Version
2.54.1 }

Language
CPP

Segment
{Robot/Smartphone/VR/AR/others }

Issue Description
I'm basically having the same issue as https://github.com/IntelRealSense/librealsense/issues/2685 (though for a different dependency).
I have the rs-hello-realsense.cpp example in a directory (~/src/realsense/rs-hello-realsense.cpp).
I run cmake, then from the build directory make and get the Undefined symbols for architecture x86_64 error.
╭─ ~/src/realsense                                                                                  ✔  9099  16:06:39
╰─ cmake -S . -B build
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/dave/src/realsense/build
╭─ ~/src/realsense                                                                                  ✔  9100  16:06:42
╰─ cd build
╭─ ~/src/realsense/build                                                                                  ✔  9101  16:06:46
╰─ make
[ 50%] Linking CXX executable rs-hello-realsense
Undefined symbols for architecture x86_64:
  "_rs2_create_context", referenced from:
      rs2::context::context() in rs-hello-realsense.cpp.o
  "_rs2_create_pipeline", referenced from:
      rs2::pipeline::pipeline(rs2::context) in rs-hello-realsense.cpp.o
  "_rs2_delete_context", referenced from:
      rs2::context::context() in rs-hello-realsense.cpp.o
  "_rs2_delete_pipeline", referenced from:
      rs2::pipeline::pipeline(rs2::context) in rs-hello-realsense.cpp.o
  "_rs2_delete_pipeline_profile", referenced from:
      rs2::pipeline::start() in rs-hello-realsense.cpp.o
  "_rs2_depth_frame_get_distance", referenced from:
      rs2::depth_frame::get_distance(int, int) const in rs-hello-realsense.cpp.o
  "_rs2_embedded_frames_count", referenced from:
      rs2::frameset::frameset(rs2::frame const&) in rs-hello-realsense.cpp.o
  "_rs2_extract_frame", referenced from:
      void rs2::frameset::foreach_rs<rs2::frameset::first_or_default(rs2_stream, rs2_format) const::'lambda'(rs2::frame)>(rs2::frameset::first_or_default(rs2_stream, rs2_format) const::'lambda'(rs2::frame)) const in rs-hello-realsense.cpp.o
  "_rs2_frame_add_ref", referenced from:
      rs2::frame::add_ref() const in rs-hello-realsense.cpp.o
  "_rs2_free_error", referenced from:
      rs2::error::error(rs2_error*) in rs-hello-realsense.cpp.o
  "_rs2_get_error_message", referenced from:
      rs2::error::error(rs2_error*) in rs-hello-realsense.cpp.o
  "_rs2_get_failed_args", referenced from:
      rs2::error::error(rs2_error*) in rs-hello-realsense.cpp.o
  "_rs2_get_failed_function", referenced from:
      rs2::error::error(rs2_error*) in rs-hello-realsense.cpp.o
  "_rs2_get_frame_height", referenced from:
      rs2::video_frame::get_height() const in rs-hello-realsense.cpp.o
  "_rs2_get_frame_stream_profile", referenced from:
      rs2::frame::get_profile() const in rs-hello-realsense.cpp.o
  "_rs2_get_frame_width", referenced from:
      rs2::video_frame::get_width() const in rs-hello-realsense.cpp.o
  "_rs2_get_librealsense_exception_type", referenced from:
      rs2::error::handle(rs2_error*) in rs-hello-realsense.cpp.o
      rs2::error::error(rs2_error*) in rs-hello-realsense.cpp.o
  "_rs2_get_stream_profile_data", referenced from:
      rs2::stream_profile::stream_profile(rs2_stream_profile const*) in rs-hello-realsense.cpp.o
  "_rs2_is_frame_extendable_to", referenced from:
      rs2::frameset::frameset(rs2::frame const&) in rs-hello-realsense.cpp.o
      rs2::depth_frame::depth_frame(rs2::frame const&) in rs-hello-realsense.cpp.o
      rs2::video_frame::video_frame(rs2::frame const&) in rs-hello-realsense.cpp.o
  "_rs2_is_stream_profile_default", referenced from:
      rs2::stream_profile::stream_profile(rs2_stream_profile const*) in rs-hello-realsense.cpp.o
  "_rs2_pipeline_start", referenced from:
      rs2::pipeline::start() in rs-hello-realsense.cpp.o
  "_rs2_pipeline_wait_for_frames", referenced from:
      rs2::pipeline::wait_for_frames(unsigned int) const in rs-hello-realsense.cpp.o
  "_rs2_release_frame", referenced from:
      rs2::frame::reset() in rs-hello-realsense.cpp.o
      rs2::frame::~frame() in rs-hello-realsense.cpp.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [rs-hello-realsense] Error 1
make[1]: *** [CMakeFiles/rs-hello-realsense.dir/all] Error 2
make: *** [all] Error 2

From #2685 it looks like I'm maybe missing linking some libraries, but the g++ ... command suggested doesn't work:
g++ -I /Users/dave/src/librealsense/include/ -I ../ -std=c++11 ../rs-hello-realsense.cpp -o rs-hello-realsense -lglfw -lrealsense2 -framework OpenGL
ld: library not found for -lglfw
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Suggestions for how to resolve this?

discovered that glfw didn't exist in /usr/local/lib, so installed it using brew install glfw and now I"m getting:
╭─ ~/src/realsense/build                                                                                  2 ↵  9124  19:25:15
╰─ g++ -I /Users/dave/src/librealsense/include/ -I ../ -std=c++11 ../rs-hello-realsense.cpp -o rs-hello-realsense -lglfw -lrealsense2 -framework OpenGL
ld: library not found for -lrealsense2
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Realized I missed the last step, maybe? Ran the install_name_tool command from the "Packaging your Application" step, but I'm unclear about where to put the resulting dylib file.

Hi @caddac  Do you have an M1 model of Mac, please?  If you do then the M1 processor chip is based on Arm architecture, whilst older Macs with Intel-based processors are based on x86/x64 architecture.  The error message Undefined symbols for architecture x86_64 suggests to me that your Mac is not an old Intel-based model.
The link below has a guide for both Intel and M1 Macs about building the librealsense SDK.
https://lightbuzz.com/realsense-macos/
The SDK can alternatively be installed for Intel and M1 Macs and MacOS versions up to Ventura with brew install librealsense
https://formulae.brew.sh/formula/librealsense

Hi @MartyG-RealSense Sorry for not specifying, I am running Monterey on an Intel Mac.
I'll take a look at the lightbuzz article and report back!
Thank you for the suggestions!

I ran into needing to upgrade Xcode, but my HDD is full. Its going to take me a while to get a new HDD and update Xcode. Closing this for now and will re-open when I get back to this.

That's no problem at all to re-open later.  Thanks very much for the update.  Good luck!


Nucleo Apps doesnt work in firefox

I bought Dashboar Paper Pro 2 and when I start any example page in firefox (release 59.0) I have been getting bad request for Nucleo Apps Icon. I was taking look in head calls and I realized that nc is not called that. Could it be a problem? How could I deal with that?

Hello @photonauta35, thank you for using our products, this may be an issue with Firefox.  On local, Firefox doesn't know the correct path to the icons. Try to set your path like C:\User\Desktop\YourProject..., not a relative one; or you can run your project on a server. As you can see, the icons from our product(Paper Dashboard) on livePreview is working fine.

Hope these informations helps you.
All the best,
Stefan

Did you also tried to run your project on npm? here are some steps that you have to follow: https://demos.creative-tim.com/paper-dashboard-2-pro/docs/1.0/components/colors.html

Ran into the same issue, use this to properly configure Firefox: https://stackoverflow.com/a/3704578

That's great! Thank you. =)


Maps not displaying without  Google Maps API Key

Need to add a variable to include your google maps API key for the map to work now. (At least for me).
I added to line 109 of session.php
..... src="https://maps.googleapis.com/maps/api/js?key=GOOGLEMAPSAPIKEY">
I removed the sensor parameter as it looks like it's deprecated.
I got my key from here
https://developers.google.com/maps/documentation/javascript/
You will need a google id to get the key, then copy where GOOGLEMAPSAPIKEY is above.

I can confirm this. For the sake of source code improvements the KEY should not be stored in session.php, but in the creds.php instead, because that file is ignored.


Upgrade dependencies

npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none was installed.
Can we get eslint-plugin-mocha updated so this warning will go away?

Hey @niftylettuce would you mind creating a PR for this?

Thar you go @bromanko

@bromanko @yjhuoh can someone here please release a new version to NPM and bump package.json version?

Done


docs: en: Add Dragonboard820c installation guide

This PR adds a guide for installing Autoware on 96Boards Dragonboard820c.
Signed-off-by: Manivannan Sadhasivam<EMAIL_ADDRESS>

@yk-fujii please also create a PR targeting develop to keep both branches synced


Add automatic theme convert

https://stackoverflow.com/a/45205645/528136

:tada: This issue has been resolved in version 0.12.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


"unknown rank error" in tensorflow.keras.layers.Layer with tensorflow.py_function output

Issue Type -> Bug
Have you reproduced the bug with TF nightly? -> No
Tensorflow Version -> v2.11.0-rc2-17-gd5b57ca93e5,  2.11.0
Custom Code -> Yes
OS Platform and Distribution -> Google colab
Standalone code to reproduce the issue
import numpy as np,tensorflow as tf
@tf.function
def sample(inputs:tf.Tensor):                                                   # some sample function

  def numpy_method(inputs:np.ndarray):
    # some complex method
    return np.sqrt(inputs)# example code

  inputs=tf.numpy_function(numpy_method,[inputs],inputs.dtype,name='outputs')   # make apply numpy method
  inputs=tf.keras.layers.AvgPool2D()(inputs)                                    # ERROR !!!!!

  return inputs

sample(tf.random.uniform((1,32,32,3)))

Relevant log output

.....

ValueError: in user code:

    File "<ipython-input-189-8001acaa2b43>", line 10, in sample  *
        inputs=tf.keras.layers.AvgPool2D()(inputs)                                    # ERROR !!!!!
    File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 70, in error_handler  **
        raise e.with_traceback(filtered_tb) from None

    ValueError: Exception encountered when calling layer 'average_pooling2d' (type AveragePooling2D).

    Cannot take the length of shape with unknown rank.

    Call arguments received by layer 'average_pooling2d' (type AveragePooling2D):
      • inputs=tf.Tensor(shape=<unknown>, dtype=float32)

But works fine if we reverse the order (i.e. tf.numpy_function cannot properly cast back outputs to tensors)
import numpy as np,tensorflow as tf
@tf.function
def sample(inputs:tf.Tensor):                                                   # some sample function

  def numpy_method(inputs:np.ndarray):
    # some complex method
    return np.sqrt(inputs)# example code

  inputs=tf.keras.layers.AvgPool2D()(inputs)                                    # works fine
  inputs=tf.numpy_function(numpy_method,[inputs],inputs.dtype,name='outputs')   # make apply numpy method

  return inputs

sample(tf.random.uniform((1,32,32,3)))

Output
<tf.Tensor: shape=(1, 16, 16, 3), dtype=float32, numpy=
array([[[[0.7783269 , 0.6260437 , 0.81261575] ......

This issue was also mention here -> #37193 "Cannot take the length of shape with unknown rank" error in but not resolved.

I was able to reproduce this issue in TF 2.11 and TF Nightly 2.13.0-dev20230219 . Please find the gist here.
@SuryanarayanaY Could you please look into this issue. Thanks.

Can we update something here -> https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/script_ops.py#L768
somthing like this ->
# ......
def numpy_function(func, inp, Tout, outputs_shapes:tuple=None, stateful=True, name=None):

  ''' docs ------
  '''

  outputs = py_func_common(func, inp, Tout, stateful=stateful, name=name)# line 768
  if isinstance(outputs,tf.Tensor) and (outputs.shape==None):
    if outputs_shapes==None:
      if isinstance(inp,(list,tuple)): outputs.set_shape(inp[0].shape)
      else: outputs.set_shape(inp.shape)
    else: outputs.set_shape(outputs_shapes)
  #elif isinstance(outputs,(list,tuple)): .....
  return outputs

# .....

Hi @MegaCreater ,
I have tested the proposed changes and seems working fine for this case.Please refer to attached tested gist. At least the changes works fine with and without @tf.function. I am not sure whether this covers all cases or not. Also it involves the structural changes in API as it adds new argument also.This may needs to be reviewed properly for short falls if any.
The proposed changes relies completely on input_shape which we are assuming that output_shape also being same as input_shape. What about the case when there is change in shapes that actually happens inside the numpy_method ?
Could we cover this to all use cases ?  Thanks!

Yes we can. When the shapes of output is same then user wont have to pass output_shape variable (i.e output_shape=None, but when output_shape changes then output_shape has to be provided. Give me some time to work on it more. I will update you.
But still with this update rank issue is not resolved. I am look for that also.

@MegaCreater ,
If output_shape shape has to be passed by the user explicitly when output_shape changes based on computation then the same thing can be done by inputs.set_shape([None,32,32,3])  that proposed in earlier workaround right.
I would like to understand how it is different from earlier workaround. For me the earlier work around using set_shape might work for all cases.Is there any exception for this ?

I feel like this should be documented officially. I just wasted a few hours pulling my hairs :(


Modify cg-site font sizes for UX

The font sizes on the billboard displays for the main site are too
large. There's been usability problems on smaller screens and
devices.
Ref https://favro.com/card/1e11108a2da81e3bd7153a7a/18F-2886

Hmm, billboard uses $sans-s9, not serif, fyi.

I could change those too


[FALSE-POSITIVE]

Domains or links
Please list any domains and links listed here which you believe are a false positive.
pxl.to
https://pxl.to/si-dhl
More Information
How did you discover your web site or domain was listed here?
Incorrectly marked as Phishing on Phishtank or OpenPhish?
Have you requested removal from other sources?
Please include all relevant links to your existing removals / whitelistings.
We've requested removal from the following vendors:
CyRadar
Fortinet
Netcraft
Sophos
CRDF
ESET
Kaspersky
Phishing
VIPRE
alphaMountain
Additional context
Add any other context about the problem here.
We're a link shortening service, and it was correctly reported several days ago that a URL created with our service was redirecting to a phishing website. We have now terminated hosting for this URL.
As of now, the URL redirects to our generic 404 page and is no longer malicious.
:exclamation:
We understand being listed on a Phishing Database like this can be frustrating and embarrassing for many web site owners. The first step is to remain calm. The second step is to rest assured one of our maintainers will address your issue as soon as possible. Please make sure you have provided as much information as possible to help speed up the process.
Send a Pull Request for faster removal
Users who understand github and creating Pull Requests can assist us with faster removals by sending a PR to mitchellkrogza/phishing repository, on the falsepositive.list file

https://github.com/mitchellkrogza/phishing/blob/main/falsepositive.list
Please include the same above information to help speed up the whitelisting process.

https://www.virustotal.com/gui/url/cf8faa590e8b45d45338f241bc94886c31e899854770925812ce1f384e4923a2?nocache=1

when the blocked virus will be removed phishing database?


Update to v1.13, Refreshed Recipe

Checklist

[ ] Used a fork of the feedstock to propose changes
[ ] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[ ] Ensured the license file is being packaged.

@conda-forge-admin, please rerender

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.

@conda-forge-admin, please restart ci

@conda-forge-admin, please rerender

@conda-forge-admin, please rerender

@conda-forge-admin, please rerender

@conda-forge-admin, please restart ci

@conda-forge-admin, please rerender


Change Event also occurs at the initialisation.

Hi,
After some tests I can say that the change event is also triggered at the initialisation of the plugin and not only at "on choosing option" as it should.
Thanks in advance.

@joselrio to disable this event trigger set silentInitialValueSet: true

Oh... Thanks!
Already tried and it works well. On my case I'm not setting a default value that way I didn't try it. That said, as a suggestion you could improve the documentation by adding this extra info as well :)
Again, thanks for your work!


Remove duplicated assert in test

Description
This line can be safely removed because the two lines following this assert test exactly the same condition.
Related Issues
n/a
Check List

~[ ] New functionality includes testing.~

~[ ] All tests pass~

~[ ] New functionality has been documented.~

~[ ] New functionality has javadoc added~

[ ] Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
[x] Commits are signed per the DCO using --signoff
~[ ] Commit changes are listed out in CHANGELOG.md file (See: Changelog)~
~[ ] Public documentation issue/PR created~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

This change does not require a CHANGELOG.md addition (it is a small modification of a test). How can I get rid of the failing check then?

Created new flaky test issues:

https://github.com/opensearch-project/OpenSearch/issues/12619
https://github.com/opensearch-project/OpenSearch/issues/12620
https://github.com/opensearch-project/OpenSearch/issues/12622


Check for VK_AMD_device_coherent_memory support at the runtime

There is an EnforceCoherentMemory variable which controls whether to enforce this extension, however it would be better to check if extension is supported at the run time.
I found this tool to be useful while debugging hangs on Turnip driver (open-source Vulkan driver for Adreno) and we don't support VK_AMD_device_coherent_memory.

It makes sense to do this check, it's just cleaner.
I assume that Turnip supported the buffer marker extension then?

I assume that Turnip supported the buffer marker extension then?

It's not yet merged but yes (https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13553).


#173 + dedicated Tor relays

#173 + add 3 dedicated tor relays

TODO: update commit msg "tor: add 3 relay nodes to docker "

Closing for now


Update pytket, qiskit, and qiskit-ibm-provider

Updating qiskit only to 0.46, not to 1.0 yet.
Closes #249 .

CI tests are stalled because of queue size.

It seems that the error on CI was caused by the upgrade to qiskit-ibm-runtime 0.20.0; downgrading to 0.19.0 fixes the issue.


Add HTML example stories

It is probably worth documenting base USWDS styles that apply to just plain old HTML, without creating React components for them. For example, what does a ul, ol, dl look like with the USWDS CSS applied? We should add Storybook stories to illustrate these elements.
For a comprehensive list of HTML elements, see https://developer.mozilla.org/en-US/docs/Web/HTML/Element

It is probably worth documenting base USWDS styles that apply to just plain old HTML, without creating React components for them. For example, what does a ul, ol, dl look like with the USWDS CSS applied? We should add Storybook stories to illustrate these elements.
For a comprehensive list of HTML elements, see https://developer.mozilla.org/en-US/docs/Web/HTML/Element

It is probably worth documenting base USWDS styles that apply to just plain old HTML, without creating React components for them. For example, what does a ul, ol, dl look like with the USWDS CSS applied? We should add Storybook stories to illustrate these elements.
For a comprehensive list of HTML elements, see https://developer.mozilla.org/en-US/docs/Web/HTML/Element

The original USWDS documentation does this already.
https://designsystem.digital.gov/components/
Does this statisfy your proposed need? If not, help me understand the benefit of including HTML examples in this React library.

Late here but @plummer-flex point makes sense to me.  Our goal for the storybook is to provide examples of using our React component -  rather than replicating the main USWDS design documentation which will always be more up to date.  Closing this issue.


re-deploy landkid

re-deploy landkid to fix vulns

Downgrading?

Yeah,artifact-publish-token downgraded to 1.0.1, there's no 1.0.4 anymore


Code coverage

Blocked by https://github.com/benhoyt/goawk/issues/182

The GoAWK bug is now fixed by https://github.com/benhoyt/goawk/pull/201 and in GoAWK v1.24.0.

Thank you @benhoyt! I will try the new version.

Thanks all works.


Local filtering with server side rendering

I am using the react-table  ^6.7.6 version. I have requirement to get the data from server but to apply filter on given page & sorting within the fetch result i.e. local sorting and filtering.
Can some one please point me to an example. I followed this example https://react-table.js.org/#/story/server-side-data as reference but no luck so far.

This feature/issue has been tagged as one that will likely be fixed or improved in the next major release of React-Table. This release is expected to be completed in the next 6 - 8 months. Please see #1143 for more detail. If you believe this feature can be easily fixed, we invite you to fork the project, attempt the fix, and use it until the latest version is available. If the fix is simple enough, we will also consider it as a PR to the current version.


Missing indexes in loans_items

Most of the columns in derived table loans_items do not appear to be indexed.

Here's a version with indexes (indices?), in case it's helpful. I don't want to commit to my current fork and mess up the PR.
loans_items.sql.zip

Here's a version with indexes (indices?), in case it's helpful. I don't want to commit to my current fork and mess up the PR.
loans_items.sql.zip


How to display point cloud in image plane?

Hello, with the default kitti velodyne data, I was able to project the point cloud onto the image plane using this code:
   vld = np.fromfile(os.path.join(args.kitti_loc, 'velodyne', velo_files[seg_counter]), dtype=np.float32)
   vld = vld.reshape((-1, 4)).T

    # Reshape calibration files
    P2 = clb['P2']
    R0 = np.eye(4)
    R0[:-1, :-1] = clb['R0_rect']
    Tr = np.eye(4)
    Tr[:-1, :] = clb['Tr_velo_to_cam']

    # Prepare 3d points
    pts3d = vld[:, vld[-1, :] > 0].copy()
    pts3d[-1, :] = 1

    # Project 3d points
    pts3d_cam = np.matmul(np.matmul(R0, Tr), pts3d)
    mask = pts3d_cam[2, :] >= 0  # Z >= 0
    pts2d_cam = np.matmul(P2, pts3d_cam[:, mask])
    pts2d = (pts2d_cam / pts2d_cam[2, :])[:-1, :].T

    # draw points on img
    height, width, channels = mask_img.shape
    blank_image = np.zeros((height, width, channels), np.uint8)
    for x in range(pts2d.shape[0]):
        xcolor = int(pts2d[x, 0])
        ycolor = int(pts2d[x, 1])
        if 0 <= xcolor < img.shape[1] and 0 <= ycolor < img.shape[0]:
            blank_image[ycolor, xcolor, :] = colors['pink']

    cv2.imshow('', blank_image)
    cv2.waitKey(100)

This displays the point cloud correctly on the image plane:

However, when I use the same code to display your point cloud, I get this:

I know that the depth map was generated correctly (and shockingly accurate might I add) so can you please tell me how to project your point cloud onto the image plane? Thanks!

Hi
You can use this script to generate depth maps from point clouds https://github.com/mileyan/pseudo_lidar/blob/master/preprocessing/generate_disp.py.
But you have to comment out this line https://github.com/mileyan/pseudo_lidar/blob/master/preprocessing/generate_disp.py#L25.


disable_all_colors() not working?

It's probably me being silly and it using it correctly, but from what i could gather from the docs, calling it anywhere should just disable all colors;
>>> from colorclass import disable_all_colors
>>> print string
←[31mbla←[39m
>>> disable_all_colors()
>>> print string
←[31mbla←[39m
>>>

What am I doing wrong? tried it both on Windows 7, 10 and Ubuntu...

It only disables colors used by ColorClass instances, not system wide (except on Windows where it does disable colors until you close the cmd.exe window).

Well I only tried to use it with ColorClass obviously, but did not see that it worked,  not on windows and not on Linux, like I said.

Ah, didn't see how you created that variable. The function disables colors for any new Color instance created afterwards, not current ones.
>>> string = Color.red('test')
>>> str(string)
'\x1b[31mtest\x1b[39m'
>>> disable_all_colors()
>>> string = Color.red('test')
>>> str(string)
'test'
>>>

Ah, I see. I figured I was misusing it. It would be nice if it was documented though. I'll give this a try and report if I still have issues. Thanks!


Add more options for notifications

I would like to have notifications for:

[ ] When something is said in a channel.
I think this is very useful if you have an important channel, but with very little traffic
[ ] When a nick says something
This can be useful if you want to be notified when a user is active.

Might also be useful to have some sort of rate limiting.

I haven't implemented "When a nick says something", but that is not so important to me.


Build tcc fails because the related tag is absent in git repo

Describe the bug
vladimir@Veriton:~/Project2/tilck$ ./scripts/build_toolchain -s build_tcc
ARCH=i386
HOST_ARCH=x86_64
GCC_TC_VER=8.4.0
ARCH_GCC_TC_VER=stable-2020.02-2
CI Detected: None [Assuming user machine]
[build_toolchain] Executing single function 'build_tcc'
NOTE: Skipping GCC TOOLCHAIN (i386 - musl)
*** TCC ***
+ local url=git://repo.or.cz/tinycc.git
+ local tag=15e9b7384e7091200aa33499b1ddcbf6e2f6a959
+ local tarname=tcc.tgz
+ local dirname=tcc
+ '[' -f /home/vladimir/Project2/tilck/toolchain2/cache/tcc.tgz ']'
+ '[' -z 15e9b7384e7091200aa33499b1ddcbf6e2f6a959 ']'
+ git clone --branch 15e9b7384e7091200aa33499b1ddcbf6e2f6a959 --depth 1 git://repo.or.cz/tinycc.git tcc
Cloning into 'tcc'...
warning: Could not find remote branch 15e9b7384e7091200aa33499b1ddcbf6e2f6a959 to clone.
fatal: Remote branch 15e9b7384e7091200aa33499b1ddcbf6e2f6a959 not found in upstream origin
+ tar cfz tcc.tgz tcc
tar: tcc: Cannot stat: No such file or directory
tar: Exiting with failure status due to previous errors
+ mv tcc.tgz /home/vladimir/Project2/tilck/toolchain2/cache
+ rm -rf tcc
+ set +x
Extracting tcc.tgz...
Extracting completed.
/home/vladimir/Project2/tilck/scripts/bash_includes/tc/tcc: line 40: cd: tcc: No such file or directory
sed: can't read Makefile: No such file or directory
Running command: ./configure --enable-cross --cross-prefix=i686-linux- --enable-static --cpu=i386 --prefix=/ --extra-ldflags='-Wl,--oformat=elf32-i386 -static' --crtprefix=/lib/i686-tilck-musl --libpaths=/lib/i686-tilck-musl &> configure.log
Running command: make cross-i386 &> build.log
i686-linux-strip: 'i386-tcc': No such file
[build_toolchain] Executing single function: DONE
Host configuration

Linux distro and version
Linux Veriton 5.13.0-41-generic 20.04.1-Ubuntu SMP Wed Apr 20 13:16:21 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Compiler (name and version)
gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)

Tilck commit: please indicate the exact commit used to reproduce the problem
b4ba5e7b1c04bcedbe233252b79e964bc937a201

Type of issue (select one)

Toolchain build problem

P.S. tcc.tgz in cahe is empty.
P.P.S. It's desirable to have minimal error processing in scripts to exit early after the critical errors.
Best regards,
Vladimir

Thank you for the bug report, Vladimir.
I've just fixed the issue with commit d03e87636f604551c13020e652e52db52acce69c.
It looks like the TCC repo changed something server side and now it doesn't accept fetch of untagged refs. Since TCC has no recent tags, I have no choice but to point to a specific commit SHA. That's not supported by their server anymore, maybe because they switched to BitBucket, I don't know. See this: https://stackoverflow.com/a/51002078/2077198
My fix does full git clone and then checks out the pointed commit, in case the direct clone fails and we referred to an untagged commit.
Note: in order to try TCC on Tilck, you'll need to re-run ./scripts/cmake_run (to discover the new package in the TC) and then to run ccmake to enable EXTRA_TCC option that now will be available in your build. In alternative to ccmake, you can enable the option with:  ./scripts/cmake_run -DEXTRA_TCC=1. Clearly, after that, run make.
Enojy!

Thank you, it works like a charm!
P.S. Though the version information does not look convenient to use and support:
Host:
vladimir@Veriton:~$ busybox | head -1
BusyBox v1.30.1 (Ubuntu 1:1.30.1-4ubuntu6.4) multi-call binary.
vladimir@Veriton:~$ tcc -v
tcc version 0.9.27 (x86_64 Linux)
Tilck:

There is no limit to perfection to have a version number and the latest commit tag for the git based apps :-)

@duselguy  You're welcome!
What do you mean with "the version information does not look convenient to use and support"? Tilck's build scripts do nothing in particular to add meta-information to the version of each supported app, if that's what you mean. If you refer to the TCC version that looks weird, consider that 0.9.27 has been released several years ago, and since then nobody tagged a commit with a new version. Tilck is using a much more recent version of TCC, which has no tags, just a commit SHA. Probably that version (commit 15e9b7384e709120) is more recent than the one you have on your host machine, as Linux distros tend to stick with the latest tagged version, no matter what. In my case, that didn't work, because I use do a cross-build, which is not properly supported by TCC. With commit 15e9b7384e709120 some of the bugs in 0.9.27 have been fixed, but it still required hacks to build.

My previous comment was rhetorical (and not about tilck project). The output of -v is (traditionally) incomplete and non-standard to give sufficient information about the state of the product. And the date/MSK (busybox) and d03e8763 (tilck) values look odd (to me) as they refer to the product's environment rather than being a useful addition to the version information of the product itself.

@duselguy I fixed that problem as well in commit 0afb60c2dab87caa620e34b0cbb55259241f6b56.
Now, after pulling the latest commit, do:
1. rm -rf toolchain2/i386/tcc
2. rm toolchain2/cache/tcc-15e9b7384e709120.tgz
3. ./scripts/build_toolchain -s tcc
4. make rem

And you'll see that tcc -v will show its own commit SHA.

@vvaltchev
Thank you, is ok now.


enable per frame delay

Hey there,
I think this is a pretty straight-forward change that enables some great use cases.
Cheers,
xUMi

if we use the setDelay function before adding a frame then we can achieve the same thing. Cant we?(just confirming anyways this way would be more user friendly but it seems the library has been abandoned)


Allow access to billing page without talking to stripe.

Run ./manage.py populate_billing_realms. Avoid running the server while doing so to avoid any weird behaviour.

Avoid running the server while doing so to avoid any weird behaviour

What kind of weird behavior should we expect? Maybe better to reply in chat.zulip.org for greater visibility.

This seems like progress, merged, thanks @amanagr!


Expose ParseUser.SessionToken in the public API

The SessionToken is currently exposed as an internal function. But it is useful to pass the SessionToken to the BecomeAsync(string sessionToken) function to log in as a particular user.
Both the iOS and Javascript APIs already expose the sessionToken, so the .NET API should as well.

This is solved and was implemented in the past.
Unfortunately this issue never received an update on this but the SessionToken of an ParseUser instance is publicly accessible.


Distance model via quadratic regression

This uses the calibration step to fit quadratic regression coefficients on the homography matrix.

The merge commit was due to a conflict in vision.py which I thought was worth resolving in this branch.


Discussion: Autotools will be deprecated in 2.0.0

As per 2.0.0 I will deprecate the use of autotools (autoconf, automake). As it is not as cross-platform minded as CMake. And I am unable to maintain two build systems. Installing on windows autotools is a pain and I want to create IDE or console projects. The reasons why I only want to support CMake:

No obsecure m4-macro system, and have a decent programming language (no wrapped shell commands) for the build system

Loops
Conditionals
Variables

Easy to maintain different targets with different compiler flags
Generate files (from templates) in a cross-platform fashion (not using sed,grep awk etc)
Builtin testing facility (CTest)
Builtin packaging facility (CPack)
Multiple output generators (Make, Xcode (OSX), Visual Studio)

One current use-case is:
Generate a version.h from version.h.in template and use it to notify users which version they run. Completely automatic created from git tags or version.txt file.

I agree with you. So lets deprecate autotools, unless there is a strong
argument about keeping it.
2016-06-07 19:24 GMT+02:00 Jerry Jacobs<EMAIL_ADDRESS>
As per 2.0.0 I will deprecate the use of autotools (autoconf, automake).
As it is not as cross-platform minded as CMake. And I am unable to maintain
two build systems. Installing on windows autotools is a pain and I want to
create IDE or console projects. The reasons why I only want to support
CMake:

No obsecure m4-macro system, and have a decent programming language
for the build system
Easy to maintain different targets with different compiler flags
Generate files (from templates) in a cross-platform fashion (not
using sed,grep awk etc)
Builtin testing facility (CTest)
Builtin packaging facility (CPack)
Multiple output generators (Make, Xcode (OSX), Visual Studio)

One current use-case is:
Generate a version.h from version.h.in template and use it to notify
users which version they run. Completely automatic created from git tags or
version.txt file.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/texane/stlink/issues/431, or mute the thread
https://github.com/notifications/unsubscribe/AAEeH3XDBnmV08vu78bHzfUhaKTGUqO0ks5qJalfgaJpZM4IwKam
.

Thank you @texane for the feedback, there are tons of other nice features why we should stick with cmake. I will keep things posted here when I need some discussion.

👍  for CMake, I see no point in keeping autotools as well.

Fixed by PR #434


Support for Multi Label Classification

🚀 Feature
Currently Classification assumes that each image only has one label this is not always the case we should brainstorm how to support multi label classification scenarios.

I need support submitting a patch for this while keeping consistent with our current guidlines.


Implement restore command

Description
This is another easy to implement command that shouldn't require writing data files. It makes the history command much more useful!
See docs: https://docs.databricks.com/delta/delta-utility.html#restore-delta-table
Use Case
Related Issue(s)

Hello,
I will start working on this.
I will use a similar approach as for Vacuum implementation.

Hi @Bernolt - there is a PR open (#852) which will intorduce some changes to the operations APIs. It is similar to the way how vacuum is implemented right now, but leverages the IntoFuture trait stabilized in 1.64. Not sure how far along you are, but if still feasible maybe consider this :).

Is this issue still available?

Yes, we haven't gotten a PR for this yet, but would be happy to review one. I haven't heard from Bernolt recently.

OK.I will look into it and try to file a PR.

Been busy with other things lately, now back to this feature. I've already read delta's implement, and will try to file recent days.

@wjones127 it's available in rust and python now, so we can close it :)


Bundler spams Gemfile lock with the same gem over and over

This is the same as #3216, I just didn't want to comment in that closed request.
I am having the issue using Bundler 1.7.12.
Gemfile.lock
treetop (1.4.15)
polyglot

 polyglot

 polyglot

... snip ...

 polyglot (>= 0.3.1)

 polyglot (>= 0.3.1)

Please post the full output from bundle env (cutting out any private credentials if they appear)

Closing for lack of information.


perf: Speed up compile times with nom

This is using a short-lived fork nom until we can get the changes upstreamed.  Note that combine requires using attempt to backtrack while nom backtracks by default and requires cut to not backtrack.
I find the straight functions much easier to understand what is happening and this allows us to intermix procedural with declarative logic (e.g. in string processing I skipped combinators to make it easier to not require an allocation).
Regarding that allocation, we still do it but this opens the door for us to use InternalString for user values which might give us more of a performance boost (previously, the forced allocation made it a moot point to measure it).
Running cargo clean -p toml_edit && time cargo check, I get 3s for building toml_edit with combine and 0.5s for nom.
For runtime performance

Parsing cargo inits generated manifest took 4% less time
Parsing cargos manifest took 2% less time
10 tables took 37% less time
100 tables took 41% less time
Array of 10 tables took 38% less time
Array of 100 tables took 40% less time

(some of this might be because we use dispatch more than before and some because our error reporting is less allocation heavy)
This is with Rust 1.66 on a i9-12900H processor under WSL2
Fixes #327

Also, compare CI times

Before: https://github.com/toml-rs/toml/actions/runs/3754649195
After: https://github.com/toml-rs/toml/actions/runs/3767930124


Switch onCheckedChange Handler Not Triggering After Closing Dialog

🐛 Bug report
I have a controlled Switch and controlled Dialog. Clicking the Switch prompts up a warning Dialog. If the user accepts the warning, the Switch turns on. If the user declines the warning, the Switch remains off.
The bug is that if the user declines the warning and closes the Dialog, the Switch needs to be clicked twice for the change handler to retrigger and open the Dialog again.
💥 Steps to reproduce

Go to reproduction
Click Switch
Click cancel button on Dialog
Click Switch again (nothing will happen, this is the bug)
Click Switch again (Dialog finally opens)

💻 Link to reproduction

https://stackblitz.com/edit/h4fpxy?file=src%2FApp.tsx
🧐 Expected behavior

The Switch should invoke the onCheckedChange handler on click after closing the Dialog, but it isn't doing that.
🧭 Possible Solution
🌍 System information

Software
Version(s)

Zag Version
0.77.1

Browser
Chrome

Operating System
Windows

📝 Additional information

If I comment out the Dialog component completely, i can click the Switch as many times as I want, and the correct controlled state is reported back through onCheckedChange handler every time.

I've looked at the code. You should probably wait for the next tick to synchronize the state.
 const onCheckedChange = useCallback(({ checked }: { checked: boolean }) => {
    queueMicrotask(() => {
      if (checked) {
        setIsConfirmDialogOpen(true);
      } else {
        setIsChecked(false);
      }
    });
  }, []);

https://stackblitz.com/edit/h4fpxy-mrlo2h?file=src%2FApp.tsx


Added_Node.js_Playlist

Hello Nirbhay,
I have added a Node.js course playlist by Andrew Mead under the youtube tutorials section.
Hope many node.js enthusiasts benefit from this awesome list including myself.
Regards,
Ashish Mishra

Thanks for adding.