⚠ SolarOscp has degraded performance

In b367430, SolarOscp ($SOLAROSCP_URL) experienced degraded performance:

HTTP code: 200
Response time: 438 ms

Resolved: SolarOscp performance has improved in d73a11d after 10 minutes.


Anerdw: Watch nativeindianorganics.com

Anerdw requests the watch of the watch_keyword nativeindianorganics\.com. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

nativeindianorganics\.com has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by Cow in Charcoal HQ


README.MD missing BreakpointEvent moved to core

Hi,
I had some trouble following the README.MD when it came to using BreakpointEvent as it seems to have moved from ps4debug.BreakpointEvent to ps4debug.core.BreakpointEvent. As I'm new to it and only reached that point so far while not exactly copying the code from the README.MD I opted to not pull request an untested change. Could do it later though if you want me to.
Best,
b2ag

Also looks like set_breakpoint is a coroutine and has to be awaited.
async with ps4.debugger(pid, resume=True) as debugger:
    debugger.set_breakpoint(0, True, address, on_hit=callback)

Hi, excuse me for the late reply. You are correct, I must have missed those two examples when refactoring to an async framework. I'll adjust the README.md soon.

With ae2f426a1f19724eaa5fb78d45ae05a4b3c6b15d the README.md should be adjusted to the actual codebase now. Thanks for using the library!


WID-115. Add siibra components

This PR relates to issue WID-115 and includes 2 new components for interacting with siibra:

The first component uses the code from tvb-root to get the structural&functional connectivities from siibra. Here multiple connectivities can be retrieved

The second one lays under TVB_CONNECTIVITY and uses the output from the first component to select just 1 connectivity, which can be later used in a simulation scenario.

Note: For the moment we don't do anything with the Functional Connectivities.

@romina1601 could you take care of rebasing this PR?

@romina1601 could you take care of rebasing this PR?

yes, will do it today.

We decided for the moment to do a refactor to the workflow that uses siibra, such that only one component is necessary. This was done to make other workflows cleaner (ex. an entire simulation workflow).
In the future, however, it might make sense to divide the retrieval of connectivities using siibra and their selection for simulation into multiple components (especially if/when we decide to also use the functional connectivities from siibra).


The scroll of the code block

@mittsuu Hi!
I would like to add the option to enable the scroll of the code block. ⛳

That sounds great idea!
Waiting for your pull request.

Hi, ＠ykyouhei
The code scroll enable was added by v1.0.4 last.
Please try.
Release Note v1.0.4

@mittsuu Hi!
very thanks! I try it!


php and go script should take option for temperature and hunidity calibration

Your scripts are working very well but unfortunatelly my sensor has an offset between real and measured temperature.
An option for calibration both temperature and humidity would be an great enhancement.

The Go tool now also has calibration options. Please give it a try.


Update sbt-tpolecat to 0.1.19

Updates io.github.davidgregory084:sbt-tpolecat from 0.1.18 to 0.1.19.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "io.github.davidgregory084", artifactId = "sbt-tpolecat" } ]

labels: sbt-plugin-update, semver-patch

Superseded by #141.


Trying to train with HTC, 2 bugs found

Describe the bug
First I get bug #977 Not clear why that line changes the file name. If removed, then I get this exception:
Traceback (most recent call last):
  File "tools/train.py", line 108, in <module>
    main()
  File "tools/train.py", line 104, in main
    logger=logger)
  File "/home/eduardo/dev/mmdetection/mmdet/apis/train.py", line 60, in train_detector
    _non_dist_train(model, dataset, cfg, validate=validate)
  File "/home/eduardo/dev/mmdetection/mmdet/apis/train.py", line 221, in _non_dist_train
    runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/mmcv-0.2.12-py3.7-linux-x86_64.egg/mmcv/runner/runner.py", line 358, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/mmcv-0.2.12-py3.7-linux-x86_64.egg/mmcv/runner/runner.py", line 264, in train
    self.model, data_batch, train_mode=True, **kwargs)
  File "/home/eduardo/dev/mmdetection/mmdet/apis/train.py", line 38, in batch_processor
    losses = model(**data)
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/eduardo/dev/mmdetection/mmdet/core/fp16/decorators.py", line 49, in new_func
    return old_func(*args, **kwargs)
  File "/home/eduardo/dev/mmdetection/mmdet/models/detectors/base.py", line 86, in forward
    return self.forward_train(img, img_meta, **kwargs)
  File "/home/eduardo/dev/mmdetection/mmdet/models/detectors/htc.py", line 189, in forward_train
    loss_seg = self.semantic_head.loss(semantic_pred, gt_semantic_seg)
  File "/home/eduardo/dev/mmdetection/mmdet/core/fp16/decorators.py", line 127, in new_func
    return old_func(*args, **kwargs)
  File "/home/eduardo/dev/mmdetection/mmdet/models/mask_heads/fused_semantic_head.py", line 104, in loss
    loss_semantic_seg = self.criterion(mask_pred, labels)
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/loss.py", line 916, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/torch/nn/functional.py", line 1995, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/torch/nn/functional.py", line 1826, in nll_loss
    ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: 1only batches of spatial targets supported (non-empty 3D tensors) but got targets of size: : [1, 100, 128, 3]

Reproduction

What command or script did you run?

python tools/train.py cfg/htc_x101_32x4d_fpn_20e.py

Did you make any modifications on the code or config? Did you understand what you have modified?

Only pointed the configuration to my dataset paths.

What dataset did you use?

I am using my own COCO formatted dataset. It worked fine for Retinanet but does not work with HTC (config htc_x101_32x4d_fpn_20e).
With Retinanet I didn't provide segmentation. Here it seems mandatory (crashes if not provided), so I fill it with 4 coordinates of the bbox rectangle.
Environment

OS: Ubuntu 18.04.2
GCC 7.4.0
PyTorch version 1.2.0 (conda)
GPU model: 2080 ti
CUDA 10 and CUDNN 7.6.0

In config I changed all of these from True to False:
with_mask=False
with_crowd=False
with_label=False
with_semantic_seg=False

And the new error is:
File "/home/eduardo/dev/capilos/mmdetection/mmdet/models/detectors/base.py", line 86, in forward
    return self.forward_train(img, img_meta, **kwargs)
TypeError: forward_train() missing 1 required positional argument: 'gt_labels

Does that make any sense? Reading #1024 it seems that mask is required, how should I provide that for each bbox rectangle?

I guess the mask must be the png file and original image jpg?
I have mixed images in my dataset, so should I convert all of them to jpg and provide the mask in png always?

HTC is for instance segmentation. Similar to Mask R-CNN, the mask of each instance should be annotated. If your dataset does not provide such annotation, HTC/Mask R-CNN is not proper for usage. Furthermore, HTC can also utilize the semantic segmentation annotation, in the config directory, there are two kind of settings that with or without semantic segmentation annotations.

As this is the third time I see this question, perhaps it should be in the
readme? A better error message will also be good.
On Fri, Aug 9, 2019, 18:09 Kai Chen<EMAIL_ADDRESS>wrote:

HTC is for instance segmentation. Similar to Mask R-CNN, the mask of each
instance should be annotated. If your dataset does not provide such
annotation, HTC/Mask R-CNN is not proper for usage. Furthermore, HTC can
also utilize the semantic segmentation annotation, in the config directory,
there are two kind of settings that with or without semantic segmentation
annotations.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/open-mmlab/mmdetection/issues/1154?email_source=notifications&email_token=AC7IWC5LKFUG3CG2A6KH2SDQDUCYXA5CNFSM4IKNZPMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD35WIGI#issuecomment-519791641,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AC7IWC3HIM3QTUC5DVBIMLLQDUCYXANCNFSM4IKNZPMA
.

Thank you for your help.
I was trying to improve my object detection model (now using retinanet_x101_32x4d_fpn_1x) with HTC because it seems it would produce a better bounding box precision than retinanet.
If I don't care about the mask, does it make sense if I add simple bbox mask data to my dataset and try?
Or should I use another model instead?
Thanks,
Eduardo

If you do not have mask annotations, you may try object detection methods instead of instance segmentation models. There are lots of supported detection methods in mmdetection.

@hellock Thank you, I am trying other models.
I tried GHM and it worked correctly but didn't get an improvement.
Later I have tried other models such as cascade_rcnn_x101_32x4d_fpn_1x and ga_faster_x101_32x4d_fpn_1x but I am getting this error:
Traceback (most recent call last):
  File "./tools/train.py", line 108, in <module>
    main()
  File "./tools/train.py", line 104, in main
    logger=logger)
  File "/home/eduardo/dev/mmdetection/mmdet/apis/train.py", line 58, in train_detector
    _dist_train(model, dataset, cfg, validate=validate)
  File "/home/eduardo/dev/mmdetection/mmdet/apis/train.py", line 186, in _dist_train
    runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/mmcv-0.2.12-py3.7-linux-x86_64.egg/mmcv/runner/runner.py", line 358, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/mmcv-0.2.12-py3.7-linux-x86_64.egg/mmcv/runner/runner.py", line 271, in train
    self.call_hook('after_train_iter')
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/mmcv-0.2.12-py3.7-linux-x86_64.egg/mmcv/runner/runner.py", line 229, in call_hook
    getattr(hook, fn_name)(self)
  File "/home/eduardo/dev/mmdetection/mmdet/core/utils/dist_utils.py", line 53, in after_train_iter
    runner.outputs['loss'].backward()
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/torch/tensor.py", line 118, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [512, 256, 7, 7]], which is output 0 of IndexPutBackward, is at version 4; expected version 3 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
Traceback (most recent call last):
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/torch/distributed/launch.py", line 246, in <module>
    main()
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/torch/distributed/launch.py", line 242, in main
    cmd=cmd)

Any idea of the possible cause?
My env is this:
Package          Version
---------------- --------------
absl-py          0.7.1
addict           2.2.1
certifi          2019.6.16
cffi             1.12.3
chardet          3.0.4
Click            7.0
cycler           0.10.0
Cython           0.29.12
decorator        4.4.0
Flask            1.1.1
Flask-Cors       3.0.8
grpcio           1.16.1
gunicorn         19.9.0
idna             2.8
imagecorruptions 1.0.0
imageio          2.5.0
itsdangerous     1.1.0
Jinja2           2.10.1
kiwisolver       1.1.0
Markdown         3.1.1
MarkupSafe       1.1.1
matplotlib       3.1.1
mkl-fft          1.0.12
mkl-random       1.0.2
mmcv             0.2.12
mmdet            1.0rc0+3180621
networkx         2.3
numpy            1.16.4
olefile          0.46
opencv-python    <IP_ADDRESS>
Pillow           6.1.0
pip              19.1.1
protobuf         3.8.0
pycocotools      2.0.0
pycparser        2.19
pyparsing        2.4.2
python-dateutil  2.8.0
PyWavelets       1.0.3
PyYAML           5.1.2
requests         2.22.0
scikit-image     0.15.0
scipy            1.3.0
setuptools       41.0.1
six              1.12.0
tensorboard      1.14.0
tensorboardX     1.8
terminaltables   3.1.0
torch            1.2.0
torchvision      0.4.0
urllib3          1.25.3
Werkzeug         0.15.4
wheel            0.33.4

Thanks,
Eduardo

Please refer to #1160 for PyTorch 1.2.

@hellock Thank you I will try that :)

Describe the bug
First I get bug #977 Not clear why that line changes the file name. If removed, then I get this exception:
Traceback (most recent call last):
  File "tools/train.py", line 108, in <module>
    main()
  File "tools/train.py", line 104, in main
    logger=logger)
  File "/home/eduardo/dev/mmdetection/mmdet/apis/train.py", line 60, in train_detector
    _non_dist_train(model, dataset, cfg, validate=validate)
  File "/home/eduardo/dev/mmdetection/mmdet/apis/train.py", line 221, in _non_dist_train
    runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/mmcv-0.2.12-py3.7-linux-x86_64.egg/mmcv/runner/runner.py", line 358, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/mmcv-0.2.12-py3.7-linux-x86_64.egg/mmcv/runner/runner.py", line 264, in train
    self.model, data_batch, train_mode=True, **kwargs)
  File "/home/eduardo/dev/mmdetection/mmdet/apis/train.py", line 38, in batch_processor
    losses = model(**data)
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/eduardo/dev/mmdetection/mmdet/core/fp16/decorators.py", line 49, in new_func
    return old_func(*args, **kwargs)
  File "/home/eduardo/dev/mmdetection/mmdet/models/detectors/base.py", line 86, in forward
    return self.forward_train(img, img_meta, **kwargs)
  File "/home/eduardo/dev/mmdetection/mmdet/models/detectors/htc.py", line 189, in forward_train
    loss_seg = self.semantic_head.loss(semantic_pred, gt_semantic_seg)
  File "/home/eduardo/dev/mmdetection/mmdet/core/fp16/decorators.py", line 127, in new_func
    return old_func(*args, **kwargs)
  File "/home/eduardo/dev/mmdetection/mmdet/models/mask_heads/fused_semantic_head.py", line 104, in loss
    loss_semantic_seg = self.criterion(mask_pred, labels)
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/loss.py", line 916, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/torch/nn/functional.py", line 1995, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/eduardo/miniconda2/envs/pytorch/lib/python3.7/site-packages/torch/nn/functional.py", line 1826, in nll_loss
    ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: 1only batches of spatial targets supported (non-empty 3D tensors) but got targets of size: : [1, 100, 128, 3]

Reproduction

What command or script did you run?

python tools/train.py cfg/htc_x101_32x4d_fpn_20e.py

Did you make any modifications on the code or config? Did you understand what you have modified?

Only pointed the configuration to my dataset paths.

What dataset did you use?

I am using my own COCO formatted dataset. It worked fine for Retinanet but does not work with HTC (config htc_x101_32x4d_fpn_20e).
With Retinanet I didn't provide segmentation. Here it seems mandatory (crashes if not provided), so I fill it with 4 coordinates of the bbox rectangle.
Environment

OS: Ubuntu 18.04.2
GCC 7.4.0
PyTorch version 1.2.0 (conda)
GPU model: 2080 ti
CUDA 10 and CUDNN 7.6.0

Since facing the same problems when training with myself COCO-like datasets and HTC, I finally found that there is a line of code in mmdetection/mmdet/datasets/coco.py, which reads the segmentation maps with "*.png" format on your host disk.
This means it needs not only "segmentation" fields in your coco-like json, but also the ground truth mask images.
Thus, you may need to write a method to load data into the seg_map.

Hi i got the same RuntimeError: 1only batches of spatial targets supported (non-empty 3D tensors) but got targets of size problem but i just use the coco datasets with no modification. and this is my data structure:

coco
annotations
---instances_{trail,val}2017.json
---stuff_{train,val}2017.json
train2017
---0000.jpg
val2017
---0000.jpg
stuffthingmaps
---train2017
------0000.png
---val2017
------0000.png

can someone help me? thanks in advance.

met the same error here with HTC and coco2017, mmdet2.8.0
File "tools/train.py", line 193, in <module>
    main()
  File "tools/train.py", line 189, in main
    meta=meta)
  File "/home/work/JXF/mmdetection/mmdet/apis/train.py", line 150, in train_detector
    runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
  File "/home/work/.virtualenvs/mmdetection/lib/python3.6/site-packages/mmcv/runner/epoch_based_runner.py", line 125, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/work/.virtualenvs/mmdetection/lib/python3.6/site-packages/mmcv/runner/epoch_based_runner.py", line 50, in train
    self.run_iter(data_batch, train_mode=True)
  File "/home/work/.virtualenvs/mmdetection/lib/python3.6/site-packages/mmcv/runner/epoch_based_runner.py", line 30, in run_iter
    **kwargs)
  File "/home/work/.virtualenvs/mmdetection/lib/python3.6/site-packages/mmcv/parallel/data_parallel.py", line 67, in train_step
    return self.module.train_step(*inputs[0], **kwargs[0])
  File "/home/work/JXF/mmdetection/mmdet/models/detectors/base.py", line 236, in train_step
    losses = self(**data)
  File "/home/work/.virtualenvs/mmdetection/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/work/.virtualenvs/mmdetection/lib/python3.6/site-packages/mmcv/runner/fp16_utils.py", line 84, in new_func
    return old_func(*args, **kwargs)
  File "/home/work/JXF/mmdetection/mmdet/models/detectors/base.py", line 176, in forward
    return self.forward_train(img, img_metas, **kwargs)
  File "/home/work/JXF/mmdetection/mmdet/models/detectors/two_stage.py", line 164, in forward_train
    **kwargs)
  File "/home/work/JXF/mmdetection/mmdet/models/roi_heads/htc_roi_head.py", line 256, in forward_train
    loss_seg = self.semantic_head.loss(semantic_pred, gt_semantic_seg)
  File "/home/work/.virtualenvs/mmdetection/lib/python3.6/site-packages/mmcv/runner/fp16_utils.py", line 164, in new_func
    return old_func(*args, **kwargs)
  File "/home/work/JXF/mmdetection/mmdet/models/roi_heads/mask_heads/fused_semantic_head.py", line 105, in loss
    loss_semantic_seg = self.criterion(mask_pred, labels)
  File "/home/work/.virtualenvs/mmdetection/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/work/.virtualenvs/mmdetection/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 948, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/home/work/.virtualenvs/mmdetection/lib/python3.6/site-packages/torch/nn/functional.py", line 2422, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/work/.virtualenvs/mmdetection/lib/python3.6/site-packages/torch/nn/functional.py", line 2220, in nll_loss
    ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: 1only batches of spatial targets supported (3D tensors) but got targets of size: : [2, 100, 144, 3]

The error comes from seg map label files, the .png label files should be read with PIL.Image, just like the VOC dataset, these label images are in palette  mode.
If we just read it with opencv or mmcv, we will get labels shape like H,W,3 which should be H,W, so the torch.nn.CrossEntropy loss function went into error.

The error comes from seg map label files, the .png label files should be read with PIL.Image, just like the VOC dataset, these label images are in palette  mode. If we just read it with opencv or mmcv, we will get labels shape like H,W,3 which should be H,W, so the torch.nn.CrossEntropy loss function went into error.

Following the explanation you stated, how can I modify in the code?


Add custom operator support

This is a replacement of the following PR (it can't be reopen after force push...):
https://github.com/apache/incubator-heron/pull/3049
This is the implementation of CustomOperator based on this design doc:
https://docs.google.com/document/d/1XzF0IlfuaaW8Gx3cPx1xLtP-kgCFK0TRNS5aAzuMuMg/edit#
CustomOperator and CustomOperatorOutput are included in this PR.
Future works:

add/refactor windowed custom operator
refactor/simplify existing operators (v2 design in doc)

@nwangtw can you please provide an example topology that uses the custom operator?  Also, isn't the point of the custom operator to provide some sort of compatibility with the "Storm" API?  Instead of forcing users to copy their code to the corresponding "prepare" and "execute" methods in the custom operator, couldn't we just do something like CustomOperator(Bolt)?  So that the user that just use their existing spout or bolt implementation? and the CustomOperator is just a wrapper for bolts which allows them to be used in streamlet topologies.

@nwangtw can you please provide an example topology that uses the custom operator?
Sure.

Also, isn't the point of the custom operator to provide some sort of compatibility with the "Storm" API? Instead of forcing users to copy their code to the corresponding "prepare" and "execute" methods in the custom operator, couldn't we just do something like CustomOperator(Bolt). So that the user that just use their existing spout or bolt implementation? and the CustomOperator is just a wrapper for bolts which allows them to be used in streamlet topologies.

Not really in this step/PR.
The goal of this step is to provide a way for users to write their own operators when needed. The low level Bolt is supported by previous PRs.

It does blur the line a bit to allow users to write their own operators with all functions available in low level API. However these operators still has type safety and can be used the same way as other pre-defined operators. Therefore, from API point of view, I don't think it "bring the streamlet API back to the bolt and spout API." Instead, it brings more powers from the bolt and spout API to streamlet.
Basically Streamlet is a functional API, and function API is overall nice and elegant. However I believe, a functional API is HARDLY a general purpose API and it is only good for "nice" problems. It might enforce uses to change the way of thinking and solving problems, which can be good sometimes (use general/standard operations) and bad in other times (more like hacks). Therefore I think supporting custom operator is important for Heron.
That being said, the original feature I needed was to reuse existing Bolts and Spouts which has been completed. This design doc and PR are just an extension from the original feature so it is not as urgent and we still have time to discuss.


"get access" and "share access" tabs show the same content until you've added a peer

Closing as this is not a priority right now.


Duplicate records in syslog

/etc/rsyslog.d/50-default.conf in an overlay for stretch has practically same content as default /etc/rsyslog.conf, leading to most records printed twice in the logs:
Jul 27 16:41:21 z liblogging-stdlog: action 'action 9' suspended, next retry is Thu Jul 27 16:41:51 2017 [v8.24.0 try http://www.rsyslog.com/e/2007 ]
Jul 27 16:41:21 z liblogging-stdlog: action 'action 9' suspended, next retry is Thu Jul 27 16:41:51 2017 [v8.24.0 try http://www.rsyslog.com/e/2007 ]
Jul 27 16:44:08 z liblogging-stdlog: action 'action 9' suspended, next retry is Thu Jul 27 16:44:38 2017 [v8.24.0 try http://www.rsyslog.com/e/2007 ]
Jul 27 16:44:08 z liblogging-stdlog: action 'action 9' suspended, next retry is Thu Jul 27 16:44:38 2017 [v8.24.0 try http://www.rsyslog.com/e/2007 ]
Jul 27 16:45:01 z liblogging-stdlog: action 'action 9' suspended, next retry is Thu Jul 27 16:45:31 2017 [v8.24.0 try http://www.rsyslog.com/e/2007 ]
Jul 27 16:45:01 z liblogging-stdlog: action 'action 9' suspended, next retry is Thu Jul 27 16:45:31 2017 [v8.24.0 try http://www.rsyslog.com/e/2007 ]
Jul 27 16:45:01 z CRON[6610]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jul 27 16:45:01 z CRON[6610]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jul 27 16:52:23 z liblogging-stdlog: action 'action 9' suspended, next retry is Thu Jul 27 16:52:53 2017 [v8.24.0 try http://www.rsyslog.com/e/2007 ]
Jul 27 16:52:23 z liblogging-stdlog: action 'action 9' suspended, next retry is Thu Jul 27 16:52:53 2017 [v8.24.0 try http://www.rsyslog.com/e/2007 ]

Please consider removing this file.

I'm also seeing many duplicates of this message rsyslogd-2007: action 'action 9' suspended, next retry is Fri Mar  2 08:36:47 2018 [v8.16.0 try http://www.rsyslog.com/e/2007 ] in our log files at Scaleway (this is on 16.04 on an ARM64-2GB node).

Same issue

Instance type: Barematal C1
Disrib: Debian GNU/Linux 9.4 (stretch)
Bootscript: armv7l 4.10.8 std #1 (stable)
Kernel: 4.10.8-std-1

Apr 10 22:05:01 zz CRON[7604]: pam_unix(cron:session): session opened for user root by (uid=0)
Apr 10 22:05:01 zz liblogging-stdlog[3617]: action 'action 9' suspended, next retry is Tue Apr 10 22:06:01 2018 [v8.24.0 try http://www.rsyslog.com/e/2007 ]
Apr 10 22:05:01 zz CRON[7605]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Apr 10 22:05:01 zz CRON[7604]: pam_unix(cron:session): session closed for user root

Any update on a official fix for this?

Hello,
I am preparing a new build of the stretch image. I will look into fixing this.


Can't use custom decoder or default encoding for ZIP without UTF8 supporting

After fix https://github.com/adamhathcock/sharpcompress/pull/363 it became impossible to use custom decoder or default encoding for ZIP with null bit 11. However, archives created by Windows 10 tools do not use the IBM 437 encoding. It looks like they use the current user encoding. Previously, you could specify it as the default encoding and the names were decoded successfully. But now it doesn't work.

Example.zip
See this example zip archive.

You're right....I will fix this now.

@saragulov look at the PR and see if this works for you...I can merge and do a bugfix release if so.  I believe using Forced to override seems correct.

Thank you for fix again. Can you do a bugfix release?

Thank you very much! Been waiting for the RAR5 support


Please provide more detailed and correct documentation

Please provide more detailed and correct documentation about the installation steps. There is as lots of stuff in the instructions that is missing. For example: to make FOSCommentBundle work you also need to activate FOS\RestBundle\FOSRestBundle() in Symfony's AppKernel.php. This step is missing!

There's always room for improvements. Feel free to make some suggestions (or even better: open a PR).


Battery API

Read battery status using the (deprecated) battery api, so we can eventually send that in

We have the data now, next to use it
closed by d5ef8863666ea1c8c42f5fcfdea722adebcf2574


Use the default primary color inside buttons

Use light grey instead of dark one in buttons to increase visibility.

@SimonDegraeve is this still a valid pull request? Why switch to the different class name? Do you have a before/after screenshot?


Android 7.1 and higher allow halfways safe use  of "Use System CAs" and the UI checks should reward it for that

From a mail from Ralf Paffrath on the eduroam SG list:
'on ANDROID 7.1.1 nougat you can configure “Use system certificates” and you can although
choose the domain (domain_suffix_match) e.g. radius server name the first time. If you choose Use system certificates you must
choose a domain_suffix_match also otherwise you can not go further.
So it is possible for an eduroam user to configure eduroam securely by the user interface without using eduroamCAT App if the IdP uses a PKI which RootCA is part of the system
certificates in ANDROID 7.'
But if you have installed eduroamCAT App already and use the user interface to configure eduroam securely the eduroamCAT App notifies there is something wrong with the eduroam configuration, some red checks: No CA certificate found and Server Subject Match missing. In both cases the notification are wrong because of the red checks and this is misleading for the eduroam user.
Expected behaviour:
if a system config has set both "Use System CAs" and "domain_suffix_match" then the CA info should not be an error (red), but a warning (yellow) stating:
"(!) All device CAs are trusted for this connection"
Also, if domain_suffix_match is set from end-user UI, the field "Server Subject Match" should go to green state.

hint: the android version is not a good identifier for this exception i've seen some Android 6 devices with this functionality and some Android 8 without... some do differ in extent of the features ege. some even allow to select a specific CA from storage (this maybe only Lineage OS but i'm not sure)...


Add combine filter functionality in MongoFilterGenerator

Closes #360
This PR adds the ability to combine two separate filters into one filter. Essentially, it performs an AND operation on two filters to combine them together. Now, we will be able to match on multiple properties.
For example:
username: "user1"
AND
password: "verysecurepassword!"
To use the CombineFilters method, you will need to first create two separate filters using the GetEqualsFilter method. Then, pass the two filters in to the CombineFilters method.

All open issues have been resolved


Poor memory usage in scipy.sparse to DMatrix conversion

When indices / indptr are an array of np.int64 the following line causes an error:
https://github.com/dmlc/xgboost/blob/78c4188cec31425f708d238160ea3afb67a7250a/python-package/xgboost/core.py#L160-L162
Traceback (most recent call last):
  File "run_xgboost.py", line 142, in <module>
    main()
  File "run_xgboost.py", line 116, in main
    dtrain = xgb.DMatrix(X, label=y, weight=weights)
  File "/usr/local/lib/python3.5/dist-packages/xgboost-0.6-py3.5.egg/xgboost/core.py", line 276, in __init__
    self._init_from_csc(data)
  File "/usr/local/lib/python3.5/dist-packages/xgboost-0.6-py3.5.egg/xgboost/core.py", line 322, in _init_from_csc
    c_array(ctypes.c_uint, csc.indices),
  File "/usr/local/lib/python3.5/dist-packages/xgboost-0.6-py3.5.egg/xgboost/core.py", line 162, in c_array
    return (ctype * len(values))(*values)
SystemError: ../Objects/tupleobject.c:71: bad argument to internal function

It is easy to reproduce with this minimal bit of code:
>>> import ctypes
>>> import numpy as np
>>> a = np.ones(np.iinfo(np.int32).max + 1, dtype=np.uint32)
>>> a[-1] = np.iinfo(np.int32).max + 123
>>> start = time.time()
>>> (ctypes.c_uint * len(a))(*a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: ../Objects/tupleobject.c:71: bad argument to internal function
>>> print(time.time() - start)
130.310668945331865

Also for my input array, which consumes 8GB of memory, the c_array function causes an explosion of memory usage up to 75GB, and it takes a horribly long 130 seconds at 100% cpu usage before failure.
This doesn't appear to be an out of memory issue as the machine that I am on still has 25GB of memory left when the error occurs.
This happens in python 3.5 and python 3.6
I have a potential solution:
(ctypes.c_uint * len(values)).from_buffer_copy(values)

If the input array is 8GB this solution only doubles the memory to 16GB, and it takes only 7 seconds.
>>> import time
>>> start = time.time(); x = (ctypes.c_uint * len(a)).from_buffer_copy(a); print(time.time() - start)
7.331865310668945

probably needs some checks on the dtype of the incoming array for buffer length and change the buffer size of the np.array if it doesn't match and can be converted


CLOUDSTACK-8964: Fix KVM snap to vol and snap to templ, no delegate for Ovm3..

Snapshot to volume, snapshot to template was tested with KVM.
Ovm3 does not have this functionality yet. The Ovm3HypervisorGuru was however giving an incorrect answer to a delegation request, which should never have been implemented.

Eh that's odd, it's supposed to wait for 2 seconds, waited for 1.002 and then fails because it waited too short ? what am I missing here ? :)

Thanks for taking this up @snuf. I am out of office and will be able to test it only on Monday.
@NuxRo Can you try this please?

I've built EL6 RPMs from https://github.com/snuf/cloudstack/archive/6efe627e8e11c18acf62ff02b85007de122a917c and "yum reinstall" management and agent.
(am I testing the right thing?)
No go.
Volume creation from snapshot still fails with this: http://fpaste.org/282421/44552050/raw/
Template creation from snapshot works OK.
Do note I have removed the Ovm3Guru from the hypervisor guru, excluding it again does not make a difference.

@NuxRo interesting... I'm checking out my commit again and rebuilding to test again, I'm running both at the same time though... so that might explain something perhaps..

@snuf The org.apache.cloudstack.utils.hypervisor.HypervisorUtilsTest.checkVolumeFileForActivityTest(HypervisorUtilsTest.java:70) test is IMHO a known issue. Are you building against JDK 8 or 7?

@borisroman problem is that some of our jenkins build slaves ("our" are the project ones) fail, so "restarting" it and hoping you'll end up on a build slave that does work is kind of what i did.... cough.

@NuxRo tested all the scenarios and I'm getting different results as before and different results than you, this is an excerpt of the log testing the various options: http://fpaste.org/282859/45595257/raw/
Oddly enough almost none of the OVM3 options work anymore, but all the KVM ones do in my case, I think we have a different problem, as in it's not one guru that's screwing up, but that's just my hunch will investigate further.

@Snuf is there a list of all hv gurus? I'd like to exclude all of them (bar kvm), see if that helps.

@NuxRo here's a list, there is no KVM Guru... ?
funs@cs-mgmt:~/cloudstack/plugins/hypervisors$ find . -type f -name *.xml | xargs grep -A 1 Guru | grep property | awk -F: '{ print $2 }' | uniq

@NuxRo @karuturi Superseded by #975 and in . Closing this one!

@snuf @NuxRo KVMGuru and LXCGuru are defined at https://github.com/apache/cloudstack/blob/master/server/resources/META-INF/cloudstack/server-compute/spring-server-compute-context.xml


Code runs OK in REPL but fails when compiled as script

Following code runs OK in amm REPL but fails when compiled as script:
val home = root/RelPath(System.getenv("HOME").substring(1))
val dir = home/'tmp/'amm/'bckp
val fileSeq = ls! dir
println(fileSeq);
val dirSeq : Seq[Path] = fileSeq.sorted |? ( f => f.isDir)
val dirSeq2 = dirSeq |? grep! "[0-9][0-9]".r
println(dirSeq2);

It gives following compilation error:
exp01.sc:17: type mismatch;
found   : ammonite.runtime.tools.grep.![scala.util.matching.Regex]
 required: ammonite.ops.Path => Boolean
val dirSeq2 = dirSeq |? grep! "[0-9][0-9]".r

@jk-1 I was able to get this to compile and run after importing
import ammonite.ops._ and import pprint.Config.Defaults._.
I got the idea for the pprint import from here. I believe this is due to changing what is loaded in scripts in version 0.7.2.
The APIs available by default in the REPL and in scripts is different, so some imports have changed.

Yes, you have to import ammonite.ops._ if you want ammonite ops APIs
As for grep, that's probably more meant for the REPL than in scripts. I have thus removed it from the script imports in https://github.com/lihaoyi/Ammonite/commit/b8fb39b94f7ecbc90508875bd92ff15ea14fb59d, which should avoid the confusing error message

http://www.lihaoyi.com/Ammonite/#0.9.0 is out


CB-7643 temporarily disable deterministic uid/gid script

Looks good. I think when we re-introduce it again then we shall refactor a bit to avoid copy paste and just use the include directive.

Looks good. I think when we re-introduce it again then we shall refactor a bit to avoid copy paste and just use the include directive.

The content is different. During upgrade we need to migrate otherwise it's just an init.

Right, to be honest, I had to go through two times to spot that one line has a -m flag...


Speech v1 GAPIC.

LGTM, let @geigerj approve as well.

Just merged googleapis/toolkit#1152 so we could get tests also. :-)


Need *.webp type images reader and writer

As more and more *.webp type images popping up everywhere, I hope commons imaging will help us to read and write this type of images.
As an example, https://i.guancha.cn/bbs/2019/07/10/20190710110612113 is pretty exciting.
Thanks very much!

Sounds like a good idea. I think there was some merge/rebase issue somewhere. There are way too many commits in this PR. You may want to review your commit log, rebase onto master, and make sure there are fewer changes in the PR, as that makes it easier to be reviewed/merged.
Thanks!

Commits still needing to be squashed. Right now in GitHub UI it shows 59 commits in this PR.

Hmmm, we can't really merge the PR until it is properly rebased. Where it is now saying in GitHub that "changbaishan wants to merge 120 commits into..." it should actually say something like "changbaishan wants to merge 1 commits into", because normally PR's contain just a few commits. I don't think these 120 commits are related to webp type images.
@changbaishan could you look into it, please? Happy to review once that's sorted.

Is there way I can release myself from this PR? I witnessed wonderful works you did. However, I opened it just to suggest a useful feature. I had no idea of what it actually meant. I still love your software. Just don't want to prolong a mistake.

Is there way I can release myself from this PR? I witnessed wonderful works you did. However, I opened it just to suggest a useful feature. I had no idea of what it actually meant. I still love your software. Just don't want to prolong a mistake.

Your action got the right reaction then :-) we definitely need a webp parser, and I will create a JIRA issue for that. And it was no mistake at all. Feel free to raise issues, pull requests, etc, for whatever new ideas or bugs you may have.
Cheers
Bruno

FYI https://issues.apache.org/jira/browse/IMAGING-57

Thanks very much! I really appreciate you and your work. Just that I don't know enough to handle the work. Sorry again. Looking forward to use your new release.


fix bug where quick pick always returns empty string

This fixes https://github.com/NVIDIA/bluebazel/issues/27

@jabbottn Hi, a few of us at work were starting to experiment with this project (thanks!) and we found this bug related to Pick inputs and hoped to get it merged soon.


Disable https on Android SDK fetch

Apparently it no longer works

@alexcrichton: no appropriate reviewer found, use r? to override

@bors: r+

:pushpin: Commit bccba4a has been approved by alexcrichton

:hourglass: Testing commit bccba4a996c04b62538ed7c65a1c274961f272e9 with merge c959b16f4f844ea4075969ce9f84c50991b33c4c...

:sunny: Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing c959b16f4f844ea4075969ce9f84c50991b33c4c to master...


Transloco MessageFormat Error: Localisation function not found for locale

I'm submitting a...

[ ] Regression (a behavior that used to work and stopped working in a new release)
[x] Bug report
[ ] Performance issue
[ ] Feature request
[ ] Documentation issue or request
[ ] Support request
[ ] Other... Please describe:

Current behavior

We use "@ngneat/transloco": "2.18.3" with "@ngneat/transloco-messageformat": "1.2.3"
We faced with issue that "transloco-messageformat" works wrong with locale named "default".
How we can fix it?
An error:
ERROR Error: Localisation function not found for locale "default"
    at w (messageformat.js:1)
    at new t (messageformat.js:1)
    at mfFactory (ngneat-transloco-messageformat.js:27)
    at MessageFormatTranspiler.setLocale (ngneat-transloco-messageformat.js:87)
    at MessageFormatTranspiler.onLangChanged (ngneat-transloco-messageformat.js:80)
    at TranslocoService.setActiveLang (ngneat-transloco.js:830)
    at app-localization.service.ts:24
    at ZoneDelegate.invoke (zone-evergreen.js:364)
    at Object.onInvoke (core.js:27147)
    at ZoneDelegate.invoke (zone-evergreen.js:363)

Expected behavior

"transloco-messageformat" should works correctly with any locale.
Minimal reproduction of the problem with instructions
Sample: https://github.com/aerialist7/transloco-bug-polygon
Run ng serve and open browser console on http://localhost:4200/

app-localization.service.ts contains language loading logic
assets/l10n.json - contains translations
assets/user.json - contains user language data

Environment

Angular version: 10.0.5

Browser:
- [x] Chrome (desktop) version XX
- [ ] Chrome (Android) version XX
- [ ] Chrome (iOS) version XX
- [ ] Firefox version XX
- [ ] Safari (desktop) version XX
- [ ] Safari (iOS) version XX
- [ ] IE version XX
- [ ] Edge version XX

For Tooling issues:
- Node version: 12.16.1
- Platform: Windows

P.S. It's strange, but if I change "default" to "en" all works!

@aerialist7 I see that the "default" is coming from your AppLocalizationService from the languagePreloader function so this bug isn't related to this library


Runtime integration for static web assets

Adds an extension method to IWebHostBuilder to plug-in static web
assets.
Plugs-in static web assets in development by default.

Can someone catch me up on the quick ~2 sentence reason for this new integration? Are static web assets part of single-file deployment or something else?

Can someone catch me up on the quick ~2 sentence reason for this new integration? Are static web assets part of single-file deployment or something else?

This is part of an E2E for multi-project development with static assets for Razor/Blazor. For us to grow a library ecosystem for Blazor, we need provide a way for developers to package assets like css or js with their libraries.
This works by:

Copying files to the output directory on publish
Providing a set of extra paths to server for local builds (run without publish)

The mechanism in this case is a file that's produced by the build, but not included in publishing.
The experience with this is really pretty nice because it will just work:

As a package author
As a person who uses multiple projects for any ASP.NET Core development
As a person who uses rando nuget packages

That's more than two sentence tho 😢

Cool, that's exactly what I was looking for!

Insufficient. If there's not a beahavioral difference then don't check IsDevelopment.

I think of this feature in the same way I think of user secrets and user secrets is keyed of development. I'm not sure how I feel about the app using this feature in a non development environment.
We don't want to enable this in publish if people copy the file to the publish output by carelessly updating a <content /> item in their project (or by some other SDK/props/targets including it by mistake)
Keyeing it off development ensures we don't run into those situations.

Are these ever going to be called from outside of ConfigureWebDefaults? If so then they need a different home.
No that I'm aware off (there is only a single extension method on IWebHostBuilder if you want to enable the feature unconditionally).
I'm not opposed to move this to a new package (in fact it was my first thought) but I won't be doing this on preview6 as its risky. Based on @rynowak lets wait until we find a good reason to move it into a new package.

@HaoK Helix is failing due to a completely unrelated test. Do we care?

@javiercn I don't believe helix-tests are a required check so I don't think you need to block on those, cc @aspnet/build to confirm that

@Tratcher I can't seem to be able to comment on the right thing

You don't need FilePathComparison, BasePath isn't considered as part of the file path, it's treated as a case insensitive prefix.

We do need to maintain the case sensitivity of the OS to avoid discrepancies with the published outcome.
At publish time the files end up as physical files under wwwroot in the published app. For example, if the base path is _content/library and there is a file in <<library>>\wwwroot\js\file.js. The file ends up in wwwroot\_content\library\js\file.js on the published app folder.
If we are in a case-sensitive file system and we don't respect the sensitivity, then during development a relative path to /_CONTENT/LIBRARY/js/file.js will work, and when the app its published and deployed it will start failing. To avoid that, we need to replicate the case sensitivity of the OS.
Keep in mind that the base path here represents a prefix to where the file ends up on disk, and not a base path in the URL sense.
@Tratcher Does that help clarify things?
Is there anything else to address or is it good to merge?

We do need to maintain the case sensitivity of the OS to avoid discrepancies with the published outcome.

That assumes your dev and production environments use the same OS and casing restrictions. Even with this check it could work fine on windows and fail when published to Linux. This check only ends up protecting Unix -> Unix, but I guess some safety is better than none.

Keep in mind that the base path here represents a prefix to where the file ends up on disk, and not a base path in the URL sense.

Ah, nothing in this PR conveys the meaning of BasePath, it's an arbitrary field in the manifest. You should add some doc comments to that effect.

That assumes your dev and production environments use the same OS and casing restrictions. Even with this check it could work fine on windows and fail when published to Linux. This check only ends up protecting Unix -> Unix, but I guess some safety is better than none.

That's the point, it offers the same guarantees. I've added a doc comment explaining the purpose and workings of the base path, and there will be documentation in the official docs

/Azure-pipelines run

This PR is on Preview 7 now, but I cannot find out any instructions to use it. Could someone help us?

@minhhieugma https://docs.microsoft.com/en-us/aspnet/core/razor-pages/ui-class?view=aspnetcore-3.0&tabs=visual-studio#create-an-rcl-with-static-assets


chore: remove legacy treemap/area/boxplot/pie

SUMMARY

Remove unused viz-plugins

Legacy TreeMap
NVD3 Area
NVD3 Pie
Legacy BoxPlot

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

N/A
TESTING INSTRUCTIONS

CI
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

@zhaoyongjie are we 100% sure these are unused? If not it seems like a migration would be necessary conditional on the fact that parity exists between the old NVD3 and new ECharts charts.

@john-bodley thanks for the tip, I updated PR, and the legacy plugins were been kept.


added minlength / maxlength to dictionaries

The minlength / maxlength rule, formerly only usable with strings
and lists, is now compatible with arbitrary dictionaries.
Tests and custom field in scheme were added as well.
Updated docs accordingly.

Hello, thanks for this. I'm willing to merge as soon as the note by @funkyfuture has been taken care of.

@fin-8 would you reapply your changes the current master's tip?
i tried to rebase it, but it's a pita.
alternatively, i could do it, but that would lose your credit in the commit.

@funkyfuture I think you can proceed with the rebase.


Split a router in multiple files or variables

Hi,
I'm unable to split a router into multiple files or variables. I prefer not to create nested routers.
I'm encountering a TypeScript error because getPost is not recognized.
const s = initServer();
const getPost = async ({ params: { id } }) => {
    return {
        status: 200,
        body: {
            id: "1",
            title: "my title",
            body: "000"
        }
    };
};
const createPost = async ({ body }) => {
    return {
        status: 201,
        body: {
            id: "1",
            title: "my title",
            body: "000"
        }
    };
};
const router = s.router(contract, {
    getPost,
    createPost
});
createExpressEndpoints(contract, router, app);

This code is fine, but I prefer not to have multiple endpoints in a single file.
    const s = initServer();

    const router = s.router(contract, {
        // move getPost in routers/post/get.ts
        getPost: async ({ params: { id } }) => {
            console.log(id);
            return {
                status: 200,
                body: {
                    id: "1",
                    title: "my title",
                    body: "000"
                }
            };
        },
        // move getPost in routers/post/create.ts
        createPost: async ({ body }) => {
            return {
                status: 201,
                body: {
                    id: "1",
                    title: "my title",
                    body: "000"
                }
            };
        }
    });

    createExpressEndpoints(contract, router, app);

The code below works, but doesn't seem very conventional.
const router1 = s.router(
    { getPost: contract.getPost },
    {
        getPost: async ({ params: { id } }) => {
            console.log(id);
            return {
                status: 200,
                body: {
                    id: "1",
                    title: "my title",
                    body: "000"
                }
            };
        }
    }
);
const router2 = s.router(
    { createPost: contract.createPost },
    {
        createPost: async ({ body }) => {
            return {
                status: 201,
                body: {
                    id: "23",
                    title: "toto",
                    body: "jhjkhk"
                }
            };
        }
    }
);
createExpressEndpoints(contract, { ...router1, ...router2 }, app);

Think you will have success in using the AppRouteImplementation type. Also you have the added benefit of having the request and response typed in your routes definitions 😉
import { type AppRouteImplementation } from '@ts-rest/express';

const s = initServer();

const getPost: AppRouteImplementation<typeof contract.getPost> = async ({ params: { id } }) => {
    return {
        status: 200,
        body: {
            id: "1",
            title: "my title",
            body: "000"
        }
    };
};
const createPost: AppRouteImplementation<typeof contract.createPost> = async ({ body }) => {
    return {
        status: 201,
        body: {
            id: "1",
            title: "my title",
            body: "000"
        }
    };
};
const router = s.router(contract, {
    getPost,
    createPost
});

createExpressEndpoints(contract, router, app);

Thanks @toteto is what I was looking for.

Think you will have success in using the AppRouteImplementation type. Also you have the added benefit of having the request and response typed in your routes definitions 😉
import { type AppRouteImplementation } from '@ts-rest/express';

const s = initServer();

const getPost: AppRouteImplementation<typeof contract.getPost> = async ({ params: { id } }) => {
    return {
        status: 200,
        body: {
            id: "1",
            title: "my title",
            body: "000"
        }
    };
};
const createPost: AppRouteImplementation<typeof contract.createPost> = async ({ body }) => {
    return {
        status: 201,
        body: {
            id: "1",
            title: "my title",
            body: "000"
        }
    };
};
const router = s.router(contract, {
    getPost,
    createPost
});

createExpressEndpoints(contract, router, app);

Works well enough, thanks!!

Example of how to type the middleware when doing kind of code splitting:
const getUserById: {
	middleware: [TsRestRequestHandler<typeof contract.getUserById>];
	handler: AppRouteImplementation<typeof contract.getUserById>;
} = {
	middleware: [
		(req, res, next) => {
			...
		},
	],
	handler: async ({ params: { id } }) => {
		...
	},
};

I have found that the optimal way of having good code splitting with ts-rest is to create multiple instances of whatever you need across different files (be it router or contract), and create separate implementations there for each as if they are all a standalone entity (this way you will also have TypeScript support, pulling all of the built-in types into you code, and not having to explicitly define them like I was doing in my comment above), and then merging all of that into a main router/contract in one file at the end.
Hope this example can help you understand what I am trying to say:
// userRouter.ts
import { initServer } from '@ts-rest/express';

const s = initServer();

const getUsersRoute = s.router(
	{ getUsers: contract.getUsers },
	{
		getUsers: async () => {
			// Handler logic here...
		},
	}
);

const getUserByIdRoute = s.router(
    { getUserById: contract.getUserById },
    {
        getUserById: {
            middleware: [
                (req, res, next) => {
                    // Middleware logic here...
                    next();
                },
            ],
            handler: async ({ params: { id } }) => {
                // Handler logic here...
            },
        },
    }
);

const userRoutes = {
    getUserById: getUserByIdRoute.getUserById,
    getUsers: getUsersRoute.getUsers,
};

export default userRoutes;

// postsRouter.ts
import { initServer } from '@ts-rest/express';

const s = initServer();

const getPostsRoute = s.router(
    { getPosts: contract.getPosts },
    {
        getPosts: async () => {
            // Handler logic here...
        },
    }
);

const postsRoutes = {
    getPosts: getPostsRoute.getPosts,
};

export default postsRoutes;

// mainRouter.ts
import { initServer } from '@ts-rest/express';
import userRoutes from './userRouter';
import postsRoutes from './postsRouter';

const s = initServer();

const mainRouter = s.router(contract, {
    ...userRoutes,
    ...postsRoutes,
    // Add other routes here...
});

export default mainRouter;

// server.ts
import express from 'express';
import { createExpressEndpoints } from '@ts-rest/express';
import mainRouter from './mainRouter';
import { contract } from './contract'; // Assume this is already defined

const app = express();

createExpressEndpoints(contract, mainRouter, app);

app.listen(3000, () => {
    console.log('Server is running on port 3000');
});

Use this same logic in the contract.
Keep in mind, this pseudocode is just an example of the thought process behind the implementations. If you can understand logically how it works, then you are good to go.
// usersContract.ts
import { initContract } from '@ts-rest/core';

const c = initContract();

const getUsersContract = c.router({
    getUsers: {
        method: 'GET',
        path: '/users',
        responses: {
            200: { description: 'List of users' },
            404: { description: 'Not found' },
            500: { description: 'Internal server error' },
        },
        summary: 'Get all users',
    },
});

const getUserByIdContract = c.router({
    getUserById: {
        method: 'GET',
        path: '/users/:id',
        responses: {
            200: { description: 'User details' },
            404: { description: 'User not found' },
            500: { description: 'Internal server error' },
        },
        summary: 'Get a user by ID',
    }
});

export const usersContract = {
	getUsers: getUsersContract.getUsers,
	getUserById: getUserByIdContract.getUserById,
};

// postsContract.ts

const getPostsContract = c.router({
    getPosts: {
        method: 'GET',
        path: '/posts',
        responses: {
            200: { description: 'List of posts' },
            404: { description: 'Not found' },
            500: { description: 'Internal server error' },
        },
        summary: 'Get all posts',
    },
});

const postsContract = {
    getPosts: getPostsContract.getPosts,
};

// mainContract.ts

export const contract = c.router({
    ...usersContract,
    ...postsContract,
});

@Gabrola This should really be in the docs, someplace easy to find.

@ElhanM I agree, we need to have a separate clear section in the docs about contract and router splitting - mainly to document the capabilities rather than be opinionated on how to split and re-combine the contracts or routers.
Regarding your code snippet, you can use c.query or c.mutation (currently undocumented and needs to be added to the docs) to define singular route definitions rather than have redundant names by doing const getUserByIdContract = c.router({ getUserById: { ... } }). However, I do not see the point in splitting up the definitions if they are still going to be kept in the same file like in your example - seems like extra effort to me, but anyone should do it however they please. Hence, why we'd just prefer to just document the capability and not be opinionated on the organization of the code.
There is a similar helper function in all server libraries to do the same for route implementations using s.route instead of s.router.

@Gabrola They are not in the same file, that is the whole point. Coding the routes/contracts this way allows us to define them in different files, and import/merge all of them together in the end. I put all of the code route/contract code into the same code block for simplicity, but I have comments in place that indicate they should be different files.
I now notice that I forgot to add exports to the routes/contracts, maybe that part was confusing (I put this together quickly, and did not bother going over it too much)
I will update my original comment with exports in place


Running go tests in virtual environment

Since running go get for the coverage tests needs sudo permission, the
tests can now run in a virtual environment to remove this dependency

If this works, works for me!

lgtm

same, lgtm


Spyder crashes during startup since upgrade to version 4

Hello,
Since update to version 4 with pip my spyder IDE is crashing at start up. I don't understand what's happen because I did the update with pip install --upgrade spyder and saw ERROR due to requirement incompatibility for others packages:
Successfully built python-language-server diff-match-patch watchdog intervaltree python-jsonrpc-server ujson pathtools
ERROR: spyder-kernels 1.8.1 has requirement pyzmq>=17, but you'll have pyzmq 16.0.2 which is incompatible.
ERROR: jupyter-client 5.3.4 has requirement jupyter-core>=4.6.0, but you'll have jupyter-core 4.4.0 which is incompatible.
ERROR: helpdev 0.6.10 has requirement psutil>=5.6, but you'll have psutil 5.4.2 which is incompatible.
ERROR: autopep8 1.4.4 has requirement pycodestyle>=2.4.0, but you'll have pycodestyle 2.3.1 which is incompatible.
ERROR: flake8 3.7.9 has requirement pycodestyle<2.6.0,>=2.5.0, but you'll have pycodestyle 2.3.1 which is incompatible.
ERROR: flake8 3.7.9 has requirement pyflakes<2.2.0,>=2.1.0, but you'll have pyflakes 1.6.0 which is incompatible.
Installing collected packages: parso, jedi, jupyter-client, ipykernel, spyder-kernels, qtconsole, more-itertools, zipp, importlib-metadata, helpdev, qdarkstyle, pluggy, ujson, python-jsonrpc-server, autopep8, entrypoints, flake8, snowballstemmer, pydocstyle, yapf, python-language-server, diff-match-patch, pyxdg, argh, pathtools, watchdog, sortedcontainers, intervaltree, atomicwrites, spyder
  Found existing installation: parso 0.1.1
    Uninstalling parso-0.1.1:
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: 'normalizer.cpython-36.pyc'
Consider using the `--user` option or check the permissions.

As recommanded I tried with the  --user and the update completed but still with the ERROR incompatibility for the other packages. I decided to update all these packages with pip install --upgrade packageName --user and everything went fine, but unfortunately Spyder crashes at startup :-(
I noticed the only spyder binary on my Ubuntu was spyder3 while I see the new Spyder 4 popup during startup. This is the error returned during startup:
abcdefg@pseudo-lenovo-ideapad-710s-13isk:~$ spyder3
Attribute Qt::AA_ShareOpenGLContexts must be set before QCoreApplication is created.
Attribute Qt::AA_UseSoftwareOpenGL must be set before QCoreApplication is created.
Traceback (most recent call last):
  File "/home/aaz/.local/lib/python3.6/site-packages/spyder/app/mainwindow.py", line 3718, in main
    mainwindow = run_spyder(app, options, args)
  File "/home/aaz/.local/lib/python3.6/site-packages/spyder/app/mainwindow.py", line 3559, in run_spyder
    main.setup()
  File "/home/aaz/.local/lib/python3.6/site-packages/spyder/app/mainwindow.py", line 1010, in setup
    from spyder.plugins.ipythonconsole.plugin import IPythonConsole
  File "/home/aaz/.local/lib/python3.6/site-packages/spyder/plugins/ipythonconsole/plugin.py", line 52, in <module>
    from spyder.plugins.ipythonconsole.widgets import ClientWidget
  File "/home/aaz/.local/lib/python3.6/site-packages/spyder/plugins/ipythonconsole/widgets/__init__.py", line 16, in <module>
    from .debugging import DebuggingWidget
  File "/home/aaz/.local/lib/python3.6/site-packages/spyder/plugins/ipythonconsole/widgets/debugging.py", line 22, in <module>
    from IPython.core.inputtransformer2 import TransformerManager
ModuleNotFoundError: No module named 'IPython.core.inputtransformer2'

So I decided to install Spyder with sudo apt install spyder and now I have both spyder and spyder3. The new binary spyder launch without problem but it is still version 3, and when I follow the upgrade instruction and run pip install --pre -U spyder it returns no error but it start version 3 again.
So how can I manage to upgrade to version 4?
Thank you,

You have a serious dependencies problem and I don't know how you ended up in that state. Please remove this directory:
/home/aaz/.local/lib/python3.6/

and use a virtualenv/pyenv to install Spyder instead.
If you are unable to fix the problem after that, please install Anaconda instead. Unfortunately, we don't have the resources nor the time to help people with custom installations, so it's up to you to figure this one out.
This is because custom installations are for experts or very knowledgeable people who really know what they are doing and are able to easily solve problems like this.

So I decided to install Spyder with sudo apt install spyder

Also remove that package too.

Thank you for your help Cordoba I appreciate. spyder3 starts without error after removing the folder above, and I deleted the spyder apt package. But from here, wht is the best way to upgrade to version 4? Shall I follow the recommandation and run pip install --pre -U spyder?

Hello, just to let you know that Spyder 4 is working great now. I created a
virtual environnment and installed in it with pip3 instead of pip.
Maybe it was the issue
Le jeu. 2 janv. 2020 à 19:56, Carlos Cordoba<EMAIL_ADDRESS>a
écrit :

So I decided to install Spyder with sudo apt install spyder
Also remove that package too.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/spyder-ide/spyder/issues/11215?email_source=notifications&email_token=AD3JLD6ADLNBEKVKM6BJ7E3Q3ZBGPA5CNFSM4KCF3S7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH7HIAY#issuecomment-570323971,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AD3JLD66FBXP537ZLUV4CRTQ3ZBGPANCNFSM4KCF3S7A
.

I had the same issue, I solved mine by uninstalling pylint and reinstalling it.


Dapper doesn't work - EIP-1102

Looks like we need to clean up our web3 interface.
Attached are the docs for Dapper.
https://app.zenhub.com/files/142967010/b2696c24-d185-434b-89d1-cda15b9b0323/download
@joeykrug @chwy -- do we wanna priority high this and try to get it into 1.12.1?

Sure. Since this is the last v1 release let’s do it

It looks like it DOES work.
Closing.


Print multiple copies of labels

Is your feature request related to a problem? Please describe.
I often store my filament unopened, in its manufacturer's vacuum bag until its needed (i keep several in stock of common colours).  To make my life easier I print 2 labels, one to put on it, and one to keep with it for when i open it, so as not to get lazy and forget to label it.
Describe the solution you'd like
A number of copies field on the print dialog, to have it print 2x or 3x of each label as needed.
Describe alternatives you've considered
printing twice, but then the collation is a pain in the butt, if it was done in software, it can keep the labels together on the page, as it prints.
Additional context
I may work on a PR for this when i have some time, but didnt want to forget about the need, and if someone else has more time/motivation...

Great suggestion, should be easy to implement :)


SAML & WS-Fed Support

Why was the SAML and WS-Fed support removed from version 3.x? Is there a specific reason for this?
According to the README.md "It includes OpenID Connect, WS-Federation, and SAML-P authentication and authorization."

We will investigate this issue and update here when we have a definitive answer, thanks for your patience.


update to verify 17

Thanks Simon!

no worries. given the breaking change effects most extensions, figured i would save u the hassle


distributed training  van.cc Check failed

Description
i used this shell to test  distributed training  but it shows this error
../../tools/launch.py -n 2 -H hosts --launcher ssh python image_classification.py --dataset cifar10 --model vgg11 epochs 1 --kvstore dist_sync
File "/usr/local/lib/python3.6/dist-packages/mxnet/kvstore_server.py", line 82, in _init_kvstore_server_module
server.run()
File "/usr/local/lib/python3.6/dist-packages/mxnet/kvstore_server.py", line 73, in run
check_call(_LIB.MXKVStoreRunServer(self.handle, _ctrl_proto(self._controller()), None))
File "/usr/local/lib/python3.6/dist-packages/mxnet/base.py", line 252, in check_call
raise MXNetError(py_str(LIB.MXGetLastError()))
mxnet.base.MXNetError: [08:18:33] src/van.cc:291: Check failed: (my_node.port) != (-1) bind failed
Environment info (Required)
unbutu 16.04
python3.6.7
1 ps
1 worker(2 gpu)
Steps to reproduce
(Paste the commands you ran that produced the error.)
1.git clone incubator-mxnet and git clone dmlc-core
2.cd dmlc-core && make
3  cd incubator-mxnet/example/gluon/
../../tools/launch.py -n 2 -H hosts --launcher ssh python image_classification.py --dataset cifar10 --model vgg11 epochs 1 --kvstore dist_sync

@Davdi What is your host file looking like?

@Davdi https://github.com/Davdi your make instruction should use
USE_DIST_KVSTORE=1
I guess you probably did it, else you would have got different error.
Other thing is, Try using IP address of hosts in Hostfile rather than
hostnames.
On Tuesday, December 4, 2018, Lanking<EMAIL_ADDRESS>wrote:

@Davdi https://github.com/Davdi Looks like the build instruction is not
right. Have you followed this one?
http://mxnet.incubator.apache.org/install/ubuntu_setup.html
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/apache/incubator-mxnet/issues/13526#issuecomment-444310391,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABu8WZXINh3H8ZO34n3d3t0erA7WPgwsks5u1xG5gaJpZM4ZAcmw
.

--
Sent from iPhone

@Davdi https://github.com/Davdi your make instruction should use USE_DIST_KVSTORE=1 I guess you probably did it, else you would have got different error. Other thing is, Try using IP address of hosts in Hostfile rather than hostnames.
…
On Tuesday, December 4, 2018, Lanking @.***> wrote: @Davdi https://github.com/Davdi Looks like the build instruction is not right. Have you followed this one? http://mxnet.incubator.apache.org/install/ubuntu_setup.html — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#13526 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/ABu8WZXINh3H8ZO34n3d3t0erA7WPgwsks5u1xG5gaJpZM4ZAcmw .
-- Sent from iPhone

hosts file is this
<IP_ADDRESS>:10004

@Davdi Looks like the build instruction is not right. Have you followed this one?
http://mxnet.incubator.apache.org/install/ubuntu_setup.html

the env is 1 ps and 1 worker and should i follow the instruction on all the 2 container ?

Are you running in containers?
You said "the env is 1 ps and 1 worker and should i follow the instruction on all the 2 container ?"
but host file only has one entry, shouldn't it be 2 entries!
Can you try these steps -

Make sure that there is no python process running on any of the host.
Make sure that for all the entries in host file, you can ssh to those host from master node(the node where you ran launch.py)
if you are using ec2 instance, try using private ip in hosts file
Launch distributed training without port: <IP_ADDRESS> , mxnet will automatically chose port for workers

If error persist -
4) Paste the output of :
echo $env
5) cat hosts
6) For each entry in hosts file , ssh to host and paste output of :
ps -efl | grep python
7) paste launch command and Paste the entire log that you get after running launch.py

Are you running in containers?
You said "the env is 1 ps and 1 worker and should i follow the instruction on all the 2 container ?"
but host file only has one entry, shouldn't it be 2 entries!
Can you try these steps -

Make sure that there is no python process running on any of the host.
Make sure that for all the entries in host file, you can ssh to those host from master node(the node where you ran launch.py)
if you are using ec2 instance, try using private ip in hosts file
Launch distributed training without port: <IP_ADDRESS> , mxnet will automatically chose port for workers

If error persist -
4) Paste the output of :
echo $env
5) cat hosts
6) For each entry in hosts file , ssh to host and paste output of :
ps -efl | grep python
7) paste launch command and Paste the entire log that you get after running launch.py
I run the command on the container .but not aws ,
4)echo $env  it shows nothing

cat hosts
<IP_ADDRESS>:10004 this is ip of the worker container
ssh -i application_1543732416493_0027 -p 10004<EMAIL_ADDRESS>and it is successfully
application_1543732416493_0027  is the primary key

root@ecbf58635533:/userhome/incubator-mxnet/example/gluon# ps -efl | grep python 0 S root        236      1  0  80   0 -  1126 wait   07:22 pts/1    00:00:00 /bin/sh -c python image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root        239    236  0  80   0 - 1671321 hrtime 07:22 pts/1  00:00:21 python image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root        402    239  0  80   0 -  7441 pipe_w 07:22 pts/1    00:00:00 /usr/local/bin/python -c from multiprocessing.semaphore_tracker import main;main(3) 0 S root        435      1  0  80   0 -  1126 wait   07:23 pts/1    00:00:00 /bin/sh -c python image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root        437    435  0  80   0 - 1671320 hrtime 07:23 pts/1  00:00:19 python image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root        601    437  0  80   0 -  7441 pipe_w 07:23 pts/1    00:00:00 /usr/local/bin/python -c from multiprocessing.semaphore_tracker import main;main(3) 0 S root        637      1  0  80   0 -  1126 wait   07:25 pts/1    00:00:00 /bin/sh -c python image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root        639    637  0  80   0 - 1671321 hrtime 07:25 pts/1  00:00:19 python image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root        799    639  0  80   0 -  7441 pipe_w 07:25 pts/1    00:00:00 /usr/local/bin/python -c from multiprocessing.semaphore_tracker import main;main(3) 0 S root        996      1  0  80   0 -  1126 wait   07:31 pts/1    00:00:00 /bin/sh -c python /userhome/incubator-mxnet/example/gluon/image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root        999    996  0  80   0 - 1671321 hrtime 07:31 pts/1  00:00:27 python /userhome/incubator-mxnet/example/gluon/image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root       1158    999  0  80   0 -  7441 pipe_w 07:31 pts/1    00:00:00 /usr/local/bin/python -c from multiprocessing.semaphore_tracker import main;main(3) 0 S root       1416      1  0  80   0 -  1126 wait   07:41 pts/1    00:00:00 /bin/sh -c python /userhome/incubator-mxnet/example/gluon/image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root       1419   1416  0  80   0 - 1671321 hrtime 07:41 pts/1  00:00:30 python /userhome/incubator-mxnet/example/gluon/image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root       1582   1419  0  80   0 -  7441 pipe_w 07:41 pts/1    00:00:00 /usr/local/bin/python -c from multiprocessing.semaphore_tracker import main;main(3) 0 S root      17930      1  0  80   0 -  1126 wait   08:01 pts/1    00:00:00 /bin/sh -c python /userhome/incubator-mxnet/example/gluon/image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root      17933  17930  0  80   0 - 1671381 hrtime 08:01 pts/1  00:00:24 python /userhome/incubator-mxnet/example/gluon/image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root      18092  17933  0  80   0 -  7441 pipe_w 08:01 pts/1    00:00:00 /usr/local/bin/python -c from multiprocessing.semaphore_tracker import main;main(3) 0 S root      18183      1  0  80   0 -  1126 wait   08:07 pts/1    00:00:00 /bin/sh -c python /userhome/incubator-mxnet/example/gluon/image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root      18187  18183  0  80   0 - 1671380 hrtime 08:07 pts/1  00:00:29 python /userhome/incubator-mxnet/example/gluon/image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root      18349  18187  0  80   0 -  7441 pipe_w 08:07 pts/1    00:00:00 /usr/local/bin/python -c from multiprocessing.semaphore_tracker import main;main(3) 0 S root      18412      1  0  80   0 -  1126 wait   08:11 pts/1    00:00:00 /bin/sh -c python /userhome/incubator-mxnet/example/gluon/image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root      18416  18412  0  80   0 - 1671382 hrtime 08:11 pts/1  00:00:27 python /userhome/incubator-mxnet/example/gluon/image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root      18574  18416  0  80   0 -  7441 pipe_w 08:12 pts/1    00:00:00 /usr/local/bin/python -c from multiprocessing.semaphore_tracker import main;main(3) 0 S root      18746      1  0  80   0 -  1126 wait   08:24 pts/1    00:00:00 /bin/sh -c python /userhome/incubator-mxnet/example/gluon/image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root      18749  18746  0  80   0 - 1671380 hrtime 08:24 pts/1  00:00:18 python /userhome/incubator-mxnet/example/gluon/image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root      18908  18749  0  80   0 -  7441 pipe_w 08:24 pts/1    00:00:00 /usr/local/bin/python -c from multiprocessing.semaphore_tracker import main;main(3) 0 S root      18928      1  0  80   0 -  1126 wait   08:25 pts/1    00:00:00 /bin/sh -c python /userhome/incubator-mxnet/example/gluon/image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root      18931  18928  0  80   0 - 1671381 hrtime 08:25 pts/1  00:00:17 python /userhome/incubator-mxnet/example/gluon/image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root      19089  18931  0  80   0 -  7441 pipe_w 08:25 pts/1    00:00:00 /usr/local/bin/python -c from multiprocessing.semaphore_tracker import main;main(3) 0 S root      19228      1  0  80   0 -  1126 wait   08:26 pts/1    00:00:00 /bin/sh -c python /userhome/incubator-mxnet/example/gluon/image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root      19231  19228  1  80   0 - 1671381 hrtime 08:26 pts/1  00:00:23 python /userhome/incubator-mxnet/example/gluon/image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root      19390  19231  0  80   0 -  7441 pipe_w 08:27 pts/1    00:00:00 /usr/local/bin/python -c from multiprocessing.semaphore_tracker import main;main(3) 0 S root      19678      1  0  80   0 -  1126 wait   08:49 pts/1    00:00:00 /bin/sh -c python /userhome/incubator-mxnet/example/gluon/image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root      19681  19678  5  80   0 - 1671380 hrtime 08:49 pts/1  00:00:29 python /userhome/incubator-mxnet/example/gluon/image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root      19840  19681  0  80   0 -  7441 pipe_w 08:49 pts/1    00:00:00 /usr/local/bin/python -c from multiprocessing.semaphore_tracker import main;main(3) 0 S root      19900      1  0  80   0 -  1126 wait   08:53 pts/1    00:00:00 /bin/sh -c python /userhome/incubator-mxnet/example/gluon/image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root      19903  19900  7  80   0 - 1671380 hrtime 08:53 pts/1  00:00:26 python /userhome/incubator-mxnet/example/gluon/image_classification.py --model vgg11 epochs 1 --kvstore dist_sync 0 S root      20062  19903  0  80   0 -  7441 pipe_w 08:53 pts/1    00:00:00 /usr/local/bin/python -c from multiprocessing.semaphore_tracker import main;main(3) 0 S root      20144    169  0  80   0 -  2821 pipe_w 08:59 pts/1    00:00:00 grep --color=auto python
../../tools/launch.py -n 2 -H hosts --launcher ssh python image_classification.py --dataset cifar10 --model vgg11 epochs 1 --kvstore dist_sync
the log is

Traceback (most recent call last):
File "/userhome/incubator-mxnet/tools/launch.py", line 128, in
main()
File "/userhome/incubator-mxnet/tools/launch.py", line 109, in main
raise RuntimeError('Unknown submission cluster type %s' % args.cluster)
RuntimeError: Unknown submission cluster type ssh
root@ecbf58635533:/userhome/incubator-mxnet/example/gluon# /userhome/incubator-mxnet/tools/launch.py -n 1 -H hosts python /userhome/incubator-mxnet/example/gluon/image_classification.py --model vgg11 epochs 1 --kvstore dist_sync
/usr/local/lib/python3.6/dist-packages/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:47: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
[08:53:21] src/van.cc:290: Bind to role=scheduler, id=1, ip=<IP_ADDRESS>, port=9103, is_recovery=0
/usr/local/lib/python3.6/dist-packages/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:47: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
Traceback (most recent call last):
File "/userhome/incubator-mxnet/example/gluon/image_classification.py", line 23, in
import mxnet as mx
File "/usr/local/lib/python3.6/dist-packages/mxnet/init.py", line 57, in
from . import kvstore_server
File "/usr/local/lib/python3.6/dist-packages/mxnet/kvstore_server.py", line 85, in
_init_kvstore_server_module()
File "/usr/local/lib/python3.6/dist-packages/mxnet/kvstore_server.py", line 82, in _init_kvstore_server_module
server.run()
File "/usr/local/lib/python3.6/dist-packages/mxnet/kvstore_server.py", line 73, in run
check_call(_LIB.MXKVStoreRunServer(self.handle, _ctrl_proto(self._controller()), None))
File "/usr/local/lib/python3.6/dist-packages/mxnet/base.py", line 252, in check_call
raise MXNetError(py_str(LIB.MXGetLastError()))
mxnet.base.MXNetError: [08:54:25] src/van.cc:291: Check failed: (my_node.port) != (-1) bind failed

Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python3.6/dist-packages/mxnet/libmxnet.so(+0x382d4a) [0x7f191e65bd4a]
[bt] (1) /usr/local/lib/python3.6/dist-packages/mxnet/libmxnet.so(+0x383381) [0x7f191e65c381]
[bt] (2) /usr/local/lib/python3.6/dist-packages/mxnet/libmxnet.so(+0x31c1bca) [0x7f192149abca]
[bt] (3) /usr/local/lib/python3.6/dist-packages/mxnet/libmxnet.so(+0x31cbaca) [0x7f19214a4aca]
[bt] (4) /usr/local/lib/python3.6/dist-packages/mxnet/libmxnet.so(+0x31bcdb9) [0x7f1921495db9]
[bt] (5) /usr/local/lib/python3.6/dist-packages/mxnet/libmxnet.so(+0x2c84b63) [0x7f1920f5db63]
[bt] (6) /usr/local/lib/python3.6/dist-packages/mxnet/libmxnet.so(MXKVStoreRunServer+0x88) [0x7f1920d4d2b8]
[bt] (7) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7f19becbce40]
[bt] (8) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x2eb) [0x7f19becbc8ab]
[bt] (9) /usr/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(_ctypes_callproc+0x2c6) [0x7f19beed09e6]
/usr/local/lib/python3.6/dist-packages/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:47: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
usage: image_classification.py [-h] [--dataset DATASET] [--data-dir DATA_DIR]
[--num-worker NUM_WORKERS]
[--batch-size BATCH_SIZE] [--gpus GPUS]
[--epochs EPOCHS] [--lr LR]
[--momentum MOMENTUM] [--wd WD] [--seed SEED]
[--mode MODE] --model MODEL [--use_thumbnail]
[--batch-norm] [--use-pretrained]
[--prefix PREFIX] [--start-epoch START_EPOCH]
[--resume RESUME] [--lr-factor LR_FACTOR]
[--lr-steps LR_STEPS] [--dtype DTYPE]
[--save-frequency SAVE_FREQUENCY]
[--kvstore KVSTORE]
[--log-interval LOG_INTERVAL] [--profile]
[--builtin-profiler BUILTIN_PROFILER]
image_classification.py: error: unrecognized arguments: epochs 1
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/dist-packages/dmlc_tracker/ssh.py", line 62, in run
subprocess.check_call(prog, shell = True)
File "/usr/lib/python3.6/subprocess.py", line 291, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'ssh -o StrictHostKeyChecking=no <IP_ADDRESS> -p 10001 'export LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64; export DMLC_NUM_WORKER=1; export DMLC_NUM_SERVER=1; export DMLC_PS_ROOT_URI=<IP_ADDRESS>; export DMLC_PS_ROOT_PORT=9103; export DMLC_ROLE=server; export DMLC_NODE_HOST=<IP_ADDRESS>; cd /userhome/incubator-mxnet/example/gluon/; python /userhome/incubator-mxnet/example/gluon/image_classification.py --model vgg11 epochs 1 --kvstore dist_sync'' returned non-zero exit status 1.
Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.6/dist-packages/dmlc_tracker/ssh.py", line 62, in run
subprocess.check_call(prog, shell = True)
File "/usr/lib/python3.6/subprocess.py", line 291, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'ssh -o StrictHostKeyChecking=no <IP_ADDRESS> -p 10001 'export LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64; export DMLC_NUM_WORKER=1; export DMLC_NUM_SERVER=1; export DMLC_PS_ROOT_URI=<IP_ADDRESS>; export DMLC_PS_ROOT_PORT=9103; export DMLC_ROLE=worker; export DMLC_NODE_HOST=<IP_ADDRESS>; cd /userhome/incubator-mxnet/example/gluon/; python /userhome/incubator-mxnet/example/gluon/image_classification.py --model vgg11 epochs 1 --kvstore dist_sync'' returned non-zero exit status 2.

As I see there are 3 different issues here:
File "/usr/local/lib/python3.6/dist-packages/mxnet/base.py", line 252, in check_call
raise MXNetError(py_str(LIB.MXGetLastError()))
mxnet.base.MXNetError: [08:54:25] src/van.cc:291: Check failed: (my_node.port) != (-1) bind failed

Host file -
if you say -n 2, there will be 2 worker and 2 server. If you have only one line with host and port, all of the processes will try to launch on same port.
So work around is same as what I suggested earlier. Please use only host and let mxnet chose port.
If you want chose port yourself, find 4 different ports which are not used and use 4 entries in host file.

Ideally you should have multiple hosts for distributed training.
`Traceback (most recent call last):
File "/userhome/incubator-mxnet/tools/launch.py", line 128, in
main()
File "/userhome/incubator-mxnet/tools/launch.py", line 109, in main
raise RuntimeError('Unknown submission cluster type %s' % args.cluster)
RuntimeError: Unknown submission cluster type ssh

This seems like a launch script issue. Can you try not giving --launcher option in command line, and using and use full host file path in -H option
usage: image_classification.py [-h] [--dataset DATASET] [--data-dir DATA_DIR]
[--num-worker NUM_WORKERS]
[--batch-size BATCH_SIZE] [--gpus GPUS]
[--epochs EPOCHS] [--lr LR]
[--momentum MOMENTUM] [--wd WD] [--seed SEED]
[--mode MODE] --model MODEL [--use_thumbnail]
[--batch-norm] [--use-pretrained]
[--prefix PREFIX] [--start-epoch START_EPOCH]
[--resume RESUME] [--lr-factor LR_FACTOR]
[--lr-steps LR_STEPS] [--dtype DTYPE]
[--save-frequency SAVE_FREQUENCY]
[--kvstore KVSTORE]
[--log-interval LOG_INTERVAL] [--profile]
[--builtin-profiler BUILTIN_PROFILER]
image_classification.py: error: unrecognized arguments: epochs 1

This is problem with training code. If it is coming from examples this needs to be fixed.

As I see there are 3 different issues here:
File "/usr/local/lib/python3.6/dist-packages/mxnet/base.py", line 252, in check_call
raise MXNetError(py_str(LIB.MXGetLastError()))
mxnet.base.MXNetError: [08:54:25] src/van.cc:291: Check failed: (my_node.port) != (-1) bind failed

Host file -
if you say -n 2, there will be 2 worker and 2 server. If you have only one line with host and port, all of the processes will try to launch on same port.
So work around is same as what I suggested earlier. Please use only host and let mxnet chose port.
If you want chose port yourself, find 4 different ports which are not used and use 4 entries in host file.

Ideally you should have multiple hosts for distributed training.
`Traceback (most recent call last):
File "/userhome/incubator-mxnet/tools/launch.py", line 128, in
main()
File "/userhome/incubator-mxnet/tools/launch.py", line 109, in main
raise RuntimeError('Unknown submission cluster type %s' % args.cluster)
RuntimeError: Unknown submission cluster type ssh

This seems like a launch script issue. Can you try not giving --launcher option in command line, and using and use full host file path in -H option
usage: image_classification.py [-h] [--dataset DATASET] [--data-dir DATA_DIR]
[--num-worker NUM_WORKERS]
[--batch-size BATCH_SIZE] [--gpus GPUS]
[--epochs EPOCHS] [--lr LR]
[--momentum MOMENTUM] [--wd WD] [--seed SEED]
[--mode MODE] --model MODEL [--use_thumbnail]
[--batch-norm] [--use-pretrained]
[--prefix PREFIX] [--start-epoch START_EPOCH]
[--resume RESUME] [--lr-factor LR_FACTOR]
[--lr-steps LR_STEPS] [--dtype DTYPE]
[--save-frequency SAVE_FREQUENCY]
[--kvstore KVSTORE]
[--log-interval LOG_INTERVAL] [--profile]
[--builtin-profiler BUILTIN_PROFILER]
image_classification.py: error: unrecognized arguments: epochs 1

This is problem with training code. If it is coming from examples this needs to be fixed.
ok thanks ,i modify the hosts file  and the content is this

ps-0
worker-0
worker-1
this is ip of ps and worker ,
and under the folder /root/.ssh/config
Host ps-0   HostName <IP_ADDRESS>   Port 10015   User root   StrictHostKeyChecking no   UserKnownHostsFile /dev/null   IdentityFile /root/.ssh/application_1544059068811_0001 Host worker-0   HostName <IP_ADDRESS>   Port 10016   User root   StrictHostKeyChecking no   UserKnownHostsFile /dev/null   IdentityFile /root/.ssh/application_1544059068811_0001 Host worker-1   HostName <IP_ADDRESS>   Port 10023   User root   StrictHostKeyChecking no   UserKnownHostsFile /dev/null   IdentityFile /root/.ssh/application_1544059068811_0001
but when i run command
../../tools/launch.py -n 2 -H hosts --launcher ssh python image_classification.py --dataset cifar10 --model vgg11 --epochs 1 --kvstore dist_sync
it shows the error
Warning: Permanently added '<IP_ADDRESS>' (ECDSA) to the list of known hosts.
Warning: Permanently added '<IP_ADDRESS>' (ECDSA) to the list of known hosts.
Warning: Permanently added '<IP_ADDRESS>' (ECDSA) to the list of known hosts.
Warning: Permanently added '<IP_ADDRESS>' (ECDSA) to the list of known hosts.
<EMAIL_ADDRESS>password<EMAIL_ADDRESS>password<EMAIL_ADDRESS>password<EMAIL_ADDRESS>password:
it seems that i need password but i use the primary key and no password ,and when i use ssh worker-0
it login succesfully

is this ec2 or your own host?
please share the command you use to ssh-ing to master instance?
1/ use ssh -A to login to master
2/ on your master instance,run below command (this enables master to use your agent to login to workers)
setup ssh-forwarding.
sed -i "s/^#(\s+)ForwardAgent(\s+)no/\ \1ForwardAgent\2yes/g" /etc/ssh/ssh_config
3/ In host file use ip address instead.
if you want to launch with 1 ps and 2 worker
your host file should look like:
<IP_ADDRESS>
<IP_ADDRESS>
<IP_ADDRESS>
4/use full path for host file
5/ launch with command: -s option says that use 1 server and -n is 2 worker . 1st host in host file will be server, 2nd host will be worker, 3rd host will be worker
launch.py -s 1 -n 2 ........

@Davdi hope your issue was solved.

@Davdi hope your issue was solved.
You should try this link to use https://github.com/awslabs/dynamic-training-with-apache-mxnet-on-aws#how-to-setup-dynamic-training-with-a-cloud-formation-template
In the same page there is a link to mxnet binary for cuda9.2, in case you don't want to build from source:
pip install --user --force-reinstall -e https://s3.amazonaws.com/us-east-1-aws-dl-cfn/mxnet_cu92mkl-1.3.1-py2.py3-none-manylinux1_x86_64.whl
thanks ! the reason of this error is that ip's port of the hosts is not default 22 .  so when excute the command ,it shows this error ,
so when the ip's port of host file is not 22 ,example 1000,so what should i set the parameter of the lanch.py .thanks!


Add tool to sync runbooks from kubevirt/monitoring

This PR adds a tool to automate PR creation for newly updated CNV runbooks in https://github.com/kubevirt/monitoring.
GitHub action to execute the tool will be created in a follow up PR.
Results:

if a runbook does not exist in openshift/runbooks, a PR to add the file is created
if a runbook exists in both repos, it compares the last update date for the runbook, if it is newer in kubevirt/monitoring, a PR to update it in openshift/runbooks is created
removed/deprecated PRs are disabled for now (needing final decision on the process)

If a create/update PR is closed because we might want different wording in this repo, the automation will not try to create a new PR anymore until a new change is made downstream
example PRs on my personal repo: https://github.com/machadovilaca/runbooks/pulls
jira-ticket: https://issues.redhat.com/browse/CNV-28200
/cc @sradco @ctomasko @ousleyp

/hold

deprecated in favor of https://github.com/kubevirt/monitoring/pull/230


feat(deps): upgrade runtime dependencies

Upgrades project dependencies. See details in workflow run.

Automatically created by projen via the "upgrade-runtime-dependencies-2.x" workflow

⚪ Backport skipped
The pull request was not backported as there were no branches to backport to. If this is a mistake, please apply the desired version labels or run the backport tool manually.
Manual backport
To create the backport manually run:
backport --pr 1388

Questions ?
Please refer to the Backport tool documentation