"windows machine file copy" "Clean target" copy is slow when a large number of files are present in the target folder

"windows machine file copy" "Clean target" copy is slow when a large number of files are present in the target folder.
Tested on TFS2015
to reproduce:

copy to a remote machine a zip to an empty folder (this is quick)
unzip it
repeat - copy is now slow and full of "Extra fir" "Extra file"

also

copy zip / unzip it/ copy single small file - this too is slow. Should be quick.

Cheers,

Thank you for your feedback. I will pass this on to the product team and update the topic based on their response.

@queBurro  Which task version are you using ? Can you please share file size in target and how much this operation is taking to complete ? Please share the debug logs to<EMAIL_ADDRESS>

I believe what's happening is that the "remove-item" is being run on the agent's host against the remote host. This means that the files are being moved across the nw before being deleted. I've swapped this "delete task" for some in-line powershell that runs the "remove-item" on the target machine and the time taken is as it should be.

I'm using version TFS2015.3. I've not got any logs from 2 months ago.

@queBurro We are taking mirror image with empty folder through robocopy to clean target. We are just cleaning the empty folder in last which is on agent box Link. We had already done fixes. If you still face any issue please write to<EMAIL_ADDRESS>

@queBurro I am closing this GitHub issue - if the previous comments don't resolve your issue, please send email to<EMAIL_ADDRESS>as described in the previous comment. Thank you.


overflow intrinc can only be an argument of extractvalue instruction

wouldn't it be easier to do everything in one place at Parser.cpp:507?

Oops! I need to handle this for phi instruction as well. Updating my patch now.

Updated. Thanks.

this is too many repetitions of the same case statement, the overflow check needs to be factored out into a separate function

also if there's only one case you don't really want a switch

Sure, I will fix the code. Thanks.

LGTM


GgetPolicyBlocks() throws error.

Guardener::GgetPolicyBlocks(accessToken = accTknG,  policyId = idPolicyVnH) throws error:
Error in `map()`:
ℹ In index: 1.
Caused by error in `GgetPolicy()`:
! object 'AT' not found
Run `rlang::last_error()` to see where the error occurred.

Sien 6f7a34b..65f45d0


Add TestCheckForMisbehaviour using mock vm for 08-wasm

Summary
Add TestCheckForMisbehaviour. See this commented out test. There is currently no equivalent test with the grandpa client

For Admin Use

[ ] Not duplicate issue
[ ] Appropriate labels applied
[ ] Appropriate contributors tagged/assigned

Closed by #4901


update test scenarios

update test scenarios
update example policies with how to test each policy

Codecov Report

Merging #674 (8c2848e) into main (bf6087f) will decrease coverage by 0.57%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #674      +/-   ##
==========================================
- Coverage   35.96%   35.38%   -0.58%
==========================================
  Files          25       25
  Lines        9018     9018
==========================================
- Hits         3243     3191      -52
- Misses       5325     5381      +56
+ Partials      450      446       -4

Impacted Files
Coverage Δ

KubeArmor/monitor/systemMonitor.go
38.48% <0.00%> (-9.95%)
:arrow_down:

KubeArmor/feeder/policyMatcher.go
37.56% <0.00%> (+0.20%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bf6087f...8c2848e. Read the comment docs.

Small nit, Rest LGTM 👌🏽
Side note, What do you think about adding a CI to run multiubuntu test scenario on PRs, but since this is an extended test suit it should only run if manually triggered.

The reason that we use github test scenarios instead of multiubuntu ones is that it takes too much time.


prmiles cleanup docs

Pull Request Test Coverage Report for Build 117

18 of 20 (90.0%)  changed or added relevant lines in 4 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage decreased (-0.1%) to 98.09%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pymcmcstat/MCMC.py
8
9
88.89%

pymcmcstat/chain/ChainStatistics.py
3
4
75.0%

Files with Coverage Reduction
New Missed Lines
%

pymcmcstat/chain/ChainStatistics.py
1
95.16%

Totals

Change from base Build 112:
-0.1%

Covered Lines:
2054

Relevant Lines:
2094

💛  - Coveralls


test(test-tooling): add container image builder utilities

Currently our integration tests depend on pre-published container
images to be on the official registry (ghcr.io). This has pros and cons.
The pro is that we can pin the tests to a specific ledger version and
then have confidence that the test code works with that specific image.
On the other hand if the image itself has problems we won't know it until
after it was published and then tests were executed with it (unless we
perform manual testing which is a lot of effrot as it requires the
manual modification of the test cases).
In order to gives us the ability to test against the container image
definitions as they are in the current revision of the source code,
we are adding here a couple of utility functions to streamline writing
test cases that build the container images for themselves as part of the
test case.

An example of how to use it in a test case:
const imgConnectorJvm = await buildImageConnectorCordaServer({
    logLevel,
});

// ...

connector = new CordaConnectorContainer({
    logLevel,
    imageName: imgConnectorJvm.imageName,
    imageVersion: imgConnectorJvm.imageVersion,
    envVars: [envVarSpringAppJson],
});

Signed-off-by: Peter Somogyvari<EMAIL_ADDRESS>Pull Request Requirements

[x] Rebased onto upstream/main branch and squashed into single commit to help maintainers review it more efficient and to avoid spaghetti git commit graphs that obfuscate which commit did exactly what change, when and, why.
[x] Have git sign off at the end of commit message to avoid being marked red. You can add -s flag when using git commit command. You may refer to this link for more information.
[x] Follow the Commit Linting specification. You may refer to this link for more information.

Character Limit

[x] Pull Request Title and Commit Subject must not exceed 72 characters (including spaces and special characters).
[x] Commit Message per line must not exceed 80 characters (including spaces and special characters).

A Must Read for Beginners
For rebasing and squashing, here's a must read guide for beginners.

This is great, LGTM, thank you!

@outSH Thank you! For now I only have the ready-made utility functions for the corda images, but we can add more, ideally all images would be possible to be built by the tests like this in the future.


Implement new ruleset changes

[x] New Tolerance by percent, not unit
- The existing system checks for a direct offset. This needs to instead use percent error.
[x] Modify the default scoring ruleset
[x] Modify the default multiplier ruleset
- The default game parameters should match those in the gameplay document.

All done in #144


Add generated file note to Options.swift

Updating makeOptions to include note that options.swift is generated so it should not be edited by hand.
Adding a note to the file so that folks are reminded not to edit it by hand.

@swift-ci please test

I should probably add the generated header comment to Options.swift.

@swift-ci please test

@swift-ci please test Windows


Cannot reproduce results on cascade hrnet

Python: 3.7.7 (default, Mar 26 2020, 15:48:22) [GCC 7.3.0]
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.3.1
TorchVision: 0.4.2
OpenCV: 4.2.0
MMCV: 0.4.3
MMDetection: 1.1.0+2452761
MMDetection Compiler: GCC 7.5
MMDetection CUDA Compiler: 10.0

I have trained faster_rcnn_r50_fpn_1x、cascade_rcnn_hrnetv2p_w32_20e、cascade_rcnn_r50_fpn_1x on my dataset. Performance of cascade_rcnn_hrnetv2p_w32_20e is much higher than faster_rcnn_r50_fpn_1x but slightly lower than cascade_rcnn_r50_fpn_1x.
I have evaluated cascade_rcnn_hrnetv2p_w32_20e on coco dataset with pretrained model, everything seems fine. Then I trained cascade_rcnn_hrnetv2p_w32_20e from scrath on coco2017 dataset with 8 gpu:
sh ./tools/dist_train.sh ${config_file} 8 --work_dir ${OUTPUT_DIR}/ --autoscale-lr

AP of the model is 0.212.
Is any suggestions?

You do not need to use --autoscale-lr since the config already set the learning rate for 8 GPU setting. So it is very likely that your LR is too large.

You do not need to use --autoscale-lr since the config already set the learning rate for 8 GPU setting. So it is very likely that your LR is too large.

It may have some bugs in --autoscale-lr. I found that lr was always 0.00250, one eighth of the value I set in config.py(0.02) whether I am using 2 GPUs or 8 GPUs. Now I am training from scrath without --autoscale-lr, lr is 0.02. After 11 epoches, the loss is around 1.0, which I think is too large.

training from scrath is a little bit vague, usually it means you train a model without using pre-trained backbone, did you do that? Or you just simply rerun a config without --autoscale-lr?
It is quite normal for cascade methods have loss around 1.0, some logs can be found in our model zoo for your reference, e.g., cascade_mask_rcnn_r50_fpn_1x. We will provide model models and logs in the model zoo.

training from scrath is a little bit vague, usually it means you train a model without using pre-trained backbone, did you do that? Or you just simply rerun a config without --autoscale-lr?
It is quite normal for cascade methods have loss around 1.0, some logs can be found in our model zoo for your reference, e.g., cascade_mask_rcnn_r50_fpn_1x. We will provide model models and logs in the model zoo.

I cannot train cascade_rcnn_hrnetv2p_w32_20e on my own dataset and I suspect it has something to do with training. So I trained it on coco dataset without using pre-trained backbone and tried to reproduce the results.
I have trained another cascade hrnetv2p on my own dataset without --autoscale-lr . Again, performance of the model slight lower than that of cascade_rcnn_r50_fpn_1x

New training without --autoscale-lr failed again, AP is 0.25.

Evaluation on coco2017:
Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.255
Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.401
Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.270
Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.128
Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.265
Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.362
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.257
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.399
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.415
Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.210
Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.445
Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.565

The performance reported in the model zoo is produced with ImageNet pre-trained models.
Training without using a pre-trained backbone usually obtain worse results with a 1x schedule and you can check details in here.

The performance reported in the model zoo is produced with ImageNet pre-trained models.
Training without using a pre-trained backbone usually obtain worse results with a 1x schedule and you can check details in here.

Many thanks.
But how can I be sure that there is no problem in training process? Or is that normal that the performance of cascade_rcnn_hrnetv2p_w32 is slower than that of cascade_rcnn_r50_fpn_1x?

Usually, if you can see the training logs print out normally, it is OK.
About the training speed, cascade_rcnn_hrnetv2p_w32 should be slower than cascade_rcnn_r50_fpn.

Usually, if you can see the training logs print out normally, it is OK.
About the training speed, cascade_rcnn_hrnetv2p_w32 should be slower than cascade_rcnn_r50_fpn.

Sorry for my unclear expression. What I mean is that the AP of cascade_rcnn_hrnetv2p_w32_20e is much higher than that of faster_rcnn_r50_fpn_1x but slightly lower than that of cascade_rcnn_r50_fpn_1x on my dataset. Now I am working on finding out the reason.

Hi @Mabinogiysk ,
If these configs on other datasets do not exhibit the same trends they have on the coco dataset, then there may be some bug or non-trivial things.

Thanks

I have trained faster_rcnn_hrnetv2p_w32_1x、cascade_rcnn_hrnetv2p_w32_20e and cascade_rcnn_r50_fpn_1x on my dataset. faster_rcnn_hrnetv2p_w32_1x and cascade_rcnn_r50_fpn_1x both work fine but the AP of cascade_rcnn_hrnetv2p_w32_20e is slightly lower. All model have trained with pre-trained model and the only difference in config.py is the path of ann_file and img_prefix in training process.
Now I will try it on mmdetection2.0.


When installing on Raspberry Pi 3, wheel needed to be installed before pycryptodomex

Bug report
Your Environment

Netflix add-on version: 1.18.4.+matrix.1
Operating system version/name: OSMC running Kodi 19.3, OSMC 2021.12-1, kernel 5.10.78-2-osmc
Device model:  Raspberry Pi 3

Used Operating system:

[ ] Android
[ ] iOS
[ ] Linux
[ ] OSX
[X] Raspberry-Pi
[ ] Windows

Describe the bug

When following the installation instructions from https://github.com/CastagnaIT/plugin.video.netflix/wiki/How-install-the-addon for OSMC, the Kodi 19 variant, on a fresh OSMC install, I got an error when installing pycryptodomex. The issue was fixed when I installed wheel and retried installation of pycryptodomex.
Expected behavior

pycryptodomex should install when following the instructions
Actual behavior

pycryptodomex  failed to install with the following error message, which actually makes it seem like the installation was succesful, but the addon would not run:
osmc@osmc:~$ sudo pip3 install pycryptodomex
Collecting pycryptodomex
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x75411910>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/pycryptodomex/
  Downloading https://files.pythonhosted.org/packages/24/40/e249ac3845a2333ce50f1bb02299ffb766babdfe80ca9d31e0158ad06afd/pycryptodomex-3.14.1.tar.gz (3.4MB)
|████████████████████████████████| 3.4MB 78kB/s
Building wheels for collected packages: pycryptodomex
  Running setup.py bdist_wheel for pycryptodomex ... error
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-lpuqsaq1/pycryptodomex/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-71thlbxj --python-tag cp37:
  Testing support for clang
  Target does not support clang
  Testing support for gcc
  Target does support gcc
  Testing support for stdint.h header
  Target does support stdint.h header
  Testing support for 128-bit integer
  Target does not support 128-bit integer
  Testing support for cpuid.h header
  Target does not support cpuid.h header
  Testing support for intrin.h header
  Target does not support intrin.h header
  Testing support for posix_memalign
  Target does support posix_memalign
  Testing support for SSE2(intrin.h)
  Target does not support SSE2(intrin.h)
  Testing support for SSE2(x86intrin.h)
  Target does not support SSE2(x86intrin.h)
  Testing support for SSE2(emmintrin.h)
  Target does not support SSE2(emmintrin.h)
  Testing support for gcc
  Target does support gcc
  Warning: compiler does not support AESNI instructions
  Warning: compiler does not support CLMUL instructions
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for pycryptodomex
  Running setup.py clean for pycryptodomex
Failed to build pycryptodomex
Installing collected packages: pycryptodomex
  Running setup.py install for pycryptodomex ... done
Successfully installed pycryptodomex

Steps to reproduce the behavior

Install OSMC on Raspberry Pi 3
Follow installation instructions from https://github.com/CastagnaIT/plugin.video.netflix/wiki/How-install-the-addon

Possible fix

Runing sudo pip3 install wheel before running sudo pip3 install pycryptodomex again fixed the issue for me.
Debug log

The debug log can be found here:
I did not try to file a bug report before resolving the issue, so I haven't thought to turn the logging on.

Additional context or screenshots (if appropriate)
Other information

Screenshots

thanks added