feat: NZIM-v36-1022-20 NagPack

Fixes #1067

On the change to the Rules.md file, i can do some of the work now, and then come back and add the Links once the docs are published.

@dontirun ,   I've added a couple of sample lines to the the RULES.md,    Before i do all of them, can we check that the sample lines are in the right format?

It looks like there are seven new rules to create. Some of these will be easy, others less so.
A temporary file ( nzismRulestoCreate )
cloudfront-default-root-object-configured
ebs-snapshot-public-restorable-check
ec2-imdsv2-check
ec2-instance-managed-by-systems-manager
elb-custom-security-policy-ssl-check
lambda-function-public-access-prohibited
vpc-sg-open-only-to-authorized-ports

EC2SecurityGroupOnlyTcp443 checks security groups to see if they only have tcp 443 open

EC2SecurityGroupOnlyTcp443 checks security groups to see if they only have tcp 443 open

Is this an implementation of
vpc-sg-open-only-to-authorized-ports?
I don't think we should be implementing this rule since it's highly dependent on business logic.
The config rule takes in an input list because it is supposed to be tuned to business logic

EC2SecurityGroupOnlyTcp443 checks security groups to see if they only have tcp 443 open

Is this an implementation of vpc-sg-open-only-to-authorized-ports?
I don't think we should be implementing this rule since it's highly dependent on business logic.
The config rule takes in an input list because it is supposed to be tuned to business logic

The nzism specifically calls out that only tcp/443 shoudl be open.
    Controls: ['3205']
    Type: AWS::Config::ConfigRule
    Properties:
      ConfigRuleName: vpc-sg-open-only-to-authorized-ports
      InputParameters:
        authorizedTcpPorts: '443'
      Scope:
        ComplianceResourceTypes:
        - AWS::EC2::SecurityGroup
      Source:
        Owner: AWS
        SourceIdentifier: VPC_SG_OPEN_ONLY_TO_AUTHORIZED_PORTS
      Description: "SHOULD 18.1.13.C.02[CID:3205]| Network security/Network Management/Limiting network access"```

Tests for EC2IMDv2 are not yet completed.

@dontirun, I think this is ready to go now.

Andrew / Arun -  Is there an ETA on when this PR may be finalised?  I have a couple of organizations interested in using this feature.

@chindaws There is no ETA. This is a community led contribution by @mrpackethead.
I'll give @mrpackethead a chance to respond, but if this is something you would like and @mrpackethead is no longer working on this then I would say that this PR is available for someone else to pick up and continue working on

Andrew / Arun - Is there an ETA on when this PR may be finalised? I have a couple of organizations interested in using this feature.

@chindaws , get in touch with me if theres a reason to progress this. I could be convinced to do it if there was a need.


Why are the beta packages of v4 not tagged with beta in npm?

Is it intentional that the beta versions of v4 were not tagged with "beta" in npm?
npm publish --tag beta

Actually if you install jodit using npm installit automatically installs the latest beta package. Is that intended?
If the beta packages were tagged with beta npm install would install the latest v3 package. I think it would better for people to install the v3 by default and v4 beta if they install it using npm install jodit@beta because v3 is a stable version.

Unfortunately I didn't know that, sorry

No problem :)


Enable dependabot to upgrade libs

We can rely on GitHub's dependabot to open PRs with version upgrades and CVEs fixes to avoid future problems.

DCO is failing because I opened this PR via the project's settings.


SFTP.DownloadFiles added cancellationToken

Added cancellation token to the method listing the source files.
Modified method's name from GetSourceFiles to ListSourceFiles.

Can you link the issue you are solving? Is it #123 ?


Be able to run Github checker standalone

...without depending on reviewdog.app. I'm afraid my employer will not accept us using a third party service. :-/

Same here, @haya14busa do you consider to release doghouse as FOSS app?

@mavimo I think your proposal is the way to go. It's similar to the way bors-ng recommends doing it and has worked for us when we've set that application up ourselves.

@JensRantil I'm working on PoC related to use Checks feature with the previous reported approach, I'm going also to investigate if the specified approach do not break any GH policy (we are going to use GH Application to build a "out-of-standard" app 😄)

Thanks for the comments.

I'm going also to investigate if the specified approach does not break any GH policy

yes, that's a concern. Maybe it's good to support standalone checker but wondering is it really ok or not.
For example, I may register reviewdog GitHub apps to GitHub market place or https://github.com/works-with, and we should check the policy.

Now reviewdog can report via Check API if it's running on GitHub Actions. https://medium.com/@haya14busa/automated-code-review-on-github-actions-with-reviewdog-for-any-languages-tools-20285e04448e


Independence from the tools like npm, systemctl

Currently boss-cli is tightly coupled with tools like npm, systemctl etc.

Resolved in https://github.com/kabirbaidhya/boss-cli/pull/52. Boss is no longer coupled with systemctl and npm. User can use what ever service or dependency manager using the custom scripts now.


Slim 873 implement get configuration object request

SonarQube analysis reported 3 issues
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 ActionMapperService.java#L150: Private method 'postConstruct' is never used.
 ConfigurationService.java#L492: Remove this unused method parameter "getFirmwareVersion".
 ActionMapperService.java#L105: Private method 'postConstruct' is never used.

All is well.

SonarQube analysis reported 3 issues
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 ActionMapperService.java#L150: Private method 'postConstruct' is never used.
 ConfigurationService.java#L492: Remove this unused method parameter "getFirmwareVersion".
 ActionMapperService.java#L105: Private method 'postConstruct' is never used.

All is well.

SonarQube analysis reported 3 issues
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 ActionMapperService.java#L150: Private method 'postConstruct' is never used.
 ConfigurationService.java#L492: Remove this unused method parameter "getFirmwareVersion".
 ActionMapperService.java#L105: Private method 'postConstruct' is never used.

All is well.

SonarQube analysis reported 5 issues
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 ActionMapperResponseService.java#L104: Private method 'postConstruct' is never used.
 ActionMapperService.java#L150: Private method 'postConstruct' is never used.
 ConfigurationService.java#L494: Remove this unused method parameter "getFirmwareVersion".
 ActionMapperResponseService.java#L77: Private method 'postConstruct' is never used.
 ActionMapperService.java#L105: Private method 'postConstruct' is never used.

All is well.

This pull request makes the build fail.

SonarQube analysis reported 5 issues
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 ActionMapperResponseService.java#L104: Private method 'postConstruct' is never used.
 ActionMapperService.java#L150: Private method 'postConstruct' is never used.
 ConfigurationService.java#L494: Remove this unused method parameter "getFirmwareVersion".
 ActionMapperResponseService.java#L77: Private method 'postConstruct' is never used.
 ActionMapperService.java#L105: Private method 'postConstruct' is never used.

All is well.

SonarQube analysis reported 5 issues
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 ActionMapperResponseService.java#L107: Private method 'postConstruct' is never used.
 ActionMapperService.java#L153: Private method 'postConstruct' is never used.
 ConfigurationService.java#L494: Remove this unused method parameter "getFirmwareVersion".
 ActionMapperResponseService.java#L78: Private method 'postConstruct' is never used.
 ActionMapperService.java#L106: Private method 'postConstruct' is never used.

All is well.

SonarQube analysis reported 5 issues
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 ActionMapperResponseService.java#L107: Private method 'postConstruct' is never used.
 ActionMapperService.java#L153: Private method 'postConstruct' is never used.
 ConfigurationService.java#L494: Remove this unused method parameter "getFirmwareVersion".
 ActionMapperResponseService.java#L78: Private method 'postConstruct' is never used.
 ActionMapperService.java#L106: Private method 'postConstruct' is never used.

All is well.

SonarQube analysis reported 5 issues
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 ActionMapperResponseService.java#L107: Private method 'postConstruct' is never used.
 ActionMapperService.java#L153: Private method 'postConstruct' is never used.
 ConfigurationService.java#L494: Remove this unused method parameter "getFirmwareVersion".
 ActionMapperResponseService.java#L78: Private method 'postConstruct' is never used.
 ActionMapperService.java#L106: Private method 'postConstruct' is never used.

All is well.

SonarQube analysis reported 5 issues
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 ActionMapperResponseService.java#L107: Private method 'postConstruct' is never used.
 ActionMapperService.java#L153: Private method 'postConstruct' is never used.
 ConfigurationService.java#L495: Remove this unused method parameter "getFirmwareVersion".
 ActionMapperResponseService.java#L78: Private method 'postConstruct' is never used.
 ActionMapperService.java#L106: Private method 'postConstruct' is never used.

All is well.

SonarQube analysis reported 8 issues

 7 major
 1 minor

Watch the comments in this conversation to review them.
6 extra issues
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 ActionMapperResponseService.java#L107: Private method 'postConstruct' is never used.
 ActionMapperService.java#L153: Private method 'postConstruct' is never used.
 ConfigurationService.java#L495: Remove this unused method parameter "getFirmwareVersion".
 DeviceFirmwareConverter.java#L20: Override this superclass' "equals" method.
 ActionMapperResponseService.java#L78: Private method 'postConstruct' is never used.
 ActionMapperService.java#L106: Private method 'postConstruct' is never used.

All is well.

SonarQube analysis reported 7 issues

 7 major

Watch the comments in this conversation to review them.
6 extra issues
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 ActionMapperResponseService.java#L107: Private method 'postConstruct' is never used.
 ActionMapperService.java#L153: Private method 'postConstruct' is never used.
 ConfigurationService.java#L495: Remove this unused method parameter "getFirmwareVersion".
 DeviceFirmwareConverter.java#L20: Override this superclass' "equals" method.
 ActionMapperResponseService.java#L78: Private method 'postConstruct' is never used.
 ActionMapperService.java#L106: Private method 'postConstruct' is never used.

All is well.

SonarQube analysis reported 6 issues

 6 major

Watch the comments in this conversation to review them.
5 extra issues
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 ActionMapperService.java#L153: Private method 'postConstruct' is never used.
 ConfigurationService.java#L495: Remove this unused method parameter "getFirmwareVersion".
 DeviceFirmwareConverter.java#L20: Override this superclass' "equals" method.
 ActionMapperResponseService.java#L77: Private method 'postConstruct' is never used.
 ActionMapperService.java#L106: Private method 'postConstruct' is never used.

All is well.

SonarQube analysis reported 6 issues

 6 major

Watch the comments in this conversation to review them.
5 extra issues
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 ActionMapperService.java#L153: Private method 'postConstruct' is never used.
 ConfigurationService.java#L495: Remove this unused method parameter "getFirmwareVersion".
 DeviceFirmwareConverter.java#L20: Override this superclass' "equals" method.
 ActionMapperResponseService.java#L77: Private method 'postConstruct' is never used.
 ActionMapperService.java#L106: Private method 'postConstruct' is never used.

All is well.


"Mesh" and "Hpo" linkers give the same result

Hi,
I'm trying to annotate data using Scispacy. Loading "mesh" and "hpo" gives the exact same results no matter what is the input.
For example:

I tried on many texts and both linkers plotted the same results.

Hi, there are two components related to entity recognition and linking in scispacy. One is the Named Entity Recognition (NER) component, which identifies textual spans that are likely to be entities (and depending on which scipsacy model, also their broad type). This information can be accessed as you've done via doc.ents and doc.ents[0].ent_type_. The second is the Entity Linking component, which is the one you specify mesh/hpo for. That component takes in the textual spans selected by the NER component and attempts to link them to an entity from the knowledge base. That information can be accessed via doc.ents[0]._.kb_ents. Hope that helps!

Thanks for the quick response, yes it does help.
I see now that the Entities linking are different

But I couldn't find a way to map back from id e.g ('C0346073') to the name of the entity at the knowledge base ('mesh'/'hpo')

I have a similar question. In the above example itself, in spite of using hpo as the linker, the id returned is C0346073 instead of [HP:0012329](https://hpo.jax.org/app/browse/term/HP:0012329) as we'd expect from the mapping shown here. I tried go as well and yet same result. Am I missing something?

All of the ontology options are implemented as subsets of UMLS. We don't have any cross mapping to the root ontology identifier. You would have to get that from UMLS or another source. The entity information available from UMLS in scispacy can be accessed as in the example code
linker = nlp.get_pipe("scispacy_linker") for umls_ent in entity._.kb_ents: 	print(linker.kb.cui_to_entity[umls_ent[0]])

Then how do linkers like hpo and go change the output?

They link to subsets of UMLS that are more specific than the full UMLS. This can be useful for two reasons (at least two that come to mind) if you know that you just want entities that fall into one of those subsets, 1) the downloaded file is much smaller and memory usage is less 2) the results will be higher precision because you won't get links to entities of a different type that you are not interested in.

Is there any way to map back 'mesh' or 'hpo' linkers back the to relevant UMLS Entities ?
In other words, if I'm using the umls linker can I filter which are 'mesh' related and which are 'hpo' related ?
e.g.

The mesh and hpo linker entities should contain the exact same information as the umls linker entities since they are just a subset.


How can I specify the concrete type for java.util.Set?

Question
I encounter a problem that I have a DTO must convert to a VO.
public class ReportCriteriaDto implements Serializable {
//...
    private Set<ReportOrderDto> reportOrder = new LinkedHashSet<ReportOrderDto>();
//...
}

But... after modelmapper.map(), the reportOrder didn't remain in order and I find out that the concrete type of reportOrder becomes java.util.HashSet.
So, I must write a post converter in TypeMap in order to sort the reportOrder.  It's working but not so convenient.
Can ModelMapper fix this in future?

Tried to use you solution with Provider<Set>, but still have HashSet  as destination property.
During debug I found my settings was overridden by MappingContextHelper in CollectionConverter
Do I need any additional setup?


Enable dependabot

Dependabot does not support poetry. Related: https://github.com/dependabot/feedback/issues/57


Plugin installation fails.

NodeBB version:
v1.7.5
NodeBB git hash:
7df1914
Database type: mongo, redis, or postgres
mongo
Database version:
3.4.4
Exact steps to cause this issue:
Start from scratch. Follow the guidelines as expected in NodeBB installation guide.
Install https://github.com/akhoury/nodebb-plugin-import using their installation guide.
What you expected:
The plugin to be seen.
What happened instead:=
The plugin is not seen.

any error logs? anywhere?


⚠ ClamAV has degraded performance

In 919aab8, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 107 ms

Resolved: ClamAV performance has improved in 27b5d60 after 535 days, 23 hours, 43 minutes.


Investigate alignment configuration with paddings set

Was fixed by: https://github.com/rubensousa/DpadRecyclerView/issues/93


[BUG] The code for deepspeed.comm.comm.monitored_barrier()

takes a "timeout" parameter but then INTERNALLY calls a function
*.barrier() that does NOT take the "timeout" paramter.

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
ds_report output
Please run ds_report to give us details about your setup.
Screenshots
If applicable, add screenshots to help explain your problem.
System info (please complete the following information):

OS: [e.g. Ubuntu 18.04]
GPU count and types [e.g. two machines with x8 A100s each]
Interconnects (if applicable) [e.g., two machines connected with 100 Gbps IB]
Python version
Any other relevant info about your setup

Launcher context
Are you launching your experiment with the deepspeed launcher, MPI, or something else?
Docker context
Are you using a specific docker image that you can share?
Additional context
Add any other context about the problem here.

@phalexo thanks for bringing this to our attention. I believe that barrier was mistakenly used instead of monitored_barrier here:
https://github.com/microsoft/DeepSpeed/blob/604d701e35548e5407b017c088bdc3760832c9e0/deepspeed/comm/comm.py#L419
@Quentin-Anthony you were the last one to touch this line. Can you comment? Thanks!

@phalexo thanks for bringing this to our attention. I believe that barrier was mistakenly used instead of monitored_barrier here:
https://github.com/microsoft/DeepSpeed/blob/604d701e35548e5407b017c088bdc3760832c9e0/deepspeed/comm/comm.py#L419
@Quentin-Anthony you were the last one to touch this line. Can you comment? Thanks!

Thank you for looking at this. The initial problem for me is that there does not seem to be a way to change the timeout. I went as far as changing the hardcoded value of "default_pg_time" to 5 hours, but for some reason at 30 minutes an exception is raised anyway.
There must be some way to control the timeout for the default group.

Fixing this bug may or not allow to change the timeout. See below.
#pragma once
#include <condition_variable>
#include
#include
#include
#include <unordered_map>
#include
#include
#include <ATen/ATen.h>
#include <c10/macros/Macros.h>
#include <torch/csrc/distributed/c10d/Work.hpp>
#include <torch/csrc/distributed/c10d/Types.hpp>
#include <torch/csrc/distributed/c10d/Utils.hpp>
#include <torch/csrc/distributed/c10d/debug.h>
#include <torch/csrc/distributed/c10d/sequence_num.hpp>
constexpr auto kBackendDefaultTimeout =
std::chrono::milliseconds(30 * 60 * 1000); // This is mostly likely the culprit, nested in several layers of include files.
namespace c10d {
class TORCH_API Backend : public torch::CustomClassHolder {
public:
// Backend Options is a base struct that defines the basic options
// when constructing a Backend. Each Backend subclass should
// extend this struct and define its options if it wants to provide more
// config options (beyond basic ones defined here) to end user.
FILE: ~/pytorch/torch/csrc/distributed/c10d/Backend.hpp

@phalexo thanks for bringing this to our attention. I believe that barrier was mistakenly used instead of monitored_barrier here:
https://github.com/microsoft/DeepSpeed/blob/604d701e35548e5407b017c088bdc3760832c9e0/deepspeed/comm/comm.py#L419
@Quentin-Anthony you were the last one to touch this line. Can you comment? Thanks!

@mrwyattii -- You're correct! Looks like a typo. I introduced a PR in https://github.com/microsoft/DeepSpeed/pull/4496.
@phalexo -- I believe the cause of your issue is that torch.distributed.barrier() doesn't have a timeout arg, so your deepspeed.comm.monitored_barrier() call dropped the timeout arg.  I was able to verify this myself with the nccl backend and timeouts of both 5 and 45 minutes. Give it a try and let me know if it works for you!

@phalexo thanks for bringing this to our attention. I believe that barrier was mistakenly used instead of monitored_barrier here:
https://github.com/microsoft/DeepSpeed/blob/604d701e35548e5407b017c088bdc3760832c9e0/deepspeed/comm/comm.py#L419
@Quentin-Anthony you were the last one to touch this line. Can you comment? Thanks!

@mrwyattii -- You're correct! Looks like a typo. I introduced a PR in #4496.
@phalexo -- I believe the cause of your issue is that torch.distributed.barrier() doesn't have a timeout arg, so your deepspeed.comm.monitored_barrier() call dropped the timeout arg.  I was able to verify this myself with the nccl backend and timeouts of both 5 and 45 minutes. Give it a try and let me know if it works for you!

It seems only the deepspeed/comm/torch.py has been modified, but deepspeed/comm/comm.py remains the same.
I will test with both and see if it makes a difference.

The exception happens deep inside the C/C++ code in pytorch/nccl, @Quentin-Anthony
Somehow one has to be able to adjust the timeout inside the backend.
frame #29: PyEval_EvalCodeEx + 0x39 (0x5568d06e5fd9 in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #30: PyEval_EvalCode + 0x1b (0x5568d06e5f9b in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #31:  + 0x1eb929 (0x5568d0706929 in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #32:  + 0x1ea923 (0x5568d0705923 in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #33:  + 0x9a00f (0x5568d05b500f in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #34: PyRun_SimpleFileExFlags + 0x364 (0x5568d05b4b13 in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #35:  + 0x8cfdc (0x5568d05a7fdc in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #36: Py_BytesMain + 0x39 (0x5568d06d97b9 in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #37: __libc_start_main + 0xf3 (0x7fca24cf9083 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #38:  + 0x1be6bd (0x5568d06d96bd in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
. This may indicate a possible application crash on rank 0 or a network set up issue.
Traceback (most recent call last):
File "fine-tune.py", line 263, in
train()
File "fine-tune.py", line 201, in train
barrier()
File "/home/developer/.local/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3313, in barrier
work = default_pg.barrier(opts=opts)
RuntimeError: [3] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: Socket Timeout
Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:604 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f298cf424d7 in /home/developer/.local/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, char const*) + 0x68 (0x7f298cf0c434 in /home/developer/.local/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #2: c10d::TCPStore::doWait(c10::ArrayRefstd::string, std::chrono::duration<long, std::ratio<1l, 1000l> >) + 0xd8 (0x7f28fc3d87c8 in /home/developer/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #3: c10d::TCPStore::doGet(std::string const&) + 0x22 (0x7f28fc3d9472 in /home/developer/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #4: c10d::TCPStore::get(std::string const&) + 0x59 (0x7f28fc3d94f9 in /home/developer/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f28fc3989c1 in /home/developer/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f28fc3989c1 in /home/developer/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f28fc3989c1 in /home/developer/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f28ab3a7eef in /home/developer/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, std::vector<c10::Device, std::allocatorc10::Device > const&, c10d::OpType, int, bool) + 0x201 (0x7f28ab3abba1 in /home/developer/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #10:  + 0x1322ccd (0x7f28ab3b2ccd in /home/developer/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #11: c10d::ProcessGroupNCCL::allreduce_impl(std::vector<at::Tensor, std::allocatorat::Tensor >&, c10d::AllreduceOptions const&) + 0x21 (0x7f28ab3b40d1 in /home/developer/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #12: c10d::ProcessGroupNCCL::allreduce(std::vector<at::Tensor, std::allocatorat::Tensor >&, c10d::AllreduceOptions const&) + 0x39d (0x7f28ab3b6d8d in /home/developer/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #13: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x851 (0x7f28ab3c5ce1 in /home/developer/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #14:  + 0x54861d9 (0x7f28fc38d1d9 in /home/developer/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #15:  + 0x5489f2a (0x7f28fc390f2a in /home/developer/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #16:  + 0x5498d90 (0x7f28fc39fd90 in /home/developer/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #17:  + 0xb7409e (0x7f2910fd009e in /home/developer/.local/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #18:  + 0x3bfba0 (0x7f291081bba0 in /home/developer/.local/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #19: PyCFunction_Call + 0x52 (0x5605b928ffd2 in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #20: _PyObject_MakeTpCall + 0x3db (0x5605b927b29b in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #21:  + 0x136d7d (0x5605b928fd7d in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #22: _PyEval_EvalFrameDefault + 0x11bb (0x5605b92734ab in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #23:  + 0x1cbe00 (0x5605b9324e00 in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #24: _PyEval_EvalFrameDefault + 0x3aa (0x5605b927269a in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #25: _PyEval_EvalCodeWithName + 0x2f1 (0x5605b9271261 in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #26: _PyFunction_Vectorcall + 0x19c (0x5605b928289c in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #27: _PyEval_EvalFrameDefault + 0x3aa (0x5605b927269a in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #28: _PyEval_EvalCodeWithName + 0x2f1 (0x5605b9271261 in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #29: PyEval_EvalCodeEx + 0x39 (0x5605b9323fd9 in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #30: PyEval_EvalCode + 0x1b (0x5605b9323f9b in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #31:  + 0x1eb929 (0x5605b9344929 in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #32:  + 0x1ea923 (0x5605b9343923 in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #33:  + 0x9a00f (0x5605b91f300f in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #34: PyRun_SimpleFileExFlags + 0x364 (0x5605b91f2b13 in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #35:  + 0x8cfdc (0x5605b91e5fdc in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #36: Py_BytesMain + 0x39 (0x5605b93177b9 in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
frame #37: __libc_start_main + 0xf3 (0x7f2992fe4083 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #38:  + 0x1be6bd (0x5605b93176bd in /home/developer/mambaforge/envs/FinGPT/bin/python3.8)
. This may indicate a possible application crash on rank 0 or a network set up issue.

@phalexo thanks for bringing this to our attention. I believe that barrier was mistakenly used instead of monitored_barrier here:
https://github.com/microsoft/DeepSpeed/blob/604d701e35548e5407b017c088bdc3760832c9e0/deepspeed/comm/comm.py#L419
@Quentin-Anthony you were the last one to touch this line. Can you comment? Thanks!

@mrwyattii -- You're correct! Looks like a typo. I introduced a PR in #4496.
@phalexo -- I believe the cause of your issue is that torch.distributed.barrier() doesn't have a timeout arg, so your deepspeed.comm.monitored_barrier() call dropped the timeout arg.  I was able to verify this myself with the nccl backend and timeouts of both 5 and 45 minutes. Give it a try and let me know if it works for you!

As an experiment I replaced "barrier" calls with "all_reduce" since essentially from the timing point of view it does the same synchronization as "barrier" calls.
I got exactly the same error, with the socket timeout, except the function name in the trace was now "all_reduce" instead of "barrier".
Could you post your code to experiment with 5, 45 minute timeouts?

It seems only the deepspeed/comm/torch.py has been modified, but deepspeed/comm/comm.py remains the same.

I don't think this is correct? Here's the PR change: https://github.com/microsoft/DeepSpeed/pull/4496/files#diff-2bf8bf2cd0ea4637cdd3397e6e27d38ce59e0ebc8daafcc583dbfcd8c092b0beR419

So, torch.distributed.monitored_barrier() is NOT implemented for "nccl', just for 'gloo'

Yep you're right. I was using an older custom version of torch with monitored_barrier implemented for NCCL in a hacky way (spawns an MPI process group to track ranks instead of Gloo), but I realize that could be confusing so I apologize.
What I ran was essentially the following:
import deepspeed
import deepspeed.comm as dist

if __name__ == "__main__":
	deepspeed.init_distributed("gloo", dist_init_required=True)
	if dist.get_rank() != 1:
		dist.monitored_barrier(timeout=timedelta(minutes=5))

Can you try this?
Also, I think your socket timeouts:

RuntimeError: [3] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: Socket Timeout

Are completely unrelated to the monitored_barrier timeout arg. This is a generic error message and the error source is unclear from your trace. I suspect you would get the same error by calling the pure torch.distributed calls without DeepSpeed, especially if you get the same error with AllReduce.

I will try it with "gloo" backend. That said, what I think happens is this:
if rank > 0:
barrier()
 // lengthy training data processing that exceeds 30 minutes.

if rank == 0:
barrier()
All ranks above 0 are stopped at the barrier() above, waiting for rank 0 to finish preprocessing and rejoin at the second barrier(). When about 30 minutes hit, they assume rank 0 is dead and some kind of exception processing happens. I have looked at some settings for sockets and the timeout is 10 minutes, but there are 3 retries for sockets. So maybe 30 minutes come from that.
Why isn't monitored_barrier implemented for nccl? I thought it is faster than gloo.
if dist.get_rank() != 1:  // Should this be 1? or -1, just a choice? or 0?

It seems only the deepspeed/comm/torch.py has been modified, but deepspeed/comm/comm.py remains the same.

I don't think this is correct? Here's the PR change: https://github.com/microsoft/DeepSpeed/pull/4496/files#diff-2bf8bf2cd0ea4637cdd3397e6e27d38ce59e0ebc8daafcc583dbfcd8c092b0beR419

So, torch.distributed.monitored_barrier() is NOT implemented for "nccl', just for 'gloo'

Yep you're right. I was using an older custom version of torch with monitored_barrier implemented for NCCL in a hacky way (spawns an MPI process group to track ranks instead of Gloo), but I realize that could be confusing so I apologize.
What I ran was essentially the following:
import deepspeed
import deepspeed.comm as dist

if __name__ == "__main__":
	deepspeed.init_distributed("gloo", dist_init_required=True)
	if dist.get_rank() != 1:
		dist.monitored_barrier(timeout=timedelta(minutes=5))

Can you try this?
Also, I think your socket timeouts:

RuntimeError: [3] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: Socket Timeout

Are completely unrelated to the monitored_barrier timeout arg. This is a generic error message and the error source is unclear from your trace. I suspect you would get the same error by calling the pure torch.distributed calls without DeepSpeed, especially if you get the same error with AllReduce.

Why isn't monitored_barrier implemented for nccl? I thought it is faster than gloo.

Because monitored_barrier uses a CPU process group to monitor ranks and NCCL Is GPU-only.

if dist.get_rank() != 1: // Should this be 1? or -1, just a choice? or 0?

Just a choice. My code is just triggering the case where a single rank (1 in this case) doesn't enter the barrier.

I have looked at some settings for sockets and the timeout is 10 minutes, but there are 3 retries for sockets. So maybe 30 minutes come from that.

Wouldn't the 30-minute timeout just be from the default pytorch pg timeout?
https://github.com/microsoft/DeepSpeed/blob/78c518ed979b4cf1df1816bdbe52b0d21531bee4/deepspeed/comm/comm.py#L608
and
https://github.com/pytorch/pytorch/blob/5c3955200c3ca1916f669a32d99cd7976f3d4930/torch/csrc/distributed/c10d/ProcessGroup.hpp#L25

Why isn't monitored_barrier implemented for nccl? I thought it is faster than gloo.

Because monitored_barrier uses a CPU process group to monitor ranks and NCCL Is GPU-only.

if dist.get_rank() != 1: // Should this be 1? or -1, just a choice? or 0?

Just a choice. My code is just triggering the case where a single rank (1 in this case) doesn't enter the barrier.

I have looked at some settings for sockets and the timeout is 10 minutes, but there are 3 retries for sockets. So maybe 30 minutes come from that.

Wouldn't the 30-minute timeout just be from the default pytorch pg timeout?
https://github.com/microsoft/DeepSpeed/blob/78c518ed979b4cf1df1816bdbe52b0d21531bee4/deepspeed/comm/comm.py#L608
and
https://github.com/pytorch/pytorch/blob/5c3955200c3ca1916f669a32d99cd7976f3d4930/torch/csrc/distributed/c10d/ProcessGroup.hpp#L25

default_pg_timeout itself is NOT set to an actual value, but instead is set to another constant.
It is still a total puzzle to me why values defined in Backend.hpp and ProcessGroup.hpp are not controlling all the other values, including default_pg_timeout.
I built pytorch from source after modifying those timeout values by 10x, but it had no effect.
If nccl is strictly GPU side, then why can't one have CPU side functions for barriers? Why do I need to change the backend? I am not quite getting this. I am not trying to synch the GPU processes, just the CPU side preprocessing.

Still a problem, even with GLOO backend.
The argument trust_remote_code is to be used with Auto classes. It has no effect here and is ignored.
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:27<00:00, 13.69s/it]
Using pad_token, but it is not set yet.
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:28<00:00, 14.35s/it]
Using pad_token, but it is not set yet.
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:30<00:00, 15.12s/it]
Using pad_token, but it is not set yet.
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:30<00:00, 15.45s/it]
Using pad_token, but it is not set yet.
Map (num_proc=14):  70%|██████████████████████████████████████████████████████████████████▌                            | 652383/930514 [6:39:53<4:31:33, 17.07 examples/s][E ProcessGroupGloo.cpp:138] Rank 3 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
Traceback (most recent call last):
File "fine-tune.py", line 305, in
train()
File "fine-tune.py", line 232, in train
dist.monitored_barrier(timeout=datetime.timedelta(minutes=600))
File "/home/developer/pytorch/torch/distributed/distributed_c10d.py", line 3401, in monitored_barrier
return group_to_use.monitored_barrier(timeout, wait_all_ranks=wait_all_ranks)
RuntimeError: Rank 3 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
Original exception:
[/home/developer/pytorch/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:133] Timed out waiting 24000000ms for send operation to complete
Map (num_proc=14):  70%|██████████████████████████████████████████████████████████████████▌                            | 652391/930514 [6:39:54<4:45:07, 16.26 examples/s][E ProcessGroupGloo.cpp:138] Rank 1 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
Traceback (most recent call last):
File "fine-tune.py", line 305, in
train()
File "fine-tune.py", line 232, in train
dist.monitored_barrier(timeout=datetime.timedelta(minutes=600))
File "/home/developer/pytorch/torch/distributed/distributed_c10d.py", line 3401, in monitored_barrier
return group_to_use.monitored_barrier(timeout, wait_all_ranks=wait_all_ranks)
RuntimeError: Rank 1 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
Original exception:
[/home/developer/pytorch/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:133] Timed out waiting 24000000ms for send operation to complete
Map (num_proc=14):  70%|██████████████████████████████████████████████████████████████████▌                            | 652417/930514 [6:39:56<7:05:10, 10.90 examples/s][E ProcessGroupGloo.cpp:138] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
Traceback (most recent call last):
File "fine-tune.py", line 305, in
train()
File "fine-tune.py", line 232, in train
dist.monitored_barrier(timeout=datetime.timedelta(minutes=600))
File "/home/developer/pytorch/torch/distributed/distributed_c10d.py", line 3401, in monitored_barrier
return group_to_use.monitored_barrier(timeout, wait_all_ranks=wait_all_ranks)
RuntimeError: Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
Original exception:
[/home/developer/pytorch/third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:133] Timed out waiting 24000000ms for send operation to complete
Map (num_proc=14):  70%|██████████████████████████████████████████████████████████████████▌                            | 652430/930514 [6:39:57<6:55:04, 11.17 examples/s][2023-10-17 06:15:53,453] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 47356
[2023-10-17 06:15:54,809] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 47357
[2023-10-17 06:15:54,810] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 47358
[2023-10-17 06:15:54,811] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 47390


PR: Airplay support

Willing to make a PR for airplay support. Wonder if this would be a welcome contribution.

There is no Airplay support since at the time I was creating the package there was no good way to implement it. Don't know if there is one right now.
Feel free to experiment with it :)


Allow pre-allocated grpc_byte_buffer to be passed down for recv ops

Currently we insist upon a grpc_byte_buffer** for a recv op in the public API. Consider relaxing this so that a pre-allocated grpc_byte_buffer can be passed down, avoiding an allocation.

Will be solved with direct placement work


moe backend fixes

IOS Preferences
fix java.lang.ClassCastException: java.lang.String cannot be cast to ios.foundation.NSString
Using NSAutoreleasePool on moe going to hangs
IOSInput fix some copy-paste bugs, merge with robovm backend

Thanks for the contribution, I'd like to fix these in different ways. Will be pushing fixes shortly.
Not sure what your problem was with CoreMotion, could you explain why you removed it for the deprecated UIAccelleromter?

UIAccelleromter its just copy-paste from robovm.
Main fix it remove breaks at this
`public boolean isCatchMenuKey() {
return false;
@@ -544,15 +561,12 @@ void processEvents () {
if (event.phase == UITouchPhase.Began) {
if (inputProcessor != null) inputProcessor.touchDown(event.x, event.y, event.pointer, Buttons.LEFT);
if (numTouched == 1) justTouched = true;

 			break;
 		}
 		if (event.phase == UITouchPhase.Cancelled || event.phase == UITouchPhase.Ended) {
 			if (inputProcessor != null) inputProcessor.touchUp(event.x, event.y, event.pointer, Buttons.LEFT);

 			break;
 		}
 		if (event.phase == UITouchPhase.Moved || event.phase == UITouchPhase.Stationary) {
 			if (inputProcessor != null) inputProcessor.touchDragged(event.x, event.y, event.pointer);

 			break;
 		}
 	}
 	touchEventPool.freeAll(touchEvents);`

and save touches touchDown[event.pointer] = touch; merge it only, now input is broken without this fixs

Yes, will be pushing that soon, just testing.  Accel is already implemented through CMMotionManager


Use of actual ids rather than accessibility identifiers for native apps

The matcher by.id(id) matches elements with the specified accessibility identifier. Would it be possible to match elements by their actual ids?
For example my Android native app, I already have the following to define a button:
android:id="@+id/bt_login"
I now need to add the following additional line in order to match it with Detox:
android:tag="bt_login"
This is causing concerns from our developers who are not happy about adding additional lines to their code to support automation testing
Since it is possible with Espresso to match with the actual id, and since Detox now uses Espresso, are there any plans on allowing for matching with actual identifiers?

@chirag-timetac could you elaborate on your use case? Do you have intermixed screens with native and react-native views?
Would you consider this approach for your case? Detox-Native

@d4vidi Thanks I was using detox with our native app, I will try with Detox-Native.

In any case, I think we might be able to add a by.nativeId() or something similar to the JS API - regardless.

@chirag-timetac would you mind sharing what you solution of choice was? Have you decided on Espresso / Espresso+Detox-Native?

After a discussion on this, we decided not to support this. Here's the rationale:

For native apps: Using Detox is somewhat pointless. We currently don't support that on Android (#2543).
For greenfield RN apps: No control over native ID's, which again makes it pointless.

So the only relevant use case is greenfield RN apps, which, as it appears, the OP hasn't requested.
That said, had we had a simple solution for it (tracing back the resource ID's to their original name), we might have considered executing it nonetheless. However ATM, is appears as something that would require research, and perhaps would not be at all feasible.


Nginx custom log

Hello,
I have a problem and need help.
1.How to filter a custom log type?
like this:
access_log logs/access.log mylog;
2.By default, the child configuration block overrides the parent configuration block and does not inherit the configuration of the parent configuration block. So what happens if the child configuration block normally logs all logs and the parent configuration block configures the filter?
like this:
server{access_log_bypass_if ($status = 200);access_log logs/access_if.log;
location /test {access_log logs/access.log;}}

thanks all.

in your case, are you going to filter "mylog"? you can not do like that. you should add different "access_log" configurations and use them separately in different location or server blocks.

because "access_log_bypass_if" and "access_log" are two directives, they are two configurations, so "access_log" in location block overrides it in server block, however, there is no "access_log_bypass_if" in  location block, its server configuration is inherited.

Ok, I understand, thank you.


store: playing with json models

Just some playing around in here with a RegisterJSON method on the Store. I didn't work past the getEntityID` panic, but seems doable. Thoughts? Worth pursuing?

Didn't know about dynamicstruct library, pretty neat.
This idea seems to unlock the necessary to use the API as is, so I agree that it seems doable.
Internally, I mentioned another possible approach that I'll try to sketch too and see how if feels.

Superseded by #99


Misleadin docsting for suppressDragLeaveHidesColumns argument of AgGrid class in v2.0.0a5

What is default value?
- suppressDragLeaveHidesColumns (boolean; default True):
    If True, when you drag a column out of the grid (e.g. to the group
    zone) the column is not hidden. Default Value: False.

It is True. Thanks for pointing this out. 😊


fix: revert @oclif/core to v1 [FTL-9076]

:tada: This PR is included in version 1.61.15 :tada:
The release is available on:

GitHub release
npm package (@latest dist-tag)

Your semantic-release bot :package::rocket:


TextInput: long text overflows textinput's border

Software Versions

Python: 3.7.5
OS: Android, Windows (happens independently of OS)
Kivy: 1.11.1
Kivy installation method: (not sure?)

Describe the bug
With single line textinputs (multiline: False) long texts will overflow and the last letter typed in will not be visible.
To me, it seems to be an issue of the cursor. When the text overflows, if I press the right arrow (i.e. move the cursor right) I am then able to read the whole text. So the text does fit into the textinput, it is the cursor that seems to misbehave (instead of moving after the n typed letter, it stays at n-1).
Expected behavior
The last letter typed should be clearly visible.
To Reproduce
from kivy.app import App
from kivy.core.window import Window
from kivy.lang import Builder
from kivy.uix.gridlayout import GridLayout
from kivy.uix.screenmanager import Screen, ScreenManager

class MainGrid(GridLayout):
    pass

kv = Builder.load_string(
'''
MainGrid:

<MainGrid>:
    size_hint: 1, 1
    StackLayout:
        size_hint: 1,1
        TextInput:
            size_hint: 1, 1
            multiline: False
''')

class TrialApp(App):
    def build(self):
        return kv

if __name__ == "__main__":
    TrialApp().run()

Additional context
None

Can you try with Kivy==2.0.0rc3?

Kivy==2.0.0rc3

You mean putting that in the buildozer.spec?

Yes and post here if it solved the issue.

Just tested it. Still happening with Kivy==2.0.0rc3


Missing networkInactivityEnabled on appgatesdp_saml_identity_provider

Not filling in most of the Issue Template because this is just a missing attribute in the provider.
The 6.1 Release contains a new Inactivity Timeout feature which takes into account whether there's actually traffic flowing over the tunnel in addition to OS Input Activity.
Unfortunately, this can't currently be managed via terraform because a corresponding attribute isn't found in the provider. Can this be added and released?
Thanks in advance!

👋 @danijeel. :)
Thanks again for the quick turnaround on this.
I'm using 1.0.4 against a 6.1 Collective and I just got the following:
2023-02-28T13:30:43.135-0500 [DEBUG] [aws-sdk-go] {}
╷
│ Error: network_inactivity_timeout_enabled is only available in 6.1 or higher
│
│   with module.identity_providers.appgatesdp_saml_identity_provider.google,
│   on identity_providers/google.tf line 1, in resource "appgatesdp_saml_identity_provider" "google":
│    1: resource "appgatesdp_saml_identity_provider" "google" {
│
╵

Any ideas what would cause that?
Looking at the logs on my Collective I don't even see an attempt to POST to /admin/identity-providers so it seems like the Provider itself is concluding that it shouldn't POST?
$ journalctl --since -1d -g 'POST /admin/identity-providers'
-- Logs begin at Mon 2023-02-20 20:52:44 UTC, end at Tue 2023-02-28 18:49:16 UTC. --
-- No entries --

On another Collective running an identical version of both the Provider (1.0.4) and Appgate (6.1.0-31746-release) I see the success POST going through…
Any help would be greatly appreciated. :)

Also note that removing the attributes network_inactivity_timeout_enabled and inactivity_timeout_minutes from the config results in a successful POST against the misbehaving Collective:
… [28/Feb/2023:18:55:06 +0000] "POST /admin/identity-providers …

Can you see what client_version  you are using, either in the provider appgatesdp block or in your statefile? it should be 18 for version 6.1
provider "appgatesdp" {
  client_version = XX// ??
}

or test  with environment variable, for example:
APPGATE_CLIENT_VERSION=18 terraform apply -auto-approve

The provider block makes no mention of client_version (anywhere in my config).
Setting APPGATE_CLIENT_VERSION=18 during plan and apply does seem to have fixed things. But I haven't needed to set that anywhere else in my config.

One thing I noticed is that if I successfully create the IDP I can then modify it to include the inactivity settings with no change (including setting client version).

Shouldn't client_version default to 18 in 1.0.4 since that adds network_inactivity_timeout_enabled and it seems to be required?

yes, You are right. We have forgotten to update it, we will fix it.

Shouldn't client_version default to 18 in 1.0.4 since that adds network_inactivity_timeout_enabled and it seems to be required?

@danijeel I just noticed the following in my Controller Logs.

Warning from Controller
4/24/2023, 5:20:51 PM 21 occurrences
Delete Message
API version 'application/vnd.appgate.peer-v13+json' is still in use by "Terraform/1.3.9 (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 appgatesdp/1.2.0 - <IP_ADDRESS>" (current version is 'application/vnd.appgate.peer-v18+json'). Please update any external scripts as support for this version will be removed.

I guess that means that this was never fixed up through provider version 1.2.0?

It's most likely because the provider config is missing client version and it's computed automatically. This error message is harmless and will be generated each time you run terraform apply|refresh|destroy. If you want to avoid this error message, add client_version to your provider configuration or set the environment variable APPGATE_CLIENT_VERSION.


git changes not showing after branch merge

VSCode Version: 1.52.1 (user setup)
OS Version: Windows_NT x64 10.0.18363

Steps to Reproduce:

Open a git repository
Make a merge branch

The issue is only specific to this git project, I tried to delete it and run git clone, didn't fix it

Does this issue occur when all extensions are disabled?: Yes
git output:

Looking for git in: C:\Program Files\Git\cmd\git.exe
Using git 2.29.2.windows.3 from C:\Program Files\Git\cmd\git.exe
git rev-parse --git-dir
Open repository: c:\myproject
git fetch
git status -z -u
git symbolic-ref --short HEAD
git rev-parse develop
git rev-parse --symbolic-full-name develop@{u}
git rev-list --left-right develop...refs/remotes/origin/develop
git for-each-ref --sort -committerdate --format %(refname) %(objectname) %(*objectname)
git remote --verbose
Warning: Failed to watch ref 'c:\myproject.git\refs\remotes\origin\develop', is most likely packed.
git status -z -u
git symbolic-ref --short HEAD
git rev-parse develop
git rev-parse --symbolic-full-name develop@{u}
git rev-list --left-right develop...refs/remotes/origin/develop
git for-each-ref --sort -committerdate --format %(refname) %(objectname) %(*objectname)
git remote --verbose
Warning: Failed to watch ref 'myproject.git\refs\remotes\origin\develop', is most likely packed.

Screenshot:

Same for me (not always).
MacOs 10.15.7, VsCode Version: 1.52.1
Commit: ea3859d4ba2f3e577a159bc91e3074c5d85c0523
Date: 2020-12-16T16:30:02.420Z (1 day ago)

Extentions
coenraads.bracket-pair-colorizer-2-0.2.0        ms-azuretools.vscode-docker-1.9.0               spoonscen.es6-mocha-snippets-0.2.2
eamodio.gitlens-11.0.6                          ms-python.python-2020.12.424452561              tomoki1207.pdf-1.1.0
esbenp.prettier-vscode-5.8.0                    ms-toolsai.jupyter-2020.12.414227025            wilson-godoi.wg-getters-and-setters-2.0.0
gruntfuggly.todo-tree-0.0.192                   ms-vscode.powershell-2020.6.0                   yzhang.markdown-all-in-one-3.4.0
mohsen1.prettify-json-0.0.3                     pkief.material-icon-theme-4.4.0                 zainchen.json-2.0.2

I have the same issue here. any fixes ??

I have the same issue here. any fixes ??
Hey, I downgraded to the previous version temporary, it's 100% fix :)

I have the same issue here. any fixes ??

Hey, I downgraded to the previous version temporary, it's 100% fix :)

you're right, thanks.

Same here.
When upgraded to version 1.52 (as well as 1.52.1), I noticed that changes are not shown when performing certain merges, which is really confusing.

#112713
#112697

Seems that rename/delete conflicts are mistakenly treated as a file.

The temporary solution is turn off the git.openDiffOnClick command.

When this option is turned off, corresponding conflicts are correctly displayed.

Corresponding Git blame:
https://github.com/microsoft/vscode/blame/0c947cb4ee4a35f04aa6c5117a9727be35b3c828/extensions/git/src/repository.ts#L586

Duplicate of #112537 (which will be fixed in tomorrow's insiders)

Duplicate of #112537 (which will be fixed in tomorrow's insiders)


Net package ip.To4() function does not validates the IP address and only checks for byte length

Go version
1.22 && 1.21.3
Output of go env in your module/workspace:
GO111MODULE='on'
GOARCH='arm64'
GOBIN='/Users/shubham/go/bin'
GOCACHE='/Users/shubham/Library/Caches/go-build'
GOENV='/Users/shubham/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/shubham/go/pkg/mod'
GONOPROXY='github.com/spectrocloud'
GONOSUMDB='github.com/spectrocloud'
GOOS='darwin'
GOPATH='/Users/shubham/go'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.21.3/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.21.3/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.21.3'
GCCGO='gccgo'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='0'
GOMOD='/Users/shubham/go/spectrocloud/mogoco/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/hv/8fj3n4r533v5plclxkqpw88m0000gn/T/go-build29931247=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?
https://go.dev/play/p/hp26BIPZvGN
What did you see happen?
I saw a name with a length of four being converted to net.IP
for ex:
Name: mady
IP: <IP_ADDRESS>
What did you expect to see?
IP address is invalid

that is a valid IP address.

Sorry it might be a noob question but do you care to elaborate why is it correct.

We don't use the issue tracker for questions.  Please see https://go.dev/wiki/Questions.
That said, why would that IP address not be correct.  But don't answer here!  Explain why when you ask the question on a forum.  Thanks.


[Proposal]Solution for the top 100 places in the test network to be controlled by one team

💥 Proposal

It's already clear that there's some team occupying a top 100 spot right now.
And I think the aleo team is having a hard time catching these people right now because they can turn on firewalls for miners and mining pools to not be accessible to the public and then go through a few client nodes to broadcast the data.
So I offer a solution
The first step is to immediately turn on the registration system and allow only registered members to participate in this test network. The registration form has two required fields, the name used as kyc, and the participating wallet.
Step 2, work on kyc for the first 200 people. And require video interviews, video interviews will be technically oriented, the same as mina had done, this effect is the best, no one.
The third step, after completing 200 people kyc interview, snapshot the current ranking, and then if a new wallet is found to produce five 10 blocks or more in a day, then a one-on-one video interview will be conducted within three days.
To be honest, I really need the aleo team to get me back on message about aleo, I love aleo, made my own videos and am promoting it to my community, but there are so many issues at the moment.
I think it's a very negative move to let it go like this, even if the aleo team eventually sends out a public saying that the first 100 people are normal, but will the community members believe it? 6p/s or so of arithmetic power to theoretically produce a block a day, and that 6ps of arithmetic power requires about 15 7282x2 machines to reach, and those 15 machines need 300$ a month each There is too much evidence in the discord for everyone to see.
I have no objection to rewarding the geek, even if he alone mines 99 percent of the blocks, which is his ability, and those of us who can go for the rest of the blocks. But now all changed, he not only wants top5, even top100 he all want, which touches everyone's interests.
So please team seriously consider this proposal, this proposal has no burden for the normal participating players and it is great protection for them. The only thing that hurts is the people who break the rules.

I couldn't agree with you more.👍 While the world of blockchain is free, these measures are necessary to implement if a harmonious and stable blockchain environment is to be created.

I agree with you that if there is no fair competition, we will lose our meaning here

Agree +1
MINA was the best testnet experience!!!
And there was everything very clear for the community.
Please VOTE: ThumbUP or ThumbDown

It will be adequate if the project team at least somehow marks and rewards the nodes without mined blocks. These nodes participated in the project, people helped the project, although not as actively as those who have access to super-powerful GPUs.

I completely agree. Testers with 32+ core servers cannot mine even one block. With the current state of affairs, 90% of testers will be left without rewards. It's not fair.

I fully agree with all the above statements and complaints.
It would be fair to give testers at least a priority, not a guaranteed allocation of course, but a priority. Because after all, this team did not take into account the factors that led to the disappointment that many are experiencing.

Closing, as this doesn't seem to be applicable to the current testnet.


Fix aggregation excluding NULL values

Description
Shortcut: (link)
Formula widget was displaying incorrect values because NULL values were considered as 0. Now we're ignoring this items instead.
BEFORE:

NOW:

Type of change

Fix

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

14 of 14 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.2%) to 73.562%

Totals

Change from base Build<PHONE_NUMBER>:
0.2%

Covered Lines:
1313

Relevant Lines:
1685

💛  - Coveralls


CRM: allow overriding/extending app environment

It would be convenient to be able to create a single image and then use it in different environments (production, staging, qa, ...). This is currently not possible because there is no way to parametrize the executables which are running inside a container. The executable, its arguments, and environment variables are defined in the ImageManifest and the CRM can not override those.
I propose adding a new field to CRM apps.app with the name environment and value same as ImageManifest app.environment. The executor would then merge the CRM app environment into the IM app environment (allowing the former to override the environment of the later).

Actually, it might be useful to have a more strictly defined way to parametrize the executable, similar to how mount points are defined. A set of inputs (parameters) which must be supplied by the outside (executor, user) and without which the executable can not function.
This would make it possible to:

Prompt the user for input while trying to create a container (rkt run ..).
Validate the container before actually executing the apps within.
Automatically generate forms in a web based container management UI.

@wereHamster I think this is covered by #84 (at least what I had in mind), do you mind if we close this out and move the discussion there?

@wereHamster I updated the OP of #84 to include mention of environment

Actually, it might be useful to have a more strictly defined way to parametrize the executable, similar to how mount points are defined. A set of inputs (parameters) which must be supplied by the outside (executor, user) and without which the executable can not function.

I am not really sure what you are getting at here. The executor (and user) can provide parameters to applications through environment variables (and the metadata service), is that not sufficient? Can you give me an example of parameters "without which the executable can not function"?

I believe this has been adequately resolved by https://github.com/appc/spec/commit/9aafd97bcebd31d7f82b7f04fb1d2ce4661ec3b8


Use same libnetcdf on Windows and *nix

@conda-forge-admin, please rerender

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

All these are coming from defaults in OS X. We need to investigate this before merging it.
asn1crypto                0.24.0                   py36_0    defaults
cffi                      1.11.5           py36h342bebf_0    defaults
chardet                   3.0.4            py36h96c241c_1    defaults
conda-env                 2.6.0                h36134e3_0    defaults
cryptography              2.2.2            py36h1de35cc_0    defaults
idna                      2.6              py36h8628d0a_1    defaults
libcxx                    4.0.1                h579ed51_0    defaults
libcxxabi                 4.0.1                hebd6815_0    defaults
libedit                   3.1.20170329         hb402a30_2    defaults
libffi                    3.2.1                h475c297_4    defaults
ncurses                   6.1                  h0a44026_0    defaults
pip                       10.0.1                   py36_0    defaults
pycosat                   0.6.3            py36hee92d8f_0    defaults
pycparser                 2.18             py36h724b2fc_1    defaults
pyopenssl                 18.0.0                   py36_0    defaults
pysocks                   1.6.8                    py36_0    defaults
python                    3.6.5                hc167b69_1    defaults
python.app                2                        py36_8    defaults
readline                  7.0                  hc1231fa_4    defaults
requests                  2.18.4           py36h4516966_1    defaults
ruamel_yaml               0.15.37          py36h1de35cc_2    defaults
setuptools                39.2.0                   py36_0    defaults
six                       1.11.0           py36h0e22d5e_1    defaults
sqlite                    3.23.1               hf1716c9_0    defaults
tk                        8.6.7                h35a86e2_3    defaults
urllib3                   1.22             py36h68b9469_0    defaults
wheel                     0.31.1                   py36_0    defaults
xz                        5.2.4                h1de35cc_4    defaults
yaml                      0.1.7                hc338f04_2    defaults
zlib                      1.2.11               hf3cbc9b_2    defaults

Waiting on the outcome of https://github.com/conda-forge/libnetcdf-feedstock/pull/50#issuecomment-411218622 before we re-start things here.

Hi! This is the friendly automated conda-forge-linting service.
I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.
Here's what I've got...
For recipe:

The "source" section was expected to be a dictionary or a list, but got a ruamel.yaml.comments.CommentedMap.
The "build" section was expected to be a dictionary, but got a CommentedMap.
The "requirements" section was expected to be a dictionary, but got a CommentedMap.
The "test" section was expected to be a dictionary, but got a CommentedMap.
The "about" section was expected to be a dictionary, but got a CommentedMap.
The "extra" section was expected to be a dictionary, but got a CommentedMap.
The "package" section was expected to be a dictionary, but got a CommentedMap.
The home item is expected in the about section.
The license item is expected in the about section.
The summary item is expected in the about section.
The recipe could do with some maintainers listed in the extra/recipe-maintainers section.
The recipe must have a build/number section.
Recipe name has invalid characters. only lowercase alpha, numeric, underscores, hyphens and dots allowed
The "package" section was expected to be a dictionary, but got a CommentedMap.
The "extra" section was expected to be a dictionary, but got a CommentedMap.
The "package" section was expected to be a dictionary, but got a CommentedMap.
The "source" section was expected to be a dictionary or a list, but got a ruamel.yaml.comments.CommentedMap.
The "build" section was expected to be a dictionary, but got a CommentedMap.
The "requirements" section was expected to be a dictionary, but got a CommentedMap.
The "test" section was expected to be a dictionary, but got a CommentedMap.
The "about" section was expected to be a dictionary, but got a CommentedMap.
The "extra" section was expected to be a dictionary, but got a CommentedMap.

Hi! This is the friendly automated conda-forge-linting service.
I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.
Here's what I've got...
For recipe:

The "source" section was expected to be a dictionary or a list, but got a ruamel.yaml.comments.CommentedMap.
The "build" section was expected to be a dictionary, but got a CommentedMap.
The "requirements" section was expected to be a dictionary, but got a CommentedMap.
The "test" section was expected to be a dictionary, but got a CommentedMap.
The "about" section was expected to be a dictionary, but got a CommentedMap.
The "extra" section was expected to be a dictionary, but got a CommentedMap.
The "package" section was expected to be a dictionary, but got a CommentedMap.
The home item is expected in the about section.
The license item is expected in the about section.
The summary item is expected in the about section.
The recipe could do with some maintainers listed in the extra/recipe-maintainers section.
The recipe must have a build/number section.
Recipe name has invalid characters. only lowercase alpha, numeric, underscores, hyphens and dots allowed
The "package" section was expected to be a dictionary, but got a CommentedMap.
The "extra" section was expected to be a dictionary, but got a CommentedMap.
The "package" section was expected to be a dictionary, but got a CommentedMap.
The "source" section was expected to be a dictionary or a list, but got a ruamel.yaml.comments.CommentedMap.
The "build" section was expected to be a dictionary, but got a CommentedMap.
The "requirements" section was expected to be a dictionary, but got a CommentedMap.
The "test" section was expected to be a dictionary, but got a CommentedMap.
The "about" section was expected to be a dictionary, but got a CommentedMap.
The "extra" section was expected to be a dictionary, but got a CommentedMap.

Hi! This is the friendly automated conda-forge-linting service.
I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.
Here's what I've got...
For recipe:

The "source" section was expected to be a dictionary or a list, but got a ruamel.yaml.comments.CommentedMap.
The "build" section was expected to be a dictionary, but got a CommentedMap.
The "requirements" section was expected to be a dictionary, but got a CommentedMap.
The "test" section was expected to be a dictionary, but got a CommentedMap.
The "about" section was expected to be a dictionary, but got a CommentedMap.
The "extra" section was expected to be a dictionary, but got a CommentedMap.
The "package" section was expected to be a dictionary, but got a CommentedMap.
The home item is expected in the about section.
The license item is expected in the about section.
The summary item is expected in the about section.
The recipe could do with some maintainers listed in the extra/recipe-maintainers section.
The recipe must have a build/number section.
Recipe name has invalid characters. only lowercase alpha, numeric, underscores, hyphens and dots allowed
The "package" section was expected to be a dictionary, but got a CommentedMap.
The "extra" section was expected to be a dictionary, but got a CommentedMap.
The "package" section was expected to be a dictionary, but got a CommentedMap.
The "source" section was expected to be a dictionary or a list, but got a ruamel.yaml.comments.CommentedMap.
The "build" section was expected to be a dictionary, but got a CommentedMap.
The "requirements" section was expected to be a dictionary, but got a CommentedMap.
The "test" section was expected to be a dictionary, but got a CommentedMap.
The "about" section was expected to be a dictionary, but got a CommentedMap.
The "extra" section was expected to be a dictionary, but got a CommentedMap.

Hi! This is the friendly automated conda-forge-linting service.
I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.
Here's what I've got...
For recipe:

The "source" section was expected to be a dictionary or a list, but got a ruamel.yaml.comments.CommentedMap.
The "build" section was expected to be a dictionary, but got a CommentedMap.
The "requirements" section was expected to be a dictionary, but got a CommentedMap.
The "test" section was expected to be a dictionary, but got a CommentedMap.
The "about" section was expected to be a dictionary, but got a CommentedMap.
The "extra" section was expected to be a dictionary, but got a CommentedMap.
The "package" section was expected to be a dictionary, but got a CommentedMap.
The home item is expected in the about section.
The license item is expected in the about section.
The summary item is expected in the about section.
The recipe could do with some maintainers listed in the extra/recipe-maintainers section.
The recipe must have a build/number section.
Recipe name has invalid characters. only lowercase alpha, numeric, underscores, hyphens and dots allowed
The "package" section was expected to be a dictionary, but got a CommentedMap.
The "extra" section was expected to be a dictionary, but got a CommentedMap.
The "package" section was expected to be a dictionary, but got a CommentedMap.
The "source" section was expected to be a dictionary or a list, but got a ruamel.yaml.comments.CommentedMap.
The "build" section was expected to be a dictionary, but got a CommentedMap.
The "requirements" section was expected to be a dictionary, but got a CommentedMap.
The "test" section was expected to be a dictionary, but got a CommentedMap.
The "about" section was expected to be a dictionary, but got a CommentedMap.
The "extra" section was expected to be a dictionary, but got a CommentedMap.


Provide demo

There is a good documentation but no live link demo. It could be interesting to add it.

we can use this plunkr.
https://plnkr.co/edit/O3fP71Yc2uMApv5aGot6

demo added


Add CAS Auth Type

Adds CAS Auth type support
Breaking Changes
None
Review Checklist

Tests were added to cover all code changes
Documentation was added / updated
Code and tests follow standards in CONTRIBUTING.md

Codecov Report

Merging #927 into master will decrease coverage by 0.27%.
The diff coverage is 23.07%.

@@            Coverage Diff             @@
##           master     #927      +/-   ##
==========================================
- Coverage   90.19%   89.92%   -0.28%
==========================================
  Files          48       48
  Lines        6413     6439      +26
==========================================
+ Hits         5784     5790       +6
- Misses        629      649      +20

Flag
Coverage Δ

#python
89.92% <23.07%> (-0.28%)
:arrow_down:

Impacted Files
Coverage Δ

api/auth/authproviders.py
87.76% <23.07%> (-10.39%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4958909...cee3a97. Read the comment docs.

@nagem Environmental var SCITRAN_SITE_INACTIVITY_TIMEOUT is now working. Could we change the name to make it obvious the value represents seconds, or at a minimum document that in sample.config? Thanks.


add sidebar/nav and routes Co-authored-by: Amr Ahmed

Changes introduced via this PR: 👇

[x] Add a navigation panel with links
[x] Add routes

@AlexRS90 Thank you for the review ❤


Log which Garbage Collector is used

The answer is G1.

The answer is G1.

huh. You already worked on it in the past. Then we don't need it :)


can't parse iQIYI for video

Unable to parse video, especially mp4 format.  Use a browser to simulate phone grabbing.
You can catch the interface to get the mp4 video link. But many parameters of the interface do not know how to get, I hope you can update iQIYI video parsing.
Thank you very much.

Excuse me, what do you mean? Why is this problem closed?Are you updating iqiyi's parsing?
------------------ 原始邮件 ------------------
发件人: "Sergey<EMAIL_ADDRESS>发送时间: 2019年8月15日(星期四) 上午10:13
收件人<EMAIL_ADDRESS>抄送: "袁琦"2743579313<EMAIL_ADDRESS>主题: Re: [ytdl-org/youtube-dl] can't parse iQIYI for video (#22099)
Closed #22099.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.


Create 1582-special-positions-in-a-binary-matrix.py, 2482-difference-between-ones-and-zeros-in-row-and-column.py, 1475-final-prices-with-a-special-discount-in-a-shop.py

File(s) Modified: 1582-special-positions-in-a-binary-matrix.py, 2482-difference-between-ones-and-zeros-in-row-and-column.py,  1475-final-prices-with-a-special-discount-in-a-shop.py
Language(s) Used: python
Submission URL: _https://leetcode.com/problems/special-positions-in-a-binary-matrix/submissions/1118605153/
https://leetcode.com/problems/difference-between-ones-and-zeros-in-row-and-column/submissions/1119313515/
https://leetcode.com/problems/final-prices-with-a-special-discount-in-a-shop/submissions/1121032165/_

@MITHUNTR7 Thanks for your contribution 👍


[Issue] WebPack warning (React app)

I see this when running react app
(node:166812) [DEP_WEBPACK_DEV_SERVER_ON_AFTER_SETUP_MIDDLEWARE] DeprecationWarning: 'onAfterSetupMiddleware' option is deprecated. Please use the 'setupMiddlewares' option.
(Use `node --trace-deprecation ...` to show where the warning was created)
(node:166812) [DEP_WEBPACK_DEV_SERVER_ON_BEFORE_SETUP_MIDDLEWARE] DeprecationWarning: 'onBeforeSetupMiddleware' option is deprecated. Please use the 'setupMiddlewares' option.

I think this can be closed as it's an upstream issue.


Polyglot Notebook: [DevExE2E] Stop Cell Execution will not work if we run all test -> stop Execution.

Describe the bug
Using extension Polyglot Notebook in VS Code
Stop Cell Execution will not work if repeat do for multi times.
Detailed Repro Build info: VSCode 1.77.3 + dotnet-interactive-vscode-1.0.4227011.vsix
VS Code Version: 1.77.3 (user setup)
Commit: 704ed70d4fd1c6bd6342c436f1ede30d1cff4710
Date: 2023-04-12T09:16:02.548Z
Electron: 19.1.11
Chromium: 102.0.5005.196
Node.js: 16.14.2
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.22621
Sandboxed: No
dotnet-interactive-vscode-1.0.4227011.vsix
build link: https://dev.azure.com/dnceng/internal/_build/results?buildId=2168897&view=artifacts&pathAsName=false&type=publishedArtifacts
sdk info: dotnet-sdk-7.0.203-win-x64.exe
Note:
1.It's a not a regression issue. It also repro on VSCode 1.77.3 + dotnet-interactive-vscode-1.0.4219021.vsix.
2.Repro VM: <IP_ADDRESS>
Steps:

Open the notebook from https://github.com/dotnet/interactive/blob/main/NotebookTestScript.dib
In top menu, click Run all -> click Stop Execution
Click Go To, click "Stop Cell Execution". (Repeat this step)

Expected Results:
"Stop Cell Execution" can work all the time, with no error.
Actual Results:
Some tests are still running, and there's no response when clicking the "Stop Cell Execution".

Please complete the following:
Which version of .NET Interactive are you using? (In a notebook, run the #!about magic command. ):

OS

[√] Windows 11
[ ] Windows 10
[ ] macOS
[ ] Linux (Please specify distro)
[ ] iOS
[ ] Android

Browser

[ ] Chrome
[ ] Edge
[ ] Firefox
[ ] Safari

Frontend

[ ] Jupyter Notebook
[ ] Jupyter Lab
[ ] nteract
[√] Visual Studio Code
[ ] Visual Studio Code Insiders
[ ] Visual Studio
[ ] Other (please specify)

Screenshots
If applicable, add screenshots to help explain your problem.

Stopping execution won't fix the prompt, execution is blocked at that point . Should we investigate a way to "recall" and input request?

+1 here, after queuing multiple cells or clicking the "Run All" button, if I then click "Stop Cell Execution" the cells that are supposed to execute after the current one get stuck in a non-executable state. The only way to resolve this issue is to close and reopen VSCode.


Upgrade spdlog to 0.13.0.

Upgrade to spdlog 0.13.0 (release notes).

@moderation I assume you tested compile with the new version? Not sure if there are any breaking changes?

@mattklein123 yes, I've tested the upgraded spdlog for about a week of use and it passes tests.


Old connection pool not cleaned up after token refresh?

We are using Azure AD auth and setting the token on the SQL connection using SQLConnection.AccessToken. It is my understanding that the token is part of the key for the connection pool so that when the token is refreshed a new connection pool is created. On our system it seems like the old connection pool remains with all its TCP connections open and seems to grow every time the token is refreshed. This from looking at the active-hard-connections performance counter. The performance counter number-of-active-connection-pools also keeps growing.
As a workaround I have tried calling SQLConnection.ClearPool(...) on a connection that belongs to the old pool a few minutes after the token is refreshed and this seems to work to close down the old pool properly.
Further technical details
Microsoft.Data.SqlClient version: 5.1.0
.NET target: .NET 7
SQL Server version: Azure SQL
Operating system: ubuntu-22.04

@mamort the token is cached inside the connection. Renewing the token does not clear the connection pool and does not create new connections. However, clearing connection pool will force acquiring new tokens as well. This is by design.

Thanks for the quick response. I wondered if this was taken care of in a better way when using the "Active Directory Default" auth mode. Might be a good idea to switch, but I wasn't sure if it supported WorkloadIdentity (?)
However, the way we do it now should work and the old pool does not get cleaned up after 2-4 minutes. I see the connections remain for days and do not seem to ever seem to get closed. We create new connections for every query and dispose them right after using Dapper. Could be a bug in our code and I will check again, but it seemed unlikely (not many lines of code).
I guess the only way to know is to create a minimal example. Just too bad there is no way to force a token to be refreshed.

@mamort I am not familiar with WorkloadIdentity. Is it Azure Kubernetes Service related application?

Yes it is. It is described here:
https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview
Since DefaultAzureCredential is used in SqlClient I think it should just work, but we need to be able to configure clientId among other things which might be possible by using environment variables

@JRahnama Any comments about this? Is this behavior just by design and something that must be handled by ourselves if we use the direct approach by setting the access token on the SqlConnection? Seems like bad design to not prune the old app pool using an expired token (when MinPoolSize > 0), but I guess it might be difficult to know that it has expired.
What happens when using "Active Directory Default" auth? Do you actively close down the old connection pool if the token has expired?

I guess it might be difficult to know that it has expired.

each token is good for almost 10 hours.

What happens when using "Active Directory Default" auth? Do you actively close down the old connection pool if the token has expired?

A new access token is acquired and assigned to the old connection inside the pool. Basically creating new connection is more expensive than acquiring a new access token, so the SqlClient driver sends a request to AAD to get a new access token and assigns it to the previously created connections inside the pool.

What happens when using "Active Directory Default" auth? Do you actively close down the old connection pool if the token has expired?

A new access token is acquired and assigned to the old connection inside the pool. Basically creating new connection is more expensive than acquiring a new access token, so the SqlClient driver sends a request to AAD to get a new access token and assigns it to the previously created connections inside the pool.

This isn't quite right. When the token has expired and a new connection is requested that matches an existing pool, the new connection request will initiate fetching of a new token. That token will be used for a new connection to the server and saved with the pool to be used with subsequent new connections for that pool. Any old connections in the pool that used the expired token will be pruned from the pool as they are closed/disposed/freed by the application. There is no updating of tokens on connections that are already established to the server.

I just want to add to what's happening here. AccessToken is part of the pool key, as you said. Since you have set min pool size to 10, you will get a pool of 10 connections with each unique access token. When you set a new access token, you get a new pool. The old pool is not related to the new one, so there is no "replacement" of the connections from one to the other. It's an unfortunate behavior, but by design. When you refresh your token, you should call ClearPool on your previous pool with the old token to make sure it gets cleaned up (since you have set a min pool size).

Thanks for the clarifications. It seems it would be ideal to "warm up" the new pool before the old token expires. I have now added logic to our app that does this since we typically get a new token a few minutes before the old one expire. Then, 1 minute before the old token expire we switch to using the new warmed up pool and call SqlConnection.ClearPool() on a connection with the old token. Seems to work.
This should be better documented, especially the unfortunate behavior with the combination of setting the access token on the SqlConnection + configuring MinPoolSize > 0. It would have been really nice to be able to set a Azure.Core.AccessToken on the SqlConnection instead of a string token. Then the pooling would be able to know when the token would expire and the logic I implemented above could be baked into the SqlClient-library.

It would have been really nice to be able to set a Azure.Core.AccessToken on the SqlConnection instead of a string token.

@mamort You might be pleasantly surprised to know almost exactly that is currently available in the 5.2 preview 3 release (see #1260 for a code example). 😃

That sounds perfect! 😃

@mamort I am closing the issue as resolved. Feel free to reopen the issue or create a new issue if you think there is more to be done here.


DW-198 Retry support for extract

This add support for extract to retry on a per-table basis. The number of retry attempts it uses is controlled by a new configuration setting, and has a default value of 1 retry.
This was tested out using the FakeSqoopExtractor; so it does work in the case where Sqoop returns a non-zero exit code.
Open questions/issues for feedback:

Are there any known failure cases today where Sqoop returns a non-zero error code and we do not want to retry? The current implementation is a very blunt instrument, and just always retries Sqoop.
The Static and ManifestOnly extractors both simply interact with S3 via Boto. Boto internally handles retries for various transient errors, so it didn't seem necessary to add explicit retries in user code on top of that. Thoughts?
The Spark extractor seems to be broken, based on the comments in spark.py on lines 156 and 157. Because of that, the complexity involved in creating a test environment I could use to test retry code for Spark errors, and not finding some readily available documentation on the exceptions/errorrs thrown by Spark, I didn't test this out on Spark. Should Spark be out of scope for this?

RE Spark -- yes, it's out of scope since Spark extracts are only used for debugging. The spark extract isn't really that broken ... there are some edge cases where the load would fail.  Meaning that using Spark extract to test any logic of this step locally is still valuable.  (At least, to me.)


Stepper motor will only move in small steps

This is a link from the previous LED issue. This time its with a stepper motor and for some reason when I try to run the program, although the stepper motor does move, it takes really small steps. I ran into this issue with some Bluetooth code I was trying to get to work and I still have not figured that one out.

We might not need to fix it because in the app code, it continuously sends the data since it checks if the for example forwards button is clicked and it stays toggled on which means its constantly sending data but it depends how it does it


Update docs about the default Eip and VIP mode

Signed-off-by: Patrick-LuoYu<EMAIL_ADDRESS>Please take a look. Thanks!
/assign @zheng1 @renyunkang

/approve

Please also include @chaunceyjiang Xingyan as a reviewer, he has helped us to implement the default EIP mode in OpenELB.

Further optimization. Please take a look. Thanks!
/assign @zheng1 @renyunkang @chaunceyjiang

Thank you for your work.
/LGTM

/LGTM

@Patrick-LuoYu The Vercel preview looks good now. Thanks for your updates!
/LGTM
/Approve

/cc @renyunkang @zheng1

/lgtm
/approve

/approve


Backup/restore not working as expected

I have attempted to restore a backup of all dashboards of a Grafana installation using sqlite3 for its storage. Now I want to migrate to Postgres due to database locking issues. My plan was to fire up a new instance of Grafana, configure it to use Postgres and then restore the dashboards from the backup to the new instance. To make things easier I decided to just use the http api.
I'm using a grafcli.ini in the same directory where I'm executing grafcli and where the backup is stored. The crendentials are provided in the grafcli.ini-file.
However, grafcli complains about getting a 404 from the new Grafana instance:
grafcli restore backup.tgz remote/newinstance
Traceback (most recent call last):
  File "/usr/local/bin/grafcli", line 27, in <module>
    sys.exit(main())
  File "/usr/local/bin/grafcli", line 13, in main
    result = cli.execute(*sys.argv[1:])
  File "/usr/local/lib/python3.6/site-packages/climb/core.py", line 79, in execute
    return self._commands.execute(command, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/climb/commands.py", line 26, in execute
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/grafcli/commands.py", line 205, in restore
    self.file_import(file_path, doc_path)
  File "/usr/local/lib/python3.6/site-packages/grafcli/commands.py", line 237, in file_import
    self._resources.save(path, document)
  File "/usr/local/lib/python3.6/site-packages/grafcli/resources/resources.py", line 51, in save
    return manager.save(document, *parts)
  File "/usr/local/lib/python3.6/site-packages/grafcli/resources/common.py", line 46, in save
    origin_document = self.get(dashboard_name, row_name, panel_name)
  File "/usr/local/lib/python3.6/site-packages/grafcli/resources/common.py", line 33, in get
    dashboard = self._storage.get(dashboard_name)
  File "/usr/local/lib/python3.6/site-packages/grafcli/storage/api.py", line 37, inget
    source = self._call('GET', 'dashboards/db/{}'.format(dashboard_id))
  File "/usr/local/lib/python3.6/site-packages/grafcli/storage/api.py", line 29, in_call
    response.raise_for_status()
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 937, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://newinstance.corp.tld/api/dashboards/db/dashboard_to_restore

Am I missing something or is this a bug? Let me know if you need me to provide additional information.

I can reproduce this with the latest docker image of Grafana on localhost. The backup is working fine but restoring isn't possible.

Hey. It looks like the restore tries to get a dashboard that does not exist. I'll look into it, but if you'd like a quick workaround, you could try creating dashboard with this name on the new instance.

Thanks. I've tried to create an empty dashboard with the same name of the dashboard from the backend, which grafcli recognized and prompted if it should overwrite its content:

Overwrite new-dashboard-copy? [y/n]:

Unfortunately, I'm still getting this error after approving this:
Traceback (most recent call last):
  File "/usr/local/bin/grafcli", line 27, in <module>
    sys.exit(main())
  File "/usr/local/bin/grafcli", line 13, in main
    result = cli.execute(*sys.argv[1:])
  File "/usr/local/lib/python3.6/site-packages/climb/core.py", line 79, in execute
    return self._commands.execute(command, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/climb/commands.py", line 26, in execute
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/grafcli/commands.py", line 205, in restore
    self.file_import(file_path, doc_path)
  File "/usr/local/lib/python3.6/site-packages/grafcli/commands.py", line 237, in file_import
    self._resources.save(path, document)
  File "/usr/local/lib/python3.6/site-packages/grafcli/resources/resources.py", line 51, in save
    return manager.save(document, *parts)
  File "/usr/local/lib/python3.6/site-packages/grafcli/resources/common.py", line 46, in save
    origin_document = self.get(dashboard_name, row_name, panel_name)
  File "/usr/local/lib/python3.6/site-packages/grafcli/resources/common.py", line 33, in get
    dashboard = self._storage.get(dashboard_name)
  File "/usr/local/lib/python3.6/site-packages/grafcli/storage/api.py", line 37, in get
    source = self._call('GET', 'dashboards/db/{}'.format(dashboard_id))
  File "/usr/local/lib/python3.6/site-packages/grafcli/storage/api.py", line 29, in _call
    response.raise_for_status()
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 937, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://localhost:3000/api/dashboards/db/new-dashboard-copy

So it seems this was due to silly lack of exception mapping in the API storage. Can you try it with the most recent version?

Looks good, I just tested it locally. I'll attempt to migrate our production Grafana today. I'm expecting that it'll work now. Thanks!

My local tests succeed, but in production I could nut fully restore from the backup. After restoring 5 dashboards I'm hitting a 500:
grafcli restore backup.tgz remote/remote.tld                         ] 2:58 PM
Traceback (most recent call last):
  File "/usr/local/bin/grafcli", line 27, in <module>
    sys.exit(main())
  File "/usr/local/bin/grafcli", line 13, in main
    result = cli.execute(*sys.argv[1:])
  File "/usr/local/lib/python3.6/site-packages/climb/core.py", line 79, in execute
    return self._commands.execute(command, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/climb/commands.py", line 26, in execute
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/grafcli/commands.py", line 205, in restore
    self.file_import(file_path, doc_path)
  File "/usr/local/lib/python3.6/site-packages/grafcli/commands.py", line 237, in file_import
    self._resources.save(path, document)
  File "/usr/local/lib/python3.6/site-packages/grafcli/resources/resources.py", line 51, in save
    return manager.save(document, *parts)
  File "/usr/local/lib/python3.6/site-packages/grafcli/resources/common.py", line 69, in save
    self._storage.save(dashboard.id, dashboard)
  File "/usr/local/lib/python3.6/site-packages/grafcli/storage/api.py", line 63, in save
    self._call('POST', 'dashboards/db', data)
  File "/usr/local/lib/python3.6/site-packages/grafcli/storage/api.py", line 30, in _call
    response.raise_for_status()
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 937, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://remote.tld/api/dashboards/db

I checked the Grafana pod's logs and it seems it's missing the data sources and that something's off with the alert of one of our dashboards:
t=2017-07-31T12:58:59+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/dashboards/db/dashboard_1 status=404 remote_addr=$OUR_OFFICE_IP time_ms=130 size=33 referer=
t=2017-07-31T12:59:00+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/dashboards/db/consul-cluster status=404 remote_addr=$OUR_OFFICE_IP time_ms=174 size=33 referer=
t=2017-07-31T12:59:01+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/dashboards/db/dashboard_3 status=404 remote_addr=$OUR_OFFICE_IP time_ms=78 size=33 referer=
t=2017-07-31T12:59:01+0000 lvl=eror msg="Invalid alert data. Cannot save dashboard" logger=context userId=1 orgId=1 uname=admin error="Data source not found"
t=2017-07-31T12:59:01+0000 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/dashboards/db status=500 remote_addr=$OUR_OFFICE_IP time_ms=114 size=55 referer=
t=2017-07-31T12:59:15+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=1 uname= method=GET path=/login/github status=302 remote_addr=$OUR_OFFICE_IP time_ms=11 size=299 referer="https://remote.tld/login?redirect=%2F"
t=2017-07-31T12:59:16+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=1 uname= method=GET path=/login/github status=302 remote_addr=$OUR_OFFICE_IP time_ms=1315 size=24 referer="https://remote.tld/login?redirect=%2F"
t=2017-07-31T12:59:56+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/dashboards/db/dashboard_4 status=404 remote_addr=$OUR_OFFICE_IP time_ms=64 size=33 referer=
t=2017-07-31T12:59:56+0000 lvl=eror msg="Invalid alert data. Cannot save dashboard" logger=context userId=1 orgId=1 uname=admin error="Data source not found"
t=2017-07-31T12:59:56+0000 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/dashboards/db status=500 remote_addr=$OUR_OFFICE_IP time_ms=113 size=55 referer=

According to the Grafana docs, exporting a dashboard should also include the data sources it depends on. Doesn't grafcli behave the same?
Thanks!

Oh, right. This might be not so trivial then. It's connected to #8.
When grafcli was first created, grafana didn't have any export mechanism or API support yet, so all operations were based on direct SQL manipulation. That's why there are some differences now when working with the current API. I will definitely look into this soon.
Once again, if you're in a hurry, you could try creating manually the data source with the same name. I'm not sure about the alert-related error though.